Сrash on bad vhost configuration #1439

krizhanovsky · 2020-08-11T13:05:26Z

Scope

I tried to configure HSTS as described in the Wiki https://github.com/tempesta-tech/tempesta/wiki/HTTP-security#http-strict-transport-security-hsts:

listen 192.168.100.4:443 proto=https;
listen 192.168.100.4:80;

srv_group default {
	server 127.0.0.1:8080 conns_n=4;
}

vhost debian {
	tls_certificate /root/tempesta/etc/tfw-root.crt;
	tls_certificate_key /root/tempesta/etc/tfw-root.key;

	resp_hdr_set Strict-Transport-Security "max-age=31536000; includeSubDomains"

	proxy_pass default;
}

cache 1;
cache_fulfill * *;

block_action attack reply;

http_chain {
	-> debian;
}

The configuration not only doesn't work as declared in the Wiki, but leads to the crash (I tried twice and it's reproduced in both the cases):

[   21.690027] [tempesta fw] Initializing Tempesta FW kernel module...
[   21.715934] [tempesta fw] Warning: Listening for HTTP/1.1 protocol is compatibility mode and may lack of performance.
[   21.721954] [tempesta fw] ERROR: resp_hdr_set: Invalid number of values.
[   21.723563] [tempesta fw] ERROR: configuration parsing error:
[   21.723563]   12:    resp_hdr_set Strict-Transport-Security "max-age=31536000; includeSubDomains"
[   21.723563]       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[   21.727700] [tempesta fw] Warning: Configuration parsing has failed. Clean up...
[   21.729370] [tempesta tls] Warning: Too large MPI pool was allocated (order=1 used=120)
[   21.731511] [tempesta tls] Warning: Too large MPI pool was allocated (order=1 used=152)
[   21.733882] ------------[ cut here ]------------
[   21.735014] kernel BUG at /root/tempesta/tempesta_fw/vhost.c:2220!
[   21.736356] invalid opcode: 0000 [#1] SMP PTI
[   21.737033] Modules linked in: tempesta_fw(O) tempesta_db(O) tempesta_tls(O) tempesta_lib(O) bochs_drm ttm drm_kms_helper crct10dif_pclmul drm ppdev fb_sys_fops syscopyarea sysfillrect sysimgblt serio_raw parport_pc sg parport button binfmt_misc ip_tables x_tables ext4 crc16 mbcache jbd2 fscrypto sd_mod sr_mod cdrom ata_generic virtio_net ata_piix libata psmouse virtio_pci virtio_ring virtio i2c_piix4 scsi_mod
[   21.743342] CPU: 0 PID: 1060 Comm: sysctl Tainted: G           O    4.14.32-kdump+ #155
[   21.745612] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.13.0-2.fc32 04/01/2014
[   21.747743] task: ffff9bdde5558000 task.stack: ffff9bddf338c000
[   21.748962] RIP: 0010:tfw_cfgop_vhost_begin+0x1ef/0x220 [tempesta_fw]
[   21.751777] RSP: 0018:ffff9bddf338fa38 EFLAGS: 00010286
[   21.753938] RAX: ffffffffc03d9e40 RBX: ffffffffc0475418 RCX: 0000000000000001
[   21.755901] RDX: 0000000000000001 RSI: ffff9bddf338fb10 RDI: ffffffffc0475418
[   21.757799] RBP: ffff9bddf338fb10 R08: 0000000000000000 R09: 0000000000000074
[   21.759704] R10: ffff9bddeeabb4c5 R11: f000000000000000 R12: ffffffffc0475500
[   21.761677] R13: ffffffffc0475418 R14: ffff9bddf338fb10 R15: ffffffffc04754c0
[   21.763621] FS:  00007f91756b27c0(0000) GS:ffff9bddffc00000(0000) knlGS:0000000000000000
[   21.765784] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   21.767472] CR2: 0000559d5c58caf0 CR3: 00000000721c2001 CR4: 00000000003606f0
[   21.769800] Call Trace:
[   21.771336]  tfw_cfg_handle_children+0x74/0x1f0 [tempesta_fw]
[   21.773129]  spec_handle_entry+0x7a/0x90 [tempesta_fw]
[   21.774935]  tfw_cfg_parse_mods+0x188/0x220 [tempesta_fw]
[   21.778429]  ? parse_cfg_entry+0x1b8/0x670 [tempesta_fw]
[   21.781821]  tfw_cfg_parse+0x45/0x90 [tempesta_fw]
[   21.784394]  tfw_ctlfn_state_io+0xf9/0x450 [tempesta_fw]
[   21.787664]  ? mutex_lock+0x9/0x30
[   21.790352]  ? tfw_ctlfn_state_io+0x3f/0x450 [tempesta_fw]
[   21.792827]  ? cap_inode_getsecurity+0x220/0x220
[   21.795321]  proc_sys_call_handler+0xe8/0x110
[   21.797538]  __vfs_write+0x31/0x170
[   21.799560]  vfs_write+0xab/0x190
[   21.801526]  SyS_write+0x4d/0xb0
[   21.803553]  do_syscall_64+0x61/0x110
[   21.808024]  entry_SYSCALL_64_after_hwframe+0x3d/0xa2

Testing

Please make a functional test for simultaneous accept HTTP and HTTPS connections with and w/o HSTS redirections (#856), at least Strict-Transport-Security must work for now, i.e. an HTTP connection must be redirected to HTTPS.

The text was updated successfully, but these errors were encountered:

krizhanovsky · 2020-08-11T13:36:49Z

The problem was with missing ; at the end of resp_hdr_set ... line. With ; I can confirm that Tempesta properly sets the header. So the problem is in crashing on wrong configuration option only.

Also I observed that my browser just ignores the header and HSTS is mostly on the client side, so there is no sense to make a functional test for Tempesta.

krizhanovsky · 2020-08-11T14:42:42Z

With more experiments on #1440 I faced that this configuration

listen 192.168.100.4:443 proto=https;
listen 192.168.100.4:80;

srv_group default {
	server 127.0.0.1:8080 conns_n=4;
}

#vhost debian {
vhost default {
	tls_certificate /root/tempesta/etc/tfw-root.crt;
	tls_certificate_key /root/tempesta/etc/tfw-root.key;

	resp_hdr_set Strict-Transport-Security "max-age=31536000; includeSubDomains";

	frang_limits {
		http_methods GET;
		http_uri_len 512;
		http_resp_code_block 400 403 404 3 10;
		client_body_timeout 10;
		http_header_chunk_cnt 10;
		http_body_chunk_cnt 0;
	}

	proxy_pass default;
}

cache 0;
#cache_fulfill * *;

http_uri_brange 0x2f 0x41-0x5a 0x61-0x7a 0x30-0x39 0x2d 0x5f; # /a-zA-Z0-9-_

frang_limits {
	client_header_timeout 20;
}

block_action attack reply;

http_chain {
	#-> debian;
	-> default;
}

also leads to the crash. The reason for the rejected configuration is that client_body_timeout must be in global configuration section.

krizhanovsky · 2020-08-13T15:11:55Z

Just tried to remove ';' for resp_hdr_set vhost setting in the configuration file for tempesta-tech.com and got bit different crash on current master as of c68cf24 . I got the crash on clear build with just --start.

[   81.491505] [tempesta fw] Warning: Configuration parsing has failed. Clean up...
[   81.494203] [tempesta tls] Warning: Too large MPI pool was allocated (order=1 used=120)
[   81.496161] [tempesta tls] Warning: Too large MPI pool was allocated (order=1 used=152)
[   81.499475] ------------[ cut here ]------------
[   81.501058] kernel BUG at /root/tempesta/tempesta_fw/vhost.c:1502!
[   81.502974] invalid opcode: 0000 [#1] SMP PTI
[   81.506242] Modules linked in: tempesta_fw(O) tempesta_db(O) tempesta_tls(O) tempesta_lib(O) bochs_drm ttm drm_kms_helper crct10dif_pclmul ppdev drm fb_sys_fops syscopyare
a sg sysfillrect sysimgblt serio_raw parport_pc parport button binfmt_misc ip_tables x_tables ext4 crc16 mbcache jbd2 fscrypto sd_mod sr_mod cdrom ata_generic virtio_net ata_
piix psmouse virtio_pci virtio_ring virtio i2c_piix4 libata scsi_mod
[   81.517268] CPU: 0 PID: 1100 Comm: sysctl Tainted: G           O    4.14.32-kdump+ #155
[   81.519616] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.13.0-2.fc32 04/01/2014
[   81.522106] task: ffff9fd173b12140 task.stack: ffff9fd18f1b8000
[   81.524217] RIP: 0010:tfw_cfgop_out_sticky_begin+0x25/0x50 [tempesta_fw]
[   81.525979] RSP: 0018:ffff9fd18f1bba38 EFLAGS: 00010286
[   81.527513] RAX: ffffffffc0470c90 RBX: ffffffffc050e3a8 RCX: ffff9fd1498a61ec
[   81.529974] RDX: 0000000000000001 RSI: ffff9fd18f1bbb10 RDI: ffffffffc050e3a8
[   81.532053] RBP: ffff9fd18f1bbb10 R08: 0000000000000079 R09: 0000000000000079
[   81.534832] R10: ffff9fd19c698786 R11: 0000000000000020 R12: ffffffffc0511e20
[   81.538702] R13: ffffffffc050e3a8 R14: ffff9fd18f1bbb10 R15: ffffffffc050e4e0
[   81.541504] FS:  00007f581804b7c0(0000) GS:ffff9fd1bfc00000(0000) knlGS:0000000000000000
[   81.544218] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   81.545982] CR2: 000055b0bac6baf0 CR3: 000000005c5b0001 CR4: 00000000003606f0
[   81.548679] Call Trace:
[   81.549978]  tfw_cfg_handle_children+0x74/0x1f0 [tempesta_fw]
[   81.551786]  spec_handle_entry+0x7a/0x90 [tempesta_fw]
[   81.553700]  tfw_cfg_parse_mods+0x188/0x220 [tempesta_fw]
[   81.555430]  ? parse_cfg_entry+0x1b8/0x670 [tempesta_fw]
[   81.557368]  tfw_cfg_parse+0x45/0x90 [tempesta_fw]
[   81.559667]  tfw_ctlfn_state_io+0xf9/0x450 [tempesta_fw]
[   81.561614]  ? mutex_lock+0x9/0x30
[   81.563267]  ? tfw_ctlfn_state_io+0x3f/0x450 [tempesta_fw]
[   81.567427]  ? cap_inode_getsecurity+0x220/0x220
[   81.569961]  proc_sys_call_handler+0xe8/0x110
[   81.572270]  __vfs_write+0x31/0x170
[   81.574040]  vfs_write+0xab/0x190
[   81.575531]  SyS_write+0x4d/0xb0
[   81.577067]  do_syscall_64+0x61/0x110
[   81.579009]  entry_SYSCALL_64_after_hwframe+0x3d/0xa2

krizhanovsky · 2020-08-13T15:40:46Z

In both the cases this is BUG_ON(tfw_vhost_entry). I analyzed the first crash little bit more and it's quite strange that while the configuration error was in resp_hdr_set line inside the vhost section, we crash on BUG_ON at tfw_cfgop_vhost_begin() - the function processing start of vhost section. So the first thing to check is the sequence of calls for the vhost section begin processing and ingress configuration options. Note that I couldn't reproduce the crashes for other sections like frang_limits.

Next, the BUG_ON()probably means that there tfw_vhost_entry is left dirty on error path.

krizhanovsky added bug TLS Tempesta TLS module and related issues labels Aug 11, 2020

krizhanovsky added this to the 0.7 HTTP/2 milestone Aug 11, 2020

krizhanovsky assigned avbelov23 Aug 11, 2020

krizhanovsky mentioned this issue Aug 11, 2020

TLS: further performance improvements and cleanups #1064

Open

14 tasks

krizhanovsky added the low priority label Aug 11, 2020

krizhanovsky changed the title ~~HSTS configuration doesn work and crash on vhost~~ Сrash on bad vhost configuration Aug 11, 2020

krizhanovsky mentioned this issue Aug 11, 2020

Confusing vhost and defatult frang limits #1440

Closed

krizhanovsky removed the low priority label Aug 11, 2020

avbelov23 mentioned this issue Aug 14, 2020

Zero some vars as expected by the reconfiguration procedure #1443

Merged

avbelov23 closed this as completed Aug 14, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Сrash on bad vhost configuration #1439

Сrash on bad vhost configuration #1439

krizhanovsky commented Aug 11, 2020 •

edited

Loading

krizhanovsky commented Aug 11, 2020 •

edited by vankoven

Loading

krizhanovsky commented Aug 11, 2020

krizhanovsky commented Aug 13, 2020 •

edited

Loading

krizhanovsky commented Aug 13, 2020 •

edited

Loading

Сrash on bad vhost configuration #1439

Сrash on bad vhost configuration #1439

Comments

krizhanovsky commented Aug 11, 2020 • edited Loading

Scope

Testing

krizhanovsky commented Aug 11, 2020 • edited by vankoven Loading

krizhanovsky commented Aug 11, 2020

krizhanovsky commented Aug 13, 2020 • edited Loading

krizhanovsky commented Aug 13, 2020 • edited Loading

krizhanovsky commented Aug 11, 2020 •

edited

Loading

krizhanovsky commented Aug 11, 2020 •

edited by vankoven

Loading

krizhanovsky commented Aug 13, 2020 •

edited

Loading

krizhanovsky commented Aug 13, 2020 •

edited

Loading