Kernel panic on simple HTTP/2 request #1549

byko3y · 2022-01-12T21:20:23Z

Scope

Simple HTTP/2 request
curl -Lvk --http2-prior-knowledge --tlsv1.2 --tls-max 1.2 https://debian:444
with Tempesta FW of revision a671734 configured as a proxy for an almost default nginx setup

cat etc/tempesta_fw.conf

listen 127.0.0.1:443 proto=https;
listen 127.0.0.1:444 proto=h2;
listen 127.0.0.1:80;

srv_group default {
    server 127.0.0.1:81 conns_n=4;
}

vhost debian {
    tls_certificate /home/user/tempesta/etc/ECDSA/tfw-root.crt;
    tls_certificate_key /home/user/tempesta/etc/ECDSA/tfw-root.key;

    proxy_pass default;
}

cache 1;
cache_fulfill * *;
cache_resp_hdr_del Perma;

block_action attack reply;

http_chain {
    -> debian;
}

fragment of nginx config for this URL

        location / {
                try_files $uri $uri/ =404;
                add_header Remove-me ' ';
        }

leads to a kernel panic:

linux-5.10.35/scripts/decode_stacktrace.sh linux-5.10.35/vmlinux linux-5.10.35/ tempesta/fw/tempesta_fw.ko < panic.log

[14164.533351] Modules linked in: tempesta_fw(OE) tempesta_db(OE) tempesta_tls(OE) tempesta_lib(OE) sha256_ssse3(E) sha512_ssse3(E) sha512_generic(E) uinput(E) intel_rapl_msr(E) intel_rapl_common(E) intel_pmc_core_pltdrv(E) intel_pmc_core(E) kvm_intel(E) snd_hda_codec_generic(E) kvm(E) ledtrig_audio(E) irqbypass(E) snd_hda_intel(E) 9p(E) snd_intel_dspcfg(E) fscache(E) soundwire_intel(E) ghash_clmulni_intel(E) soundwire_generic_allocation(E) snd_soc_core(E) aesni_intel(E) libaes(E) rfkill(E) crypto_simd(E) snd_compress(E) cryptd(E) soundwire_cadence(E) glue_helper(E) rapl(E) snd_hda_codec(E) snd_hda_core(E) snd_hwdep(E) 9pnet_virtio(E) soundwire_bus(E) iTCO_wdt(E) joydev(E) intel_pmc_bxt(E) snd_pcm(E) iTCO_vendor_support(E) 9pnet(E) watchdog(E) serio_raw(E) snd_timer(E) virtio_console(E) sg(E) virtio_balloon(E) snd(E) evdev(E) soundcore(E) pcspkr(E) qemu_fw_cfg(E) configfs(E) fuse(E) virtio_rng(E) rng_core(E) ip_tables(E) x_tables(E) autofs4(E) ext4(E) crc32c_generic(E) crc16(E) mbcache(E)
[14164.533401]  jbd2(E) hid_generic(E) usbhid(E) hid(E) sr_mod(E) cdrom(E) virtio_blk(E) virtio_net(E) net_failover(E) failover(E) qxl(E) drm_ttm_helper(E) ttm(E) crct10dif_pclmul(E) crct10dif_common(E) crc32_pclmul(E) ahci(E) crc32c_intel(E) libahci(E) drm_kms_helper(E) libata(E) xhci_pci(E) cec(E) psmouse(E) xhci_hcd(E) drm(E) scsi_mod(E) i2c_i801(E) i2c_smbus(E) lpc_ich(E) virtio_pci(E) usbcore(E) virtio_ring(E) virtio(E) button(E) [last unloaded: tempesta_lib]
[14164.547750] CPU: 3 PID: 482 Comm: nginx Tainted: G           OE     5.10.35 #2
[14164.548434] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.14.0-2 04/01/2014
[14164.549209] RIP: 0010:tfw_hpack_write (/home/user/tempesta/fw/hpack.c:2443 (discriminator 1)) tempesta_fw
[14164.549853] Code: e8 d5 fd ff ff c3 0f 1f 44 00 00 41 54 55 53 48 89 fb 8b 57 18 41 89 d4 4c 89 e0 48 0b 47 10 0f 94 c0 0f b6 c0 48 85 c0 74 02 <0f> 0b 48 85 c0 75 46 f6 43 1c 01 74 02 0f 0b 85 d2 75 06 4c 8d 63
All code
========
   0:	e8 d5 fd ff ff       	callq  0xfffffffffffffdda
   5:	c3                   	retq   
   6:	0f 1f 44 00 00       	nopl   0x0(%rax,%rax,1)
   b:	41 54                	push   %r12
   d:	55                   	push   %rbp
   e:	53                   	push   %rbx
   f:	48 89 fb             	mov    %rdi,%rbx
  12:	8b 57 18             	mov    0x18(%rdi),%edx
  15:	41 89 d4             	mov    %edx,%r12d
  18:	4c 89 e0             	mov    %r12,%rax
  1b:	48 0b 47 10          	or     0x10(%rdi),%rax
  1f:	0f 94 c0             	sete   %al
  22:	0f b6 c0             	movzbl %al,%eax
  25:	48 85 c0             	test   %rax,%rax
  28:	74 02                	je     0x2c
  2a:*	0f 0b                	ud2    		<-- trapping instruction
  2c:	48 85 c0             	test   %rax,%rax
  2f:	75 46                	jne    0x77
  31:	f6 43 1c 01          	testb  $0x1,0x1c(%rbx)
  35:	74 02                	je     0x39
  37:	0f 0b                	ud2    
  39:	85 d2                	test   %edx,%edx
  3b:	75 06                	jne    0x43
  3d:	4c                   	rex.WR
  3e:	8d                   	.byte 0x8d
  3f:	63                   	.byte 0x63

Code starting with the faulting instruction
===========================================
   0:	0f 0b                	ud2    
   2:	48 85 c0             	test   %rax,%rax
   5:	75 46                	jne    0x4d
   7:	f6 43 1c 01          	testb  $0x1,0x1c(%rbx)
   b:	74 02                	je     0xf
   d:	0f 0b                	ud2    
   f:	85 d2                	test   %edx,%edx
  11:	75 06                	jne    0x19
  13:	4c                   	rex.WR
  14:	8d                   	.byte 0x8d
  15:	63                   	.byte 0x63
[14164.551595] RSP: 0018:ffffc90000120808 EFLAGS: 00010202
[14164.552100] RAX: 0000000000000001 RBX: ffffc90000120870 RCX: ffff888022328115
[14164.552778] RDX: 0000000000000000 RSI: ffff8881621b4181 RDI: ffffc90000120870
[14164.553467] RBP: 0000000000000009 R08: ffff88802232810d R09: 0000000000001000
[14164.554198] R10: ffff8881621b4020 R11: 0000000000000eb8 R12: 0000000000000000
[14164.554948] R13: 00000000000001bb R14: ffffc90000120a18 R15: 0000000000000029
[14164.555709] FS:  00007f417e08cb80(0000) GS:ffff88817bd80000(0000) knlGS:0000000000000000
[14164.556504] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[14164.557048] CR2: 0000556732eb4307 CR3: 0000000124da2005 CR4: 0000000000370ee0
[14164.557731] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[14164.558406] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[14164.559071] Call Trace:
[14164.559324]  <IRQ>
[14164.559551] tfw_hpack_add_node (/home/user/tempesta/fw/hpack.c:2979 /home/user/tempesta/fw/hpack.c:3126) tempesta_fw
[14164.560073] tfw_hpack_encoder_index (/home/user/tempesta/fw/hpack.c:3204) tempesta_fw
[14164.560625] tfw_hpack_encode (/home/user/tempesta/fw/hpack.c:3729) tempesta_fw
[14164.561150] tfw_h2_resp_adjust_fwd (/home/user/tempesta/fw/http.c:4869) tempesta_fw
[14164.561800] ? tfw_h2_resp_adjust_fwd (/home/user/tempesta/fw/http.c:5605) tempesta_fw
[14164.562367] tfw_http_resp_cache_cb (/home/user/tempesta/fw/http.c:5613) tempesta_fw
[14164.562903] tfw_cache_add (/home/user/tempesta/fw/cache.c:1889) tempesta_fw
[14164.563368] ? tfw_h2_resp_adjust_fwd (/home/user/tempesta/fw/http.c:5605) tempesta_fw
[14164.563926] ? tfw_h2_resp_adjust_fwd (/home/user/tempesta/fw/http.c:5605) tempesta_fw
[14164.564561] tfw_cache_do_action (/home/user/tempesta/fw/cache.c:2372) tempesta_fw
[14164.565102] tfw_cache_process (/home/user/tempesta/fw/cache.c:2431) tempesta_fw
[14164.565603] ? tfw_gfsm_move (/home/user/tempesta/fw/gfsm.c:254) tempesta_fw
[14164.566078] tfw_http_resp_cache (/home/user/tempesta/fw/http.c:5795) tempesta_fw
[14164.566693] tfw_http_resp_process (/home/user/tempesta/fw/http.c:6038) tempesta_fw
[14164.567236] tfw_http_msg_process_generic (/home/user/tempesta/fw/http.c:6104 (discriminator 1)) tempesta_fw
[14164.567828] tfw_http_msg_process (/home/user/tempesta/fw/http.c:6131 (discriminator 1)) tempesta_fw
[14164.568340] __gfsm_fsm_exec (/home/user/tempesta/fw/gfsm.c:186 (discriminator 2)) tempesta_fw
[14164.568814] ? ss_skb_queue_coalesce_tail (/home/user/tempesta/fw/ss_skb.c:1345) tempesta_fw
[14164.569411] tfw_gfsm_dispatch (/home/user/tempesta/fw/gfsm.c:205) tempesta_fw
[14164.569901] tfw_connection_recv (/home/user/tempesta/fw/connection.c:135) tempesta_fw
[14164.570405] ss_tcp_process_skb (/home/user/tempesta/fw/sock.c:794 (discriminator 3)) tempesta_fw
[14164.570916] ss_tcp_process_data (/home/user/tempesta/fw/sock.c:851 (discriminator 3)) tempesta_fw
[14164.571431] ss_tcp_data_ready (/home/user/tempesta/fw/sock.c:900) tempesta_fw
[14164.571934] tcp_data_queue (/home/user/shared/linux-5.10.35/net/ipv4/tcp_input.c:4984) 
[14164.572303] tcp_rcv_established (/home/user/shared/linux-5.10.35/./include/linux/skbuff.h:1833 /home/user/shared/linux-5.10.35/./include/net/tcp.h:1802 /home/user/shared/linux-5.10.35/./include/net/tcp.h:1875 /home/user/shared/linux-5.10.35/net/ipv4/tcp_input.c:5379 /home/user/shared/linux-5.10.35/net/ipv4/tcp_input.c:5874) 
[14164.572707] tcp_v4_do_rcv (/home/user/shared/linux-5.10.35/net/ipv4/tcp_ipv4.c:1721) 
[14164.573063] tcp_v4_rcv (/home/user/shared/linux-5.10.35/net/ipv4/tcp_ipv4.c:2061) 
[14164.573406] ip_protocol_deliver_rcu (/home/user/shared/linux-5.10.35/net/ipv4/ip_input.c:204 (discriminator 1)) 
[14164.573838] ip_local_deliver_finish (/home/user/shared/linux-5.10.35/./include/linux/rcupdate.h:72 /home/user/shared/linux-5.10.35/./include/linux/rcupdate.h:702 /home/user/shared/linux-5.10.35/net/ipv4/ip_input.c:232) 
[14164.574259] __netif_receive_skb_one_core (/home/user/shared/linux-5.10.35/net/core/dev.c:5361 (discriminator 4)) 
[14164.574719] process_backlog (/home/user/shared/linux-5.10.35/./include/linux/rcupdate.h:72 /home/user/shared/linux-5.10.35/./include/linux/rcupdate.h:702 /home/user/shared/linux-5.10.35/net/core/dev.c:6373) 
[14164.575087] net_rx_action (/home/user/shared/linux-5.10.35/./arch/x86/include/asm/jump_label.h:25 /home/user/shared/linux-5.10.35/./include/linux/jump_label.h:200 /home/user/shared/linux-5.10.35/./include/trace/events/napi.h:14 /home/user/shared/linux-5.10.35/net/core/dev.c:6817 /home/user/shared/linux-5.10.35/net/core/dev.c:6886) 
[14164.575448] __do_softirq (/home/user/shared/linux-5.10.35/./arch/x86/include/asm/jump_label.h:25 /home/user/shared/linux-5.10.35/./include/linux/jump_label.h:200 /home/user/shared/linux-5.10.35/./include/trace/events/irq.h:142 /home/user/shared/linux-5.10.35/kernel/softirq.c:304) 
[14164.575791] asm_call_irq_on_stack (/home/user/shared/linux-5.10.35/arch/x86/entry/entry_64.S:796) 
[14164.576195]  </IRQ>
[14164.576409] do_softirq_own_stack (/home/user/shared/linux-5.10.35/./arch/x86/include/asm/irq_stack.h:27 /home/user/shared/linux-5.10.35/./arch/x86/include/asm/irq_stack.h:77 /home/user/shared/linux-5.10.35/arch/x86/kernel/irq_64.c:77) 
[14164.576812] do_softirq (/home/user/shared/linux-5.10.35/kernel/softirq.c:351 /home/user/shared/linux-5.10.35/kernel/softirq.c:338) 
[14164.577141] __local_bh_enable_ip (/home/user/shared/linux-5.10.35/kernel/softirq.c:196) 
[14164.577537] ip_finish_output2 (/home/user/shared/linux-5.10.35/net/ipv4/ip_output.c:240) 
[14164.577923] __ip_queue_xmit (/home/user/shared/linux-5.10.35/./include/net/dst.h:443 /home/user/shared/linux-5.10.35/net/ipv4/ip_output.c:126 /home/user/shared/linux-5.10.35/net/ipv4/ip_output.c:532) 
[14164.578298] __tcp_transmit_skb (/home/user/shared/linux-5.10.35/net/ipv4/tcp_output.c:1408 (discriminator 4)) 
[14164.578757] tcp_write_xmit (/home/user/shared/linux-5.10.35/net/ipv4/tcp_output.c:2754) 
[14164.578806] ? ext4_read_block_bitmap_nowait+0x192/0x630 ext4
[14164.578808] __tcp_push_pending_frames (/home/user/shared/linux-5.10.35/net/ipv4/tcp_output.c:2934) 
[14164.578810] tcp_setsockopt (/home/user/shared/linux-5.10.35/net/ipv4/tcp.c:3270 /home/user/shared/linux-5.10.35/net/ipv4/tcp.c:3213 /home/user/shared/linux-5.10.35/net/ipv4/tcp.c:3365) 
[14164.578812] __sys_setsockopt (/home/user/shared/linux-5.10.35/net/socket.c:2140) 
[14164.578816] __x64_sys_setsockopt (/home/user/shared/linux-5.10.35/net/socket.c:2149 /home/user/shared/linux-5.10.35/net/socket.c:2146 /home/user/shared/linux-5.10.35/net/socket.c:2146) 
[14164.578818] do_syscall_64 (/home/user/shared/linux-5.10.35/arch/x86/entry/common.c:46) 
[14164.578820] entry_SYSCALL_64_after_hwframe (/home/user/shared/linux-5.10.35/arch/x86/entry/entry_64.S:127) 
[14164.578821] RIP: 0033:0x7f417e18d41a
[14164.578824] Code: ff ff ff c3 0f 1f 40 00 48 8b 15 71 ea 0b 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b7 0f 1f 00 49 89 ca b8 36 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 46 ea 0b 00 f7 d8 64 89 01 48
All code
========
   0:	ff                   	(bad)  
   1:	ff                   	(bad)  
   2:	ff c3                	inc    %ebx
   4:	0f 1f 40 00          	nopl   0x0(%rax)
   8:	48 8b 15 71 ea 0b 00 	mov    0xbea71(%rip),%rdx        # 0xbea80
   f:	f7 d8                	neg    %eax
  11:	64 89 02             	mov    %eax,%fs:(%rdx)
  14:	48 c7 c0 ff ff ff ff 	mov    $0xffffffffffffffff,%rax
  1b:	eb b7                	jmp    0xffffffffffffffd4
  1d:	0f 1f 00             	nopl   (%rax)
  20:	49 89 ca             	mov    %rcx,%r10
  23:	b8 36 00 00 00       	mov    $0x36,%eax
  28:	0f 05                	syscall 
  2a:*	48 3d 01 f0 ff ff    	cmp    $0xfffffffffffff001,%rax		<-- trapping instruction
  30:	73 01                	jae    0x33
  32:	c3                   	retq   
  33:	48 8b 0d 46 ea 0b 00 	mov    0xbea46(%rip),%rcx        # 0xbea80
  3a:	f7 d8                	neg    %eax
  3c:	64 89 01             	mov    %eax,%fs:(%rcx)
  3f:	48                   	rex.W

Code starting with the faulting instruction
===========================================
   0:	48 3d 01 f0 ff ff    	cmp    $0xfffffffffffff001,%rax
   6:	73 01                	jae    0x9
   8:	c3                   	retq   
   9:	48 8b 0d 46 ea 0b 00 	mov    0xbea46(%rip),%rcx        # 0xbea56
  10:	f7 d8                	neg    %eax
  12:	64 89 01             	mov    %eax,%fs:(%rcx)
  15:	48                   	rex.W
[14164.578824] RSP: 002b:00007fff0d4e23a8 EFLAGS: 00000246 ORIG_RAX: 0000000000000036
[14164.585011] RAX: ffffffffffffffda RBX: 00005567338baba0 RCX: 00007f417e18d41a
[14164.585682] RDX: 0000000000000003 RSI: 0000000000000006 RDI: 000000000000000f
[14164.586344] RBP: 00005567338998c0 R08: 0000000000000004 R09: 0000000000000000
[14164.587004] R10: 00007fff0d4e23b4 R11: 0000000000000246 R12: 00007f417af85498
[14164.587670] R13: 00005567338aefb8 R14: 0000556733848ec0 R15: 0000000000000000
[14164.588337] ---[ end trace fd1b4ccbbda7de67 ]---
[14164.588778] BUG: kernel NULL pointer dereference, address: 0000000000000000
[14164.589432] #PF: supervisor read access in kernel mode
[14164.589916] #PF: error_code(0x0000) - not-present page
[14164.590398] PGD 0 P4D 0
[14164.590644] Oops: 0000 [#1] SMP PTI
[14164.590976] CPU: 3 PID: 482 Comm: nginx Tainted: G        W  OE     5.10.35 #2
[14164.591652] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.14.0-2 04/01/2014
[14164.592412] RIP: 0010:tfw_hpack_hdr_inplace (/home/user/tempesta/fw/hpack.c:3654) tempesta_fw
[14164.593014] Code: 00 00 48 89 84 24 90 00 00 00 4c 89 e2 48 8d b4 24 80 00 00 00 48 89 ef e8 32 72 01 00 85 c0 0f 85 c0 00 00 00 48 8b 44 24 20 <4c> 8b 20 4c 89 e2 48 89 e6 48 89 ef e8 26 72 01 00 85 c0 74 34 e9
All code
========
   0:	00 00                	add    %al,(%rax)
   2:	48 89 84 24 90 00 00 	mov    %rax,0x90(%rsp)
   9:	00 
   a:	4c 89 e2             	mov    %r12,%rdx
   d:	48 8d b4 24 80 00 00 	lea    0x80(%rsp),%rsi
  14:	00 
  15:	48 89 ef             	mov    %rbp,%rdi
  18:	e8 32 72 01 00       	callq  0x1724f
  1d:	85 c0                	test   %eax,%eax
  1f:	0f 85 c0 00 00 00    	jne    0xe5
  25:	48 8b 44 24 20       	mov    0x20(%rsp),%rax
  2a:*	4c 8b 20             	mov    (%rax),%r12		<-- trapping instruction
  2d:	4c 89 e2             	mov    %r12,%rdx
  30:	48 89 e6             	mov    %rsp,%rsi
  33:	48 89 ef             	mov    %rbp,%rdi
  36:	e8 26 72 01 00       	callq  0x17261
  3b:	85 c0                	test   %eax,%eax
  3d:	74 34                	je     0x73
  3f:	e9                   	.byte 0xe9

Code starting with the faulting instruction
===========================================
   0:	4c 8b 20             	mov    (%rax),%r12
   3:	4c 89 e2             	mov    %r12,%rdx
   6:	48 89 e6             	mov    %rsp,%rsi
   9:	48 89 ef             	mov    %rbp,%rdi
   c:	e8 26 72 01 00       	callq  0x17237
  11:	85 c0                	test   %eax,%eax
  13:	74 34                	je     0x49
  15:	e9                   	.byte 0xe9
[14164.594757] RSP: 0018:ffffc90000120878 EFLAGS: 00010246
[14164.595290] RAX: 0000000000000000 RBX: ffff8880350f1020 RCX: ffffc9000012091c
[14164.595977] RDX: 0000000000000009 RSI: ffffc9000012091c RDI: ffff888022328070
[14164.596641] RBP: ffff8880350f1130 R08: ffffc9000012091c R09: 0000000000000001
[14164.597306] R10: 0000000000000001 R11: 0000000000000000 R12: ffff88802232810d
[14164.597978] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[14164.598640] FS:  00007f417e08cb80(0000) GS:ffff88817bd80000(0000) knlGS:0000000000000000
[14164.599394] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[14164.600005] CR2: 0000000000000000 CR3: 0000000124da2005 CR4: 0000000000370ee0
[14164.600663] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[14164.601320] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[14164.601975] Call Trace:
[14164.602210]  <IRQ>
[14164.602416] tfw_hpack_encode (/home/user/tempesta/fw/hpack.c:3822) tempesta_fw
[14164.602914] tfw_h2_resp_adjust_fwd (/home/user/tempesta/fw/http.c:4869) tempesta_fw
[14164.603471] ? tfw_h2_resp_adjust_fwd (/home/user/tempesta/fw/http.c:5605) tempesta_fw
[14164.604026] tfw_http_resp_cache_cb (/home/user/tempesta/fw/http.c:5613) tempesta_fw
[14164.604549] tfw_cache_add (/home/user/tempesta/fw/cache.c:1889) tempesta_fw
[14164.605002] ? tfw_h2_resp_adjust_fwd (/home/user/tempesta/fw/http.c:5605) tempesta_fw
[14164.605557] ? tfw_h2_resp_adjust_fwd (/home/user/tempesta/fw/http.c:5605) tempesta_fw
[14164.606110] tfw_cache_do_action (/home/user/tempesta/fw/cache.c:2372) tempesta_fw
[14164.606608] tfw_cache_process (/home/user/tempesta/fw/cache.c:2431) tempesta_fw
[14164.607100] ? tfw_gfsm_move (/home/user/tempesta/fw/gfsm.c:254) tempesta_fw
[14164.607568] tfw_http_resp_cache (/home/user/tempesta/fw/http.c:5795) tempesta_fw
[14164.608083] tfw_http_resp_process (/home/user/tempesta/fw/http.c:6038) tempesta_fw
[14164.608615] tfw_http_msg_process_generic (/home/user/tempesta/fw/http.c:6104 (discriminator 1)) tempesta_fw
[14164.609201] tfw_http_msg_process (/home/user/tempesta/fw/http.c:6131 (discriminator 1)) tempesta_fw
[14164.609706] __gfsm_fsm_exec (/home/user/tempesta/fw/gfsm.c:186 (discriminator 2)) tempesta_fw
[14164.610176] ? ss_skb_queue_coalesce_tail (/home/user/tempesta/fw/ss_skb.c:1345) tempesta_fw
[14164.610753] tfw_gfsm_dispatch (/home/user/tempesta/fw/gfsm.c:205) tempesta_fw
[14164.611240] tfw_connection_recv (/home/user/tempesta/fw/connection.c:135) tempesta_fw
[14164.611740] ss_tcp_process_skb (/home/user/tempesta/fw/sock.c:794 (discriminator 3)) tempesta_fw
[14164.612249] ss_tcp_process_data (/home/user/tempesta/fw/sock.c:851 (discriminator 3)) tempesta_fw
[14164.612756] ss_tcp_data_ready (/home/user/tempesta/fw/sock.c:900) tempesta_fw
[14164.613239] tcp_data_queue (/home/user/shared/linux-5.10.35/net/ipv4/tcp_input.c:4984) 
[14164.613601] tcp_rcv_established (/home/user/shared/linux-5.10.35/./include/linux/skbuff.h:1833 /home/user/shared/linux-5.10.35/./include/net/tcp.h:1802 /home/user/shared/linux-5.10.35/./include/net/tcp.h:1875 /home/user/shared/linux-5.10.35/net/ipv4/tcp_input.c:5379 /home/user/shared/linux-5.10.35/net/ipv4/tcp_input.c:5874) 
[14164.614002] tcp_v4_do_rcv (/home/user/shared/linux-5.10.35/net/ipv4/tcp_ipv4.c:1721) 
[14164.614356] tcp_v4_rcv (/home/user/shared/linux-5.10.35/net/ipv4/tcp_ipv4.c:2061) 
[14164.614686] ip_protocol_deliver_rcu (/home/user/shared/linux-5.10.35/net/ipv4/ip_input.c:204 (discriminator 1)) 
[14164.615112] ip_local_deliver_finish (/home/user/shared/linux-5.10.35/./include/linux/rcupdate.h:72 /home/user/shared/linux-5.10.35/./include/linux/rcupdate.h:702 /home/user/shared/linux-5.10.35/net/ipv4/ip_input.c:232) 
[14164.615528] __netif_receive_skb_one_core (/home/user/shared/linux-5.10.35/net/core/dev.c:5361 (discriminator 4)) 
[14164.615983] process_backlog (/home/user/shared/linux-5.10.35/./include/linux/rcupdate.h:72 /home/user/shared/linux-5.10.35/./include/linux/rcupdate.h:702 /home/user/shared/linux-5.10.35/net/core/dev.c:6373) 
[14164.616404] net_rx_action (/home/user/shared/linux-5.10.35/./arch/x86/include/asm/jump_label.h:25 /home/user/shared/linux-5.10.35/./include/linux/jump_label.h:200 /home/user/shared/linux-5.10.35/./include/trace/events/napi.h:14 /home/user/shared/linux-5.10.35/net/core/dev.c:6817 /home/user/shared/linux-5.10.35/net/core/dev.c:6886) 
[14164.616759] __do_softirq (/home/user/shared/linux-5.10.35/./arch/x86/include/asm/jump_label.h:25 /home/user/shared/linux-5.10.35/./include/linux/jump_label.h:200 /home/user/shared/linux-5.10.35/./include/trace/events/irq.h:142 /home/user/shared/linux-5.10.35/kernel/softirq.c:304) 
[14164.617095] asm_call_irq_on_stack (/home/user/shared/linux-5.10.35/arch/x86/entry/entry_64.S:796) 
[14164.617493]  </IRQ>
[14164.617697] do_softirq_own_stack (/home/user/shared/linux-5.10.35/./arch/x86/include/asm/irq_stack.h:27 /home/user/shared/linux-5.10.35/./arch/x86/include/asm/irq_stack.h:77 /home/user/shared/linux-5.10.35/arch/x86/kernel/irq_64.c:77) 
[14164.618088] do_softirq (/home/user/shared/linux-5.10.35/kernel/softirq.c:351 /home/user/shared/linux-5.10.35/kernel/softirq.c:338) 
[14164.618402] __local_bh_enable_ip (/home/user/shared/linux-5.10.35/kernel/softirq.c:196) 
[14164.618792] ip_finish_output2 (/home/user/shared/linux-5.10.35/net/ipv4/ip_output.c:240) 
[14164.619179] __ip_queue_xmit (/home/user/shared/linux-5.10.35/./include/net/dst.h:443 /home/user/shared/linux-5.10.35/net/ipv4/ip_output.c:126 /home/user/shared/linux-5.10.35/net/ipv4/ip_output.c:532) 
[14164.619549] __tcp_transmit_skb (/home/user/shared/linux-5.10.35/net/ipv4/tcp_output.c:1408 (discriminator 4)) 
[14164.619941] tcp_write_xmit (/home/user/shared/linux-5.10.35/net/ipv4/tcp_output.c:2754) 
[14164.620319] ? ext4_read_block_bitmap_nowait+0x192/0x630 ext4
[14164.620872] __tcp_push_pending_frames (/home/user/shared/linux-5.10.35/net/ipv4/tcp_output.c:2934) 
[14164.621306] tcp_setsockopt (/home/user/shared/linux-5.10.35/net/ipv4/tcp.c:3270 /home/user/shared/linux-5.10.35/net/ipv4/tcp.c:3213 /home/user/shared/linux-5.10.35/net/ipv4/tcp.c:3365) 
[14164.621672] __sys_setsockopt (/home/user/shared/linux-5.10.35/net/socket.c:2140) 
[14164.622045] __x64_sys_setsockopt (/home/user/shared/linux-5.10.35/net/socket.c:2149 /home/user/shared/linux-5.10.35/net/socket.c:2146 /home/user/shared/linux-5.10.35/net/socket.c:2146) 
[14164.622438] do_syscall_64 (/home/user/shared/linux-5.10.35/arch/x86/entry/common.c:46) 
[14164.622776] entry_SYSCALL_64_after_hwframe (/home/user/shared/linux-5.10.35/arch/x86/entry/entry_64.S:127) 
[14164.623259] RIP: 0033:0x7f417e18d41a
[14164.623596] Code: ff ff ff c3 0f 1f 40 00 48 8b 15 71 ea 0b 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b7 0f 1f 00 49 89 ca b8 36 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 46 ea 0b 00 f7 d8 64 89 01 48
All code
========
   0:	ff                   	(bad)  
   1:	ff                   	(bad)  
   2:	ff c3                	inc    %ebx
   4:	0f 1f 40 00          	nopl   0x0(%rax)
   8:	48 8b 15 71 ea 0b 00 	mov    0xbea71(%rip),%rdx        # 0xbea80
   f:	f7 d8                	neg    %eax
  11:	64 89 02             	mov    %eax,%fs:(%rdx)
  14:	48 c7 c0 ff ff ff ff 	mov    $0xffffffffffffffff,%rax
  1b:	eb b7                	jmp    0xffffffffffffffd4
  1d:	0f 1f 00             	nopl   (%rax)
  20:	49 89 ca             	mov    %rcx,%r10
  23:	b8 36 00 00 00       	mov    $0x36,%eax
  28:	0f 05                	syscall 
  2a:*	48 3d 01 f0 ff ff    	cmp    $0xfffffffffffff001,%rax		<-- trapping instruction
  30:	73 01                	jae    0x33
  32:	c3                   	retq   
  33:	48 8b 0d 46 ea 0b 00 	mov    0xbea46(%rip),%rcx        # 0xbea80
  3a:	f7 d8                	neg    %eax
  3c:	64 89 01             	mov    %eax,%fs:(%rcx)
  3f:	48                   	rex.W

Code starting with the faulting instruction
===========================================
   0:	48 3d 01 f0 ff ff    	cmp    $0xfffffffffffff001,%rax
   6:	73 01                	jae    0x9
   8:	c3                   	retq   
   9:	48 8b 0d 46 ea 0b 00 	mov    0xbea46(%rip),%rcx        # 0xbea56
  10:	f7 d8                	neg    %eax
  12:	64 89 01             	mov    %eax,%fs:(%rcx)
  15:	48                   	rex.W
[14164.625299] RSP: 002b:00007fff0d4e23a8 EFLAGS: 00000246 ORIG_RAX: 0000000000000036
[14164.625994] RAX: ffffffffffffffda RBX: 00005567338baba0 RCX: 00007f417e18d41a
[14164.626653] RDX: 0000000000000003 RSI: 0000000000000006 RDI: 000000000000000f
[14164.627315] RBP: 00005567338998c0 R08: 0000000000000004 R09: 0000000000000000
[14164.627974] R10: 00007fff0d4e23b4 R11: 0000000000000246 R12: 00007f417af85498
[14164.628633] R13: 00005567338aefb8 R14: 0000556733848ec0 R15: 0000000000000000
[14164.629295] Modules linked in: tempesta_fw(OE) tempesta_db(OE) tempesta_tls(OE) tempesta_lib(OE) sha256_ssse3(E) sha512_ssse3(E) sha512_generic(E) uinput(E) intel_rapl_msr(E) intel_rapl_common(E) intel_pmc_core_pltdrv(E) intel_pmc_core(E) kvm_intel(E) snd_hda_codec_generic(E) kvm(E) ledtrig_audio(E) irqbypass(E) snd_hda_intel(E) 9p(E) snd_intel_dspcfg(E) fscache(E) soundwire_intel(E) ghash_clmulni_intel(E) soundwire_generic_allocation(E) snd_soc_core(E) aesni_intel(E) libaes(E) rfkill(E) crypto_simd(E) snd_compress(E) cryptd(E) soundwire_cadence(E) glue_helper(E) rapl(E) snd_hda_codec(E) snd_hda_core(E) snd_hwdep(E) 9pnet_virtio(E) soundwire_bus(E) iTCO_wdt(E) joydev(E) intel_pmc_bxt(E) snd_pcm(E) iTCO_vendor_support(E) 9pnet(E) watchdog(E) serio_raw(E) snd_timer(E) virtio_console(E) sg(E) virtio_balloon(E) snd(E) evdev(E) soundcore(E) pcspkr(E) qemu_fw_cfg(E) configfs(E) fuse(E) virtio_rng(E) rng_core(E) ip_tables(E) x_tables(E) autofs4(E) ext4(E) crc32c_generic(E) crc16(E) mbcache(E)
[14164.629328]  jbd2(E) hid_generic(E) usbhid(E) hid(E) sr_mod(E) cdrom(E) virtio_blk(E) virtio_net(E) net_failover(E) failover(E) qxl(E) drm_ttm_helper(E) ttm(E) crct10dif_pclmul(E) crct10dif_common(E) crc32_pclmul(E) ahci(E) crc32c_intel(E) libahci(E) drm_kms_helper(E) libata(E) xhci_pci(E) cec(E) psmouse(E) xhci_hcd(E) drm(E) scsi_mod(E) i2c_i801(E) i2c_smbus(E) lpc_ich(E) virtio_pci(E) usbcore(E) virtio_ring(E) virtio(E) button(E) [last unloaded: tempesta_lib]
[14164.641481] CR2: 0000000000000000
[14164.641804] ---[ end trace fd1b4ccbbda7de68 ]---
[14164.642246] RIP: 0010:tfw_hpack_hdr_inplace (/home/user/tempesta/fw/hpack.c:3654) tempesta_fw
[14164.642848] Code: 00 00 48 89 84 24 90 00 00 00 4c 89 e2 48 8d b4 24 80 00 00 00 48 89 ef e8 32 72 01 00 85 c0 0f 85 c0 00 00 00 48 8b 44 24 20 <4c> 8b 20 4c 89 e2 48 89 e6 48 89 ef e8 26 72 01 00 85 c0 74 34 e9
All code
========
   0:	00 00                	add    %al,(%rax)
   2:	48 89 84 24 90 00 00 	mov    %rax,0x90(%rsp)
   9:	00 
   a:	4c 89 e2             	mov    %r12,%rdx
   d:	48 8d b4 24 80 00 00 	lea    0x80(%rsp),%rsi
  14:	00 
  15:	48 89 ef             	mov    %rbp,%rdi
  18:	e8 32 72 01 00       	callq  0x1724f
  1d:	85 c0                	test   %eax,%eax
  1f:	0f 85 c0 00 00 00    	jne    0xe5
  25:	48 8b 44 24 20       	mov    0x20(%rsp),%rax
  2a:*	4c 8b 20             	mov    (%rax),%r12		<-- trapping instruction
  2d:	4c 89 e2             	mov    %r12,%rdx
  30:	48 89 e6             	mov    %rsp,%rsi
  33:	48 89 ef             	mov    %rbp,%rdi
  36:	e8 26 72 01 00       	callq  0x17261
  3b:	85 c0                	test   %eax,%eax
  3d:	74 34                	je     0x73
  3f:	e9                   	.byte 0xe9

Code starting with the faulting instruction
===========================================
   0:	4c 8b 20             	mov    (%rax),%r12
   3:	4c 89 e2             	mov    %r12,%rdx
   6:	48 89 e6             	mov    %rsp,%rsi
   9:	48 89 ef             	mov    %rbp,%rdi
   c:	e8 26 72 01 00       	callq  0x17237
  11:	85 c0                	test   %eax,%eax
  13:	74 34                	je     0x49
  15:	e9                   	.byte 0xe9
[14164.644567] RSP: 0018:ffffc90000120878 EFLAGS: 00010246
[14164.645053] RAX: 0000000000000000 RBX: ffff8880350f1020 RCX: ffffc9000012091c
[14164.645712] RDX: 0000000000000009 RSI: ffffc9000012091c RDI: ffff888022328070
[14164.646372] RBP: ffff8880350f1130 R08: ffffc9000012091c R09: 0000000000000001
[14164.647031] R10: 0000000000000001 R11: 0000000000000000 R12: ffff88802232810d
[14164.647691] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[14164.648351] FS:  00007f417e08cb80(0000) GS:ffff88817bd80000(0000) knlGS:0000000000000000
[14164.649097] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[14164.649706] CR2: 0000000000000000 CR3: 0000000124da2005 CR4: 0000000000370ee0
[14164.650376] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[14164.651042] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[14164.651712] Kernel panic - not syncing: Fatal exception in interrupt
[14164.652413] Kernel Offset: disabled
[14164.652745] ---[ end Kernel panic - not syncing: Fatal exception in interrupt ]---

The text was updated successfully, but these errors were encountered:

…#1549)

Fixes #1549

Closes the issue #1549 "Kernel panic for cached HTTP/2 response with empty header value"

byko3y added bug h2 labels Jan 12, 2022

krizhanovsky added the crucial label Jan 12, 2022

krizhanovsky modified the milestones: 0.11 - HTTP/2 perf, 0.7 - HTTP/2, fast in-kernel TLS Jan 12, 2022

krizhanovsky assigned byko3y Jan 13, 2022

byko3y added a commit that referenced this issue Jan 17, 2022

Kernel panic for cached response with empty header value (fixes #1549)

8c5d85a

byko3y added a commit that referenced this issue Jan 17, 2022

Kernel panic for cached HTTP/2 response with empty header value (fixes …

c54916d

…#1549)

byko3y added a commit that referenced this issue Jan 17, 2022

Kernel panic for cached HTTP/2 response with empty header value

134a9fa

Fixes #1549

byko3y mentioned this issue Jan 17, 2022

Kernel panic for cached HTTP/2 response with empty header value #1551

Merged

byko3y closed this as completed in #1551 Feb 8, 2022

byko3y added a commit that referenced this issue Feb 8, 2022

Fixed handling of zero-length value by tfw_hpack_hdr_inplace()

25a420c

Closes the issue #1549 "Kernel panic for cached HTTP/2 response with empty header value"

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Kernel panic on simple HTTP/2 request #1549

Kernel panic on simple HTTP/2 request #1549

byko3y commented Jan 12, 2022 •

edited

Loading

Kernel panic on simple HTTP/2 request #1549

Kernel panic on simple HTTP/2 request #1549

Comments

byko3y commented Jan 12, 2022 • edited Loading

Scope

byko3y commented Jan 12, 2022 •

edited

Loading