Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Crash on TLS stress test #1146

Closed
vankoven opened this issue Jan 11, 2019 · 2 comments · Fixed by #1160
Closed

Crash on TLS stress test #1146

vankoven opened this issue Jan 11, 2019 · 2 comments · Fixed by #1160
Assignees
@krizhanovsky
Labels
bug crucial
Milestone
0.6 Tempesta TLS

Comments

@vankoven
Copy link
Contributor

Just caught on VM during tls.test_tls_stress.StressTls.test_tls test on CI. Tempesta is at current master.

[  445.469841] Start test: tls.test_tls_stress.StressTls.test_tls
[  445.779673] [tdb] Start Tempesta DB
[  445.808655] [tdb] Opened table /opt/tempesta/db/filter.tdb: size=16777216 rec_size=20 base=ffff9c55eac00000
[  445.962656] [tdb] Opened table /opt/tempesta/db/cache.tdb: size=268435456 rec_size=0 base=ffff9c55dac00000
[  446.054149] ------------[ cut here ]------------
[  446.057006] kernel BUG at /root/tempesta/tempesta/tls/ttls.c:371!
[  446.060664] invalid opcode: 0000 [#1] SMP PTI
[  446.062998] Modules linked in: tempesta_fw(O) tempesta_db(O) tempesta_tls(O) tempesta_lib(O) fuse ata_generic intel_rapl sb_edac crct10dif_pclmul crc32_pclmul cirrus joydev xen_netfront ghash_clmulni_intel ttm drm_kms_helper intel_rapl_perf ata_piix psmouse pcspkr libata drm i2c_piix4 scsi_mod floppy button ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 crc32c_generic fscrypto ecb crc32c_intel xen_blkfront aesni_intel aes_x86_64 crypto_simd cryptd glue_helper evdev serio_raw [last unloaded: tempesta_lib]
[  446.081747] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G        W  O    4.14.0-tempesta-amd64 #1 Debian 4.14.32-tfw3-1
[  446.085941] Hardware name: Xen HVM domU, BIOS 4.8.5-pre 06/22/2018
[  446.088896] task: ffffffffac212480 task.stack: ffffffffac200000
[  446.091778] RIP: 0010:ttls_update_checksum+0x43/0x90 [tempesta_tls]
[  446.094748] RSP: 0018:ffff9c560f4039e0 EFLAGS: 00010246
[  446.097397] RAX: 0000000000000000 RBX: ffff9c55a8dbba87 RCX: ffff9c5606a5d248
[  446.100614] RDX: 00000000000000b9 RSI: ffff9c55a8dbba87 RDI: ffff9c5606a5d248
[  446.103816] RBP: 00000000000000b5 R08: ffff9c55a8dbbb3e R09: 00000000000000b5
[  446.107013] R10: 0000000000000000 R11: 0000000000000020 R12: ffff9c560f403a6c
[  446.110199] R13: 0000000000000009 R14: 0000000000000004 R15: 0000000000000004
[  446.113354] FS:  0000000000000000(0000) GS:ffff9c560f400000(0000) knlGS:0000000000000000
[  446.116807] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  446.119620] CR2: 00007ffe2894fefc CR3: 00000000b100a001 CR4: 00000000001606f0
[  446.122783] Call Trace:
[  446.124543]  <IRQ>
[  446.126164]  ttls_recv+0x32e/0x770 [tempesta_tls]
[  446.128549]  ? ttls_decrypt+0x570/0x570 [tempesta_tls]
[  446.131091]  ss_skb_process+0xb3/0x200 [tempesta_fw]
[  446.133584]  tfw_tls_msg_process+0xea/0x370 [tempesta_fw]
[  446.136196]  ? tfw_tls_msg_process+0x6f/0x370 [tempesta_fw]
[  446.138874]  __gfsm_fsm_exec+0x56/0x90 [tempesta_fw]
[  446.141370]  tfw_connection_recv+0x4e/0x70 [tempesta_fw]
[  446.143971]  ? tfw_connection_send+0x30/0x30 [tempesta_fw]
[  446.146651]  ss_tcp_process_data+0x1db/0x440 [tempesta_fw]
[  446.149332]  ss_tcp_data_ready+0x43/0x90 [tempesta_fw]
[  446.151848]  tcp_data_queue+0x4f5/0xc50
[  446.153984]  tcp_rcv_established+0x27c/0x570
[  446.156322]  tcp_v4_do_rcv+0x129/0x1d0
[  446.158476]  tcp_v4_rcv+0x947/0xa50
[  446.160507]  ip_local_deliver_finish+0x9a/0x1c0
[  446.162855]  ip_local_deliver+0x6b/0xe0
[  446.164980]  ? tcp_v4_early_demux+0x112/0x150
[  446.167242]  ? ip_rcv_finish+0x17a/0x400
[  446.169350]  ip_rcv+0x289/0x3c0
[  446.171231]  ? inet_del_offload+0x40/0x40
[  446.173353]  __netif_receive_skb_core+0x84f/0xb30
[  446.175670]  ? process_backlog+0xa3/0x160
[  446.177775]  process_backlog+0xa3/0x160
[  446.179814]  net_rx_action+0x28e/0x3f0
[  446.181813]  ? handle_irq_event+0x47/0x60
[  446.183864]  __do_softirq+0x10f/0x2a8
[  446.185815]  irq_exit+0xae/0xb0
[  446.187592]  xen_evtchn_do_upcall+0x2c/0x40
[  446.189677]  xen_hvm_callback_vector+0x7d/0x90
[  446.191785]  </IRQ>
[  446.193171] RIP: 0010:native_safe_halt+0x2/0x10
[  446.195268] RSP: 0018:ffffffffac203e98 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff0c
[  446.198265] RAX: ffffffffab897e40 RBX: ffffffffac212480 RCX: 0000000000000000
[  446.201090] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[  446.203893] RBP: 0000000000000000 R08: 0000000048a7ae32 R09: ffff9c560264b500
[  446.206702] R10: 0000000000000000 R11: 000001351bff01fb R12: ffffffffac212480
[  446.209513] R13: ffffffffac212480 R14: 0000000000000000 R15: 0000000000000000
[  446.212300]  ? __sched_text_end+0x3/0x3
[  446.214125]  default_idle+0x1a/0xf0
[  446.215838]  do_idle+0x16e/0x1f0
[  446.217471]  cpu_startup_entry+0x6f/0x80
[  446.219308]  start_kernel+0x462/0x482
[  446.221067]  secondary_startup_64+0xa5/0xb0
[  446.222971] Code: 89 fb 48 83 ec 10 48 8b 7f 10 48 83 bf 08 03 00 00 00 74 19 48 81 c7 08 03 00 00 e8 d8 a0 00 eb 85 c0 75 37 48 83 c4 10 5b f3 c3 <0f> 0b 48 81 c7 08 03 00 00 83 78 14 04 48 89 54 24 08 48 89 34
[  446.229926] RIP: ttls_update_checksum+0x43/0x90 [tempesta_tls] RSP: ffff9c560f4039e0
[  446.233024] ---[ end trace c0653e38adc50c2c ]---
[  446.235020] Kernel panic - not syncing: Fatal exception in interrupt
[  446.237733] Kernel Offset: 0x2a200000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
@vankoven vankoven added this to the 0.6 Tempesta TLS milestone Jan 11, 2019
@kazan417
Copy link

Error is about ci is null for some reason.

ttls_update_checksum(TlsCtx *tls, const unsigned char *buf, size_t len)
{
const ttls_ciphersuite_t *ci = tls->xfrm.ciphersuite_info;

Maybe writer of this maens constant pointer to variable, but not pointer to constant.
Then correct code will be
ttls_ciphersuite_t *const ci = tls->xfrm.ciphersuite_info;
Could you correct this in root/tempesta/tempesta/tls/ttls.c and check if it fixes error?

vankoven added a commit to tempesta-tech/tempesta-test that referenced this issue Jan 15, 2019
@vankoven
Temporary blacklist tls.test_tls_stress.StressTls test until tempesta…
f41da3b 
…-tech/tempesta#1146 is fixed
@vankoven
Copy link
Contributor Author

Hi, @kazan417 ,

Thank you for digging into the problem!

Maybe writer of this maens constant pointer to variable, but not pointer to constant.

No, it must be pointer to constant and no error here. tls->xfrm.ciphersuite_info is constant data, which describes cipher suite chosen by the client and the server. The proposed change from pointer to constant to constant pointer affects only on ci type, but not on it's value. The exact reason for having NULL ci here is
NULL tls->xfrm.ciphersuite_info, which means the issue somewhere at the early TLS operations.

vankoven added a commit to tempesta-tech/tempesta-test that referenced this issue Jan 22, 2019
@vankoven
Temporary blacklist tls.test_tls_stress.StressTls test until tempesta…
ed61b0c 
…-tech/tempesta#1146 is fixed
vankoven added a commit to tempesta-tech/tempesta-test that referenced this issue Jan 24, 2019
@vankoven
Revert "Temporary blacklist tls.test_tls_stress.StressTls test until t…
92fda9d 
…empesta-tech/tempesta#1146 is fixed"

This reverts commit ed61b0c.
krizhanovsky added a commit that referenced this issue Jan 24, 2019
@krizhanovsky
Fix #1146: ttls_update_checksum() can be called for chunk of ClientHe…
0bd76fc 
…llo,

while we still don't know the cipher suite. So calculate two checksums
in parallel and copy SHA256 contex if necessary when ClientHello
sets xfrm.ciphersuite_info.

The rest of the patch, besides ttls_update_checksum() changes, is
coding style adjustments.
@krizhanovsky krizhanovsky mentioned this issue Jan 24, 2019
Closed
krizhanovsky added a commit that referenced this issue Jan 26, 2019
@krizhanovsky
Fix #1146: ttls_update_checksum() can be called for chunk of ClientHe…
f7dd088 
…llo,

while we still don't know the cipher suite. So calculate two checksums
in parallel and copy SHA256 contex if necessary when ClientHello
sets xfrm.ciphersuite_info.

The rest of the patch, besides ttls_update_checksum() changes, is
coding style adjustments.
@krizhanovsky krizhanovsky mentioned this issue Jan 27, 2019
Merged
krizhanovsky added a commit that referenced this issue Jan 29, 2019
@krizhanovsky
Merge pull request #1160 from tempesta-tech/ak-1146
0763dae 
Fix #1146 + some coding style cleanups (orig PR #1156)
krizhanovsky added a commit to tempesta-tech/tempesta-test that referenced this issue Jun 23, 2019
@krizhanovsky
tempesta-tech/tempesta#1146 is fixed now, so enable the test
fa531c5
@krizhanovsky krizhanovsky mentioned this issue Jun 23, 2019
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@krizhanovsky krizhanovsky

Labels
bug crucial
Projects
None yet
Milestone
0.6 Tempesta TLS
Development

Successfully merging a pull request may close this issue.

Fix #1146 + some coding style cleanups (orig PR #1156) tempesta-tech/tempesta
3 participants
@krizhanovsky @vankoven @kazan417