[Bug] Overflow when subtracting durations

[MikeIvanichev]

Ran into an issue where the re-state server hard crashed, this is all the information I was able to salvage:

Version: Restate Server 1.2.0 (debug) (badad0c x86_64-unknown-linux-gnu 2025-02-14)
Config: default
Hardware info:

WSL, Arch

Architecture:            x86_64
  CPU op-mode(s):        32-bit, 64-bit
  Address sizes:         48 bits physical, 48 bits virtual
  Byte Order:            Little Endian
CPU(s):                  12
  On-line CPU(s) list:   0-11
Vendor ID:               AuthenticAMD
  Model name:            AMD Ryzen 5 5600H with Radeon Graphics
    CPU family:          25
    Model:               80
    Thread(s) per core:  2
    Core(s) per socket:  6
    Socket(s):           1
    Stepping:            0
    BogoMIPS:            6587.46
    Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rd
                         rand hypervisor lahf_lm cmp_legacy svm cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves clzero xsaveerptr arat npt nrip_save tsc
                         _scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold v_vmsave_vmload umip vaes vpclmulqdq rdpid fsrm

Logs:

restate-1  | on rt:pp-23
restate-1  | thread 'rs:worker-2' panicked at /rustc/e71f9a9a98b0faf423844bf0ba7438f29dc27d58/library/core/src/time.rs:1138:31:
restate-1  | overflow when subtracting durations
restate-1  | note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

restate-1  | on rt:pp-18
restate-1  | 2025-02-27T02:59:00.475284Z ERROR tracing_panic
restate-1  |   A panic occurred
restate-1  |     panic.payload: "overflow when subtracting durations"
restate-1  |     panic.location: "/rustc/e71f9a9a98b0faf423844bf0ba7438f29dc27d58/library/core/src/time.rs:1138:31"
restate-1  |     panic.backtrace: disabled backtrace
restate-1  |     panic.note: "run with RUST_BACKTRACE=1 environment variable to display a backtrace"

[tillrohrmann]

Thanks a lot for reporting this issue @MikeIvanichev. This looks like a bug we need to fix. I am looking into this now. Did you manage to reproduce the problem so that in case we have a fix you could help us trying it out?

[tillrohrmann]

Did you run any service when the server crashed? If yes, was the service doing time based operations (e.g. a delayed call, a sleep or something like this)?

[tillrohrmann]

I have found two sources that can explain the overflow. I'll fix them and include them in the upcoming 1.2.1 release that I will create today. Once this is out, you can give it a spin.