Memory Leak with Pluto Hooks - leading to termination of workers

[schlichtanders]

Hi there,

I am using PlutoHooks and occasionally running into Distributed.ProcessExitedException(3) with a request to restart the notebook.

Looking at the output of Pluto shell I get

Worker 3 terminated.
UNHANDLED TASK ERROR: EOFError: read end of file
Stacktrace:
 [1] (::Base.var"#wait_locked#715")(s::Sockets.TCPSocket, buf::IOBuffer, nb::Int64)
   @ Base ./stream.jl:947
 [2] unsafe_read(s::Sockets.TCPSocket, p::Ptr{UInt8}, nb::UInt64)
   @ Base ./stream.jl:955
 [3] unsafe_read
   @ ./io.jl:761 [inlined]
 [4] unsafe_read(s::Sockets.TCPSocket, p::Base.RefValue{NTuple{4, Int64}}, n::Int64)
   @ Base ./io.jl:760
 [5] read!
   @ ./io.jl:762 [inlined]
 [6] deserialize_hdr_raw
   @ /usr/local/julia/share/julia/stdlib/v1.9/Distributed/src/messages.jl:167 [inlined]
 [7] message_handler_loop(r_stream::Sockets.TCPSocket, w_stream::Sockets.TCPSocket, incoming::Bool)
   @ Distributed /usr/local/julia/share/julia/stdlib/v1.9/Distributed/src/process_messages.jl:172
 [8] process_tcp_streams(r_stream::Sockets.TCPSocket, w_stream::Sockets.TCPSocket, incoming::Bool)
   @ Distributed /usr/local/julia/share/julia/stdlib/v1.9/Distributed/src/process_messages.jl:133
 [9] (::Distributed.var"#103#104"{Sockets.TCPSocket, Sockets.TCPSocket, Bool})()
   @ Distributed ./task.jl:514

I got this in very different contexts, but only when using PlutoHooks so far.
No recording is yet ready, neither a notebook which can reproduce this (it seems random sofar, not sure what is causing it yet). Hence no issue, but a discussion.

Does someone have a clue what is going on?
Any help or hint is highly appreciated.

[schlichtanders]

one time I also got an accompanying error

Worker 4 terminated.
UNHANDLED TASK ERROR: IOError: read: connection reset by peer (ECONNRESET)
Stacktrace:
  [1] wait_readnb(x::Sockets.TCPSocket, nb::Int64)
    @ Base ./stream.jl:410
  [2] (::Base.var"#wait_locked#715")(s::Sockets.TCPSocket, buf::IOBuffer, nb::Int64)
    @ Base ./stream.jl:949
  [3] unsafe_read(s::Sockets.TCPSocket, p::Ptr{UInt8}, nb::UInt64)
    @ Base ./stream.jl:955
  [4] unsafe_read
    @ ./io.jl:761 [inlined]
  [5] unsafe_read(s::Sockets.TCPSocket, p::Base.RefValue{NTuple{4, Int64}}, n::Int64)
    @ Base ./io.jl:760
  [6] read!
    @ ./io.jl:762 [inlined]
  [7] deserialize_hdr_raw
    @ /usr/local/julia/share/julia/stdlib/v1.9/Distributed/src/messages.jl:167 [inlined]
  [8] message_handler_loop(r_stream::Sockets.TCPSocket, w_stream::Sockets.TCPSocket, incoming::Bool)
    @ Distributed /usr/local/julia/share/julia/stdlib/v1.9/Distributed/src/process_messages.jl:172
  [9] process_tcp_streams(r_stream::Sockets.TCPSocket, w_stream::Sockets.TCPSocket, incoming::Bool)
    @ Distributed /usr/local/julia/share/julia/stdlib/v1.9/Distributed/src/process_messages.jl:133
 [10] (::Distributed.var"#103#104"{Sockets.TCPSocket, Sockets.TCPSocket, Bool})()
    @ Distributed ./task.jl:514

[Pangoraw]

How much does the memory increase per cell run? It may not necessarely be a memory leak per say but rather that it is allocating permanent objects (for the unnamed closure in Task for example) on each run.

[schlichtanders]

like 1 MB in 6 runs

(I now also included GC.gc() up front - was forgetting it, the increase is less, but still there and steadily summing up.

[schlichtanders]

using varinfo() as suggested in Zulip it is seen that old modules are not cleaned up. Hence even if I would reduce the memory footprint of the macro, it is only a bit later that I run into OOM.

It would really be necessary to either properly garbage collect these old modules. Or to reuse the workspace without creating new ones every time.

[schlichtanders]

@Pangoraw I found that it is indeed because of a task, but haven't found any help about how to investigate this or fix this.
Do you have some tips what to do about Tasks stacking up?

[schlichtanders]

it turns out I run into a new memory leak from Julia Base. opened an issue there JuliaLang/julia#50345

[schlichtanders]

it turns out I run into a new memory leak from Julia Base. opened an issue there JuliaLang/julia#50345