You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
1s later, our app's memory usage jumps from 0.033 GB to 1.32 GB according to netdata.
Memory remains at this level until the next time we receive Error::Protocol at which point it jumps again. Every occurrence seems to produce a memory increase of ~1.2 GB. This repeats on each Protocol error until OOM kill.
This is the protocol error msg:
unknown message type: '\\0'
I captured this memory flamegraph with heaptrack:
This implies that all of the 1.2GB of memory is being allocated by the PgConnection and not released. When we encounter a protocol error, we drop the PgListener instance and create a new one. Perhaps the Drop implementation of PgListener is failing to cleanup as the PgListener is in some sort of invalid state due to the protocol error?
Minimal Reproduction
Due to the protocol error, unable to reproduce minimally.
let format = BackendMessageFormat::try_from_u8(header.get_u8())?;
let size = (header.get_u32() - 4)asusize;
let contents = self.inner.read(size).await?;
Then a subsequent call reads that data as if it were the header of a new message, reads some arbitrary bytes and interprets them as the size of the message it should read, and attempts to allocate and read a message of that size.
I'm not certain about why the memory is leaked. However, I think it's because PgListener uses a PoolConnection<Postgres> internally, and drops that connection if there's an error, but without closing it first. So PoolConnection attempts to return itself to the pool, and spawns a task to flush the data. This could hang forever if the connection is in an invalid state, making it appear to be a memory leak (what I've been calling "live" leaked memory, because it's still tracked somewhere).
Bug Description
2024-08-08T19:10:15.458367Z
:PgListener
returnsError::Protocol(msg)
.Error::Protocol
at which point it jumps again. Every occurrence seems to produce a memory increase of ~1.2 GB. This repeats on each Protocol error until OOM kill.This is the protocol error
msg
:I captured this memory flamegraph with
heaptrack
:This implies that all of the 1.2GB of memory is being allocated by the PgConnection and not released. When we encounter a protocol error, we drop the
PgListener
instance and create a new one. Perhaps theDrop
implementation ofPgListener
is failing to cleanup as thePgListener
is in some sort of invalid state due to the protocol error?Minimal Reproduction
Due to the protocol error, unable to reproduce minimally.
Info
"chrono",
"postgres",
"runtime-tokio",
"time",
"tls-rustls",
"bigdecimal",
]
rustc --version
: rustc 1.76.0 (07dca489a 2024-02-04)The text was updated successfully, but these errors were encountered: