severe memory leak with piped cat

[oconnor663]

Run the following in an interactive interpreter:

>>> sh.head(sh.cat('/dev/urandom', _piped=True), '-c', '10')                                       
b'\xc9UJh\x8e\xe7?T\xa2\x02'

Now without closing that interpreter, open top and sort by memory usage. There you'll see the Python process steadily climbing to 100%. Wait long enough and it'll lock up the machine. If you sort by CPU, you'll see that the cat process is still running.

When I try the same with /dev/zero, the call doesn't actually return. top indicates that Python is taking 100% CPU in addition to the same memory leak as above.

[amoffat]

A process internally stores all of the data the flows through it. Otherwise you would never be able to see the output of a command, since these aren't real function calls with return values, they are evaluated objects.

There a lesser known option to _piped that connects two processes directly via their file descriptors. Shame on me for not merging it into the docs yet, but it should prevent your memory from growing: https://github.com/amoffat/sh/pull/243/files

[oconnor663]

Cool. What's the case where I'd want _piped=True but not _piped="direct"?

[amoffat]

There may not be one. _piped=True came first when the piping mechanism was introduced, and _piped="direct" came years later. It kind of has the benefit of retaining the data that flowed through it, for later processing, but I've personally never taken advantage of it. It seems that more and more, people just want to pipe quickly, without memory overhead, so it may be that _piped=True should work like _piped="direct" under the hood.

[oconnor663]

Even if we keep the copying, I hope there would be a way to do something like what SIGPIPE does and stop the copying subprocess once there's nothing left for it to copy to.