[pantsd] Investigate removing forking from DaemonPantsRunner

[blorente]

There are some issues around fork safety (#7380, #6714, possibly duped by #7323), which make it worth considering how hard it would be to remove forking from that class.

There are possible workarounds for both issues, but we have no guarantees that those workarounds will continue to work in newer versions of OSX.

Therefore, it would be good to list blockers for removing forking from DaemonPantsRunner.

[cosmicexplorer]

It's very sad to me that forking isn't a viable strategy in 2019, but moving to v2 to eventually circumvent the need for forking makes me happy enough.

The DaemonPantsRunner would need to avoid exiting, but since we've parameterized exiting into the DaemonExiter, this shouldn't be incredibly hard, even with signal handling.

If we removed forking, would we end up running pants synchronously? Would we have to have some non-python way of having concurrent pailgun requests block?

[ity]

Two questions I am trying to answer in parallel (touché):

1. Why do we fork? Pros/cons?
2. How much global state do we have to manage if we weren’t to fork?

Aspects to consider to implement no-fork:

• Any Blockers?
• Possible performance loss
• Elegant way to process tasks in parallel (fork) vs the alternative
• Note: can only run 1 pants process at a time(because of the GIL)
• Estimate Effort to implement

[stuhood]

Branch from pairing session: https://github.com/twitter/pants/pull/new/twitter/pantsd-without-forking

[ity]

Some of the workarounds and investigation results to make pantsd work without forking:

1. Subsystems (are reset already, so this might not be an issue after all)
2. Exiter (removed the call to it)
3. ExceptionSink (removed the call to it)
4. Logger (maximum recursion issue as its initialized (setup_logging) in the daemon and then subsequently in LocalPantsRunner, to bypass the issue disabled the initialization of the logger in LocalPantsRunner but this would have to be addressed)
5. Concurrency:
   a) Multiple clients at the same time
   b) People piping pants into pants
   c) can conditionally disable pantsd for the 2nd run
6. Memory leaks
   a) cyclic garbage collection (read Python documentation on this)
   b) needs further investigation

[blorente]

I think this can be closed now, we opened follow-up issues.