[pantsd] Persist logs outside of the workdir

[blorente]

Problem

When users have issues with the pants daemon, the first recommendation is to gather the logs (pantsd.log, exceptions.log...), because that's the best way to diagnose issues.
However, the first instinct of many users when something goes wrong with pants is to run ./pants clean-all and try again, which means that the logs get deleted before we can get them.

Solution

Pass an optional, daemon-invalidating flag, that is a list of locations where the daemon should log things. This would be passed via env-var to the daemon at startup, here.

Then, we should modify the LOGGER in the logging crate, so that it accepts multiple streams of output for the pantsd log. We should output the log directory very obviously whenever we throw an error on the client side.

Follow-ups

Some enhancements that can also be done as part of this or separately:

• Make the pantsd logs pid-indexed, similar to the exception logs. So, per each folder, there would be:
  • A master pantsd.log file, which would have all the logs of pantsd up to that point.
  • One pantsd.<pid>.log file per daemon that was started in with that directory.

Tradeoffs

• This means that we will have potentially spread out logs, and different logfiles might have different contents. For instance, imagine this series of invocations:

$ ./pants --enable-pantsd --daemon-log-locations=['/tmp/a'] goals
$ ./pants --enable-pantsd --daemon-log-locations=['/tmp/a', '/tmp/b'] goals
$ ./pants --enable-pantsd --daemon-log-locations=['/tmp/b'] goals

This could be a bit confusing.

[stuhood]

My feeling is that this might conflate two issues:

1. we would like a pantsd instance's logs to live longer than the workdir in some cases
2. a client should be able to render logs that are specific to the request that they just made

The first thing is covered in #8128 (which this ticket seems to be closest to). The second thing is covered by #7810. AFAICT, those issues don't need to be related at all... it's possible that the reason I think this is because you mention an environment variable above... but if the flag is daemon-invalidating, then there is no need to send it across as an env var, because the daemon will restart if it is changed.

One other thing to watch out for here: directly using locations like ['/tmp/a'] would be problematic because it would cause collisions between daemons in different workspaces. An alternative would be placing things in $buildroot/.logs, but that means a proliferation of .gitignore entries (and annoyances on virtual filesystems). Doing something closer to what #8128 suggests and "just" not-deleting these logs during clean-all (and maybe consolidating them under .pants.d/logs) seems easier?

[blorente]

I had totally forgotten about #8128. This issue is intended to be a duplicate of that. I definitely just want to solve problem number 1 here.

An alternative would be placing things in $buildroot/.logs

How about putting them in a subdirectory, which is the hashed buildroot? So, for ./pants --enable-pantsd --daemon-log-locations=['/tmp/a'] goals, the logs would be somewhere like /tmp/a/123456789ABCDEF/pantsd.<pid>.log. This is not ideal because the user has to rely on us telling them where exactly the logs are stored. A solution to would be to store symlinks to the log directory in .pants.d, so

.pants.d/pantsd -> /tmp/a/123456789ABCDEF/

An alternative would be to not hash the buildroot, and do something like /tmp/a/Users_bescobar_workspace_pants/pantsd.<pid>.log.

I'm slightly weary of making exceptions in clean-all, and pretty unconfident in making changes to what pants stores in the buildroot.

Of the options you proposed, I think I'd go with special-casing clean-all, seems like a minor change.

and maybe consolidating them under .pants.d/logs

Whatever direction we take this in, I love this idea.

[stuhood]

As mentioned on #2857, it might be good for all runtracker state to be able to be relocated. That also potentially relates to ./pants server eventually being hosted out of pantsd rather than it's own process.

[benjyw]

Closing since clean-all no longer exists.