Implement conventional log rotation instead of current approach

[akamensky]

Proposal

Most systems use fixed file for logs, that is rotated in a rolling fashion. That is if original log file is some.log, then once it is rotated (by timer or by size) the old file moved (or copied) to some.log.${suffix} where suffix is either index or timestamp.

I.e. with indexes it would be:

some.log -- active log file
some.log.0 -- freshly rotated log file
some.log.1
...
some.log.10 -- oldest/subject to be removed in next cleanup

That creates a way for other applications to always refer to active log in a simple way (just as tail -F would).

What Nomad implemented is a different approach, which does not align with established conventions:

some.log.0 -- oldest
some.log.1
...
some.log.N -- current active log

Nomad current implementation does not allow any tools (sidecars) to track the current log, which may be needed.

Use-cases

In our use case we are moving legacy application to Nomad, this legacy application writes some errors in the log. We have also legacy sidecar for it that will tail the log and match error|crit, which will then be sent as an alert into monitoring system.

Attempted Solutions

• Rewriting sidecar service to use Nomad's logic -- there is no one left who knows this sidecars code, so that would be a big undertaking, there is also no notification that the current file won't be written to anymore, the only way to find that out is to
• Rewriting sidecar service to use Nomad API to streams the logs -- that will be one very very long request (essentially application lifetime), and is not an ideal approach (ideal would be to actually tail the logs).
• Searched for ANY way to send logs to beats/gelf receivers -- Nomad does not support any of that (not sure why), but can only be done by either using a sidecar inside (which again will need to tail log with this odd rotation) or send all logs for all jobs using filebeat's nomad discovery (which is a huge overkill).

PS: renamed the issue as the title unintentionally sounded passive aggressive (non native speaker here)

[tgross]

Hi @akamensky! This is something I've been hacking on on-and-off for the last little while but we don't quite have a design we're happy with yet. This work-in-progress readme contains some of the ideas. In the meantime, we've also recently merged #17196 and #16962 which will allow you to bypass Nomad's own logging system and set up whatever you'd like.

Thanks for opening this issue! I'm going to close this as a duplicate of #9211 and link from there to here to add context to that one.

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.