Modular logging

[jalaziz]

The current logging mechanisms in Janus doesn't leave much room for alternative backends.

While stdout and file logging covers most use cases, there are advanced use cases that would be possible with a plugin-based logging mechanism. For example, logging to systems like Kafka or logstash require scraping the logs from the host. Also, while still possible, handling multi-line logs or converting to structured logs is non-trivial.

It would be great if Janus could adopt a modular architecture for logging that allows custom logging backends, much like it currently does for event handlers.

[lminiero]

That's an interesting idea, but I'm a bit concerned about performance. Right now, all we have to worry about is some local I/O. The moment plugins appear to do more complex things, we risk dragging the server down as it will have to wait for a print operation to get somewhere before it can move on. I guess this would be up to plugins to implement properly, but it still is a concern.

There's also a problem in configuration and command line settings. Right now we have command line properties to log to stdout, or to file, go in background, etc. The moment this becomes plugin-based, those options will disappear (since they're modules they can't have a static setting, they might not exist in a deployment), which I'll expect will be cumbersome in some deployments. It's even more true for debugging-related settings (e.g., log level, whether timestamps should be logged, enabling lock/refcount debugging, etc.), all of whom have dedicated controls, and even Admin API commands to tweak them; again, with a plugin based approach, this would mean different settings for each of them, and a more cumbersome (and definitely not backwards compatible) way of tweaking them via Admin API.

I'll add an enhancement label but to be honest, considering things "work" right now and this change feels pretty complex, it would be low priority for us at the moment, so not sure when we'd be able to tackle this.

[jalaziz]

I totally expect it to be low priority for now.

However, I should add that performance is one of the many reasons why modular logging would be better. Logging to local I/O is full of potential performance bottlenecks. Verbose logging to a file (especially on the cloud) can easily kill IOPs capacity. Using a buffer/queue and logging asynchronously can remove the performance burden from Janus. As you said, though, it would be the up to the plugins to do the right thing. Still, there are plenty of great examples out there to model upon.

As for configuration, I understand. For normal deployments, stdout and file logging could remain as CLI options and built-in as default. I'm not suggesting those get removed. For more complex logging needs, I think it's entirely fair to expect users to provide a configuration file as is expected with transports, event handlers, etc.

Debugging-related settings can be preserved if the plugin interface mandates it. I'm not quite sure what you mean by "this would mean different settings for each of them, and a more cumbersome way of tweaking them". Assuming the plugins have some standard interface for the different debugging toggles, it should be fine, no? I'm going based off my experience with logging frameworks in Java, Python, etc, so I may be completely off-base here.

[lminiero]

I'm not quite sure what you mean by "this would mean different settings for each of them, and a more cumbersome way of tweaking them". Assuming the plugins have some standard interface for the different debugging toggles, it should be fine, no?

The way plugins are implemented right now, they all expose ways to "contact" them, but leaving freedom to what the communication should look like. With possibly wildly different logging mechanisms, it would be hard to expose a standard interface for some settings, but I guess that for some of the common ones it might make sense.

Thanks for the feedback, this will definitely be useful when we start looking into it!

[lminiero]

@jalaziz I just pushed a pull request that implements a new framework for modular logging as you suggested. Really interested in your opinion on this, so feedback welcome!

[jalaziz]

@jalaziz I just pushed a pull request that implements a new framework for modular logging as you suggested. Really interested in your opinion on this, so feedback welcome!

This is awesome! Thanks! Will check out the PR tonight.

[lminiero]

Closing as I just merged the PR. Hopefully we'll start seeing some custom loggers contributed soon!