Filter audit logs for transport layer authentication

[joshbressers]

Today we have the ability for the audit log to record all authentication successes including authentication success on the REST and transport layer. There are use cases where we see a large number of authentication messages being logged due to transport layer authentication. These messages cannot be filtered today without disabling the filtering of all authentication success events.

Disabling all authentication success events isn't a reasonable solution. A possible option is to allow the filtering of transport authentication messages.

There are of course drawbacks to filtering transport authentication messages, it could give an attacker a way to view cluster data without leaving any log messages.

[elasticmachine]

Pinging @elastic/es-security

[albertzaharovits]

I think that the logging locus that can flood is

elasticsearch/x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/authc/AuthenticationService.java

Line 433 in a3b2729

request.authenticationSuccess(authentication.getAuthenticatedBy().getName(), authentication.getUser());

The log message indicates populating the thread context with authn headers while handling ingress transport messages.

Most request, since they will involve multiple nodes, will generate multiple similar such log lines (transport authn success) on each node. This is alright from the perspective that log trails are node bound. But if all logs are collated somewhere, I agree that all of these add little value, and maybe only the original rest authn success might be valuable.

Right now there is no way to filter these because ignore policies filter on "fields", and there is no "field" denoting a rest or transport message, no field_name=[field_value]; it is a positional part in the log line. This will change in #31931 when all log line parts will be identified by a field name.

But even then, post #31931, when it will be possible to enhance ignore policies to work with rest and transport messages it is not obvious if we should do it. Right now, ignore policies filter on who (users, realms, roles) and what (index). It is natural to devise ignore policies on these terms. These terms don't concern the inner working of the system, unlike the rest and transport labels.
When ignore policies start to filter on other fields, I am afraid people will start to make suppositions about how one type of log line leads to other types; we will then be concerned to maintain these relationships.

In a few words, I think the capacity to filter on arbitrary "fields" of the log line is dangerous because one has to know very well in which cases such log entries occur; they require inner system knowledge.
I am leaning to not doing it, maybe disabling authn success events is the lesser evil.

CC @mikeh-elastic

[albertzaharovits]

My supposition about the logging locus generating the flood is probably wrong. @tvernum was right pointing to me that the line I have indicated is not executed when requests span nodes, as I supposed in my previous comment. A single authenticated request will not generate multiple similar authn success entries.

Therefore I think we need some sample of the logging deluge to reason about the best way to filter it.

[andyb-elastic]

It sounds like the feedback we're looking for here is an example of unhelpfully-verbose audit logging from @joshbressers - do I have that right @albertzaharovits?

[albertzaharovits]

@andyb-elastic Yes we need an example to go forward, but instead of Josh I would think @mikeh-elastic, who opened the ER https://github.com/elastic/enhancements/issues/4448 pointing to this issue, would be more qualified to provide one.

It's been some time, I'll be closing this in two weeks if no one fills us in with an example.

[mikeh-elastic]

@albertzaharovits
What would you like me to do to assist? We can have the customer open a support case with a sample of the logs when turned on if that is what you need.

[albertzaharovits]

@mikeh-elastic In other words we expect the audit log to be verbose. The question is if the verbosity the customer has come up against is unreasonable. If he had been fiddling with the logging settings and given a sample of the log file, we can answer if this is something we plan to change and eventually work on.

Specifically, in this instance, I do not believe transport authentication_success log entries are the entries polluting the log file. If you believe that transport layer authentication is the entry that fills the logs that the client deems as unreasonable, then we need a sample of the file: see it to believe it .

[costin]

There hasn't been any progress on this issue for quite a while - should there be more discussions around it or can it be closed?

[tvernum]

I'm going to close this issue, but we can re-open it if we get more concrete requirements.

As it stands, our view is:

• We don't believe we generate extra audit entries for transport connections that aren't present for rest connections, nor should a rest connection trigger an associated "transport" audit event. If anyone is seeing that sort of behaviour, please provide some samples so we can sort it out.
• Our view is that filtering audit logs based on which protocol is used is a dangerous step to take, and we do not want to encourage it. If you want a record of the actions someone is taking, then you don't care which protocol they used to connect.
• An exception to the above might be if you specifically want to audit when a particular user account is used on a protocol it is not intended for (e.g. a user that should be Rest-only connects using transport client) but this issue isn't asking for that.
• If anyone is seeing too much noise in their audit logs, and wants to remove superfluous entries, then we're happy to look at options for that, but we don't think the protocol is the right way to determine that. Something like audit events ignore policies: add support for "actions" attribute #37148 would be more appropriate.

We're definitely open to looking at ways to remove superfluous entries from the audit logs, but we're not convinced that protocol based filtering is the right tool.