[Filebeat] Azure Signin Module authentication_processing_details Issue

[mr1716]

The log is found in the official Elastic Repository at: https://github.com/elastic/beats/blob/main/x-pack/filebeat/module/azure/signinlogs/test/test-non-interactive-user-signin.log-expected.json

For the Azure Signin Module, the following field should have the periods in "Legacy TLS (TLS 1.0, 1.1, 3DES)" replaced with an underscore or another value. When the value is unflattened, Filebeat views the periods as subfields, which is not the intent.
"azure.signinlogs.properties.authentication_processing_details.Legacy TLS (TLS 1.0, 1.1, 3DES)": "False",

It should be something like:
"azure.signinlogs.properties.authentication_processing_details.Legacy TLS (TLS 1_0, 1_1, 3DES)": "False",

[tetianakravchenko]

@elastic/obs-cloud-monitoring fyi

[UcanInfosec]

@zmoog curious to see what is found and if changes are required. This only happens when there’s a period in that last field value key

[zmoog]

Let me run some tests to understand how these kinds of logs are ingested and indexed.

[zmoog]

I tried to ingest the test document test-non-interactive-user-signin.log you @mr1716 mentioned using Filebeat 8.6.1.

With an input document containing this snippet:

{
  "authenticationProcessingDetails": [
    {
      "key": "Legacy TLS (TLS 1.0, 1.1, 3DES)",
      "value": "False"
    },
    {
      "key": "Oauth Scope Info",
      "value": "[User.Read,Userinfo.ReadWrite]"
    },
    {
      "key": "Is CAE Token",
      "value": "False"
    }
  ]
}

The end result is the following:

{
  "authentication_processing_details": {
    "Legacy TLS (TLS 1": {
      "0, 1": {
        "1, 3DES)": "False"
      }
    },
    "Oauth Scope Info": "[User.Read,Userinfo.ReadWrite]",
    "Is CAE Token": "False"
  }
}

There is room for improvement.

[zmoog]

TIL the set processor can turn a field {"a.b": true} into {"a": {"b": true}}.

I am trying to replace the set processor in the foreach with a script processor.

Starting from:

{
  "authenticationProcessingDetails": [
    {
      "key": "Legacy TLS (TLS 1.0, 1.1, 3DES)",
      "value": "False"
    },
    {
      "key": "Oauth Scope Info",
      "value": "[User.Read,Userinfo.ReadWrite]"
    },
    {
      "key": "Is CAE Token",
      "value": "False"
    }
  ]
}

if I use this script inside the foreach processor:

{
  "script": {
    "lang": "painless",
    "source": """
      def tmp = [:];
      for (item in ctx.azure.signinlogs.properties.authentication_processing_details) {
        tmp[item.key] = item.value;
      }
      ctx.azure.signinlogs.properties.authentication_processing_details = tmp;
    """
  }
}

I get the following result:

{
  "authentication_processing_details": {
    "Oauth Scope Info": "[User.Read,Userinfo.ReadWrite]",
    "Legacy TLS (TLS 1.0, 1.1, 3DES)": "False",
    "Is CAE Token": "False"
  }
}

[UcanInfosec]

@zmoog any way to do that in JavaScript?

[zmoog]

@UcanInfosec, according to the Available scripting language, unfortunately JavaScript is not one of them.

The most interesting and used option is the Painless scripting language. It is a good option to write down expressions or small snippets of code to transform data.

[zmoog]

@mr1716 @UcanInfosec does the new document structure work for your use cases?

{
  "authentication_processing_details": {
    "Oauth Scope Info": "[User.Read,Userinfo.ReadWrite]",
    "Legacy TLS (TLS 1.0, 1.1, 3DES)": "False",
    "Is CAE Token": "False"
  }
}

I am creating a quick PR to gather feedback from other team members.

[mr1716]

@zmoog I think the question that @UcanInfosec had was how would we do this same thing if it were done in native Filebeat, not Elastic Painless. The new structure works good. Just curious how to do what you did in native filebeat

[zmoog]

@mr1716, what do you mean by 'native Filebeat'?

Here's how Filebeat collects the logs from Azure and publishes them to Elasticsearch.

┌───────────────────────────────────────┐     ┌─────────────────────────┐      ┌────────────────────────┐
│                                       │     │                         │      │                        │
│                                       │     │                         │      │                        │
│                                       │     │                         │      │                        │
│  ┌─────────────┐   ┌────────────────┐ │     │   ┌─────────────────┐   │      │  ┌───────────────────┐ │
│  │ diagnostic  │   │  appservices   │ │     │   │ azure-eventhub  │   │      │  │     eventhub      │ │
│  │   setting   │──▶│ <<event hub>>  │─┼amqp─┼──▶│    <<input>>    │───┼─http─┼─▶│  <<data stream>>  │ │
│  └─────────────┘   └────────────────┘ │     │   └─────────────────┘   │      │  └───────────────────┘ │
│                                       │     │                         │      │                        │
│                                       │     │                         │      │                        │
│                                       │     │                         │      │                        │
│                                       │     │                         │      │                        │
└─Azure─────────────────────────────────┘     └───Filebeat──────────────┘      └──Elasticsearch─────────┘

The azure-eventhub input connects to the Azure event hub and fetches the logs. The input sends the logs to the data stream, where an ingest pipeline processes them before the indexing.

The azure-eventhub input does not process the logs. It is the Filebeat adapter to access the event hub. All the processing and data transformation from the source format from Azure to the document in Elasticsearch happens in the ingest pipeline.

[UcanInfosec]

@zmoog how would the painless script be converted into a filebeat script processor?

[UcanInfosec]

Because it’s fine when there are spaces but not periods originally

[zmoog]

@zmoog how would the painless script be converted into a filebeat script processor?

For existing Filebeat modules and integrations, the processors are defined as YAML files and created in Elasticsearch during installation.

For example, here are the source for the sign-in logs ingest pipelines:

• Filebeat module
• Elastic Agent integration

If you want to add a script processor to a new or existing pipeline, you can do it in Kibana or Dev Tools.

For example, in Kibana, you can:

• visit Stack Management > Ingest Pipelines
• pick an existing pipeline or click on the 'Create pipeline' button

I you want to get started with the Painless scripting language, then Painless scripting language is a good starting point.

[zmoog]

Because it’s fine when there are spaces but not periods originally

The set processor expands the dots in field names into subfields.

For example, given the following pipeline in the Dev Tools:

PUT _ingest/pipeline/zmoog-test
{
  "processors": [
    {
      "set": {
        "field": "a.b.c",
        "value": true
      }
    }
  ]
}

If we simulate the pipeline execution using the empty test object {}:

POST _ingest/pipeline/zmoog-test/_simulate
{
  "docs": [
    {
      "_source": {}
    }
  ]
}

I get the following result:

{
  "docs": [
    {
      "doc": {
        "_index": "_index",
        "_id": "_id",
        "_version": "-3",
        "_source": {
          "a": {
            "b": {
              "c": true
            }
          }
        },
        "_ingest": {
          "timestamp": "2023-01-31T21:20:48.352056732Z"
        }
      }
    }
  ]
}

The set processor turns the {"a.b.c": True} into this one:

{
  "a": {
    "b": {
      "c": true
    }
  }
}

We proposed the fix https://github.com/elastic/integrations/pull/5129/files#diff-274e12d0961404cd66da857b49259663133debe54e6cc0b9e0832114450785c8 that replaces for foreach + set processors with one script processor running a short Painless script to avoid the expansion of the dots.

If it works for you, we'll port the fix from the integrations repo to beats.

[UcanInfosec]

As long as it fixes this, let’s do it. This needs to get fixed

[zmoog]

Great. I am finalizing the two PRs for both Beats and Elastic Agent integration.

[zmoog]

Recap:

• Expanding the dots in the field name into sub-fields of authentication_processing_details is a side-effect.
• To avoid this side-effect, we are replacing the foreach + set processors combo with a script processor.

Fixes implemented in:

• Beats: [Azure Logs] Replace foreach+set combo with a script processor #34478
• Integrations: [Azure Logs] Fix authentication_processing_details parsing in sign-in logs integrations#5129

[zmoog]

Targeting to ship this fix in the following Beats releases:

• 8.7
• 8.6.2
• 7.17.10

And in the Azure Logs integration 1.5.7