[Azure Logs]: Eventhub input not able to parse ApplicationGatewayFirewallLog due to failed to parse field [azure.eventhub.properties]

[niecore]

Integration Name

Azure Logs [azure]

Dataset Name

azure.eventhub

Integration Version

1.19.3

Agent Version

8.15.2

Agent Output Type

elasticsearch

Elasticsearch Version

8.15.2

OS Version and Architecture

elasticsearch cloud

Software/API Version

No response

Error Message

Following log message is logged in the elastic-agent running the azure logs integration and send via filebeat to elasticsearch (some information is removed):

{"log.level":"warn","@timestamp":"2024-11-19T15:42:27.592Z","message":"Cannot index event '{"@timestamp":"2024-11-19T15:42:25.405Z","message":"{\"category\":\"ApplicationGatewayFirewallLog\",\"operationName\":\"ApplicationGatewayFirewall\",\"properties\":{\"action\":\"Matched\",\"clientIp\":\"REDACTED\",\"details\":{\"data\":\"{ found within [REQUEST_HEADERS:0]} and { found within [REQUEST_METHOD:]} and { found within [REQUEST_HEADERS:]}\",\"file\":\"REQUEST-920-PROTOCOL-ENFORCEMENT.conf\",\"line\":\"1143\",\"message\":\"Equal 0; Pattern match ^OPTIONS$; Pattern match AppleWebKit Android at REQUEST_HEADERS:user-agent.\"},\"engine\":\"Azwaf\",\"hostname\":\"REDACTED\",\"instanceId\":\"appgw_4\",\"message\":\"Request Missing an Accept Header\",\"policyId\":\"24#_subscriptions_REDACTED_resourceGroups_REDACTED_providers_Microsoft.Network_ApplicationGatewayWebApplicationFirewallPolicies_policyAlarmInternet\",\"policyScope\":\"URIPath\",\"policyScopeName\":\"REDACTED\",\"requestUri\":\"REDACTED\",\"ruleGroup\":\"REQUEST-920-PROTOCOL-ENFORCEMENT\",\"ruleId\":\"920300\",\"ruleSetType\":\"OWASP CRS\",\"ruleSetVersion\":\"3.2\",\"transactionId\":\"d1358799d3e032ca93a25b1c2ff5322a\"},\"resourceId\":\"/SUBSCRIPTIONS/REDACTED/RESOURCEGROUPS/REDACTED/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/REDACTED\",\"timeStamp\":\"2024-11-19T15:41:04+00:00\"}","tags":["parse_message","azure-eventhub","forwarded"],"data_stream":{"type":"logs","dataset":"azure.eventhub","namespace":"prod"},"event":{"dataset":"azure.eventhub"},"agent":{"id":"59934f02-c408-4097-a473-e2d043dc2cf7","ephemeral_id":"7d722b70-dca5-48fd-a5bb-fc108db95377","name":"elastic-agent-azure-logs-54d674c646-hnrfk","type":"filebeat","version":"8.15.2"},"ecs":{"version":"8.0.0"},"cloud":{"account":{"id":"b3203ca9-72a1-4283-843f-a66231f50d19"},"instance":{"id":"36594b5c-9c89-4cac-94fc-2763007eb0e6","name":"aks-nodepool1-27134975-vmss_328"},"provider":"azure","machine":{"type":"Standard_D8ds_v4"},"service":{"name":"Virtual Machines"},"region":"westeurope"},"azure":{"offset":189678658248488,"sequence_number":31303060,"enqueued_time":"2024-11-19T15:41:53.187Z","eventhub":"REDACTED","consumer_group":"$Default"},"input":{"type":"azure-eventhub"},"elastic_agent":{"id":"59934f02-c408-4097-a473-e2d043dc2cf7","snapshot":false,"version":"8.15.2"}}\n' (status=400): {"type":"document_parsing_exception","reason":"[1:2640] failed to parse field [azure.eventhub.properties] of type [keyword] in document with id 'E9kWRZMBrEXF0_xo3q9r'. Preview of field's value: '{ruleSetVersion=3.2, policyScope=URIPath, ruleSetType=OWASP CRS, requestUri=REDACTED, message=Request Missing an Accept Header, transactionId=d1358799d3e032ca93a25b1c2ff5322a, hostname=REDACTED, policyScopeName=REDACTED, instanceId=appgw_4, policyId=24#_subscriptions_REDACTED_resourceGroups_REDACTED_providers_Microsoft.Network_ApplicationGatewayWebApplicationFirewallPolicies_policyAlarmInternet, engine=Azwaf, clientIp=20.123.247.131, ruleGroup=REQUEST-920-PROTOCOL-ENFORCEMENT, action=Matched, details={file=REQUEST-920-PROTOCOL-ENFORCEMENT.conf, data={ found within [REQUEST_HEADERS:0]} and { found within [REQUEST_METHOD:]} and { found within [REQUEST_HEADERS:]}, line=1143, message=Equal 0; Pattern match ^OPTIONS$; Pattern match AppleWebKit Android at REQUEST_HEADERS:user-agent.}, ruleId=920300}'","caused_by":{"type":"illegal_argument_exception","reason":"Expected text at 1:1618 but found START_OBJECT"}}, dropping event!","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"azure-eventhub-default","type":"azure-eventhub"},"log":{"source":"azure-eventhub-default"},"log.type":"event","ecs.version":"1.6.0","log.logger":"elasticsearch","log.origin":{"file.line":489,"file.name":"elasticsearch/client.go","function":"github.com/elastic/beats/v7/libbeat/outputs/elasticsearch.(*Client).applyItemStatus"},"service.name":"filebeat","ecs.version":"1.6.0"}

Event Original

No response

What did you do?

What did you see?

All ApplicationGatewayFirewall events are missing in logs-azure.eventhub-*

What did you expect to see?

Anything else?

No response

[zmoog]

Hey @niecore, from the error message you shared, I see the dataset is azure.eventhub, so I assume you are using the "generic" event hub integration (named "Collect events from Event Hub"), right?

I don't know what log categories you are sending to the event hub alongside the ApplicationGatewayFirewallLog category (so, please correct me if I'm wrong), but sometimes users get mapping errors from sending log categories with incompatible data models.

If you want to collect a diverse set of log categories with a single event hub, I suggest deploying the routing solution described at zmoog/public-notes#92.

Using the custom pipeline with routing, the pipeline will route the ApplicationGatewayFirewallLog log events to the Azure Firewall logs data stream, which is specialized for these log categories.

Note: We are working on an Azure Logs integration v2 with built-in routing that will do the same thing as the guide.

[niecore]

Yes we run the generic eventhub collector and the application gateway collector.

All other log categories are indexed correctly either in logs-azure.eventhub-* or logs-azure.application_gateway-*. Only the ApplicationGatewayFirewallLog messages fail to index at the elastic agent running the azure logs integration.

I think your solution will not help with the processing of this events, since the message fails already on the collector.

[zmoog]

All other log categories are indexed correctly either in logs-azure.eventhub-* or logs-azure.application_gateway-*. Only the ApplicationGatewayFirewallLog messages fail to index at the elastic agent running the azure logs integration.

Which data stream fails to index the ApplicationGatewayFirewallLog messages?

[niecore]

It fails to index on the dataset azure.eventhub because of the field azure.eventhub.properties

[zmoog]

We have several options here depending on the log categories: which log categories are you collecting?

[niecore]

@zmoog I upgraded to 1.20.1 and will observe the problem there. Right now everything looks good.