EQL - query language for event data in Elasticsearch

[costin]

Meta issue consolidating the EQL functionality released in Elasticsearch 7.9 as experimental.
EQL or Event Query Language is a declarative language dedicated for identifying patterns and relationships between events.

Consider using EQL if you:

• Use Elasticsearch for threat hunting or other security use cases
• Search time-series data or logs, such as network or system logs
• Want an easy way to explore relationships between events

A good intro on EQL and its purpose is available here. The language reference can be found at this address while EQL on Elasticsearch is explained at length through a dedicated chapter.

This release includes the following features:

• event queries
• sequences
• pipes

An in-depth discussion of EQL in ES scope can be found at #49581.

Full history available here.

High-level tasks

• Create EQL plugin Create EQL Plugin projects #49583
• Extract reusable components from SQL into common project Extract common/reusable components from SQL for EQL #49773
• Synchronous EQL REST API for ad-hoc querying Create Synchronous EQL querying REST API #49634
• Build query parser and plan on reusable SQL components
• EQL transpiler for stateless EQL expressions to ES Search DSL Transpile EQL stateless expressions into ES Search DSL #49589
• Response format enhancements EQL: Response format enhancements #52845
• Support existing EQL functions EQL: Add support for existing functions #51556
• Map EQL sequence parts to ES requests Map EQL sequence/join parts to ES requests #49590
• Implement pipes logic in the plugin (not using aggregations) Implement EQL Pipes in EQL Plugin #49627
• Head and Tail pipe EQL: Add Head/Tail pipe support #58536
• Sequence implementation at EQL: Sequence/Join parsing and model #54227, EQL: Introduce support for sequences #56300, EQL: Sequence improvements #56768, EQL: Introduce support for sequence maxspan #58635, EQL: Introduce until functionality #59292
• High Level REST client support for EQL API EQL: Add High Level Rest Client #51961
• Cancelling a task's grandchildren when the task is cancelled Cancelling a task's grandchildren when the task is cancelled #50990
• Convert EQL REST API to be async (similar to async search) Make EQL REST Querying API async #49638
• Create documentation for using EQL [DOCS] Document EQL support in Elasticsearch #51057
• Telemetry for EQL usage EQL usage telemetry #49630
• Escape non-alphanumeric fields for EQL Escape non-alphanumeric fields for EQL #51443
• Remove feature flags and prepare for release Prepare EQL for release #51613

[elasticmachine]

Pinging @elastic/es-ql (:Query Languages/EQL)

[costin]

Scheduled for release in Elasticsearch 7.9