ingest: Support multiple pipelines from _simulate without creating actual pipelines

[jakelandis]

In 6.5 the ingest node introduced the pipeline processor which allows one pipeline to call another pipeline. In order to call another pipeline, you need a named pipeline, a.k.a a real pipeline defined.

The _simulate API only supports a single nameless pipeline, which makes simulations against pipeline to pipeline calls awkward. For example, to use the ingest _simluate API with multiple pipelines, you must create real pipelines for the simulation. For example:

PUT _ingest/pipeline/pipeline1
{
  "processors": [
    {
      "set": {
        "field": "1",
        "value": "b"
      }
    },
    {
      "pipeline": {
        "name": "pipeline2"
      }
    }
  ]
}

PUT _ingest/pipeline/pipeline2
{
  "processors": [
        {
      "set": {
        "field": "2",
        "value": "c"
      }
    },
    {
      "pipeline": {
        "name": "pipeline3"
      }
    }
  ]
}

PUT _ingest/pipeline/pipeline3
{
  "processors": [
    {
      "set": {
        "field": "3",
        "value": "d"
      }
    }
  ]
}

POST _ingest/pipeline/pipeline1/_simulate?verbose
{
  "docs": [
    {
      "_source": {
        "0" : "a"
      }
    }
  ]
}

The above simulation works without issue, however:

• It is not well documented
• It requires creating real pipelines
  • It can pollute cluster state with in-process process pipelines
  • Others can execute the pipelines outside the scope of simulate
  • It is too easy to forget to delete the in-process pipelines
  • Any future tooling to help build pipelines would not want to have to create real pipelines just for simulation.

The proposal here is to create a new "pipelines" top level array for the _simulate API which can be used instead of the "pipeline" top level object.

[elasticmachine]

Pinging @elastic/es-core-infra

[ycombinator]

Hey @jakelandis thanks for creating this issue. We might have a need for this feature in elastic/beats#8914. Without mentioning specific releases, any ideas if this is something that'll be worked on "soon" or "later" 😄? Thanks.

[jakelandis]

"soon" or "later"

@ycombinator later. I a took a quick look and while not terribly difficult, it is not a trivial change either.

[dakrone]

This has been open for quite a while, and we haven't made much progress on this due to focus in other areas. For now I'm going to close this as something we aren't planning on implementing. We can re-open it later if needed.

[ruflin]

Mentioning the new end to end simulate API here that was built recently: #101409 One of the problems this solves is to test multiple pipelines that already exists.