Enhance the ingest node simulate verbose output #60433

jakelandis · 2020-07-29T22:45:49Z

This commit enhances the verbose output for the
_ingest/pipeline/_simulate?verbose api. Specifically
this adds the following:

the pipeline processor is now included in the output
the conditional (if) and result is now included in the output iff it was defined
a status field is always displayed. the possible values of status are
- success - if the processor ran with out errors
- error - if the processor ran but threw an error that was not ingored
- error_ignored - if the processor ran but threw an error that was ingored
- skipped - if the process did not run (currently only possible if the if condition evaluates to false)
- dropped - if the the drop processor ran and dropped the document
a processor_type field for the type of processor (e.g. set, rename, etc.)
throw a better error if trying to simulate with a pipeline that does not exist

closes #56004

===========

Example input:

POST _ingest/pipeline/_simulate?verbose
{
  "pipeline": {
    "description": "_description",
    "processors": [
      {
        "set": {
          "description": "processor_description",
          "tag": "processor_tag",
          "field": "field1",
          "value": "value1"
        }
      },
      {
        "rename": {
          "field": "dont_exist",
          "target_field": "field1",
          "ignore_failure": true
        }
      },
      {
        "rename": {
          "tag": "rename-1",
          "field": "foofield",
          "target_field": "field1",
          "on_failure": [
            {
              "set": {
                "field": "field2",
                "value": "value2"
              }
            }
          ]
        }
      },
      {
        "drop": {
          "if": "false"
        }
      },
      {
        "drop": {
          "if": "true"
        }
      }
    ]
  },
  "docs": [
    {
      "_index": "index",
      "_id": "id",
      "_source": {
        "foo": "bar"
      }
    }
  ]
}

Example output:

{
  "docs" : [
    {
      "processor_results" : [
        {
          "processor_type" : "set",
          "status" : "success",
          "description" : "processor_description",
          "tag" : "processor_tag",
          "doc" : {
            "_index" : "index",
            "_id" : "id",
            "_source" : {
              "field1" : "value1",
              "foo" : "bar"
            },
            "_ingest" : {
              "pipeline" : "_simulate_pipeline",
              "timestamp" : "2020-07-30T01:06:43.961586Z"
            }
          }
        },
        {
          "processor_type" : "rename",
          "status" : "error_ignored",
          "ignored_error" : {
            "error" : {
              "root_cause" : [
                {
                  "type" : "illegal_argument_exception",
                  "reason" : "field [dont_exist] doesn't exist"
                }
              ],
              "type" : "illegal_argument_exception",
              "reason" : "field [dont_exist] doesn't exist"
            }
          },
          "doc" : {
            "_index" : "index",
            "_id" : "id",
            "_source" : {
              "field1" : "value1",
              "foo" : "bar"
            },
            "_ingest" : {
              "pipeline" : "_simulate_pipeline",
              "timestamp" : "2020-07-30T01:06:43.961586Z"
            }
          }
        },
        {
          "processor_type" : "rename",
          "status" : "error",
          "tag" : "rename-1",
          "error" : {
            "root_cause" : [
              {
                "type" : "illegal_argument_exception",
                "reason" : "field [foofield] doesn't exist"
              }
            ],
            "type" : "illegal_argument_exception",
            "reason" : "field [foofield] doesn't exist"
          }
        },
        {
          "processor_type" : "set",
          "status" : "success",
          "doc" : {
            "_index" : "index",
            "_id" : "id",
            "_source" : {
              "field1" : "value1",
              "field2" : "value2",
              "foo" : "bar"
            },
            "_ingest" : {
              "pipeline" : "_simulate_pipeline",
              "on_failure_message" : "field [foofield] doesn't exist",
              "on_failure_processor_tag" : "rename-1",
              "timestamp" : "2020-07-30T01:06:43.961586Z",
              "on_failure_processor_type" : "rename"
            }
          }
        },
        {
          "processor_type" : "drop",
          "status" : "skipped",
          "if" : {
            "condition" : "false",
            "result" : false
          }
        },
        {
          "processor_type" : "drop",
          "status" : "dropped",
          "if" : {
            "condition" : "true",
            "result" : true
          }
        }
      ]
    }
  ]
}

For ingest node processors a per processor description was recently added. This commit displays that description in the verbose output of the pipeline simulation. related elastic#57906

jakelandis · 2020-07-30T00:58:01Z

modules/ingest-common/src/yamlRestTest/resources/rest-api-spec/test/ingest/90_simulate.yml

  - match: { docs.0.processor_results.0.doc._source.field1: "123.42 400 <foo>" }
  - match: { docs.0.processor_results.0.doc._source.status: 200 }
-  - length: { docs.0.processor_results.1: 2 }
+  - length: { docs.0.processor_results.1: 4 }


because we now always return status and processor_type

jakelandis · 2020-07-30T00:58:54Z

modules/ingest-common/src/yamlRestTest/resources/rest-api-spec/test/ingest/90_simulate.yml

@@ -731,16 +741,25 @@ teardown:
          ]
        }
 - length: { docs: 1 }
- length: { docs.0.processor_results: 3 }
+- length: { docs.0.processor_results: 5 }


because we now show the pipeline processor in the results

jakelandis · 2020-07-30T01:02:02Z

server/src/main/java/org/elasticsearch/ingest/TrackingResultProcessor.java

                        }
                        handler.accept(null, elasticsearchException);
                    }
                } else {
                    //now that we know that there are no cycles between pipelines, decorate the processors for this pipeline and execute it
                    CompoundProcessor verbosePipelineProcessor = decorate(pipeline.getCompoundProcessor(), null, processorResultList);
+                    //add the pipeline process to the results
+                    processorResultList.add(new SimulateProcessorResult(actualProcessor.getType(), actualProcessor.getTag(),
+                        actualProcessor.getDescription(), conditionalWithResult));


I think we never added the pipeline processor in the past because there would have been nothing to display, but now that we always show the type (pipeline) and status (i.e. success) there is something to display, so this line is what adds this to the output. The addition of this output caused a few tests to need to be updated

elasticmachine · 2020-07-30T01:09:57Z

Pinging @elastic/es-core-features (:Core/Features/Ingest)

jbaiera

Just one question, but otherwise LGTM

jbaiera · 2020-08-04T14:53:31Z

server/src/main/java/org/elasticsearch/ingest/TrackingResultProcessor.java

@@ -35,6 +36,7 @@
    private final ConditionalProcessor conditionalProcessor;
    private final List<SimulateProcessorResult> processorResultList;
    private final boolean ignoreFailure;
+    private Tuple<String, Boolean> conditionalWithResult = null; //null = no conditional


Seems like this could be a local variable in the execute function?

thanks @jbaiera , indeed. updated 5c26a6b

This commit enhances the verbose output for the `_ingest/pipeline/_simulate?verbose` api. Specifically this adds the following: * the pipeline processor is now included in the output * the conditional (if) and result is now included in the output iff it was defined * a status field is always displayed. the possible values of status are * `success` - if the processor ran with out errors * `error` - if the processor ran but threw an error that was not ingored * `error_ignored` - if the processor ran but threw an error that was ingored * `skipped` - if the process did not run (currently only possible if the if condition evaluates to false) * `dropped` - if the the `drop` processor ran and dropped the document * a `processor_type` field for the type of processor (e.g. set, rename, etc.) * throw a better error if trying to simulate with a pipeline that does not exist closes elastic#56004 # Conflicts: # docs/reference/ingest/apis/simulate-pipeline.asciidoc

This commit enhances the verbose output for the `_ingest/pipeline/_simulate?verbose` api. Specifically this adds the following: * the pipeline processor is now included in the output * the conditional (if) and result is now included in the output iff it was defined * a status field is always displayed. the possible values of status are * `success` - if the processor ran with out errors * `error` - if the processor ran but threw an error that was not ingored * `error_ignored` - if the processor ran but threw an error that was ingored * `skipped` - if the process did not run (currently only possible if the if condition evaluates to false) * `dropped` - if the the `drop` processor ran and dropped the document * a `processor_type` field for the type of processor (e.g. set, rename, etc.) * throw a better error if trying to simulate with a pipeline that does not exist closes #56004

jakelandis added 17 commits June 16, 2020 17:27

Per processor description for verbose simulate

a8a9d87

For ingest node processors a per processor description was recently added. This commit displays that description in the verbose output of the pipeline simulation. related elastic#57906

wip

6078c45

Merge branch 'master' into show_conditional

aacbf21

Merge remote-tracking branch 'upstream/master' into show_conditional

c5498e0

tests pass

12979f6

Merge branch 'master' into show_conditional

0e991b4

serialization fixes

cf599ee

test status

fbfc566

minor clean up

4441ff3

minor testing

5a58161

fix showing of pipeline processor

db354e4

precomit

c97e5ab

handle non-existant pipeline

e1a65e5

fix existing yaml tests

2b1f3da

add no pipeline test

79bb3fb

add a bit of coverage

a829e38

Merge remote-tracking branch 'upstream/master' into show_conditional

374b7ae

jakelandis commented Jul 30, 2020

View reviewed changes

jakelandis added :Data Management/Ingest Node Execution or management of Ingest Pipelines including GeoIP >enhancement v7.10.0 v8.0.0 labels Jul 30, 2020

jakelandis marked this pull request as ready for review July 30, 2020 01:09

elasticmachine added the Team:Data Management Meta label for data/management team label Jul 30, 2020

jakelandis requested a review from danhermann July 30, 2020 01:10

jakelandis mentioned this pull request Jul 30, 2020

[Ingest Node Pipelines] More detailed _simulate response #56004

Closed

fix doc test

cb3d4a0

jakelandis requested a review from jbaiera August 3, 2020 15:28

jbaiera approved these changes Aug 4, 2020

View reviewed changes

jakelandis added 2 commits August 4, 2020 11:20

use local variable

5c26a6b

Merge remote-tracking branch 'upstream/master' into show_conditional

dc93ec3

jakelandis merged commit 35fc997 into elastic:master Aug 4, 2020

jakelandis deleted the show_conditional branch August 4, 2020 18:09

jakelandis added the backport pending label Aug 4, 2020

jakelandis mentioned this pull request Aug 4, 2020

[7.x] Enhance the ingest node simulate verbose output (#60433) #60678

Merged

benwtrent mentioned this pull request Aug 6, 2020

[Transform] consider passing ?verbose flag to pipeline simulations in _preview #60847

Open

jakelandis removed the backport pending label Aug 27, 2020

Mpdreamz mentioned this pull request Nov 16, 2020

7.10.1 Meta Ticket elastic/elasticsearch-net#5096

Closed

61 tasks

stevejgordon mentioned this pull request Dec 17, 2020

7.11.0 Meta Ticket elastic/elasticsearch-net#5198

Closed

jakelandis added v8.0.0-alpha1 and removed v8.0.0 labels Jul 26, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enhance the ingest node simulate verbose output #60433

Enhance the ingest node simulate verbose output #60433

jakelandis commented Jul 29, 2020 •

edited by alisonelizabeth

Loading

jakelandis Jul 30, 2020

jakelandis Jul 30, 2020

jakelandis Jul 30, 2020 •

edited

Loading

elasticmachine commented Jul 30, 2020

jbaiera left a comment

jbaiera Aug 4, 2020

jakelandis Aug 4, 2020

Enhance the ingest node simulate verbose output #60433

Enhance the ingest node simulate verbose output #60433

Conversation

jakelandis commented Jul 29, 2020 • edited by alisonelizabeth Loading

jakelandis Jul 30, 2020

Choose a reason for hiding this comment

jakelandis Jul 30, 2020

Choose a reason for hiding this comment

jakelandis Jul 30, 2020 • edited Loading

Choose a reason for hiding this comment

elasticmachine commented Jul 30, 2020

jbaiera left a comment

Choose a reason for hiding this comment

jbaiera Aug 4, 2020

Choose a reason for hiding this comment

jakelandis Aug 4, 2020

Choose a reason for hiding this comment

jakelandis commented Jul 29, 2020 •

edited by alisonelizabeth

Loading

jakelandis Jul 30, 2020 •

edited

Loading