Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add verbose_pipeline documentation section in using-search-pipeline #9130

Open
wants to merge 5 commits into
base: main
Choose a base branch
from
+223 −0
Open

Add verbose_pipeline documentation section in using-search-pipeline #9130

Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
a0db134
Add verbose_pipeline documentation in using-search-pipeline.md
Jan 28, 2025
8b8d3a1
Revert "Add verbose_pipeline documentation in using-search-pipeline.m…
Jan 28, 2025
e409e49
fix error
Jan 28, 2025
8a87cfb
change the dsiplay order
Jan 28, 2025
d808f1e
edit doc based on comments
Jan 29, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
223 changes: 223 additions & 0 deletions _search-plugins/search-pipelines/debugging-search-pipeline.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,223 @@
---
layout: default
title: Debugging a search pipeline
nav_order: 25
has_children: false
parent: Search pipelines
grand_parent: Search
---


# Debugging a search pipeline

The `verbose_pipeline` parameter offers detailed insights into the flow and transformation of data through search request, search response and search phase processor in the search pipeline. It facilitates troubleshooting, pipeline optimization, and ensures transparency in the end-to-end handling of search requests and responses. This functionality is available for all three ways of using a search pipeline：

- [Specifying an existing pipeline for a request]({{site.url}}{{site.baseurl}}/search-plugins/search-pipelines/using-search-pipeline/#specifying-an-existing-search-pipeline-for-a-request).

- [Using a temporary pipeline for a request]({{site.url}}{{site.baseurl}}/search-plugins/search-pipelines/using-search-pipeline/#using-a-temporary-search-pipeline-for-a-request).

- [Setting a default pipeline for all requests in an index]({{site.url}}{{site.baseurl}}/search-plugins/search-pipelines/using-search-pipeline/#default-search-pipeline).

To enable the `verbose_pipeline` feature, add `verbose_pipeline=true` as a query parameter in your search request.

### Example request with verbose pipeline

You can specify the `verbose_pipeline` parameter in the query to enable detailed debugging. The `verbose_pipeline` parameter works seamlessly across all search pipeline configurations:

#### With Default Search Pipeline

Check failure on line 27 in _search-plugins/search-pipelines/debugging-search-pipeline.md

View workflow job for this annotation

GitHub Actions / style-job 

[vale] reported by reviewdog 🐶
[OpenSearch.HeadingCapitalization] 'With Default Search Pipeline' is a heading and should be in sentence case.

Raw Output:
{"message": "[OpenSearch.HeadingCapitalization] 'With Default Search Pipeline' is a heading and should be in sentence case.", "location": {"path": "_search-plugins/search-pipelines/debugging-search-pipeline.md", "range": {"start": {"line": 27, "column": 6}}}, "severity": "ERROR"}

To use `verbose_pipeline` with a default search pipeline, set the pipeline as the default in the index settings and include `verbose_pipeline=true` in the query:

```json
PUT /my_index/_settings
{
"index.search.default_pipeline": "my_pipeline"
}
```
{% include copy-curl.html %}

```json
GET /my_index/_search?verbose_pipeline=true
```
{% include copy-curl.html %}

#### With Specified Search Pipeline by ID

Check failure on line 44 in _search-plugins/search-pipelines/debugging-search-pipeline.md

View workflow job for this annotation

GitHub Actions / style-job 

[vale] reported by reviewdog 🐶
[OpenSearch.HeadingCapitalization] 'With Specified Search Pipeline by ID' is a heading and should be in sentence case.

Raw Output:
{"message": "[OpenSearch.HeadingCapitalization] 'With Specified Search Pipeline by ID' is a heading and should be in sentence case.", "location": {"path": "_search-plugins/search-pipelines/debugging-search-pipeline.md", "range": {"start": {"line": 44, "column": 6}}}, "severity": "ERROR"}
To use `verbose_pipeline` with a specific search pipeline, specify the pipeline ID along with verbose_pipeline=true in the query:
```json
GET /my_index/_search?search_pipeline=my_pipeline&verbose_pipeline=true
```
{% include copy-curl.html %}

#### With Temporary Search Pipeline

Check failure on line 51 in _search-plugins/search-pipelines/debugging-search-pipeline.md

View workflow job for this annotation

GitHub Actions / style-job 

[vale] reported by reviewdog 🐶
[OpenSearch.HeadingCapitalization] 'With Temporary Search Pipeline' is a heading and should be in sentence case.

Raw Output:
{"message": "[OpenSearch.HeadingCapitalization] 'With Temporary Search Pipeline' is a heading and should be in sentence case.", "location": {"path": "_search-plugins/search-pipelines/debugging-search-pipeline.md", "range": {"start": {"line": 51, "column": 6}}}, "severity": "ERROR"}
To use `verbose_pipeline` with a temporary search pipeline, define the pipeline directly in the request body and include verbose_pipeline=true in the query:
```json
POST /my_index/_search?verbose_pipeline=true
{
"query": {
"match": { "text_field": "some search text" }
},
"search_pipeline": {
"request_processors": [
{
"filter_query": {
"query": { "term": { "visibility": "public" } }
}
}
],
"response_processors": [
{
"collapse": {
"field": "category"
}
}
]
}
}
```
{% include copy-curl.html %}
### Example response with verbose pipeline

When the `verbose_pipeline` parameter is enabled, the response contains an additional `processor_results` field that details the transformations applied by each processor in the pipeline:

<details open markdown="block">
<summary>
Response
</summary>
{: .text-delta}

```json
{
"took": 27,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 1,
"relation": "eq"
},
"max_score": 0.18232156,
"hits": [
{
"_index": "my_index",
"_id": "1",
"_score": 0.18232156,
"_source": {
"notification": "This is a public message",
"visibility": "public"
}
}
]
},
"processor_results": [
{
"processor_name": "filter_query",
"tag": "tag1",
"duration_millis": 288541,
"status": "success",
"input_data": {
"verbose_pipeline": true,
"query": {
"bool": {
"adjust_pure_negative": true,
"must": [
{
"match": {
"message": {
"auto_generate_synonyms_phrase_query": true,
"query": "this",
"zero_terms_query": "NONE",
"fuzzy_transpositions": true,
"boost": 1.0,
"prefix_length": 0,
"operator": "OR",
"lenient": false,
"max_expansions": 50
}
}
}
],
"boost": 1.0
}
}
},
"output_data": {
"verbose_pipeline": true,
"query": {
"bool": {
"filter": [
{
"term": {
"visibility": {
"boost": 1.0,
"value": "public"
}
}
}
],
"adjust_pure_negative": true,
"must": [
{
"bool": {
"adjust_pure_negative": true,
"must": [
{
"match": {
"message": {
"auto_generate_synonyms_phrase_query": true,
"query": "this",
"zero_terms_query": "NONE",
"fuzzy_transpositions": true,
"boost": 1.0,
"prefix_length": 0,
"operator": "OR",
"lenient": false,
"max_expansions": 50
}
}
}
],
"boost": 1.0
}
}
],
"boost": 1.0
}
}
}
},
{
"processor_name": "rename_field",
"duration_millis": 250042,
"status": "success",
"input_data": [
{
"_index": "my_index",
"_id": "1",
"_score": 0.18232156,
"_source": {
"message": "This is a public message",
"visibility": "public"
}
}
],
"output_data": [
{
"_index": "my_index",
"_id": "1",
"_score": 0.18232156,
"_source": {
"notification": "This is a public message",
"visibility": "public"
}
}
]
}
]
}
```
</details>
Loading