Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[7.9] [Ingest Manager] Handle Legacy ES client errors (#76865) #77215

Merged
merged 3 commits into from
Sep 11, 2020
Merged

[7.9] [Ingest Manager] Handle Legacy ES client errors (#76865) #77215

merged 3 commits into from
Sep 11, 2020

Conversation

jfsiii
Copy link
Contributor

@jfsiii jfsiii commented Sep 10, 2020

Backports the following commits to 7.9:

[Ingest Manager] Handle Legacy ES client errors (#76865)
96ffe74 
## Summary

closes #75862

 1. Use an HTTP status code if ES client error provides one. extended #75862 (comment) to all 4xx-5xx errors
 1. Format Error message as described in #75862 (comment) & agreed to #75862 (comment)

### example Request/Response:
```
> curl --user elastic:changeme -X POST http://localhost:5601/api/ingest_manager/epm/packages/aws-0.2.7 -H 'kbn-xsrf: xyz'
{
  "statusCode":400,
  "error":"Bad Request",
  "message":"[parse_exception] Failed to parse content to map from ES at /_ingest/pipeline/logs-aws.cloudtrail-0.2.7: {\"error\":{\"root_cause\":[{\"type\":\"parse_exception\",\"reason\":\"Failed to parse content to map\"}],\"type\":\"parse_exception\",\"reason\":\"Failed to parse content to map\",\"caused_by\":{\"type\":\"json_parse_exception\",\"reason\":\"Duplicate field 'ListGroupsForUser'\\n at [Source: (byte[])\\\"---\\ndescription: Pipeline for AWS CloudTrail Logs\\nprocessors:\\n  - set:\\n      field: event.ingested\\n      value: '{{_ingest.timestamp}}'\\n  - rename:\\n      field: \\\"message\\\"\\n      target_field: \\\"event.original\\\"\\n  - json:\\n      field: \\\"event.original\\\"\\n      target_field: \\\"json\\\"\\n  - date:\\n      field: \\\"json.eventTime\\\"\\n      target_field: \\\"@timestamp\\\"\\n      ignore_failure: true\\n      formats:\\n        - ISO8601\\n  - rename:\\n      field: \\\"json.eventVersion\\\"\\n      target_field: \\\"aws.cloudtrail.event_versi\\\"[truncated 16425 bytes]; line: 489, column: 26]\"}},\"status\":400}"
}
```

### example Kibana Logs:
`[parse_exception] Failed to parse content to map`

<details><summary>Used `test.each` to generate tests for each 4xx - 5xx error. Call each error 3 different ways.</summary>

```
  defaultIngestErrorHandler
    use the HTTP error status code provided by LegacyESErrors
      ✓ 400 - with path & response (12ms)
      ✓ 401 - with path & response (5ms)
      ✓ 402 - with path & response (5ms)
      ✓ 403 - with path & response (6ms)
      ✓ 404 - with path & response (5ms)
      ✓ 405 - with path & response (17ms)
      ✓ 406 - with path & response (2ms)
      ✓ 407 - with path & response (3ms)
      ✓ 408 - with path & response (6ms)
      ✓ 409 - with path & response (5ms)
      ✓ 410 - with path & response (1ms)
      ✓ 411 - with path & response (1ms)
      ✓ 412 - with path & response (1ms)
      ✓ 413 - with path & response (1ms)
      ✓ 414 - with path & response (1ms)
      ✓ 415 - with path & response (1ms)
      ✓ 416 - with path & response (1ms)
      ✓ 417 - with path & response (9ms)
      ✓ 418 - with path & response (1ms)
      ✓ 421 - with path & response (1ms)
      ✓ 426 - with path & response (1ms)
      ✓ 429 - with path & response (1ms)
      ✓ 450 - with path & response (1ms)
      ✓ 494 - with path & response (1ms)
      ✓ 497 - with path & response (1ms)
      ✓ 499 - with path & response (3ms)
      ✓ 500 - with path & response (2ms)
      ✓ 501 - with path & response (1ms)
      ✓ 502 - with path & response (2ms)
      ✓ 503 - with path & response (1ms)
      ✓ 504 - with path & response (1ms)
      ✓ 505 - with path & response (8ms)
      ✓ 506 - with path & response (2ms)
      ✓ 510 - with path & response (1ms)
      ✓ 400 - with other metadata (1ms)
      ✓ 401 - with other metadata (1ms)
      ✓ 402 - with other metadata (1ms)
      ✓ 403 - with other metadata (1ms)
      ✓ 404 - with other metadata (2ms)
      ✓ 405 - with other metadata (1ms)
      ✓ 406 - with other metadata (2ms)
      ✓ 407 - with other metadata (1ms)
      ✓ 408 - with other metadata (1ms)
      ✓ 409 - with other metadata (1ms)
      ✓ 410 - with other metadata (10ms)
      ✓ 411 - with other metadata (2ms)
      ✓ 412 - with other metadata (1ms)
      ✓ 413 - with other metadata (1ms)
      ✓ 414 - with other metadata (1ms)
      ✓ 415 - with other metadata (1ms)
      ✓ 416 - with other metadata (7ms)
      ✓ 417 - with other metadata (2ms)
      ✓ 418 - with other metadata (1ms)
      ✓ 421 - with other metadata (1ms)
      ✓ 426 - with other metadata (1ms)
      ✓ 429 - with other metadata (1ms)
      ✓ 450 - with other metadata (1ms)
      ✓ 494 - with other metadata (11ms)
      ✓ 497 - with other metadata (1ms)
      ✓ 499 - with other metadata (1ms)
      ✓ 500 - with other metadata (1ms)
      ✓ 501 - with other metadata (1ms)
      ✓ 502 - with other metadata (1ms)
      ✓ 503 - with other metadata (1ms)
      ✓ 504 - with other metadata (2ms)
      ✓ 505 - with other metadata (2ms)
      ✓ 506 - with other metadata (1ms)
      ✓ 510 - with other metadata (1ms)
      ✓ 400 - without metadata (1ms)
      ✓ 401 - without metadata (1ms)
      ✓ 402 - without metadata (10ms)
      ✓ 403 - without metadata (1ms)
      ✓ 404 - without metadata (1ms)
      ✓ 405 - without metadata (1ms)
      ✓ 406 - without metadata (1ms)
      ✓ 407 - without metadata (1ms)
      ✓ 408 - without metadata (1ms)
      ✓ 409 - without metadata (1ms)
      ✓ 410 - without metadata (1ms)
      ✓ 411 - without metadata (1ms)
      ✓ 412 - without metadata (1ms)
      ✓ 413 - without metadata (1ms)
      ✓ 414 - without metadata (1ms)
      ✓ 415 - without metadata (12ms)
      ✓ 416 - without metadata (1ms)
      ✓ 417 - without metadata (2ms)
      ✓ 418 - without metadata (1ms)
      ✓ 421 - without metadata (1ms)
      ✓ 426 - without metadata (2ms)
      ✓ 429 - without metadata (2ms)
      ✓ 450 - without metadata (3ms)
      ✓ 494 - without metadata (2ms)
      ✓ 497 - without metadata (2ms)
      ✓ 499 - without metadata (1ms)
      ✓ 500 - without metadata (1ms)
      ✓ 501 - without metadata (2ms)
      ✓ 502 - without metadata (1ms)
      ✓ 503 - without metadata (10ms)
      ✓ 504 - without metadata (2ms)
      ✓ 505 - without metadata (1ms)
      ✓ 506 - without metadata (2ms)
      ✓ 510 - without metadata (1ms)
```
</details>


### Checklist

- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios


### Manual testing
<details><summary><strong>Manual testing</strong></summary>

#### Checkout the branch from elastic/package-storage#370

```
git clone https://github.com/elastic/package-storage.git
cd package-storage/
git switch -C update-aws-0.2.7-1598281986 origin/update-aws-0.2.7-1598281986
```

#### start the stack using the registry from elastic/package-storage#370
```
cd testing/environments/
docker-compose -f snapshot.yml pull
docker-compose -f snapshot.yml -f local.yml up --force-recreate
```

#### Try to install the broken package
```
> curl --user elastic:changeme -X POST http://localhost:5601/api/ingest_manager/epm/packages/aws-0.2.7 -H 'kbn-xsrf: xyz'
{"statusCode":500,"error":"Internal Server Error","message":"Bad Request"}
```
_observe the same error as #75862_

#### start only local registry with the broken package
```
# CTRL-C the stack (shell where you ran `docker-compose`)
cd ../.. # back to package-storage root
docker build .
docker run -p 8080:8080 id_from_prior_step
```

#### start the stack from this PR, pointing at the local registry from prior step
```
yarn start --no-base-path --xpack.ingestManager.registryUrl=http://localhost:8080
yarn es snapshot --license=trial  -E xpack.security.authc.api_key.enabled=true
```

#### Try to install the broken package
```
curl --user elastic:changeme -X POST http://localhost:5601/api/ingest_manager/epm/packages/aws-0.2.7 -H 'kbn-xsrf: xyz'
{
  "statusCode": 400,
  "error": "Bad Request",
  "message": "[parse_exception] Failed to parse content to map response from /_ingest/pipeline/logs-aws.cloudtrail-0.2.7: {\"error\":{\"root_cause\":[{\"type\":\"parse_exception\",\"reason\":\"Failed to parse content to map\"}],\"type\":\"parse_exception\",\"reason\":\"Failed to parse content to map\",\"caused_by\":{\"type\":\"json_parse_exception\",\"reason\":\"Duplicate field 'ListGroupsForUser'\\n at [Source: (byte[])\\\"---\\ndescription: Pipeline for AWS CloudTrail Logs\\nprocessors:\\n  - set:\\n      field: event.ingested\\n      value: '{{_ingest.timestamp}}'\\n  - rename:\\n      field: \\\"message\\\"\\n      target_field: \\\"event.original\\\"\\n  - json:\\n      field: \\\"event.original\\\"\\n      target_field: \\\"json\\\"\\n  - date:\\n      field: \\\"json.eventTime\\\"\\n      target_field: \\\"@timestamp\\\"\\n      ignore_failure: true\\n      formats:\\n        - ISO8601\\n  - rename:\\n      field: \\\"json.eventVersion\\\"\\n      target_field: \\\"aws.cloudtrail.event_versi\\\"[truncated 16425 bytes]; line: 489, column: 26]\"}},\"status\":400}"
}
```
_observe new error format_
</details>
@jfsiii
Copy link
Contributor Author

jfsiii commented Sep 10, 2020

@elasticmachine merge upstream

@kibanamachine
Copy link
Contributor

💚 Build Succeeded

Build metrics

✅ unchanged

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@jfsiii jfsiii merged commit da86e3d into elastic:7.9 Sep 11, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
backport
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@jfsiii @kibanamachine @elasticmachine