Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[EPM] Implement ingest pipeline installation from packages #49752

Closed
skh opened this issue Oct 30, 2019 · 2 comments
Closed

[EPM] Implement ingest pipeline installation from packages #49752

skh opened this issue Oct 30, 2019 · 2 comments
Assignees
Labels
Feature:EPM Fleet team's Elastic Package Manager (aka Integrations) project Team:Fleet Team label for Observability Data Collection Fleet team

Comments

@skh
Copy link
Contributor

skh commented Oct 30, 2019

This task is to implement rewriting and installation of ingest pipelines from packages.

For each input/dataset defined in a package:

  • The main ("entry") pipeline is defined in the manifest.yml of the input/dataset.
  • This main pipeline may contain references to other pipelines. The reference in the pipeline looks like this: {{IngestPipeline 'my-pipeline'}} and refers to a pipeline called my-pipeline.json in the same directory as the main pipeline in the package.
  • The main pipeline will be called like the index alias it refers to (e.g. logs-nginx-prod-access-aabbcc)
  • (40752 rewrite ingest pipeline #50627) Rewrite the content of the main pipeline, replacing every instance of {{IngestPipeline 'some-pipeline'}} with logs-nginx-prod-access-aabbcc-some-pipeline (the name of the main pipeline and the name of the referenced pipeline joined by - to one long string)
  • Install all referenced pipelines with their new names, e.g. logs-nginx-prod-access-aabbcc-some-pipeline
  • Finally install the rewritten main pipeline with the name logs-nginx-prod-access-aabbcc
  • Pipelines may also be yml-files. The same rules apply.
  • It is possible that an input/dataset doesn't have any ingest pipelines defined.
  • If the format ({{IngestPipeline 'my-pipeline'}}) turns out to be hard to parse we can still change it.
@skh skh added [zube]: In Progress Team:Infra Monitoring UI - DEPRECATED DEPRECATED - Label for the Infra Monitoring UI team. Use Team:obs-ux-infra_services Feature:EPM Fleet team's Elastic Package Manager (aka Integrations) project labels Oct 30, 2019
@skh skh self-assigned this Oct 30, 2019
@elasticmachine
Copy link
Contributor

Pinging @elastic/logs-metrics-ui (Team:logs-metrics-ui)

@elasticmachine
Copy link
Contributor

Pinging @elastic/epm (Feature:EPM)

@ruflin ruflin closed this as completed Jan 16, 2020
@jen-huang jen-huang added Team:Fleet Team label for Observability Data Collection Fleet team and removed Team:Infra Monitoring UI - DEPRECATED DEPRECATED - Label for the Infra Monitoring UI team. Use Team:obs-ux-infra_services labels Mar 26, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Feature:EPM Fleet team's Elastic Package Manager (aka Integrations) project Team:Fleet Team label for Observability Data Collection Fleet team
Projects
None yet
Development

No branches or pull requests

5 participants