Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add IP to supported types for Convert ingest processor #36145

Closed
geekpete opened this issue Dec 3, 2018 · 3 comments · Fixed by #69989
Closed

Add IP to supported types for Convert ingest processor #36145

geekpete opened this issue Dec 3, 2018 · 3 comments · Fixed by #69989
Assignees
Labels
:Data Management/Ingest Node Execution or management of Ingest Pipelines including GeoIP >feature Team:Data Management Meta label for data/management team

Comments

@geekpete
Copy link
Member

geekpete commented Dec 3, 2018

Describe the feature:

Add the ability to validate IP and convert addresses with the convert processor so that IP can be added to the list of supported types that currently include: integer, long, float, double, string, boolean, and auto.

This could use existing code doing validation checking when indexing to a mapped IP field:
"reason": "'192.168.256.1' is not an IP string literal."

Such a processor feature would allow detection handling or enrichment of ip values in inputs that have potentially mixed value types.

@geekpete geekpete added >feature :Data Management/Ingest Node Execution or management of Ingest Pipelines including GeoIP labels Dec 3, 2018
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-core-features

@andrewkroh
Copy link
Member

andrewkroh commented Apr 23, 2020

This would be helpful in all of the pipelines we build for Elastic SIEM. Elastic Common Schema declares a source.address keyword field and a source.ip ip field. You always populate source.address and then conditional populate source.ip if the value is an IP.

You shouldn't blindly copy the value from .address to .ip because this could result in a mapping exception. So if the convert processor could be used to validate the value that would help a lot.

In Beats the convert processor supports a psuedo-type for IPs. It doesn't really change the data type, but it will fail the "conversion" if the value is not an IP. https://www.elastic.co/guide/en/beats/filebeat/current/convert.html#convert

An equivalent setup with ingest node might be like:

PUT _ingest/pipeline/my-pipeline-id
{
  "processors" : [
    {
      "convert" : {
        "field" : "source.address",
        "target_field": "source.ip",
        "type": "ip",
        "ignore_failure": "true"
      }
    }
  ]
}

@geekpete
Copy link
Member Author

Epic! Thanks team!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Data Management/Ingest Node Execution or management of Ingest Pipelines including GeoIP >feature Team:Data Management Meta label for data/management team
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants