Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Multiple inputs to esimport #98

Closed
envyes opened this issue Jan 22, 2019 · 8 comments
Closed

Multiple inputs to esimport #98

envyes opened this issue Jan 22, 2019 · 8 comments
Assignees
@jasonish
Labels
elasticsearch feature

Comments

@envyes
Copy link

envyes commented Jan 22, 2019

Hello
We are planning to run multiple eve instances in our setup. In addition we would like to send some evefiles to elasticsearch using esimport. Currently, the code supports only single input file. Are there plans to add multiple inputs to esimport? Or any suggestions how we can achieve this with the current version?
Thanks in advance
Nidhi.
@jasonish
Copy link
Owner

Hi Nidhi,

Right now the only way would be to run multiple esimport processes. I will unlikely add support to esimport to follow multiple files, I think that would be better done in the evebox agent which isn't that much different than esimport, but is more my idea of a long running process that could management multiple import files. Would that work for you if I were to add support for it to the agent?

@envyes
Copy link
Author

envyes commented Jan 23, 2019 via email

@jasonish
Copy link
Owner

Makes sense since esimport goes directly to Elastic Search and the agent doesn't. I may took at look at adding this soon.

@jasonish jasonish self-assigned this Jan 23, 2019
@envyes
Copy link
Author

envyes commented Jan 24, 2019 via email

@jasonish
Copy link
Owner

I took a look at this, and I think it makes sense to add this functionality. There would be a breaking change though.

Right now I have --bookmark and --bookmark-filename, where --bookmark-filename is required if --bookmark is in use. That doesn't apply to well if multiple inputs are using. So I'd probably remove those options and simply have --bookmark-directory. This directory would contain the bookmarks, where each bookmark file is a hash of the path being used.

jasonish added a commit that referenced this issue Feb 12, 2019
@jasonish
esimport: allow multiple input files
8e8a150 
Adds a --bookmark-dir option which is required if bookmarking
multiple input files.

Github issue:
#98
@jasonish
Copy link
Owner

I've added support for multiple files to esimport.

You can either specify multiple eve files on the command line or convert the input parameter in the config file to a list.

See: https://github.com/jasonish/evebox/blob/esimport/multiple-files/doc/esimport.yaml#L1

This branch also changes the default index from evebox to logstash. So if you aren't explicitly setting it, you might want to.

Builds from this branch can be downloaded here:
https://gitlab.com/jasonish/evebox/-/jobs/160376788/artifacts/browse

I'll be testing this myself over the next few days before merging back to master.

@envyes
Copy link
Author

envyes commented Feb 14, 2019 via email

jasonish added a commit that referenced this issue Feb 19, 2019
@jasonish
esimport: allow multiple input files
c437e69 
Adds a --bookmark-dir option which is required if bookmarking
multiple input files.

Github issue:
#98
@jasonish
Copy link
Owner

jasonish commented Feb 19, 2019 •
edited
Loading

Feature merged into master.

@jasonish jasonish added this to the 0.11 milestone Nov 22, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jasonish jasonish

Labels
elasticsearch feature
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jasonish @envyes