Read from custom ES index

[psuhaj]

Hello. I wanted to ask if it is any option to read from specified elasticsearch index? I ship suricata logs to elasticsearch using filebeat (not using the filebeat module, but setting it up in filebeat.yml). If I specify the option -i filebeat then Evebox won't read from the indices created by filebeat (format is filebeat-7.11.2-YY.mm.dd), but it will create a new index in format filebeat-YY.mm.dd. Is there any solution for this? I want to rotate my indices hourly using elastic index lifecycle management, which is applied on indices created by filebeat. Thank you.

[jasonish]

The -i filebeat option should use the index pattern filebeat-* when searching.

I'm not sure why EveBox would be creating the index, that should only be done when configuring EveBox to also ship logs to Elasticsearch which is not the default.

[psuhaj]

I ran it with this command: evebox server -e http://localhost:9200 --host 0.0.0.0 -i filebeat, is it ok?

I am checking it now and you are right, it did not create new index, it could be some older index which remained. Bud it still does not read the data from filebeat indices. The indices are named like this:

[jasonish]

Can you share your filebeat configuration? Is it using json.keys_under_root: true?

[psuhaj]

Yes it is using. Here is my filebeat.yaml.

filebeat.inputs:
- type: log  
  enabled: true
  paths:
    - /var/log/suricata/eve.json
  json.keys_under_root: true
  overwrite_keys: true

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

setup.template.settings:
  index.number_of_shards: 1
  index.number_of_replicas: 0
  
setup.dashboards.enabled: false
  
output.elasticsearch:  
  hosts: ["http://localhost:9200"]
    
processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

One more question, is the option database.elasticsearch.index in evebox.yaml the same setting as specifiing the option -i? When I set the index name to filebeat in evebox.yaml then after start it creates other index, see the screenshot below. I have the option input.enabled set to false. I start filebeat before starting evebox, so the index is created before evebox is started.

[jasonish]

Yes, -i, --index, and database.elasticsearch.index are all the same. Command line takes precedence over the configuration file.

I'm curious, in your setup does Filebeat add events to an index that has the "-date" suffix on it? Or something just like "filebeat" with no suffix. The search pattern used is "-", so if your "current" index is simply "filebeat", maybe this won't work, but should pick up older events. Maybe I should just use "" which should then pickup "filebeat".

What versions are you using? Elastic moves at a breaking change pace thats hard to keep up with. My last set of comprehensive testing was with 7.10, but I haven't tested 7.11.. I am using 7.11.2 in my own install, but I'm using the EveBox method of adding events to Elastic which is closer to the legacy way of Logstash than Filebeat.

[psuhaj]

Filebeat creates index like that first in the screenshot above, so the format is filebeat-{version}-{date}-{num}, where num is starting with 000001 and is incrementing when indexes are being rotated by elastic ILM. I am using the latest versions (7.11.2).

[jasonish]

Ok, found an issue. If using the default configuration file, and the input section exists, but is enabled, it still gets loaded so EveBox will create an index and start adding alerts.

Quick fix would be to:

• Don't use the configuration, just use command line parameters.
• OR: Remove the input section from the configuration file.

Then, while I don't think this is required, but still good to clean things up:

• Remove the "filebeat-YYYY.MM.DD" indices that EveBox created (I just do this through Kibana)
• Remove the "filebeat" legacy template that was created by EveBox.

But I'll be fixing the code for this right away as well.

[psuhaj]

Thank you. As I see it is fixed now, I will give it a try. Is the 0.13 the new stable version?

[jasonish]

Thank you. As I see it is fixed now, I will give it a try. Is the 0.13 the new stable version?

Yeah, quite a few little fixes built up in there and thought that this was the tipping point for a release.

[jasonish]

Fixed in 0.13.0.