View events from a custom elasticsearch index

[satie]

I am running a 6.3.2 ELK stack with logstash sending eve logs to a custom index. When I run the evebox docker container with the -I option, specifying the index name, I do not see any events in web interface. Here is the command I am running -

$ docker run --name evebox -it -p 5636:5636 jasonish/evebox -e http://10.10.0.18:9200 -i suricata-2018.11.07

Do I need to specify another option, or change one of the options to get evebox to show the events?

BTW, the oneshot option works when I send the eve.json file to evebox.

[jasonish]

Try using -i suricata. Evebox will fill out the pattern with a wildcard.

[satie]

Thanks. No luck with that option either. It seems like evebox is looking for a template called _template/suricata. My template has a different name. Could that be causing the events to not load?

Here is the log output

2018-11-08 18:54:11 (geoip-service.go:44) <Warning> -- Failed to initialize geoip database: no database files found
2018-11-08 18:54:11 (configdb.go:52) <Info> -- Using in-memory configuration DB.
2018-11-08 18:54:11 (migrator.go:68) <Debug> -- Initializing database.
2018-11-08 18:54:11 (migrator.go:79) <Info> -- Updating database to version 0.
2018-11-08 18:54:11 (migrator.go:79) <Info> -- Updating database to version 1.
2018-11-08 18:54:11 (resources.go:50) <Warning> -- Failed to find box asset ./resources/configdb/V2.sql.
2018-11-08 18:54:11 (server.go:303) <Info> -- Configuring ElasticSearch datastore
2018-11-08 18:54:11 (server.go:304) <Info> -- Using ElasticSearch URL http://10.10.0.18:9200
2018-11-08 18:54:11 (server.go:306) <Info> -- Using ElasticSearch Index suricata.
2018-11-08 18:54:11 (elasticsearch.go:110) <Info> -- Event base index: suricata
2018-11-08 18:54:11 (elasticsearch.go:111) <Info> -- Event search index: suricata-*
2018-11-08 18:54:11 (server.go:335) <Info> -- Connected to Elastic Search (version: 6.3.2)
2018-11-08 18:54:11 (elasticsearch.go:155) <Debug> -- Fetching template _template/suricata
2018-11-08 18:54:11 (elasticsearch.go:119) <Debug> -- Decoding response (truncated at 1024 bytes): {}
2018-11-08 18:54:11 (elasticsearch.go:193) <Info> -- Found templates []
2018-11-08 18:54:11 (elasticsearch.go:196) <Debug> -- Found template version <nil>
2018-11-08 18:54:11 (elasticsearch.go:235) <Warning> -- Failed to determine Elastic Search keyword suffix, will use 'raw'.
2018-11-08 18:54:11 (server.go:131) <Info> -- Session reaper started
2018-11-08 18:54:11 (server.go:165) <Info> -- Authentication disabled.
2018-11-08 18:54:11 (server.go:276) <Info> -- Listening on 0.0.0.0:5636
2018-11-08 18:54:21 (anonymous.go:64) <Info> -- Logging in anonymous user {anonymous} from 172.17.0.1:42094
2018-11-08 18:54:21 (datastore-alertquery.go:155) <Info> -- Query elapsed time: 8.417083ms
2018-11-08 18:54:24 (anonymous.go:64) <Info> -- Logging in anonymous user {anonymous} from 172.17.0.1:42094
2018-11-08 18:54:24 (anonymous.go:64) <Info> -- Logging in anonymous user {anonymous} from 172.17.0.1:42104
2018-11-08 18:54:24 (anonymous.go:64) <Info> -- Logging in anonymous user {anonymous} from 172.17.0.1:42098
2018-11-08 18:54:24 (anonymous.go:64) <Info> -- Logging in anonymous user {anonymous} from 172.17.0.1:42106
2018-11-08 18:54:24 (anonymous.go:64) <Info> -- Logging in anonymous user {anonymous} from 172.17.0.1:42102
2018-11-08 18:54:24 (anonymous.go:64) <Info> -- Logging in anonymous user {anonymous} from 172.17.0.1:42100
2018-11-08 18:55:11 (server.go:127) <Debug> -- Reaping sessions.

[jasonish]

Ok. What’s the name? EveBox allows for a different name, but expects it to be in the default logstash style. Can you tell me how your differs from the default logstash style?

[jasonish]

Also, see if you can see events under the event tab, it’s a little more forgiving than the inbox. If so, try the —keyword option (or is it —elasticsearch-keyword) and set the value to 'keyword'.

[satie]

The template is called <companyname>-suricata. The template is created with the logstash output plugin.

The --elasticsearch-keyword set to keyword did not help either.

Thanks.

[satie]

I renamed the template to suricata and restarted logstash and evebox.
Evebox seems to find the template but does not display any events. Here is the command and log output
$ docker run -it -p 5636:5636 jasonish/evebox -e http://10.10.0.18:9200 -i suricata --elasticsearch-keyword keyword --verbose

2018-11-08 20:38:04 (server.go:176) <Info> -- This is EveBox Server version 0.9.1 (rev: 159961e); os=linux, arch=amd64
2018-11-08 20:38:04 (geoip-service.go:44) <Warning> -- Failed to initialize geoip database: no database files found
2018-11-08 20:38:04 (configdb.go:52) <Info> -- Using in-memory configuration DB.
2018-11-08 20:38:04 (migrator.go:68) <Debug> -- Initializing database.
2018-11-08 20:38:04 (migrator.go:79) <Info> -- Updating database to version 0.
2018-11-08 20:38:04 (migrator.go:79) <Info> -- Updating database to version 1.
2018-11-08 20:38:04 (resources.go:50) <Warning> -- Failed to find box asset ./resources/configdb/V2.sql.
2018-11-08 20:38:04 (server.go:303) <Info> -- Configuring ElasticSearch datastore
2018-11-08 20:38:04 (server.go:304) <Info> -- Using ElasticSearch URL http://10.10.0.18:9200
2018-11-08 20:38:04 (server.go:306) <Info> -- Using ElasticSearch Index suricata.
2018-11-08 20:38:04 (elasticsearch.go:110) <Info> -- Event base index: suricata
2018-11-08 20:38:04 (elasticsearch.go:111) <Info> -- Event search index: suricata-*
2018-11-08 20:38:04 (server.go:335) <Info> -- Connected to Elastic Search (version: 6.3.2)
2018-11-08 20:38:04 (elasticsearch.go:155) <Debug> -- Fetching template _template/suricata
2018-11-08 20:38:04 (elasticsearch.go:119) <Debug> -- Decoding response (truncated at 1024 bytes): {"suricata":{"order":0,"version":10001,"index_patterns":["suricata-*"],"settings":{"index":{"codec":"best_compression","refresh_interval":"10s","number_of_shards":"3","number_of_replicas":"1"}},"mappings":{"_default_":{"numeric_detection":true,"dynamic_templates":[{"tcp.ack":{"path_match":"tcp.ack","mapping":{"type":"boolean"}}},{"tcp.cwr":{"path_match":"tcp.cwr","mapping":{"type":"boolean"}}},{"tcp.ece":{"path_match":"tcp.ece","mapping":{"type":"boolean"}}},{"tcp.fin":{"path_match":"tcp.fin","mapping":{"type":"boolean"}}},{"tcp.psh":{"path_match":"tcp.psh","mapping":{"type":"boolean"}}},{"tcp.rst":{"path_match":"tcp.rst","mapping":{"type":"boolean"}}},{"tcp.syn":{"path_match":"tcp.syn","mapping":{"type":"boolean"}}},{"tcp.urg":{"path_match":"tcp.urg","mapping":{"type":"boolean"}}},{"string_fields":{"match_mapping_type":"string","match":"*","mapping":{"type":"keyword"}}}],"properties":{"@timestamp":{"type":"date"},"@version":{"type":"keyword"},"analytics":{"dynamic":true,"type":"object","properties":{"bytes_m
2018-11-08 20:38:04 (elasticsearch.go:193) <Info> -- Found templates [suricata]
2018-11-08 20:38:04 (elasticsearch.go:196) <Debug> -- Found template version 10001
2018-11-08 20:38:04 (elasticsearch.go:205) <Info> -- Elastic Search EVE records are using IP datatype.
2018-11-08 20:38:04 (server.go:131) <Info> -- Session reaper started
2018-11-08 20:38:04 (server.go:165) <Info> -- Authentication disabled.
2018-11-08 20:38:04 (server.go:276) <Info> -- Listening on 0.0.0.0:5636
2018-11-08 20:38:52 (anonymous.go:64) <Info> -- Logging in anonymous user {anonymous} from 172.17.0.1:59672
2018-11-08 20:38:52 (datastore-alertquery.go:155) <Info> -- Query elapsed time: 14.420391ms

[jasonish]

Regarding your last comment, Elastic Search can be weird when tweaking things without an index rollover. Sometimes this fixes itself within 24 hours. At least I don't know the proper fix, but this seems to happen.

The real fix for this issue is to allow a template name as well as an index name to be specified. Which I will add real soon.

[jasonish]

If you're willing to test, I have a build up here:

https://gitlab.com/jasonish/evebox/-/jobs/124323346/artifacts/browse

It adds a new command line option, --elasticsearch-template instead of deriving the template name from the index name.

[satie]

Thanks! Will give this a try. I will close this issue for now. Will reopen or reach out if I need help.