packetbeat requires "index" setting when using logstash output

[djschny]

It appears currently when using logstash output, there is an index setting. This seems very odd, as just because you're shipping to Logstash, does not mean Elasticsearch is involved. For example I could ship to Logstash and then from there ship to S3 or any other similar kind of store and never involve Elasticsearch. Making tight coupling between Elasticsearch and Packetbeat should be removed.

https://www.elastic.co/guide/en/beats/packetbeat/current/packetbeat-configuration.html#logstash-output

output:
  logstash:
    hosts: ["localhost:5044"]

    # index configures '@metadata.beat' field to be used by Logstash for
    # indexing. The default index name depends on the each beat.
    # For Packetbeat, the default is set to packetbeat, for Topbeat to
    # topbeat and for Filebeat to filebeat.
    index: packetbeat

My suggestion instead would be to remove this setting and then offer up a generic way to apply tags and metadata. For example:

• metadata.beat_type_field - The name of the field to use to store the type of beat. Defaults to beats.type
• metadata.beat_type_value - The value for the beat type field. Defaults to packetbeat

[tsg]

Not sure this is worth changing. "index" is meant to make the most common scenario easier, but it's not needed for operation and it's easy to remove it in Logstash if you don't want it. We have "tags" as a generic way to add metadata, and Filebeat has it's own "fields" setting where you can set metadata in a very similar way with what you propose.

If you want, we can take as an enhancement request being able to specify metadata as a key value pair for all the Beats.

[djschny]

I believe it is extremely important to change. My rationale for this because puts new users coming to a beat like this down a design path road where they think that the name of the index to which they should use (if using ES as the final persistence at the end of their LS config) should be configured/determined inside the beat. However that is a very poor place for it, as that should be controlled in the LS config and a mapping done there from generic beats metadata to index name(s).

The effort is small I'm assuming, but the clarity and benefit to folks is large.

[blanche]

as a user switching from logstash-forwarder this was very confusing to me.
it seems that the responsibilities of the different components (elasticsearch, logstash, filebeat) are being mixed up.

i thought of filebeat as a simple forwarder, but it now requires me to specify elasticsearch specific configuration in a component, that does not communicate with elasticsearch in this setup.

now which configuration is the dominant one?
the index configured in filebeat (filebeat-YYYY.MM.DD) or the one from logstash (logstash-YYYY.MM.DD)

[urso]

The setting acts as a hint for Logstash, allowing users to configure similar indexing strategy via Logstash as Packetbeat would have done itself. Logstash configurations are free to ignore the hint. The hint (and others) are send via the @metadata namespace, which will finally be removed by the outputs in Logstash. The @metadata namespace is used for internal processing.

@dedemorton Maybe we can clarify the use of this setting better in the docs. WDYT?

[dedemorton]

Agreed. I will clarify the setting in the docs.

[botelastic]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[botelastic]

This issue doesn't have a Team:<team> label.