Handle the edge case when several globs in Filebeat path match

[rdner]

Currently we can have a lot of log message at warn level produced by this

beats/filebeat/input/filestream/fswatch.go

Lines 382 to 385 in 1f4051f

	if knownFilename, exists := uniqueIDs[fileID]; exists {
	s.log.Warnf("%q points to an already known ingest target %q [%s==%s]. Skipping", fd.Filename, knownFilename, fileID, fileID)
	continue
	}

This check is supposed to protect from having symlinks resolved to the same files that are being already ingested by Filebeat.

Seems like it's triggered not just by symlinks but also by duplicate path entries.

For example, monitoring in Elastic Agent has 2 globs and one of them is a superset of another:

https://github.com/elastic/elastic-agent/blob/dfd9554406f581660747663d15476c636095fa20/internal/pkg/agent/application/monitoring/v1_monitor.go#L301-L304

This causes flood of warning messages when Elastic Agent monitoring is enabled:

{
  "log.level": "warn",
  "@timestamp": "2023-08-07T16:44:03.445Z",
  "message": "\"/opt/Elastic/Agent/data/elastic-agent-0790a4/logs/elastic-agent-watcher-20230807.ndjson\" points to an already known ingest target \"/opt/Elastic/Agent/data/elastic-agent-0790a4/logs/elastic-agent-watcher-20230807.ndjson\" [1298191-66305==1298191-66305]. Skipping",
  "component": {
    "binary": "filebeat",
    "dataset": "elastic_agent.filebeat",
    "id": "filestream-monitoring",
    "type": "filestream"
  },
  "log": {
    "source": "filestream-monitoring"
  },
  "service.name": "filebeat",
  "ecs.version": "1.6.0",
  "log.logger": "scanner",
  "log.origin": {
    "file.line": 383,
    "file.name": "filestream/fswatch.go"
  }
}
{
  "log.level": "warn",
  "@timestamp": "2023-08-07T16:44:13.445Z",
  "message": "\"/opt/Elastic/Agent/data/elastic-agent-0790a4/logs/elastic-agent-watcher-20230807-1.ndjson\" points to an already known ingest target \"/opt/Elastic/Agent/data/elastic-agent-0790a4/logs/elastic-agent-watcher-20230807-1.ndjson\" [1298202-66305==1298202-66305]. Skipping",
  "component": {
    "binary": "filebeat",
    "dataset": "elastic_agent.filebeat",
    "id": "filestream-monitoring",
    "type": "filestream"
  },
  "log": {
    "source": "filestream-monitoring"
  },
  "log.logger": "scanner",
  "log.origin": {
    "file.line": 383,
    "file.name": "filestream/fswatch.go"
  },
  "service.name": "filebeat",
  "ecs.version": "1.6.0"
}
{
  "log.level": "warn",
  "@timestamp": "2023-08-07T16:44:13.445Z",
  "message": "\"/opt/Elastic/Agent/data/elastic-agent-0790a4/logs/elastic-agent-watcher-20230807.ndjson\" points to an already known ingest target \"/opt/Elastic/Agent/data/elastic-agent-0790a4/logs/elastic-agent-watcher-20230807.ndjson\" [1298191-66305==1298191-66305]. Skipping",
  "component": {
    "binary": "filebeat",
    "dataset": "elastic_agent.filebeat",
    "id": "filestream-monitoring",
    "type": "filestream"
  },
  "log": {
    "source": "filestream-monitoring"
  },
  "log.logger": "scanner",
  "log.origin": {
    "file.line": 383,
    "file.name": "filestream/fswatch.go"
  },
  "service.name": "filebeat",
  "ecs.version": "1.6.0"
}
{
  "log.level": "warn",
  "@timestamp": "2023-08-07T16:44:23.445Z",
  "message": "\"/opt/Elastic/Agent/data/elastic-agent-0790a4/logs/elastic-agent-watcher-20230807-1.ndjson\" points to an already known ingest target \"/opt/Elastic/Agent/data/elastic-agent-0790a4/logs/elastic-agent-watcher-20230807-1.ndjson\" [1298202-66305==1298202-66305]. Skipping",
  "component": {
    "binary": "filebeat",
    "dataset": "elastic_agent.filebeat",
    "id": "filestream-monitoring",
    "type": "filestream"
  },
  "log": {
    "source": "filestream-monitoring"
  },
  "log.origin": {
    "file.line": 383,
    "file.name": "filestream/fswatch.go"
  },
  "service.name": "filebeat",
  "ecs.version": "1.6.0",
  "log.logger": "scanner"
}
{
  "log.level": "warn",
  "@timestamp": "2023-08-07T16:44:23.445Z",
  "message": "\"/opt/Elastic/Agent/data/elastic-agent-0790a4/logs/elastic-agent-watcher-20230807.ndjson\" points to an already known ingest target \"/opt/Elastic/Agent/data/elastic-agent-0790a4/logs/elastic-agent-watcher-20230807.ndjson\" [1298191-66305==1298191-66305]. Skipping",
  "component": {
    "binary": "filebeat",
    "dataset": "elastic_agent.filebeat",
    "id": "filestream-monitoring",
    "type": "filestream"
  },
  "log": {
    "source": "filestream-monitoring"
  },
  "log.logger": "scanner",
  "log.origin": {
    "file.line": 383,
    "file.name": "filestream/fswatch.go"
  },
  "service.name": "filebeat",
  "ecs.version": "1.6.0"
}

This might be due to https://pkg.go.dev/path/[email protected]#Glob returning duplicate entries (subject to testing).

We should log warning ONLY in the symlink edge case, duplicate paths from Glob should be filtered out. New test case must be introduced.

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)