Add an new input type to backfill gzipped logs

[lminaudier]

Hi,

Following this discussion on filebeat forum I would like to ask if it is possible to implement a solution to easily backfill old gzipped logs with filebeat.

The proposed solution mentioned in the topic is to add a new dedicated input_type.

It is also mentioned in the topic that when filebeat reaches the end of input on stdin it does not give you the hand back and waits for new lines which makes things hard to script to perform backfilling.

What are your thoughts on this ?

Thanks for your hard work.

[ruflin]

I would see the implementation as following:

• Prospector: A gzip prospector would be added. Harvesters would only be opened based on filenames (no inode etc.). It is assume, that the files are not renamed and don't have to be tracked. A file has a completion state and if it is once completed, it is never read again. This simplifies the implementation. If a .gz file with a new filename is found, a harvester is started.
• Harvester: A gzip harvester is added. It unzips and reads the full file only once. After finishing reading the file, the harvester stops and is never started again. The offset is only stored, if it is interrupted in the middle of reading. In a first implementation, this could even be removed for simplicity.

This would be nice to have but I think it is not on the top of our priority list. A community contribution here would be more then welcome.

For your second issue about running filebeat only until first completion, lets refer to this issue: https://github.com/elastic/filebeat/issues/219

[lminaudier]

Thanks for the fast reply and the pointer to the issue.

I will try to look at your implementation proposal. I am still quite new to Golang and the project, so I can't promise anything :)

[ruflin]

@lminaudier Always here to help.

[mryanb]

This would be a great feature addition. Currently the splunk-forwarder does something similar and will index log rotated files that have been gziped automatically.

[Ragsboss]

+1. Is anyone working on this? If not I could possibly take it up..

[ruflin]

@Ragsboss I don't think anyone is working on that. Would be great to have a PR for that to discuss the details.

[cFire]

@Ragsboss @ruflin I'm delighted to see there's someone looking to pick this up. Is this happening? The reason I ask is because it may be possible for me to spend some time helping out with this in lieu of building another solution for gziped logs to use internally.

[Ragsboss]

@cFire please feel free to take this up. I've started looking at the code familiarizing myself with Go in general and FileBeat code. But I haven't started the real work yet. I'll try to help you in anyway you want, just let me know.
Few thoughts I had. From a pure functional viewpoint, defining a new input_type doesn't seem ideal as that would force users to author a new prospector in the config file. Instead I felt it may be better for the code to automatically deal with compressed files as long as they match the given filename patterns in the config file. The code can instantiate a different harvester (IIUC) based on the file extension/type. But from implementation viewpoint if this is turning out to be difficult, I think it's ok to burden/ask the users for some extra config...

[ruflin]

From an implementation point of view I think we should go the route of having a separate input type. Currently the way filebeat is designed is that a prospector and harvester type are tightly coupled, so a prospector only starts one type of harvesters. It is ok if gzip harvester reuses lots of code from the log harvester (which I think it will) but as log tailing and reading a file completely only once are form my perspective two quite different behaviours. The question which will also be raised is if the gzip files will change their name over time (means have to be tracked based on inode / device) or it is enough to just store the filename and a read / unread flag in the registry.

[ruflin]

Here is the PR related to the above discussions: #2227

[willsheppard]

We would like filebeat to be able to read gzipped files, too.

Our main use of filebeat would be to take a set of rotated, gzipped logs that represent the previous day's events, and send them to elasticsearch.

No tailing or running as a service is required, so a "batch mode" would also be good, but other workarounds solely using filebeat would also be acceptable.

[ruflin]

@willsheppard Thanks for the details. For the batch mode you will be interested in #2456 (in progress)

[ruflin]

Now that #2456 is merged this feature got even more interesting :-)

[collabccie7]

Hello,

Has there been any update regarding the support for gzip files ? Please let know.

Thanks.

[cFire]

Idk about the others, but I've not gotten any time to work on this.

[ruflin]

No progress yet on this from my side.

[willsheppard]

This gzip input filter would be the killer feature for us. We're being forced to consider writing an Elasticsearch ingest script from scratch which writes to the Bulk API, because we need to operate on logs in-place (no space to unzip them), and we would be using batch-mode (#2456) to ingest yesterday's logs from our web clusters.

[maddazzaofdudley]

Has there been any movement on this as this too is the killer feature for me too

[woodchalk]

Throwing in my support for this feature.

[plumpNation]

Would be an awesome feature to have.

[ruflin]

There is this open PR here that still needs some work and also involves quite some discussions: #3070

[jordansissel]

Harvesters would only be opened based on filenames (no inode etc.).

@ruflin I believe inodes may to be tracked because logrotate (assuming this is a target use case) renames files and reuses file names. Unless another tracking mechanism (when is 'hello.txt.1.gz' a "new file" below, for example)

Example:

% ls -il /tmp/hello.txt*
103196 -rw-rw-r--. 1 jls jls 12 Jan 24 03:17 /tmp/hello.txt
103131 -rw-rw-r--. 1 jls jls 32 Jan 24 03:16 /tmp/hello.txt.1.gz

% cat test.conf
/tmp/hello.txt {
  rotate 5
  compress
}

% logrotate -s /tmp/example.logrotate -f test.conf

% ls -il /tmp/hello.txt*
103218 -rw-rw-r--. 1 jls jls 32 Jan 24 03:17 /tmp/hello.txt.1.gz
103131 -rw-rw-r--. 1 jls jls 32 Jan 24 03:16 /tmp/hello.txt.2.gz

^^ Above, 'hello.txt.2.gz' is the same file (inode) as previous 'hello.txt.1.gz'.

We can probably achieve this without tracking inodes (tracking modification time and only reading .gz files after they have been idle for some small time?), but I think the filename alone is not enough because file names are reused, right?

[ruflin]

I exactly hit this issues during the implementation. That is why the implementation in #3070 is not fully consistent with the initial theory in this thread.

The main difference now to a "normal" file is that it is expected, that a gz never changes, and if it changes, the complete file is read from the beginning again.

[ITSEC-Hescalona]

+1 for gzip please

[notzippy]

As an alternative, you can use mkfifo and pipe that stream into filebeat and pipe all your files into that pipe. That way you do not need to worry when you are finished ingesting a file because you can just automatically start piping the next file into it. So my zcats go into zcat ... >fb input and in another terminal, I have filebeat < fb_input

[bunnHHA]

any update on gzip files? Using zcat solve some issues but it does need to be done in manual all the time.

[elasticmachine]

Pinging @elastic/integrations-services (Team:Services)

[dwdixon]

+1 to adding this feature...is there any update on the timeline for this being added to Filebeat?

Edit: Sorry for "spam" just saw that comment... : )

[cancer13]

+1
any update?

[NemchinovSergey]

+1
parsing gzip would be great!
files with plain text are huge!

[arasic]

Hello everyone,

The feature has not been implemented even if it has been requested by many users.
The ticket #2227 was closed because there was no progress for some time. Some progess has been made, but I don't know whats remaining.

Not sure to understand if the feature was not implemented, is it because its a challenging problem or because lack of interest.

From what I understood, there are two difficulties:

• difficult to track/keep the offest of what has already been processed
• the files can be larger when extracted, like 50GBs and more?
• Did I miss any other challenging question?

So now, we realized that not only we cannot process the .gz files, but they should be excluded in exclude_files: [".gz"], otherwise we will get scrambled text.

The most helpful thing I found was posted by Vasek Šulc:
https://discuss.elastic.co/t/filebeat-on-demand-loading-compressed-gz-files-from-stdin/184159, which is to use the stdin configurations in filebeat, and redirect or pipe the compressed file outputs using zcat.

[botelastic]

This issue doesn't have a Team:<team> label.

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[jlind23]

@nimarezainia From a pure product perspective, how does this issue fit in our actual roadmap and strategy?

[nerophon]

Competitors have this feature, e.g. Splunk forwarders, and also Graphana Loki: grafana/loki#5956.

[nimarezainia]

this is tracked i the enhancements repo how ever not slated for development due to other higher priority items on the list.

[fzyzcjy]

Hi, is there any updates? Thanks

[nimarezainia]

Hi, is there any updates? Thanks

@fzyzcjy right now we don't have a development date for this.

[mathemaphysics]

This would be fabulous to have this feature. Accessing files directly would allow reading of log files directly without decompression of the file hierarchy. Text compression is generally very good, so this would be in line with preservation of disk space.

[matschaffer-roblox]

@jlind23 @nimarezainia incase it's helpful here's how I found this issue today:

We use a lot of AWS EMR where Amazon handles ingest of some logs (usually enough) but not all. Last weekend we hit a problem with one of the services in the mix that weren't ingested. We started a new EMR deployment, scp'd most of the logs (some raw, some gz) from the old deployment and terminated it. Now I'd like to get the old logs ingested for analysis.

ML's Data Visualizer does a great job on the raw files and even spits out a filebeat config for me. It'd be impressive if filebeat just worked for the .gz'd logs as well so I could slurp up everything I scp'd.

Tangentially having a filebeat option for https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html would also make this task easier.

Being able to quickly ingest and analyze a handful of local logs could be a nice on-ramp to continual ingestion as well, provided all the artifacts (filebeat config, ingest pipeline) were easily portable to something like a cloud deployment.

[botelastic]

Hi!
We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!

[philhagen]

This would be an immensely helpful feature. 👍