[Design]Automate the migration of log input states to filestream

[kvch]

Goal

Migrating from log input to filestream is painful if you want to avoid data duplication. The goal of this issue is to automatically migrate the existing state information of log input to filestream input. filestream input does not have access to log input states, and also stores information in a different data structure.

State info of log input

In the logfile

State: https://github.com/elastic/beats/blob/main/filebeat/input/file/state.go#L28-L42

{
  "k": "filebeat::logs::native::8787338-65029",
  "v": {
    "type": "log",
    "identifier_name": "native",
    "timestamp": [
      516104593874,
      1653410356
    ],
    "ttl": -1,
    "FileStateOS": {
      "device": 65029,
      "inode": 8787338
    },
    "id": "native::8787338-65029",
    "prev_id": "",
    "source": "/home/n/go/src/github.com/elastic/beats/filebeat/test.log",
    "offset": 29
  }
}

State info of filestream input

State: https://github.com/elastic/beats/blob/main/filebeat/input/filestream/input.go#L47-L49
Meta: https://github.com/elastic/beats/blob/main/filebeat/input/filestream/input.go#L51-L54

{
  "k": "filestream::my-filestream-id::native::8787338-65029",
  "v": {
    "ttl": 1800000000000,
    "updated": [
      516104569337,
      1653410356
    ],
    "cursor": {
      "offset": 29
    },
    "meta": {
      "source": "/home/n/go/src/github.com/elastic/beats/filebeat/test.log",
      "identifier_name": "native"
    }
  }
}

Differences

• ID of the state should include filestream instead of logs
• ID of the state must include a unique ID provided by the user
• type, id, prev_id and FileStateOS is not part of the filestream state
• identifier_name is part of meta
• source is part of meta
• offset is part of cursor

Note about container input

In case of conatiner input, meta also stores the stream the logs are coming from (stdout, stderr). In filestream it is concatenated to the end of the ID. Example: filestream::my-filestream-id::native::8787338-65029::stderr.

User interface

If we distribute is as part of Filebeat, I suggest we add a new subcommand.

./filebeat migrate log-to-filestream {id}

The name of the new subcommand could be migrate and log-to-filestream. It would accept one parameter, an ID or a comma separated list of IDs of filestream inputs. If no id is set, all filestream inputs are migrated.

Development

The registry consists of two files: key-value store and a log file. The selected states have to be updated in both places.

To avoid breaking existing state information of the user, we should backup the registry before doing any transformations. If something goes wrong, we can rollback to the previous version.

Testing

We have to test if the following cases in both the key-value store and in the log file:

• log input state with unfinished file
• log input state with finished file
• log input state with a TTL of -1
• log input state with a TTL of -2
• log input state with path identifier
• log input state with inode_marker identifier
• log input state where the input has an ID

The test case is successful if a valid log input state can be transformed to a valid filestream state and can be continued from the correct offset. This will need both unit tests and integration tests.

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[belimawr]

A couple of questions:

1. Regarding the user interface

The name of the new subcommand could be migrate and log-to-filestream. It would accept one parameter, an ID or a comma separated list of IDs of filestream inputs. If no id is set, all filestream inputs are migrated.

Why is the filestream input ID required here? On a first read I'd expect to indicate which log input state to migrate to filestream

2. How is the full workflow for it? I understand how the state transformation works, but I'm struggling to understand the whole workflow. Is it something like that:
   1. Filebeat is stopped
   2. filestream input configuration is added to configuration file (filebeat.yml)
   3. Migrate command is run
   4. log input configuration is removed
   5. Filebeat is restarted

[kvch]

Why is the filestream input ID required here?

It's part of the workflow I came up with. Obviously, I am open to different workflows, I tried to come up with something that involves the least amount of user interaction. Also, it is not required if you want to migrate all inputs to filestream format.

Migration workflow:

1. Stop Filebeat
2. Convert log input configuration to filestream: add an ID and rename the configuration options
3. Run migrate command, do some sanity check in the registry (maybe we could add debug logs to track the migration)
4. Restart Filebeat

In my workflow there is no intermediary step when there are both log and filestream inputs are included in the configuration.

[mguthrie88]

Just want to give a +1 to this feature. For our environment, clobbering the filebeat registry on an upgrade is not an option, and will prevent us from upgrading if a path forward isn't provided. We can't double-ship data.

[nix-power]

super important feature. +1