Skip to content

1.1.0
Compare
Choose a tag to compare
@joaolcorreia joaolcorreia released this 01 Mar 23:33
· 74 commits to main since this release
1.1.0
44f5528
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

What's New

JSON Output for all modules

In this version, we expanded support for flattened JSON output to all modules.
JSON enables easier integration with data infrastructure without creating dependencies on vendor-specific loaders.

JSON format improvements

A new entry is available in the experimental settings which allows defining the output format:

  "experimental": {
    # Whether to use an alternative output format  (either "FlattenedJson" or "EventbridgeJson")
    # - FlattenedJson: encodes the output as JSON, where unstruct_event, contexts and derived_contexts are flattened
    # - EventbridgeJson: encodes the output as JSON with extra details, intended exclusively to use with eventbridge module.
    "customOutputFormat": {
      "type": "FlattenedJson"
    }
  }

This is an example produced when enabling the FlattenedJson format (null values omitted for simplicity):

{
  "collector_tstamp" : "2011-12-03T10:15:30Z",
  "contexts_org_ietf_http_header_1" : [
    {
      "name" : "Host",
      "value" : "test"
    }
  ],
  "event_id" : "236392af-ffec-4def-a0de-86929e9615be",
}

Previously, this JSON format was available only for Kinesis but it can now be enabled in all modules.

NOTE: If you use the PII output. the JSON format is discouraged because we have not done enough tests on this.

EventbridgeJson format

This is a superset of the FlattenedJson format, it has the option to include a few extra fields in the output, it can be turned on with:

  "experimental": {
    # Whether to use an alternative output format  (either "FlattenedJson" or "EventbridgeJson")
    # - FlattenedJson: encodes the output as JSON, where unstruct_event, contexts and derived_contexts are flattened
    # - EventbridgeJson: encodes the output as JSON with extra details, intended exclusively to use with eventbridge module.
    "customOutputFormat": {
      "type": "EventbridgeJson"

      # Add a "payload" parameter with the original base64 encoded TSV. Defaults to false (Useful for EventbridgeJson only).
      "payload": true

      # Add a "collector" parameter from contexts_org_ietf_http_header_1 Host Value. Defaults to false (Useful for EventbridgeJson only).
      "collector": true
    }
  }

This is an example produced when enabling the FlattenedJson format (null values omitted for simplicity):

{
  "collector_tstamp" : "2011-12-03T10:15:30Z",
  "contexts_org_ietf_http_header_1" : [
    {
      "name" : "Host",
      "value" : "test"
    }
  ],
  "event_id" : "236392af-ffec-4def-a0de-86929e9615be",
  # produced because "collector" = true
  "collector" : "test",
  # produced because "payload" = true
  "payload" : "<<BASE64ENCODEDTDV>>,
}

NOTE: This format is intended for the eventbridge module only.

Breaking changes

If you are using the JSON output from Kinesis it means that you have these settings:

{
  ...
  output {
    good: {
      ...
      jsonOutput = true
    }

    pii: {
      ...
      jsonOutput = true
    }
    bad: {
      ...
      jsonOutput = true
    }
  }
}

These need to be removed, then, set the experimental settings like this:

  ...
  "experimental": {
    "customOutputFormat": {
      "type": "FlattenedJson"
    }
  }

If you are use eventbridge, you may have these settings:

{
  ...
  output {
    good: {
      ...
      collector = true
      payload = true
    }
  }
}

Which need to be replaced by:

{
  ...
  "experimental": {
    "customOutputFormat": {
      "type": "EventbridgeJson"
      "payload": true
      "collector": true
    }
  }
}

Changelog

New Contributors

Full Changelog: 1.0.0...1.1.0

Contributors

joaolcorreia, AlexITC, and 2 other contributors
Assets 7
Loading