OBSDATA-440 Adding SegmentMetadataEvent and publishing them via KafkaSegmentMetadataEmitter

[harinirajendran]

Adding the new SegmentMetadataEvent and publishing these segment-related metadata events into Kafka by enhancing the KafkaEmitter

Description

In this PR, we are enhancing KafkaEmitter, to emit metadata about published segments (SegmentMetadataEvent) into a Kafka topic. This segment metadata information that gets published into Kafka, can be used by any other downstream services to query Druid intelligently based on the segments published.

Old behavior of Kafka Emitter

Kafka Emitter always emits metrics and alerts and would emit requests if the config request.topic is configured.
Configs metric.topic and alert.topic are always mandatory and cannot be null.

Current behavior of Kafka Emitter [with backwards compatibility]

We introduced a new config named event.types which dictates the types of events we want the KafkaEmitter to emit. This config takes in a list of strings and can have one or more from [alerts, metrics, requests and segmentMetadata]. And based on this config, alert.topic, request.topic, metric.topic and segmentMetadata.topic should be configured and not left empty.
If no event.types is set, then by default, the kafka emitter would emit metrics and alerts. And in that case, to maintain backwards compatibility, decision to send out requests are based on if request.topic is empty or set.

Metadata format

The emitter by default would emit the metadata in json string format. If segment.metadata.format is set to protobuf, then the emitter emits it in protobuf format.

Testing Done

• Added unit tests
• Tested by running Druid and ingesting data.
• Enabled compaction and verified the fields are appropriately set in the SegmentMetadataEvent class
• Verified SegmentMetadataEvent is getting published in Kafka

This PR has:

• been self-reviewed.
• added documentation for new or modified features or behaviors.
• added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
• added or updated version, license, or notice information in licenses.yaml
• added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
• added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
• added integration tests.
• been tested in a test Druid cluster.

[kkonstantine]

@harinirajendran , fyi, if you open the PR as draft, it won't add the owners as reviewers after it's opened and will also provide a clear indication that this is a draft not ready to be merged yet.

[harinirajendran]

@harinirajendran , fyi, if you open the PR as draft, it won't add the owners as reviewers after it's opened and will also provide a clear indication that this is a draft not ready to be merged yet.

Thanks @kkonstantine . I converted this to a Draft PR :)

[lqxshay]

Thanks for the PR. I did my first pass, would have to set up druid codebase also.

[harinirajendran]

I have addressed all other comments except this one in this update. Will add a different commit addressing this comment and re-request reviews at that point.

[harinirajendran]

Updated the following

1. Removed the -protobuf suffix when the segment metata format is not JSON.
2. Fixed the double counting issue.

[xvrl]

LGTM, added a note that we should do a follow-up to add tests for the eventTypes logic

[harinirajendran]

Thanks, @xvrl for the review.

[xvrl]

before we merge this, we should make sure we enable tests for the target branch to ensure this passes all tests / validations

The base branch was changed.