Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sync protobufs/topics with sentry-kafka-schemas #103

Open
evanh opened this issue Dec 18, 2024 · 1 comment
Open

Sync protobufs/topics with sentry-kafka-schemas #103

evanh opened this issue Dec 18, 2024 · 1 comment

Comments

@evanh
Copy link
Member

evanh commented Dec 18, 2024

Currently the protobuf definitions for the taskbroker are in sentry-protos. However the kafka topic needs to be defined in sentry-kafka-schemas to be created in production, and the topic shouldn't be created with an empty schema.

Options/Ideas:

  • Have the schemas defined in both places, and try to manually keep them in sync
  • Move entirely to sentry-kafka-schemas. This will require work in that repo to make that efficient in processing protobuf encoded messages.
  • Keep the schemas in sentry-protos, and import them into sentry-kafka-schemas. This would also require some work to create that mechanism.
@markstory
Copy link
Member

However the kafka topic needs to be defined in sentry-kafka-schemas to be created in production, and the topic shouldn't be created with an empty schema.

Currently, sentry-kafka-schemas can reference a protobuf struct as a message encoding, and I've linked sentry-protos in so that the validation methods in sentry-kafka-schemas will use the generated protobuf bindings. Which is roughly the flow you mentioned.

Keep the schemas in sentry-protos, and import them into sentry-kafka-schemas. This would also require some work to create that mechanism.

Are there other gaps we still need to address?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants