Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Nuke protobuf #9207

Closed
yakkomajuri opened this issue Mar 23, 2022 · 5 comments
Closed

Nuke protobuf #9207

yakkomajuri opened this issue Mar 23, 2022 · 5 comments

Comments

@yakkomajuri
Copy link
Contributor

yakkomajuri commented Mar 23, 2022

The persons on events changes we're about to start making require sending new bits of data over through Kafka. These would require changes in the protobuf schema but honestly we might be better off just dropping it at this stage.

@macobo has done some on the initial legwork to support non-protobuf messages on #8333 but I suggest we make the move now.

This would require:

  1. Setting up a new Kafka topic (manually on Cloud, self-hosted topics are auto-created)
  2. Setting up a new set of Kafka + MV tables to consume from the topic into writable_events
  3. Begin sending all new events in a JSON encoded format to the new JSON topic
  4. Keep the protobuf topic and tables around for a release and then nuke it on the following one, so we ensure all events from that path are correctly processed
  5. Remove all the related code

A gotcha here is at step 2. What if the user isn't sharded but becomes sharded during this transition? We probably need to retroactively update the sharding async migration to drop and recreate the MV for the JSON topic as well in order to do this so that we write to the correct table.

@yakkomajuri yakkomajuri self-assigned this Mar 23, 2022
@yakkomajuri yakkomajuri changed the title Migrate away from protobuf Nuke protobuf Mar 23, 2022
@guidoiaquinti
Copy link
Contributor

More context at #8334

@yakkomajuri
Copy link
Contributor Author

Did I miss anything @macobo @guidoiaquinti @hazzadous @fuziontech

@yakkomajuri
Copy link
Contributor Author

I'll take the thumbs up as "no I didn't miss anything" rather than a "yes" response to the question and move on :D

@guidoiaquinti
Copy link
Contributor

You didn't miss anything 👍 😉 (I think)

@yakkomajuri
Copy link
Contributor Author

Well so this was done but now #9368 became a part of this work as well

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Development

No branches or pull requests

2 participants