create CDC event stream CRD

[FxKu]

The PR introduces a new operator feature to enable change data capture (CDC) streams into Zalando’s distributed event broker Nakadi. It utilizes an internal Zalando operator for creating Debezium workflows reading changes from Postgres’ logical replication slots and distribute to Nakadi (or AWS SQS)

From a new section in the manifest to list streams, the operator will create or update a custom resource that can be picked up by the CDC operator. The Postgres Operator user must specify the database tables for which logical decoding is enabled as well as event topics which are defined in the sink (Nakadi). The CDC operator is build around the outbox pattern. It’s idea is to create a de-normalized table in Postgres that resembles the structure of then event sink and perform logical decoding only on this “outbox” table. Either the app or the database is then responsible to copy change data from original tables to the outbox table. The Postgres Operator assumes that the outbox table is called like the specified table, appended with event type and _outbox suffix.

ToDo:

• fix code generation, atm it has been created manually. Running ./hack/update-codegen.sh only updates acid API
• do we need to sync DB resources? Might be the job of the CDC operator to report if specified databases and tables do not exist The operator should not deal with creating tables
• should we patch the Postgres manifest, setting wal_level to logical and creating a logical replication slot under Patroni section? Or is it up to the user to configure it before enabling streams? yes and no: the operator changes the Postgres config without the need to prepare the manifest
• operator should create a replication user to be used for all event streams

[sdudoladov]

why is this counter needed ?

[FxKu]

to fill the tableNames slice a bit quicker. I read that append (which we usually use 😃 ) is slower.

[sdudoladov]

👍

[sdudoladov]

👍

[FxKu]

👍

[achrafsahnoun1]

Hey, I want to include the streams section in the cluster manifet of postgres, I didnt find any example on how to write the tables that we want to include, this is one of my trials:

Enables change data capture streams for defined database tables

streams:

• applicationId: test-app
  database: postgres
  tables:
  data.state_pending_outbox:
  eventType: test-app.status-pending
  data.state_approved_outbox:
  eventType: test-app.status-approved
  data.orders_outbox:
  eventType: test-app.order-completed
  idColumn: o_id
  payloadColumn: o_payload
  data.links_outbox:
  eventType: test-app.order-completed
  idColumn: id
  payloadColumn: payload
  ///////////////////////////////
  In this case I want to include the links table which is in the test database, I checked if there are any publication created from the operator for that table since it should create it automatically but there is no one created. Thank you in advance.