Event Listener - create deployment

[franTarkenton]

Having created a build action that builds a containerized image, this step will define the deployment process.

• create helm chart with all the necessary templates to deploy the various components to make the listener work.

[franTarkenton]

Ticket has mushroomed in complexity. Currently considered in scope:

• helm chart to deploy listener
• persistence - mostly where time has been spent this week more detail below.
• modularity - configure so uses the same configuration objects as the code that actually does the downloads and data preparation.

Persistence

The MSC data mart AMQ server emits events. We subscribe and then consume those events. When we recieve an event, the process will cache the event into a database, and also maintain an in memory data structure. After an event has been recieved and cached it gets acknowledged (ack'd) and thus removed from our message queue. The data needs to get persisted as if the pod dies / crashes etc it needs to be able to figure out what messages it has already recieved in order to be able to pickup where it left off.

[franTarkenton]

Ticket has mushroomed in complexity. Currently considered in scope:

• helm chart to deploy listener
• persistence - mostly where time has been spent this week more detail below.
• modularity - configure so uses the same configuration objects as the code that actually does the downloads and data preparation.

Persistence

The MSC data mart AMQ server emits events. We subscribe and then consume those events. When we recieve an event, the process will cache the event into a database, and also maintain an in memory data structure. After an event has been recieved and cached it gets acknowledged (ack'd) and thus removed from our message queue. The data needs to get persisted as if the pod dies / crashes etc it needs to be able to figure out what messages it has already recieved in order to be able to pickup where it left off.

[franTarkenton]

The PR delivers v1 of a listener... remaining work:

• create a github action to perform deployments - using an oc service account, deploy the actual code. (requires some thought around different envs. Only one upstream AMQ server etc)
• figure out how to call the CMC github action via a rest call and update the code to perform this