[data dashboard backend] Add Data Dashboard Backend service

[pvannierop]

Description of the change

Data dashboard backend is a new service that exposes data from the observation table in timescaledb database (provisioned by KSQLDB service) to frontend/mobile applications.

See: https://github.com/RADAR-base/radar-data-dashboard-backend

This PR will implement the Data dashboard backend service as well as include the KSQLDB transformer logic to populate the observation table in timescaledb.

Additional information

• Transformer logic transforms messages in any kafka topic to a single observation table. For now, only questionnaire_response and questionnaire_app_events are handled. Transformation logic for other topics can be added in the future.
• Ingested topics are configured by selective uncommenting/addition in base.yaml.gotmpl file:

kafka_data_transformer:
 ksql:
   queries: |
     {{/* - readFile "../etc/cp-ksql-server/_base_observations_stream.sql"             | nindent 8 */}}
     {{/* - readFile "../etc/cp-ksql-server/questionnaire_response_observations.sql"   | nindent 8 */}}
#      {{/* - readFile "../etc/cp-ksql-server/questionnaire_app_events_observations.sql" | nindent 8 */}}

• The existing 20-grafana.yaml file name was changed to 20-dashboard.yaml to fit the expanded scope. In addition, the database name was changed from grafana-metrics to data-dashboard.

[keyvaann]

Did you change permissions of generate-secrets script? Seems like it can't run anymore
https://github.com/RADAR-base/RADAR-Kubernetes/actions/runs/9481223536/job/26123532105#step:5:39

[yatharthranjan]

Looks good. Please find some comments.

[yatharthranjan]

again it can be enabled automatically in the helmfiles using templating with if statements

[pvannierop]

Indeed. We decided to postpone this to future changes.

[yatharthranjan]

DATA_TYPE instead of topic ?

[pvannierop]

Well, that is a API question. Would we rather use radar-centric terminology (subject, topic), or expose a different API to consumers (e.g., participant, data_type). It depends on the consumers I guess. I personally think that the RADAR-base teams of The Hyve and KCL are basically the only users of this API. To me, it looks like an extra burden to juggle these translations all the time:

• participant means subject
• data_type means topic

[yatharthranjan]

ok sure, just mentioned to avoid conflicts and confusion with the KSQL topic keyword. You can use TOPIC_NAME maybe to avoid conflicts?

[pvannierop]

I renamed the column to TOPIC_NAME as suggested.

[yatharthranjan]

TIME?

[pvannierop]

I think that there is a time column created by jdbc-connector. Renaming DATE to time would cause a clash here. What would work is for instance time_of_observation or observation_time. What do you think?

[yatharthranjan]

We use TIME in our queries and works with jdbc-connector. But yes observation_time works too

[pvannierop]

I renamed DATE to OBSERVATION_TIME and END_DATE to OBSERVATION_TIME_END.

[yatharthranjan]

Maybe add a Query ID to the insert so we can identify them

[pvannierop]

Do you mean an autoincremented id field for each inserted row?

[pvannierop]

I have implemented a auto-incrementing 'id' field in data-dashboard-backend. The config in this PR has been updated accordingly.

[yatharthranjan]

Oh sorry for the confusion i just meant the QUERY_ID -- https://docs.ksqldb.io/en/latest/developer-guide/ksqldb-reference/insert-into/#query_id (it is just a name for the INSERT query)

For the database tables, i think the primary keys you had previously were good. This is not that important in case of questionnaires, but when you have passive data such as fitbit there is usually high chance of having duplicate data, so this would reduce that in the database. Also would help with querying if using primary keys (as will be indexed by default).

[pvannierop]

I added it like so:

INSERT INTO observations
WITH (QUERY_ID='questionnaire_app_event_observations')
SELECT
    ...

and

INSERT INTO observations
WITH (QUERY_ID='questionnaire_response_observations')
SELECT 
   ...

[yatharthranjan]

I think this needs to be enabled to use the sql files?

[pvannierop]

Ah well spotted. Thanks for that. For, 40-realtime-analysis.yaml this setting was set to false. In order to reuse the ksql-server, I needed to add this field to base.yaml and add a comment like so:

ksql_server:
  _install: false
  _chart_version: 0.3.1
  _extra_timeout: 0
  # -- Uncomment when using real-time analysis
  # ksql:
  #   headless: false
  # --

It is ugly, I know...

[pvannierop]

We agreed to keep the comment-based method.

[keyvaann]

I think the section about TLS can be removed since it's not used in the actual chart
https://github.com/RADAR-base/RADAR-Kubernetes/actions/runs/9515630882/job/26230258308?pr=266#step:11:38

[pvannierop]

I think this is out of scope for this PR. I created an issue out of it.

[keyvaann]

LGTM

[keyvaann]

Force pushing makes it difficult to review changes since the last time I have reviewed the code since all of the commits are pushed again.