MLOps: experiment tracking and monitoring in production

[vykozlov]

Title

MLOps: experiment tracking and monitoring in production

Description

As the field of machine learning advances, managing and monitoring intelligent models in production, also known as machine learning operations (MLOps), has become essential. Every aspect of the machine learning lifecycle, from workflow orchestration to performance monitoring, presents both challenges and opportunities.
In this unconference we would like to discuss:

• How tracking of experiments may improve the process of organizing and analyzing the results of machine learning experiments as well as team collaboration?
• What frameworks can be useful for experiment tracking (MLflow, Tensorboard, W&B, ..)?
• What are ways of publishing trained models and training datasets? (Data, model repositories?)
• What systems to choose for pipeline automation? (Jenkins, GitHub actions, ..?)
• How to monitor the performance of the machine learning model in production? Particularly the detection of drifts. Drift, which refers to the model’s learned concepts or data patterns that vary over time, can negatively influence the model inference performance, thereby necessitating the retraining

We want to discuss the topic of MLOps in an open format with you, identify most used frameworks, common solutions, and connect with interested people.

Organizational

Host(s)

Borja Esteban Sanchis
Harsh Grover
Valentin Kozlov

Format

Short Introduction talk followed by open discussion, brainstorming, etc.
As Ref.: https://cloud.google.com/architecture/mlops-continuous-delivery-and-automation-pipelines-in-machine-learning

Timeframe

1-2h

Number of participants

3-20

Material

Beamer, flipchart, markers, post-its

[BorjaEst]

[SusanneWenzel]

Dear @vykozlov,

many thanks for your contribution! We hope your session was successful and constructive!

To make this not only a nice experience during the conference but a sustainable format, we would like to release a short result on the Helmholtz AI Website. Please provide us with a little report about the discussion in your session:

• One or two intro sentences about the topic and the respective format of your session
• What was most controversial, or about what did all participants agree on?
• What were the key ideas?
• Do you see any follow up on this discussion?

Depending on the intensity of the session it can be shorter (1/3 page) or longer (one pager).

Please post your report here by end of next week. Ina already confirmed to release that at a HAICON24 subpage at Helmholtz.ai.

Best regards,
@helenehoffmann and Susanne

[SusanneWenzel]

Dear @vykozlov, Its a while ago now. Can you still provide a short report please !? ☝️