Monitoring with Prometheus (and metrics-server)

[bradenmacdonald]

Use helm subcharts to deploy Prometheus and Grafana onto the chart.

TBD: Can we have them auto-detect and start monitoring Open edX instances on the cluster as they get deployed using Tutor, like how the ingress controller works?

[antoviaque]

@bradenmacdonald Thanks for creating this task!

@lpm0073 was it the one you are taking on?

[lpm0073]

yes, this is me

[antoviaque]

@lpm0073 Alright, I'm assigning the issue to you then, if that works :)

[antoviaque]

@lpm0073 Could you post a status update on this task here? This way we could follow & discuss here async, ahead of the next meeting.

[antoviaque]

@lpm0073 Are you still interested in this task?

[antoviaque]

Recap from the meeting: now @lpm0073 is unblocked to work on this ticket, based if I understood correctly on the autoscaling work from @jfavellar90 in #2

[felipemontoya]

@lpm0073 could you give us an update on this? are you interested in pursuing this still?

[lpm0073]

i'm beginning today. i'll start here, focusing on the Karpenter dependencies, which include:

• metrics-server
• prometheus
• Vertical Pod Autoscaler

Kubecost effectively shares the same set of dependencies, so i'll use this as a guide for scaffolding purposes. Separately, when running on AWS EKS, these supporting systems will benefit from kube-proxy and coredns, so i'll look into whether we can detect if these are running, and if not then try to at least echo something to the console.

[bradenmacdonald]

@lpm0073 Thanks! Note that metrics-server and VPA are already being worked on in #17

[lpm0073]

Question to the group: aside from the helm charts, there are a few AWS resources that need to exist, and need to be provided to the helm chart:

• IAM role for service accounts
• EC2 instance profile
• EC2 tagging role and policy attachment

I have these Terraform scripts. how should i incorporate these into this repo?

[bradenmacdonald]

@lpm0073 I put some DigitalOcean example Terraform Scripts in https://github.com/openedx/openedx-k8s-harmony/tree/main/infra-example ; you could rename that folder to infra-example-do and create a new infra-example-aws folder with the AWS terraform.

[lpm0073]

confirming that pr #17 takes care of metrics-server and vpa dependencies for this issue.

[felipemontoya]

After PR#17 is merged we will need to create a new PR with the helm charts for grafana and prometheus

[adzuci]

Hi there! Though I don't have a lot of context currently, I wanted to mention that 2U has looked into the way FairwindsOps enables chart users to opt to toggle on and off the installation of Prometheus via the prometheus-metrics.installPrometheusServer flag in https://github.com/FairwindsOps/charts/tree/master/stable/insights-agent

I would be interested in talking more about how the over Slack or a call if others would.

[antoviaque]

@adzuci Did you get the information and discussions you wanted during the last meeting about this?

Note that @bradenmacdonald has also created a dedicated task to follow-up on topics of interest for you and 2U, at #28 - comments there are welcomed! Or during the meeting later today.

[felipemontoya]

@lpm0073 we are getting close to finisht the autoscaling part of the charts which were closely related to this.

Are you interested in pursuing this further?

[felipemontoya]

@lpm0073 following the meeting we will split this ticket into two less ambiguous issues. Please comment if you have a different opinion

[felipemontoya]

Now that the issues for grafana and prometheus have been created we can go ahead and close this.