Harbor should expose metrics.

[reasonerjt]

This helpful for user who host Harbor as a service.

Harbor does not expose any metrics. As the registry will be a crucial piece of our pipeline (and offering) it will be subjected to an SLA and we'll need some internal metrics published over Prometheus (our monitoring solution). We must be able to quickly diagnose problems and eventually page SRE engineers.

[jnovack]

I'm pretty sure Prometheus-compatible metrics exposure is almost a requirement in 2018 for project adoption. 🤔

[jcmartins]

+1

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[jakirpatel]

+1
This will be really helpful if harbor expose the metrics which will give insights of the harbor. @reasonerjt

[jakirpatel]

any update for this in roadmap activities..?

[ghost]

Definitely on our roadmap and we hope to get to it as soon as possible. If anyone is interested in contributing please let me know. 😄

[aylei]

@clouderati @reasonerjt I would love to help with this issue.😁

[aylei]

Hi, is there any update on this issue? I think we can introduce the prometheus format metrics to harbor as the first step (we will get the golang runtime metrics and process metrics by default), and add necessary metrics later on.
Also, I want to clarify what additional metrics we need for each harbor component, any ideas?

[salkin]

A good start would be amount of pulls / push with error codes to get more insight about unauthorized pulls.

[darkl0rd]

You can effectively get this for "free" these days, as Docker Distribution provides Prometheus metrics since v2.7.1 (See distribution/distribution@e3c37a4 and for docs: https://github.com/docker/distribution/blob/master/docs/configuration.md)

To adopt, this would imply:

a) updating the version of docker distribution (currently 2.6.4) to 2.7.1 or later.
b) an adjustment to the registry/config.yml (see linked docs)
c) updating nginx.conf (to expose the endpoint).

[YakirShriker]

A good start would be amount of pulls / push with error codes to get more insight about unauthorized pulls.

does anyone know how can I query harbor(api) for pull and push operations?

[armandorvila]

+1

[secret104278]

+1

[HaveFun83]

+1

[KarstenSiemer]

+1

[hairyhenderson]

I'm pretty sure the +1 messages are unhelpful and only serve to frustrate maintainers - use reactions (like 👍) instead to indicate that you're in support of this 😉

[hairyhenderson]

You can effectively get this for "free" these days, as Docker Distribution provides Prometheus metrics since v2.7.1

Seems that since #8003 the go.mod pulls in docker/distribution 2.7.1+, so at least part of the work is done - is anyone working on the remaining pieces?

[xaleeks]

If anyone wants to work on this issue please reach out.

[MartinForReal]

I would like to work on this issue. Thanks!

[mawl]

+1

[hrak]

I subscribed to this issue to stay informed on the developments of this feature, not to be emailed every day with another useless "+1" response kthx.

[holooooo]

I subscribed to this issue to stay informed on the developments of this feature, not to be emailed every day with another useless "+1" response kthx.

+1

[c4po]

I'm building a harbor-exporter https://github.com/c4po/harbor_exporter to dump the data from harbor API to prometheus format. All suggestions are welcome.

[pruthvirajkanmantha]

I faced the below mentioned issue while using https://github.com/c4po/harbor_exporter.

Issue:
level=error ts=2020-03-17T13:58:20.891Z caller=harbor_exporter.go:111 msg="error gathering metrics:8 error(s) occurred:\n* collected metric "harbor_quotas_count_total" { label:<name:"repo_id" value:"0" > label:<name:"repo_name" value:"" > label:<name:"type" value:"hard" > gauge:<value:-1 > } was collected before with the same name and label values\n* collected metric "harbor_quotas_count_total" { label:<name:"repo_id" value:"0" > label:<name:"repo_name" value:"" > label:<name:"type" value:"used" > gauge:<value:0 > } was collected before with the same name and label values\n* collected metric

[hrak]

@pruthvi08 then open an issue there 🤷🏼‍♂️

[Hokwang]

@xaleeks I am concerned that this issue is still in the backlog. Does that mean that this issue cannot be resolved until version 2.2?

[xaleeks]

@Hokwang We are still in planning stage and it didn't make it into the 2.1 release hopper unfortunately. To have a better chance of delivering this in 2.2, we need be finalizing some high level design pretty soon. Do you have some bandwidth to contribute to this as well?

[standup-jb]

+1
Hope provide the promethes exporter metrics

[unixzen]

+1

[bitsf]

I just saw a related article.
https://sysdig.com/blog/monitor-harbor-registry/
https://github.com/c4po/harbor_exporter

[addozhang]

+1

[DesistDaydream]

I prefer the official to provide metrics directly, rather than through exporter

[xaleeks]

Everyone please feel free to chime in with your requirements, we're working on a functional spec right now. Will share more on the next community meeting, or perhaps the one after that

[fredgate]

@xaleeks Some great metrics would be the results of the scans. So we could have alerts when vulnerabilties are found.

[xaleeks]

updated proposal is here
goharbor/community#147

[xaleeks]

metrics that will be captured in next v2.2 release are here

https://docs.google.com/document/d/1J_jiJVdV4AOB9lf9PtN5T1LoqHlHF8SjLjI2KL17b0o/edit

[kong62]

+1

[dongjiang1989]

+1

[amaters-easy]

metrics that will be captured in next v2.2 release are here

https://docs.google.com/document/d/1J_jiJVdV4AOB9lf9PtN5T1LoqHlHF8SjLjI2KL17b0o/edit

this document shows that the following metrics will be implemented:
Scan status labels: scan job id
Total vulnerabilities(p2) labels: project id

Maybe I can't find them or they are still in development?

[ninjadq]

@amaters-easy The vulnerabilities are based on artifacts. So it can become a huge number. So it isn't a good practice to expose them. It will be added in a future release if we find a proper way.

[cparadal]

@ninjadq what about replication policy status metrics? They're also in the document but i can't see them in Prometheus. Is that something that's planned to be relased in further releases?
I'm especially interestered in knowing when scheduled replication rules stop triggering (something that happens a lot after Harbor version upgrades).

Thanks!

[ninjadq]

@cparadal replication jobs are run by Job service also. you can find replication related job status with job services' metrics

[cparadal]

@cparadal replication jobs are run by Job service also. you can find replication related job status with job services' metrics

Thanks for your reply!

These are the jobservice related metrics I can see on Prometheus:

I guess I could alert if the below count doesn't increase:

harbor_jobservice_task_process_time_seconds_count{status="success",type="SCHEDULER"} 157

... even though it applies to all scheduled replications, can't see a way to get metrics for specific ones.

[reasonerjt]

@xaleeks Shall we close this one and open new issues, I believe adding/adjusting the metrics for Harbor will be a continuous effort.

[withlin]

any update?

[reasonerjt]

As per #4557 (comment) I'm closing this one.

Please open new issues when you want new metrics to be collected or make adjustment to the existing ones.