monitoring: cadvisor panels don't work at all in docker-compose/pure-docker

[bobheadxi]

in docker-compose, container names are CONTAINER, but the regex matches for _CONTAINER_ because of Kubernetes pods having loads of containers that would report different names (back when it worked anyway). So that means these dashboards don't work at all in all docker-compose/pure-docker environments! oh no. I think this might necessitate a broader rethink of how to tackle naming in cadvisor.

confirmed in managed instances:

related to https://github.com/sourcegraph/sourcegraph/issues/17069

[bobheadxi]

This should be closed by https://github.com/sourcegraph/sourcegraph/pull/17096. Will test soon

[bobheadxi]

I am was unable to get name to show up in exported cadvisor metrics in both 3.23.0 and the latest insiders (ie after patches in https://github.com/sourcegraph/sourcegraph/issues/17069#issuecomment-755950540). I fenangled with this for a good chunk of time (was worried this might be something wrong with our docker-compose cAdvisor config), which was odd since, since in e.g. devmanaged.sourcegraph.com (a 3.23.0 managed docker-compose instance), the new label matching for Grafana seems to work:

query for devmanaged.sourcegraph.com

while locally the same version does not yield any name labels exported by cAdvisor:

mid-writeup edit I finally ran across google/cadvisor#1565 and after adding TODO I got everything working (whew):

seems good to go. I'm not sure how docker-compose will play into the seeming lack of containerd compatibility noted in #17069 but FWIW I got this working on what seems to be containerd (though I'm not sure - @daxmc99 are you familiar with this?):

❯ docker info | grep runtime
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc