Core services healthchecks

This page lists the checks that should be run to validate the deployment of the core services. curl checks should be run from a container on the overlay network shared by the tested service. logs checks can be run from a manager node. The command should be run after a stabilization delay.

This is implemented in the *.test.yml of the cluster/agent/stacks stack files (part of the agent image).

order	service	depends on	context	test	command	where (node label)	delay (sec)	timeout (sec)
1	elasticsearch		single mode	cluster health should be yellow	curl -sf elasticsearch:9200/_cluster/health?wait_for_status=yellow&timeout=15s	core	10	30
1			cluster mode	cluster health should be green	curl -sf elasticsearch:9200/_cluster/health?wait_for_status=green&timeout=30s	core	15	45
2	etcd		*	endpoint health	etcdctl --endpoints "http://etcd:2379" endpoint health | grep -qw healthy	core	10	30
3	nats		*	api availability	curl -sf "nats:8222/subsz"	core	5	20
4	ampbeat	nats, elasticsearch		docker service logs amp_ampbeat 2>&1	grep -q "INFO ampbeat is running"	manager	5	20
5	kibana	elastisearch, ampbeat		UI availability	curl -sf "kibana:5601/app/kibana#/discover"	core	10	30
6	node_exporter			metrics URL	curl -sf "node-exporter:9100/metrics"	metrics	5	20
7	nats_exporter	nats		metrics URL	curl -sf "nats-exporter:7777/metrics"	metrics	5	20
8	haproxy_exporter	haproxy		metrics URL	curl -sf "haproxy-exporter:9101/metrics"	metrics	5	20
9	prometheus	node_exporter, nats_exporter, haproxy_exporter		status URL	curl -sf "prometheus:9090/status"	metrics	8	30
10	alertmanager	prometheus		metrics URL	curl -sf "alertmanager:9093/metrics"	metrics	5	20
11	grafana			org info	curl -sf "grafana:3000/api/org"	metrics	10	30
12	proxy			stats	curl -sf "http://stats:stats@proxy:1936/haproxy\?stats\;csv"	route	5	20
13	amplifier	nats, elasticsearch, etcd		metrics URL	curl -sf "amplifier:5100/metrics"	core	6	30
14	gateway	amplifier		docker service logs amp_gateway 2>&1	grep -q "gateway successfully initialized"	manager	5	30
15	agent	nats		docker service logs amp_agent 2>&1	grep -q "Connected to nats streaming successfully"	manager	5	30
16	portal	gateway		home page	curl -sf "http://portal/"	core	5	30