mbon-data uploader container dies sometimes

[dotis]

Not getting the bad gateway issue, but no updates have happened in awhile.

[7yl4r]

fknms

the data uploader died:

tylarmurray@fknms-dashboard-04:~$ docker container ls -a
CONTAINER ID   IMAGE                                      COMMAND                  CREATED         STATUS                       PORTS                                                                                  NAMES
b67bc3657ce8   mbon-dashboard-server-airflow-worker       "/usr/bin/dumb-init …"   2 months ago    Up 2 months                  8080/tcp                                                                               mbon-dashboard-server-airflow-worker-1
b70129084fae   grafana/grafana:6.7.3                      "/run.sh"                2 months ago    Up 2 months                  0.0.0.0:3000->3000/tcp, :::3000->3000/tcp                                              grafana
26d0f184d0b6   mbon-dashboard-server-nginx                "/docker-entrypoint.…"   2 months ago    Up 2 months                  0.0.0.0:80->80/tcp, :::80->80/tcp                                                      nginx
2509b6332728   mbon-dashboard-server-airflow-webserver    "/usr/bin/dumb-init …"   2 months ago    Up 2 months (healthy)        0.0.0.0:8888->8080/tcp, :::8888->8080/tcp                                              mbon-dashboard-server-airflow-webserver-1
f43587c65c77   mbon-dashboard-server-airflow-scheduler    "/usr/bin/dumb-init …"   2 months ago    Up 2 months                  8080/tcp                                                                               mbon-dashboard-server-airflow-scheduler-1
1353c06aedc5   influxdb:1.8                               "/entrypoint.sh infl…"   2 months ago    Up 4 weeks                   0.0.0.0:2003->2003/tcp, :::2003->2003/tcp, 0.0.0.0:8086->8086/tcp, :::8086->8086/tcp   influxdb
cc1654be32aa   mbon-dashboard-server-flower               "/usr/bin/dumb-init …"   2 months ago    Up 2 months (healthy)        0.0.0.0:5555->5555/tcp, :::5555->5555/tcp, 8080/tcp                                    mbon-dashboard-server-flower-1
840ec93b1ff8   mbon-dashboard-server-mbon_data_uploader   "waitress-serve --po…"   2 months ago    Exited (137) 2 weeks ago                                                                                            mbon_data_uploader
3d6bd797d4f8   mbon-dashboard-server-airflow-init         "/usr/bin/dumb-init …"   2 months ago    Exited (0) 4 weeks ago                                                                                              mbon-dashboard-server-airflow-init-1
1c51732bf7ff   postgres:13                                "docker-entrypoint.s…"   2 months ago    Up 2 months (healthy)        5432/tcp                                                                               mbon-dashboard-server-postgres-1
9f69b9b7b879   redis:latest                               "docker-entrypoint.s…"   2 months ago    Up 2 months (healthy)        0/tcp, 6379/tcp                                                                        mbon-dashboard-server-redis-1
8c1dd0547b1c   mbon-dashboard-server-erddap               "/entrypoint.sh cata…"   10 months ago   Exited (143) 10 months ago                                                                                          erddap

Things are fixed after restarting it:

tylarmurray@fknms-dashboard-04:~/mbon-dashboard-server$ docker compose up -d
[+] Building 0.0s (0/0)                                                                                                                            
WARN[0000] Found orphan containers ([erddap]) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up. 
[+] Running 11/11
 ✔ Container grafana                                    Running                                                                               0.0s 
 ✔ Container mbon_data_uploader                         Started                                                                               1.0s 
 ✔ Container influxdb                                   Running                                                                               0.0s 
 ✔ Container mbon-dashboard-server-redis-1              Healthy                                                                               0.5s 
 ✔ Container mbon-dashboard-server-postgres-1           Healthy                                                                               0.5s 
 ✔ Container nginx                                      Running                                                                               0.0s 
 ✔ Container mbon-dashboard-server-airflow-worker-1     Running                                                                               0.0s 
 ✔ Container mbon-dashboard-server-airflow-scheduler-1  Running                                                                               0.0s 
 ✔ Container mbon-dashboard-server-airflow-init-1       Started                                                                               1.0s 
 ✔ Container mbon-dashboard-server-flower-1             Running                                                                               0.0s 
 ✔ Container mbon-dashboard-server-airflow-webserver-1  Running                                                                               0.0s 

There was nothing in the logs. I am disturbed that these various docker containers seem to go offline at random. If I can't get logs then my next best idea is to add a daily cronjob that restarts them if they are dead.

---

fgbnms

tylarmurray@fgbnms-dashboard-02:~$ docker container ls -a
CONTAINER ID   IMAGE                                      COMMAND                  CREATED        STATUS                     PORTS                                                                                  NAMES
0af2dc04680d   mbon-dashboard-server-mbon_data_uploader   "waitress-serve --po…"   2 months ago   Exited (137) 2 weeks ago                                                                                          mbon_data_uploader
b79caf1ee1de   mbon-dashboard-server-flower               "/usr/bin/dumb-init …"   2 months ago   Up 2 months (healthy)      0.0.0.0:5555->5555/tcp, :::5555->5555/tcp, 8080/tcp                                    mbon-dashboard-server-flower-1
494a7caef651   influxdb:1.8                               "/entrypoint.sh infl…"   2 months ago   Up 2 months                0.0.0.0:2003->2003/tcp, :::2003->2003/tcp, 0.0.0.0:8086->8086/tcp, :::8086->8086/tcp   influxdb
299009a9e01f   grafana/grafana:6.7.3                      "/run.sh"                2 months ago   Up 2 months                0.0.0.0:3000->3000/tcp, :::3000->3000/tcp                                              grafana
2081b1933f24   mbon-dashboard-server-airflow-webserver    "/usr/bin/dumb-init …"   2 months ago   Up 2 weeks (unhealthy)     0.0.0.0:8888->8080/tcp, :::8888->8080/tcp                                              mbon-dashboard-server-airflow-webserver-1
3684df4b86b6   mbon-dashboard-server-airflow-worker       "/usr/bin/dumb-init …"   2 months ago   Up 2 months                8080/tcp                                                                               mbon-dashboard-server-airflow-worker-1
18066a67e8d9   mbon-dashboard-server-airflow-scheduler    "/usr/bin/dumb-init …"   2 months ago   Up 2 weeks                 8080/tcp                                                                               mbon-dashboard-server-airflow-scheduler-1
76e9036e4587   mbon-dashboard-server-nginx                "/docker-entrypoint.…"   2 months ago   Up 2 months                0.0.0.0:80->80/tcp, :::80->80/tcp                                                      nginx
8fb804c65c4e   mbon-dashboard-server-airflow-init         "/usr/bin/dumb-init …"   2 months ago   Exited (0) 6 weeks ago                                                                                            mbon-dashboard-server-airflow-init-1
afe81e3f9d52   postgres:13                                "docker-entrypoint.s…"   2 months ago   Up 2 months (healthy)      5432/tcp                                                                               mbon-dashboard-server-postgres-1
8e77da4e28c0   redis:latest                               "docker-entrypoint.s…"   2 months ago   Up 2 months (healthy)      0/tcp, 6379/tcp                                                                        mbon-dashboard-server-redis-1

tylarmurray@fgbnms-dashboard-02:~/mbon-dashboard-server$ docker compose up -d
[+] Running 11/11
 ⠿ Container influxdb                                   Running                                                                                   0.0s
 ⠿ Container mbon-dashboard-server-postgres-1           Healthy                                                                                   0.5s
 ⠿ Container nginx                                      Running                                                                                   0.0s
 ⠿ Container mbon_data_uploader                         Started                                                                                   1.1s
 ⠿ Container mbon-dashboard-server-redis-1              Healthy                                                                                   0.5s
 ⠿ Container mbon-dashboard-server-airflow-webserver-1  Running                                                                                   0.0s
 ⠿ Container grafana                                    Running                                                                                   0.0s
 ⠿ Container mbon-dashboard-server-flower-1             Running                                                                                   0.0s
 ⠿ Container mbon-dashboard-server-airflow-scheduler-1  Running                                                                                   0.0s
 ⠿ Container mbon-dashboard-server-airflow-init-1       Started                                                                                   1.0s
 ⠿ Container mbon-dashboard-server-airflow-worker-1     Running                                                                                   0.0s

The fact these two containers on separate VMs went offline around the same time only adds to the mystery.

[7yl4r]

fknms container went down

tylarmurray@fknms-dashboard-04:~$ docker container ls -a
CONTAINER ID   IMAGE                                      COMMAND                  CREATED         STATUS                       PORTS                                                                                  NAMES
[...]
840ec93b1ff8   mbon-dashboard-server-mbon_data_uploader   "waitress-serve --po…"   4 months ago    Exited (137) 11 days ago                                                                                            mbon_data_uploader
[...]

nothing in docker logs mbon_data_uploader.

"fixed" with docker compose up -d.

Reopening this as a known issue.
Also happened before (#51) and may be related to #52.

I don't know why these services go down randomly but I added a line to the crontab so they will restart nightly if they have crashed:

0 0 * * * cd /home/tylarmurray/mbon-dashboard-server && /bin/docker compose up -d