/beta not working on arax2.ncats.io

[edeutsch]

It was reported that https://arax2.ncats.io/beta was not functional, in fact it would not even start.

I traced the problem to a log file permission problem:

ls -al /tmp/
-rw-r--r-- 1 rt    rt       4759 Aug 21 22:49 RTX_Complete.elog
-rw-r--r-- 1 rt    rt        484 Aug 13 04:02 RTX_Complete.log
-rw-r--r-- 1 root  root    42591 Aug 21 16:26 RTX_OpenAPI_beta.elog
-rw-r--r-- 1 root  root       51 Aug 21 16:25 RTX_OpenAPI_beta.log
-rw-r--r-- 1 rt    rt      62623 Aug 22 02:09 RTX_OpenAPI_devED.elog
-rw-r--r-- 1 rt    rt         51 Aug 22 01:54 RTX_OpenAPI_devED.log
-rw-r--r-- 1 rt    rt    2557120 Aug 22 02:08 RTX_OpenAPI_devLM.elog
-rw-r--r-- 1 rt    rt         51 Aug 12 15:41 RTX_OpenAPI_devLM.log
-rw-r--r-- 1 rt    rt    2065679 Aug 22 02:08 RTX_OpenAPI_kg2.elog
-rw-r--r-- 1 rt    rt         51 Aug 12 15:39 RTX_OpenAPI_kg2.log
-rw-r--r-- 1 rt    rt    2039013 Aug 22 02:08 RTX_OpenAPI_kg2test.elog
-rw-r--r-- 1 rt    rt         51 Aug 12 15:39 RTX_OpenAPI_kg2test.log
-rw-r--r-- 1 rt    rt    2697040 Aug 22 02:08 RTX_OpenAPI_production.elog
-rw-r--r-- 1 rt    rt         51 Aug 12 15:39 RTX_OpenAPI_production.log
-rw-r--r-- 1 rt    rt    5731147 Aug 22 02:09 RTX_OpenAPI_test.elog
-rw-r--r-- 1 rt    rt       5143 Aug 22 01:59 RTX_OpenAPI_test.log

I think the service launch process first su's to user rt and then tries to launch but exploded when it tried to open the log files because they were owned by root and not world writable.

I deleted these log files and the service has been restored.
It is unclear how this happened. I suspect someone ran the startup script directly as root rather than via the service mechanism:

service RTX_OpenAPI_beta restart

This fix is documented here in case this should happen again.

The service seems operational now:
https://arax2.ncats.io/beta/?r=256946

[saramsey]

Good catch, Eric. Thank you. And also thanks for updating the issue (helpful if this happens again in a future endpoint).

[saramsey]

I've also added a note about this in the ARAX Operations & Deployment Info wiki page.

[kvnthomas98]

Thanks a lot Eric! That was me, my bad.

[edeutsch]

no problem, all is still well today, closing.