Test suite should periodically check that its launched processes are alive when waiting for ports & forward stdout/stderr

[p-mongo]

I tried running the test suite following instructions in the readme:

python setup.py test

This is the output I got:

No handlers could be found for logger "mongo_orchestration.process"
test_mongodb_auth_uri (test_sharded_clusters.ShardSSLTestCase) ... DEBUG:mongo_orchestration.servers:Server.__init__(mongos, {}, {}, None, , )
DEBUG:mongo_orchestration.servers:Creating log file for mongos: /tmp/mongo-9AN_fr/mongos.log
DEBUG:mongo_orchestration.servers:('mongos', '--version')
DEBUG:mongo_orchestration.servers:Server.__init__(mongod, {'configsvr': True, 'replSet': 'c54236fc-a7bd-4e82-86f3-2597b142d1d4'}, {'sslMode': 'requireSSL', 'sslClusterFile': '/home/w/apps/exp/mongo-orchestration/tests/lib/cluster_cert.pem', 'sslCAFile': '/home/w/apps/exp/mongo-orchestration/tests/lib/ca.pem', 'sslPEMKeyFile': '/home/w/apps/exp/mongo-orchestration/tests/lib/server.pem', 'sslAllowInvalidCertificates': True}, None, None, None)
DEBUG:mongo_orchestration.servers:Creating log file for mongod: /tmp/mongo-Z_40P4/mongod.log
DEBUG:mongo_orchestration.servers:('mongod', '--version')
DEBUG:mongo_orchestration.process:mprocess(name='mongod', config_path='/tmp/mongo-tCLOJL', port=1027, timeout=300)
DEBUG:mongo_orchestration.process:execute process: mongod --config /tmp/mongo-tCLOJL --port 1027
DEBUG:mongo_orchestration.process:wait for 1027

The test run was sitting here for a long time, I aborted to see what was happening.

It seems that the test process is waiting for something to appear on port 1027 with a timeout of 300 seconds (5 minutes), but the mongod that was launched died instantly thusly:

speed% mongod --config /tmp/mongo-UAMeTU --port 1027 
2019-06-10T20:33:18.358-0400 F CONTROL  [main] Failed global initialization: BadValue: nojournal is not allowed when configsvr is specified

There are two issues here:

1. The error report was printed to standard output (for some reason, not standard error). It appears that m-o does not capture standard output hence the error is not surfaced to the user at all.

2. M-o is making the user wait 5 minutes when it could have checked the status of the launched process, discovered that the process died, and reported the error much sooner.

[p-mongo]

Github suggested #5 as a related issue. If m-o intentionally redirects stdout & stderr to /dev/null, it is hiding error information from the user and thus makes errors difficult to troubleshoot.

[p-mongo]

I created https://jira.mongodb.org/browse/SERVER-41627 for the server sending the error message to stdout.

[ShaneHarvey]

I agree that we should fix these two issues but I'm not going to get to it anytime soon.

FYI we don't run the test suite in CI anywhere it has been defunct for years. If you have time to work on revitalizing the tests (or just fixing the issue you found) I would be happy to review the changes.

[ShaneHarvey]

Closing this ticket as it's a duplicate of #209