speed up CI by running each test app separately

[timgraham]

Since django-mongodb does not support transactions, the database must be flushed between tests which means a collection.delete_many({}) for every collection that exists. By running each test app separately, it avoids the need to flush the collections in other apps.

Current test suite run time decreased from ~100 minutes to ~8 minutes.

A downside to this approach is that test failures won't be summarized at the bottom of the logs. You'll have to search for "FAIL:", "ERROR:", or "unexpected success" to find what went wrong.

[aclark4life]

LGTM, thanks!

[Jibola]

Since django-mongodb does not support transactions, the database must be flushed between tests which means a collection.delete_many({}) for every collection that exists. By running each test app separately, it avoids the need to flush the collections in other apps.

Current test suite run time decreased from ~100 minutes to ~8 minutes.

A downside to this approach is that test failures won't be summarized at the bottom of the logs. You'll have to search for "FAIL:", "ERROR:", or "unexpected success" to find what went wrong.

Is it at all possible for us to just drop the collection? This usually is much faster than processing individual deletes.

[timgraham]

That was the approach before 8b727e9, but some tests expect to be able to introspect the presence of a collection. Even if Django provided a hook to re-create collections (along with all indexes, etc.) before each test, I don't know if that approach would perform much better. (If we take that approach without this change, it amounts to essentially dropping the database and re-creating it between tests).

[Jibola]

That was the approach before 8b727e9, but some tests expect to be able to introspect the presence of a collection. Even if Django provided a hook to re-create collections (along with all indexes, etc.) before each test, I don't know if that approach would perform much better. (If we take that approach without this change, it amounts to essentially dropping the database and re-creating it between tests).

It should be pretty fast. I imagine these tests aren't populating the local instance with much data, and getting the index warm, as well is pretty quick.

If there's not an easily leveraged setup hook already. I'm fine with this as a stop-gap solution. We'd just need a JIRA or Issue ticket raised to keep track of coming back to this. Delete many calls can definitely become cumbersome.