MongoCollection#countDocuments Performance in Driver 4.x [DATAMONGO-2669]

[spring-projects-issues]

Richard Kwasnicki opened DATAMONGO-2669 and commented

When using SimpleMongoRepository.count(), MongoDB is now performing a collscan instead of using estimated count by metadata. I had a short discussion with a MongoDB developer at https://developer.mongodb.com/community/forums/t/upgrading-java-mongodriver-3-x-to-4-x-leads-to-lock-with-aggregate-countstrategy/12754

He advised to never use a non-empty query on exact count and better go with MongoCollection#estimatedDocumentCount instead of  MongoCollection#countDocuments. 

For us it took minutes to perform a count query on a quite large collection on startup

---

Affects: 3.1.1 (2020.0.1)

[choda]

A similar issue occurs with MongoTemplate::count(Query query, ...). Under the hood, it uses an aggregation similar to this:

db.items.aggregate([
{ $match: {<query>} },
{ $group: { _id: 1, n: { $sum: 1 } } }
])

And when the query is empty, ie when counting all tasks, a full collection scan is performed, which for large collections is painfully slow... and can really impair pagination in a Spring Data Rest project.

[christophstrobl]

MongoDB db.collection.count() operates upon collection statistics that may be inaccurate.
Currently there is no support for sessions nor transactions which means that changes like inserts & removals during an active session/transaction are visible outside the session and counted.
Therefore, we'll not change the current behavior right now, but update the documentation (#3541).

Nevertheless, we're aware of the impact this has and already reached to our colleagues at MongoDB to find a solution (maybe based on $collStats: { count: { } }) for this in the future.

One workaround could be a custom MongoTemplate implementation overriding the doCount(String, Document, CountOptions) method.

@Override
protected long doCount(String collectionName, Document filter, CountOptions options) {
	
	if(!countCanBeEstimated(filter, options)) {
		return super.doCount(collectionName, filter, options);
	}

	return execute(collectionName, collection -> collection.estimatedDocumentCount());
}

private boolean countCanBeEstimated(Document filter, CountOptions options) {

	return 
		// only empty filter for estimatedCount
		filter.isEmpty() && 
		// no skip, no limit,... 
		isEmptyOptions(options) && 
		// transaction active?
		!MongoDatabaseUtils.isTransactionActive(getMongoDatabaseFactory()); 
}

The reactive variant looks a bit different due to the deferred transaction evaluation.

@Override
protected Mono<Long> doCount(String collectionName, Document filter, CountOptions options) {

	if(!countCanBeEstimated(filter, options)) {
		return super.doCount(collectionName, filter, options);
	}
	
	return ReactiveMongoDatabaseUtils.isTransactionActive(getMongoDatabaseFactory())
		.flatMap(txActive -> {

			if(txActive) {
				return <subclass-name>.super.doCount(collectionName, filter, options);
			}

			return createMono(collectionName,
				collection -> collection.estimatedDocumentCount());
	});
}

[JoergHeinicke5005]

We hit this issue before with an explicit call to count(..) with an empty query on a large collection - and could therefore quite quickly figure out the more subtle or slightly obfuscated issue when doing the following relying on Spring Data's query method derivation:

public interface EntityRepository extends MongoRepository<Entity, String> {
    Page<Entity> findBy(Pageable pageable);
}

Here the count and thereby the collection scan is triggered in PagedExecution (nested class of MongoQueryExecution), line 140:

long count = operation.matching(Query.of(query).skip(-1).limit(-1)).count();

I don't think what we are doing is so unusual but I couldn't find anybody discussing this aspect.
I would be interested if there is a similar easy workaround like above even though this is much deeper inside Spring Data MongoDB's internals.