shouldNotLoseDataWhenScalingDown ocasionally fails

[philwinder]

A couple of times I have seen this fail. Why?

Re run many times to prove stability.

[philwinder]

The issue is due to inconsistent results in the /_count aggregation across shards whilst indexing data:

[INFO] 2016-01-11 08:17:56,560 org.apache.mesos.elasticsearch.systemtest.ScalingSystemTest lambda$shouldNotLoseDataWhenScalingDown$3 - [{"_shards":{"total":5,"failed":0,"successful":5},"count":1}]
[INFO] 2016-01-11 08:17:56,667 org.apache.mesos.elasticsearch.systemtest.ScalingSystemTest lambda$shouldNotLoseDataWhenScalingDown$3 - [{"_shards":{"total":5,"failed":0,"successful":5},"count":2}]
[INFO] 2016-01-11 08:17:56,789 org.apache.mesos.elasticsearch.systemtest.ScalingSystemTest lambda$shouldNotLoseDataWhenScalingDown$3 - [{"_shards":{"total":5,"failed":0,"successful":5},"count":9}]
[INFO] 2016-01-11 08:17:56,850 org.apache.mesos.elasticsearch.systemtest.ScalingSystemTest lambda$shouldNotLoseDataWhenScalingDown$3 - [{"_shards":{"total":5,"failed":0,"successful":5},"count":2}]
[INFO] 2016-01-11 08:17:56,977 org.apache.mesos.elasticsearch.systemtest.ScalingSystemTest lambda$shouldNotLoseDataWhenScalingDown$3 - [{"_shards":{"total":5,"failed":0,"successful":5},"count":6}]
[INFO] 2016-01-11 08:17:57,096 org.apache.mesos.elasticsearch.systemtest.ScalingSystemTest lambda$shouldNotLoseDataWhenScalingDown$3 - [{"_shards":{"total":5,"failed":0,"successful":5},"count":9}]
[INFO] 2016-01-11 08:17:57,108 org.apache.mesos.elasticsearch.systemtest.ScalingSystemTest lambda$shouldNotLoseDataWhenScalingDown$3 - [{"_shards":{"total":5,"failed":0,"successful":5},"count":9}]
[INFO] 2016-01-11 08:17:57,134 org.apache.mesos.elasticsearch.systemtest.ScalingSystemTest lambda$shouldNotLoseDataWhenScalingDown$3 - [{"_shards":{"total":5,"failed":0,"successful":5},"count":10}]
[DEBUG] 2016-01-11 08:17:57,144 org.apache.mesos.elasticsearch.systemtest.ScalingSystemTest shouldNotLoseDataWhenScalingDown - Correct number of documents: 9
[DEBUG] 2016-01-11 08:17:57,181 org.apache.mesos.elasticsearch.systemtest.ScalingSystemTest shouldNotLoseDataWhenScalingDown - Scaling down to 1
[DEBUG] 2016-01-11 08:17:57,193 org.apache.mesos.elasticsearch.systemtest.ScalingSystemTest shouldNotLoseDataWhenScalingDown - Response 200
[DEBUG] 2016-01-11 08:17:58,228 org.apache.mesos.elasticsearch.systemtest.ScalingSystemTest lambda$shouldNotLoseDataWhenScalingDown$4 - Number of nodes: 1
[DEBUG] 2016-01-11 08:17:58,229 org.apache.mesos.elasticsearch.systemtest.ESTasks getTasks - Fetching tasks on http://172.17.0.7:31100/v1/tasks
[INFO] 2016-01-11 08:17:58,262 org.apache.mesos.elasticsearch.systemtest.ScalingSystemTest shouldNotLoseDataWhenScalingDown - New address: [172.17.0.1:9200]
[DEBUG] 2016-01-11 08:17:58,262 org.apache.mesos.elasticsearch.systemtest.ScalingSystemTest waitForGreen - Wating for green.
[DEBUG] 2016-01-11 08:18:49,952 org.apache.mesos.elasticsearch.systemtest.ESTasks getTasks - Fetching tasks on http://172.17.0.7:31100/v1/tasks
[DEBUG] 2016-01-11 08:18:49,962 org.apache.mesos.elasticsearch.systemtest.ScalingSystemTest lambda$waitForGreen$8 - {"cluster_name":"mesos-ha","status":"green","timed_out":false,"number_of_nodes":1,"number_of_data_nodes":1,"active_primary_shards":5,"active_shards":5,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"delayed_unassigned_shards":0,"number_of_pending_tasks":0,"number_of_in_flight_fetch":0,"task_max_waiting_in_queue_millis":0,"active_shards_percent_as_number":100.0}
[DEBUG] 2016-01-11 08:18:50,973 org.apache.mesos.elasticsearch.systemtest.ScalingSystemTest shouldNotLoseDataWhenScalingDown - New number of documents: 10

java.lang.AssertionError: 
Expected :9
Actual   :10

[philwinder]

And the test was waiting for > 9. When the number of documents == 10, it would advance on number of docs == 9.

[philwinder]

Changed again to look for exactly 10 documents. Was still unstable. bce3ed4