[CI] MultiClusterSearchYamlTestSuiteIT: Test that remote indices are subject to shard skipping

[talevy]

console logs:
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob-unix-compatibility/os=oraclelinux/1245/console

reproduce:

gradle :qa:multi-cluster-search:mixedClusterTestRunner -Dtests.seed=6F190415C02A93B3 -Dtests.class=org.elasticsearch.upgrades.MultiClusterSearchYamlTestSuiteIT -Dtests.method="test {yaml=multi_cluster/70_skip_shards/Test that remote indices are subject to shard skipping}" -Dtests.security.manager=true -Dtests.locale=ca-ES -Dtests.timezone=America/Boa_Vista -Dtests.rest.suite=multi_cluster

stacktrace:

1> [2017-07-12T21:36:11,280][INFO ][o.e.u.MultiClusterSearchYamlTestSuiteIT] [test {yaml=multi_cluster/70_skip_shards/Test that remote indices are subject to shard skipping}]: after test
  1> [2017-07-12T21:36:11,283][INFO ][o.e.u.MultiClusterSearchYamlTestSuiteIT] Stash dump on failure [{
  1>   "stash" : {
  1>     "body" : {
  1>       "took" : 39,
  1>       "timed_out" : false,
  1>       "_shards" : {
  1>         "total" : 2,
  1>         "successful" : 2,
  1>         "skipped" : 0,
  1>         "failed" : 0
  1>       },
  1>       "hits" : {
  1>         "total" : 1,
  1>         "max_score" : 1.0,
  1>         "hits" : [
  1>           {
  1>             "_index" : "skip_shards_index",
  1>             "_type" : "test_type",
  1>             "_id" : "BeuWOV0B6LpPvdNJzw-c",
  1>             "_score" : 1.0,
  1>             "_source" : {
  1>               "sort_field" : 0,
  1>               "created_at" : "2017-01-01",
  1>               "f1" : "local_cluster"
  1>             }
  1>           }
  1>         ]
  1>       }
  1>     }
  1>   }
  1> }]
FAILURE 0.21s | MultiClusterSearchYamlTestSuiteIT.test {yaml=multi_cluster/70_skip_shards/Test that remote indices are subject to shard skipping} <<< FAILURES!
   > Throwable #1: java.lang.AssertionError: Failure at [multi_cluster/70_skip_shards:37]: _shards.skipped didn't match expected value:
   >                _shards.skipped: expected [1] but was [0]
   > 	at __randomizedtesting.SeedInfo.seed([6F190415C02A93B3:E74D3BCF6ED6FE4B]:0)
   > 	at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.executeSection(ESClientYamlSuiteTestCase.java:348)
   > 	at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.test(ESClientYamlSuiteTestCase.java:328)
   > 	at java.lang.Thread.run(Thread.java:748)
   > Caused by: java.lang.AssertionError: _shards.skipped didn't match expected value:
   >                _shards.skipped: expected [1] but was [0]
   > 	at org.elasticsearch.test.rest.yaml.section.MatchAssertion.doAssert(MatchAssertion.java:87)
   > 	at org.elasticsearch.test.rest.yaml.section.Assertion.execute(Assertion.java:76)
   > 	at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.executeSection(ESClientYamlSuiteTestCase.java:344)
   > 	... 38 more

[talevy]

another occurrence: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob-unix-compatibility/os=ubuntu/1245/console

[talevy]

@s1monw, maybe you know more about this?