[CI] Netty4HeadBodyIsEmptyIT testTemplateExists failing with FIPS enabled

[mark-vieira]

This is failing in our FIPS build, it reproduced locally for me with the FIPS flag set.

Build scan:
https://gradle-enterprise.elastic.co/s/h3dxshexqrzfi/tests/:modules:transport-netty4:javaRestTest/org.elasticsearch.rest.Netty4HeadBodyIsEmptyIT/testTemplateExists

Reproduction line:
./gradlew ':modules:transport-netty4:javaRestTest' --tests "org.elasticsearch.rest.Netty4HeadBodyIsEmptyIT.testTemplateExists" -Dtests.seed=A02E26474C28612C -Dtests.security.manager=true -Dtests.locale=en-GB -Dtests.timezone=Africa/Brazzaville -Druntime.java=11 -Dtests.fips.enabled=true

Applicable branches:
master

Reproduces locally?:
Yes

Failure history:
https://gradle-enterprise.elastic.co/scans/tests?tests.container=org.elasticsearch.rest.Netty4HeadBodyIsEmptyIT&tests.test=testTemplateExists

Failure excerpt:

org.elasticsearch.client.WarningFailureException: method [PUT], host [http://[::1]:34641], URI [/_template/template], status line [HTTP/1.1 200 OK]
Warnings: [legacy template [template] has index patterns [*] matching patterns from existing composable templates [.deprecation-indexing-template,.slm-history,.watch-history-14,ilm-history,logs,metrics,synthetics] with patterns (.deprecation-indexing-template => [.logs-deprecation.*],.slm-history => [.slm-history-5*],.watch-history-14 => [.watcher-history-14*],ilm-history => [ilm-history-5*],logs => [logs-*-*],metrics => [metrics-*-*],synthetics => [synthetics-*-*]); this template [template] may be ignored in favor of a composable template at index creation time]
{"acknowledged":true}

  at __randomizedtesting.SeedInfo.seed([A02E26474C28612C:73DFDE5595D75E1F]:0)
  at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:326)
  at org.elasticsearch.client.RestClient.performRequest(RestClient.java:296)
  at org.elasticsearch.client.RestClient.performRequest(RestClient.java:270)
  at org.elasticsearch.rest.Netty4HeadBodyIsEmptyIT.testTemplateExists(Netty4HeadBodyIsEmptyIT.java:128)
  at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:-2)
  at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:566)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
  at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
  at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
  at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
  at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:375)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:824)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:475)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
  at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
  at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:375)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:831)
  at java.lang.Thread.run(Thread.java:834)

[elasticmachine]

Pinging @elastic/es-security (Team:Security)

[mark-vieira]

Since this reproduces I've muted this test in master.

[ywangd]

This is another re-occurrence of #66820 and the root cause is that we use different distributions in regular and FIPS tests. We have an issue (#70005) for not mandating default distribution for FIPS tests and it should solve this problem for good. In the meantime, we could choose to either mute the test or raise another stop-gap fix.

[valeriy42]

Other occurrences today on 7.x

• https://gradle-enterprise.elastic.co/s/dxb6cnvidytqm
• https://gradle-enterprise.elastic.co/s/e7wwzgosq5eve
• https://gradle-enterprise.elastic.co/s/f6fpotbi7uysa
• https://gradle-enterprise.elastic.co/s/l7bnzc67os5sk
• https://gradle-enterprise.elastic.co/s/y4h5rao47pb44

[ywangd]

Cherry-picked Mark's mute from master into 7.x 8b32746

[jakelandis]

This test was fixed and unmuted in the main branch but remains muted in 7.x. The fix depends on changing the way FIPS test execute (#91101) and I don't want to introduce that change in 7.x to avoid wide sweeping changes in that patch branch.