Calling some ILM APIs during rolling 6.5 to 6.6/6.7 upgrades causes 6.5 nodes to fail to join the cluster

[gwbrown]

Calling the ILM Start, Stop, or Put Lifecycle APIs in a mixed 6.5 and 6.6/6.7 cluster will cause the 6.5 nodes to fail to join the cluster with this error:

java.lang.IllegalArgumentException: Unknown NamedWriteable [org.elasticsearch.cluster.metadata.MetaData$Custom][index_lifecycle]

The cause of this is that the ILM custom metadata states that the minimum supported version is 6.5.0 (this constant was never updated when ILM was pushed back to 6.6):

elasticsearch/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/indexlifecycle/IndexLifecycleMetadata.java

Lines 108 to 110 in 4739058

	public Version getMinimalSupportedVersion() {
	return Version.V_6_5_0;
	}

This causes the 6.5 nodes to try to read the custom metadata, but because ILM is not present in 6.5, this fails.

A full cluster restart upgrade to the new version appears to be the only way to resolve the issue if a cluster gets into this state.

[elasticmachine]

Pinging @elastic/es-core-features

[gwbrown]

The minimum supported version is also incorrect in 7.0, 7.x, and master, although I don't believe it would cause the same issue, as it states a minimum version that is too high:

elasticsearch/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/indexlifecycle/IndexLifecycleMetadata.java

Lines 108 to 110 in cfa5319

	public Version getMinimalSupportedVersion() {
	return Version.V_7_0_0;
	}

[gwbrown]

Leaving this here so I don't forget: This can also happen with auto-follow patterns as the auto-follow pattern metadata uses this minimum compatability version:

elasticsearch/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/ccr/AutoFollowMetadata.java

Lines 122 to 124 in c02aa40

	public Version getMinimalSupportedVersion() {
	return Version.V_6_5_0.minimumCompatibilityVersion();
	}

Which evaluates to 5.6.0.

[gwbrown]

Fixed by #40569.