Unable to create Index pattern when 7.16 is parent build and 7.15 is child build in CCS setup.

[ghost]

Describe the bug:
Unable to create Index pattern when 7.16 is parent build and 7.15 is child build in CCS setup.
Kibana/Elasticsearch Stack version:

Version: 7.16.0
Build: 45504
Commit: 9231d806c9384df4026977ba7435a9302dc2d4ab

Pre-Conditions

1. Kibana should be up and running
2. Elastic search should be up and running
3. 7.16 build should be present as parent build and 7.15 as child build

Steps to reproduce:

1. Get the Proxy and server address of the 7.15 build from the Deployment>Security
2. Navigate to Stack management > Remote clusters under Data on 7.16 build
3. Create the remote cluster with Proxy and Server address of 7.15 build.
4. Navigate to Index pattern under Kibana and click on create Index pattern

Current behavior:
Index pattern is not getting created.

Expected behavior:
Index Pattern Should be created.

Screen Records what's not working:

Recording.12.mp4

Screen Record what's working:

1. Working when 7.15 is parent build and 7.16 is child build

Recording.13.mp4

2. Working when 7.15 is a parent build and 7.14 is a child build

remote_cluster_7.15.0.and.7.14.0.mp4

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[bhavyarm]

Hey team - can you please check this bug - #115904? We have this in works - #115942 . TLDR - is your user missing "view_index_metadata" permissions?

Thanks!

[Dosant]

Some additional observations:

• 7.16 parent and 7.16 child - works

• When calling GET /_resolve/index/7_15:* from dev tools app I get the correct result, but there are no results when going through the index pattern UI, which under the hood calls client.asCurrentUser.indices.resolveIndex

kibana/src/plugins/index_pattern_management/server/routes/resolve_index.ts

Lines 34 to 38 in 6fbda83

	const { body } = await context.core.elasticsearch.client.asCurrentUser.indices.resolveIndex({
	name: req.params.query,
	expand_wildcards: req.query.expand_wildcards || 'open',
	});
	return res.ok({ body });

For now, I don't understand what's the difference and where is the problem

[Dosant]

When calling GET /_resolve/index/7_15:* from dev tools app I get the correct result, but there are no results when going through the index pattern UI, which under the hood calls client.asCurrentUser.indices.resolveIndex

I found the difference between raw requests to elasticsearch that is causing this difference. Looks like it is x-opaque-id header. It is present when calling via elasticsearch client and is missing when calling via dev tools. When calling with with this header - then GET /_resolve/index/7_15:* doesn't return results from remote cluster.

Sample request to 7.16 cluster with remote 7.15 that works:

GET /_resolve/index/7_15:*?expand_wildcards=open HTTP/1.1
Host: localhost:9200
Authorization: Basic ZWxhc3RpYzpjaGFuZ2VtZQ==

Sample request to 7.16 cluster with remote 7.15 that returns empty result:

GET /_resolve/index/7_15:*?expand_wildcards=open HTTP/1.1
Host: localhost:9200
x-opaque-id: test
Authorization: Basic ZWxhc3RpYzpjaGFuZ2VtZQ==

This seems surreal to me...

@elastic/kibana-core, do we know of any change with x-opaque-id in 7.16.0? otherwise this might be a es bug

[Dosant]

do we know of any change with x-opaque-id in 7.16.0? otherwise this might be a es bug

Checked that 7.15 Kibana does the same request and has x-opaque-id so the issue is likely not in Kibana

I also observed the same issue when using 7.15 snapshot -> 7.14 cluster, so we will probably have the same bug in 7.15.2 release.

7.15.1 -> 7.14 works as expected.

[Dosant]

es issue is fixed elastic/elasticsearch#79948, so we should be good with the next build candidate

[ghost]

Hi Team,

We have validated this issue on 7.16.0 BC2 build and issue is still occurring, we will regress this issue again once 7.16.0 BC3 build will be available.

Build Details:

Version: 7.16.0 BC2
Build:  45662
Commit: ae8c5d7765636dd177c72ab19408a0791eba491f

Screen record:

Recording.18.mp4

Hence, we are reopening this issue.

Thanks!!

[Dosant]

@karanverma-qasource, thank you for reopening,
indeed the es commit that was supposed to fix this didn't make it to BC2

[MikePaquette]

Team, should this issue be raised as a blocker in 7.16.0 in https://github.com/elastic/dev/issues/1809 ? It's a show-stopper for users upgrading the CCS cluster with remotes at 7.15. cc: @kobelb

[kobelb]

@MikePaquette, in my opinion, this should be a blocker for 7.16, I'll add it as such.

[Dosant]

@karanbirsingh-qasource,

we will regress this issue again once 7.16.0 BC3 build will be available.

I just tested with the recent BC3 in cloud, the issue seems to be fixed, I'll leave the issue open for you to confirm and close

[ghost]

Hi Team,

We have validated the issue on 7.16.0 BC3 build and issue is fixed.

Build Details:

Version: 7.16.0 BC3
Build: 45816
Commit: acaa761f4ce46680fd7cfbeba03a652c72dc786b

Screen Records:

Recording.21.mp4

Please let us know if anything else is required from our end.

Thanks!!

[ghost]

Hi Team,

Validated the issue on other old version too and there too success in remote cluster and index pattern creation ✅

• 7.16.0 BC3 Parent and 7.14.2 Child

• 7.16.0 BC3 Child and 7.14.2 Parent

• 7.16.0 BC3 Parent and 7.13.0 Child

• 7.16.0 BC3 Child and 7.13.0 Parent
  

c.c @MadameSheema

[bhavyarm]

Thanks @karanbirsingh-qasource

[ghost]

Hi @MadameSheema,

We have validated this issue on 7.16.0 BC5 with other old versions and there too success in remote cluster and index pattern creation 🟢

Build Details:

Version:7.16.0 BC5
Build: 46061
COMMIT: f13296db8798dd0cd39ab6cc4a61a35a2a2b05cc

• 7.16.0 BC5 Parent and 7.15.0 Child

ccs.mp4

• 7.16.0 BC5 Parent and 7.14.0 Child
  

• 7.16.0 BC5 Child and 7.15.0 Parent
  

Thanks!!