[Multiple Datasource] Support serverless AWS OpenSearch in SigV4 auth type

[zhongnansu]

Problem statement

Multiple datasource should be able to connect to serverless AWS OpenSearch: https://aws.amazon.com/opensearch-service/features/serverless/

UI/UX changes (proposal only)

To create a serverless data source, there needs to be a way for user to specify serverless opensearch. I propose a new required field service type in the "create data source" flow, with options AWS OpenSearch and AWS OpenSearch serverless. Below image only explains the proposal. Please help with actual UI mock-up @shanilpa, and the correct naming.

Task breakdown

• POC & Research
• refactor test_connection to remove the use of .info() API, because .info() API is not supported in serverless. Use ping() instead
• refactor schema for datasource of sigV4 type, to add field of service, candidates values are es or aoss
• refactor UI to let user to specify serverless or regular aws opensearch, when creating data source with sigV4 auth type.
• support both legacy datasource client and new datasource client
• update functional tests: [Multiple Datasource] Add functional test cases for serverless support feature opensearch-dashboards-functional-test#665

Reference:

• Opensearch-js client 2.x has support for serverless Opensearch: [FEATURE] Add support for OpenSearch Serverless opensearch-js#356

• Similar issue: SigV4 supports different service name and other service's test connection #3718

[zhongnansu]

To support serverless for legacy datasource client is not as straight-forward as new client, because unlike new clients, legacy client package elasticsearch.js doesn't

1. provide option to sign requests using sigv4.
2. provide option to set service: aoss, when singing and sending reuqests.

For 1), we solved it in MD while releasing the sigV4 support in 2.6, by using a third party Connection handler for Amazon ES, https://github.com/TheDeveloper/http-aws-es. HttpAmazonESConnector

OpenSearch-Dashboards/src/plugins/data_source/server/legacy/configure_legacy_client.ts

Line 10 in 9d27023

import HttpAmazonESConnector from 'http-aws-es';

OpenSearch-Dashboards/src/plugins/data_source/server/legacy/configure_legacy_client.ts

Lines 197 to 208 in 9d27023

	const getAWSClient = (credential: SigV4Content, clientOptions: ConfigOptions): LegacyClient => {
	const { accessKey, secretKey, region } = credential;
	const client = new LegacyClient({
	connectionClass: HttpAmazonESConnector,
	awsConfig: new Config({
	region,
	credentials: new Credentials({ accessKeyId: accessKey, secretAccessKey: secretKey }),
	}),
	...clientOptions,
	});
	return client;
	};

For 2), since the http-aws-es package hardcoded service:es when signing and sending requests, we can't configure the client with serverless identifier directly.

Proposal

1. Short term: use a drop-in replacement package of http-aws-es, to add support of configurable service name.
2. Long term: Once Migrate from legacy client to opensearch client #2220 is completed, we'll remove the usage of legacy client, along with http-aws-es. Because new client natively have support for all we need

[KrooshalUX]

this project will be @shanilpa for UX/UI support, @zhongnansu

[kavilla]

Closing per @zhongnansu