Add integration test for workflow provisioning under multitenancy

[dbwiddis]

Is your feature request related to a problem?

We have integration tests for multitenancy for our template CRUD operations.

We have integration tests for provisioning templates.

We do not yet have integration tests to verify that the tenant ID is properly passed to the ML client as implemented in PRs for #987.

What solution would you like?

Add new, or modify existing, integration tests to validate the Create/Delete integrations for Connector, Model (register and deploy), Model Group, and Agent properly allow access only to the correct tenant.

Do you have any additional context?

You'll want to add methods to https://github.com/opensearch-project/flow-framework/blob/main/src/test/java/org/opensearch/flowframework/rest/RestWorkflowStateTenantAwareIT.java (you could do a new one but this has a lot of overlap)

The existing remote model template comes from https://github.com/opensearch-project/flow-framework/blob/main/src/test/resources/template/createconnector-registerremotemodel-deploymodel.json which does everything but the agent. Perhaps add a new json that also does the agent, or copy/simplify the one from https://opensearch.org/docs/latest/automating-configurations/workflow-tutorial/#complete-json-workflow-template

Add new code after provisioning

• fetch the list of resources from the state (you can use the stateMap.get("resources_created") to drill down to them, or possibly parse the JSON to a WorkflowState object) and get the resource IDs for all of the things you deployed
• test GET against the ML Commons REST APIs for those resource IDs, both with the correct tenant and with an incorrect tenant and with a null tenant, similar to (maybe even copy/paste) the GET code from https://github.com/opensearch-project/ml-commons/pull/2818/files
  • you should be able to GET an OK response with the correct tenantID and a FORBIDDEN response with incorrect or null

NOTE: you'll probably need to add the ml commons tenant aware setting in build.gradle in addition to the flow framework one, here:

flow-framework/build.gradle

Line 274 in d8a5a4b

systemProperty "plugins.flow_framework.multi_tenancy_enabled", "true"

Then after the deprovisioning hit the ML commons APIs and verify the resources are deleted (same PR has some delete tests... GET should return 404.)