Implement multi tenancy in Flow Framework

[dbwiddis]

Description

Uses the multitenant remote metadata client developed in ML Commons feature/multi_tenancy branch to migrate Flow Framework system indices to use the same pattern.

Submitting in logical steps to ease maintainer review. Suggestions:

1. To start, review each of the first several commits to understand how the tenant id is implemented and passed around
2. Then visit the tenant aware integ test classes to understand what it is we're trying to allow/prohibit

Check List

• New functionality includes testing.
• New functionality has been documented.
• Commits are signed per the DCO using --signoff.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[codecov]

Codecov Report

Attention: Patch coverage is 62.74256% with 288 lines in your changes missing coverage. Please review.

Project coverage is 75.51%. Comparing base (1c16b1b) to head (3550786).
Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
...framework/indices/FlowFrameworkIndicesHandler.java	65.07%	82 Missing and 6 partials ⚠️
...ework/transport/CreateWorkflowTransportAction.java	66.05%	28 Missing and 9 partials ⚠️
.../opensearch/flowframework/util/EncryptorUtils.java	63.63%	26 Missing and 10 partials ⚠️
.../transport/ReprovisionWorkflowTransportAction.java	20.00%	14 Missing and 2 partials ⚠️
...ework/transport/DeleteWorkflowTransportAction.java	55.17%	10 Missing and 3 partials ⚠️
...flowframework/transport/handler/SearchHandler.java	45.83%	13 Missing ⚠️
.../org/opensearch/flowframework/util/ParseUtils.java	43.47%	9 Missing and 4 partials ⚠️
...h/flowframework/rest/RestCreateWorkflowAction.java	60.71%	6 Missing and 5 partials ⚠️
...rk/transport/ProvisionWorkflowTransportAction.java	47.05%	7 Missing and 2 partials ⚠️
.../opensearch/flowframework/FlowFrameworkPlugin.java	50.00%	7 Missing and 1 partial ⚠️
... and 13 more

Additional details and impacted files

@@             Coverage Diff              @@
##               main     #980      +/-   ##
============================================
- Coverage     76.39%   75.51%   -0.89%     
- Complexity     1010     1050      +40     
============================================
  Files           100      101       +1     
  Lines          4881     5211     +330     
  Branches        455      498      +43     
============================================
+ Hits           3729     3935     +206     
- Misses          955     1038      +83     
- Partials        197      238      +41     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[owaiskazi19]

There are discrepancies at few places between INTERNAL_SERVER_ERROR and BAD_REQUEST rest status for the request calls

[dbwiddis]

@owaiskazi19 @amitgalitz Can we try to get this merged today? I think I've addressed all your comments or at least answered them. All tests are passing.

• Note multitenancy is not designed to be a dynamic setting or one that will be upgraded to for BWC, and even the storage option settings are set at startup. Adding a tenant_id field to every document means they are inaccessible to other tenants with the setting true, turning it off would expose all this data. And if a cluster had data on it before it was "toggled on" then none of that data would be visible to anyone.
  • When the setting is false then tenantId is set to null and the intent of most of this code is that there is no change to existing functionality in that case; while enabling per-tenant isolation in other environments; all while maintaining the same set of code.
• SdkClient initialization in the plugin class is a one method call to another external method. It only looks complex because of a 6-element map declared inline. I can potentially move that map to the settings class but can do that in a later PR; there is nothing time critical about it.
• The current thread pool default may be deleted in a future PR. Once I get this merged I will have time to work on doing that.
• I've set our threadpools back to 4/8/4 which is double what we started with. The main impact here is all our uses of actionGet() which block threads and consume the pool; there is not a lot of extra CPU involved. These should probably be made settings and I may make that change shortly as in a multitenant environment they are way too small.
• I've changed the "default tenant id" to an empty string.
• The use of a latch for master key initialization is taken from ML Commons implementation (as are a lot of the multitenancy features which were developed on a feature branch there and thoroughly tested). If you have another way to do it, feel free to rewrite that code. I know what's there works.
• @owaiskazi19 commented "There are discrepancies at few places between INTERNAL_SERVER_ERROR and BAD_REQUEST rest status for the request calls" but I'm not sure where these are. I've tried to change a couple, but many are pre-existing. Can we make a follow-up issue if you think there are any more?

[owaiskazi19]

Thanks for addressing the comments. Not blocking the PR for the ones that can be addressed later

[opensearch-trigger-bot]

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/flow-framework/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/flow-framework/backport-2.x
# Create a new branch
git switch --create backport/backport-980-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 d8a5a4bdbcb39bd45e7fb3b6aa9f800b59b1ddfb
# Push it to GitHub
git push --set-upstream origin backport/backport-980-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/flow-framework/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-980-to-2.x.