Indexer: adhoc dataset indexing API

[t83714]

Indexer: Adhoc dataset indexing API

The indexer currently subscribes (backed by registry webhook) to the dataset changes event stream and auto-indexing any dataset changes.

This process may introduce a few seconds delay between the changes made on the original dataset in the registry and the changes indexed in the search engine.

The short time delay won't cause too much trouble for general-purpose updates. But for access control related updates, we will want a proactive interface to index the changes without delay.

This ticket is about adding 2 new APIs to the indexer:

• index dataset by id: PUT /v0/dataset/:id
  • indexer will attempt to retrieve the dataset data from the registry using the ID and index the new data into search engine
• delete dataset by id: DELETE /v0/dataset/:id
  • indexer will delete the dataset with the specified id from the search engine
• Once the APIs are added, we also need to expose them via gateway & add auth to existing APIs (previously are internally only APIs). For existing APIs, we should:
• exposing the following APIs.
  • POST /reindex : trigger a proactive full index.
    • validate auth via operationUri api/indexer/reindex.
  • POST /reindex/snapshot: trigger a snapshot.
    • validate auth via operationUri api/indexer/reindex/snapshot.
    • We don't use this API at this moment as it might not fully functional due to this ticket
  • GET /reindex/in-progress: get progress info of the full index trigger.
    • validate auth via operationUri api/indexer/reindex/in-progress.
  • DELETE /dataset/{datasetId}: delete a dataset from search engine index
    • validate auth via operationUri object/dataset/delete
  • PUT /dataset/{datasetId}: reindex a single dataset
    • validate auth via operationUri object/dataset/update
• The following APIs will still stay internal only.
  • POST /registry-hook: web-hook listener
    • stay as an internal API for performance consideration
  • Status API e.g. /status/ready & /status/live
    • stay as internal API as there is for internal usage only

[t83714]

Partially done in PR: #3344
i.e.: API added but still needs to expose via gateway & add auth to existing APIs (previously are internally only APIs)

[t83714]

closed via PR #3344 and #3358