Clean up empty IOC indices created by failed source configs #1267
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Joanne Wang <[email protected]>
jowg-amazon
requested review from
amsiglan,
AWSHurneyt,
getsaurabh02,
lezzago,
praveensameneni,
sbcd90,
eirsep,
engechas,
goyamegh and
riysaxen-amzn
as code owners
AWSHurneyt
reviewed
Aug 21, 2024
| return; | ||
| } | ||
|
|
||
| String id = SecurityAnalyticsPlugin.JOB_INDEX_NAME + "-" + saTifSourceConfig.getId(); |
There was a problem hiding this comment.
Just to clarify, we don't need to convert this string to lowercase because this ID of the document in the JOB_INDEX_NAME is literally SecurityAnalyticsPlugin.JOB_INDEX_NAME + "-" + saTifSourceConfig.getId(), right?
There was a problem hiding this comment.
Yes, here's an example of what the id in the job scheduler lock index looks like .opensearch-sap--job-Q3l1dpEBFBLkuvfJmI5a
AWSHurneyt
previously approved these changes
Aug 22, 2024
engechas
previously approved these changes
Aug 22, 2024
|
|
||
| // check to make sure the job scheduler lock index exists | ||
| if (clusterService.state().metadata().hasIndex(LOCK_INDEX_NAME) == false) { | ||
| actionListener.onFailure(SecurityAnalyticsException.wrap(new OpenSearchStatusException("Threat intel job scheduler lock index does not exist", RestStatus.BAD_REQUEST))); |
There was a problem hiding this comment.
I've seen this done both ways and I think either can make sense based on the situation. We can either
- Treat the index missing as a failure since we intended to delete something and could not (current approach)
- Treat the index missing as success since we intended to delete something but it did not exist so we have fulfilled the request to delete it
It would be worth considering when this API would be invoked to decide whether 1 or 2 is the correct behavior. I could see things like operator intervention resulting in the lock index being deleted and we should be sure that would not cause permanent failures/further issues if we go with approach 1.
There was a problem hiding this comment.
Makes sense, this is only called when we want to clean up and delete everything related to the source config that's being deleted so i think the second option makes more sense in this case. Updated the code to not fail when the index doesn't exist or if the document is not found.
Comment on lines
403
to
414
| if (deleteResponse.status().equals(RestStatus.OK)) { | ||
| log.info("Deleted threat intel job scheduler lock [{}] successfully", id); | ||
| actionListener.onResponse(deleteResponse); | ||
| } else if (deleteResponse.status().equals(RestStatus.NOT_FOUND)) { | ||
| actionListener.onFailure(SecurityAnalyticsException.wrap(new OpenSearchStatusException(String.format(Locale.getDefault(), "Threat intel job scheduler lock with id [{%s}] not found", id), RestStatus.NOT_FOUND))); | ||
| } else { | ||
| actionListener.onFailure(SecurityAnalyticsException.wrap(new OpenSearchStatusException(String.format(Locale.getDefault(), "Failed to delete threat intel job scheduler lock with id [{%s}]", id), deleteResponse.status()))); | ||
| } | ||
| }, e -> { | ||
| log.error("Failed to delete threat intel job scheduler lock with id [{}]", id); | ||
| actionListener.onFailure(e); | ||
| } |
There was a problem hiding this comment.
Let's make sure we have unit tests for all of these scenarios
There was a problem hiding this comment.
Yep, will do in a follow up PR and added a github issue to increase our code cov and integ testing overall: #1268
Signed-off-by: Joanne Wang <[email protected]>
engechas
approved these changes
Aug 23, 2024
AWSHurneyt
approved these changes
Aug 23, 2024
opensearch-trigger-bot bot
pushed a commit
that referenced
this pull request
Aug 23, 2024
* cleanup empty iocs and lock Signed-off-by: Joanne Wang <[email protected]> * change action listener response Signed-off-by: Joanne Wang <[email protected]> --------- Signed-off-by: Joanne Wang <[email protected]> (cherry picked from commit 0920e47) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
opensearch-trigger-bot bot
pushed a commit
that referenced
this pull request
Aug 23, 2024
* cleanup empty iocs and lock Signed-off-by: Joanne Wang <[email protected]> * change action listener response Signed-off-by: Joanne Wang <[email protected]> --------- Signed-off-by: Joanne Wang <[email protected]> (cherry picked from commit 0920e47) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
opensearch-trigger-bot bot
pushed a commit
that referenced
this pull request
Aug 23, 2024
* cleanup empty iocs and lock Signed-off-by: Joanne Wang <[email protected]> * change action listener response Signed-off-by: Joanne Wang <[email protected]> --------- Signed-off-by: Joanne Wang <[email protected]> (cherry picked from commit 0920e47) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
jowg-amazon
pushed a commit
that referenced
this pull request
Aug 23, 2024
…1269) * cleanup empty iocs and lock * change action listener response --------- (cherry picked from commit 0920e47) Signed-off-by: Joanne Wang <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
jowg-amazon
pushed a commit
that referenced
this pull request
Aug 23, 2024
…1270) * cleanup empty iocs and lock * change action listener response --------- (cherry picked from commit 0920e47) Signed-off-by: Joanne Wang <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
jowg-amazon
pushed a commit
that referenced
this pull request
Aug 24, 2024
…1271) * cleanup empty iocs and lock * change action listener response --------- (cherry picked from commit 0920e47) Signed-off-by: Joanne Wang <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Deletes and cleans up any ioc indices that were created by failed source configs. Also manually deletes the job scheduler lock for source configs that have the job disabled. When a source config job is enabled and then the source config is deleted, the lock is automatically deleted. However, the lock is not cleaned up if the job is disabled. This PR adds a change to check whether or not the job is enabled, if not then it will manually delete the associated lock when source config is deleted.
Related Issues
Resolves #1255
Check List
--signoff.By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.