Adds implementation for the delay feature in rollup jobs #147
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
thalurur
reviewed
Sep 21, 2021
src/main/kotlin/org/opensearch/indexmanagement/indexstatemanagement/ManagedIndexRunner.kt
Outdated
Show resolved
Hide resolved
thalurur
reviewed
Sep 21, 2021
src/main/kotlin/org/opensearch/indexmanagement/rollup/RollupSearchService.kt
Outdated
Show resolved
Hide resolved
thalurur
reviewed
Sep 21, 2021
src/main/kotlin/org/opensearch/indexmanagement/rollup/util/RollupUtils.kt
Outdated
Show resolved
Hide resolved
thalurur
reviewed
Sep 21, 2021
thalurur
reviewed
Sep 21, 2021
src/main/kotlin/org/opensearch/indexmanagement/rollup/util/RollupUtils.kt
Outdated
Show resolved
Hide resolved
5 tasks
Signed-off-by: Clay Downs <[email protected]>
Signed-off-by: Clay Downs <[email protected]>
Signed-off-by: Robert Downs <[email protected]>
Signed-off-by: Clay Downs <[email protected]>
Signed-off-by: Clay Downs <[email protected]>
Signed-off-by: Clay Downs <[email protected]>
downsrob
force-pushed
the
rollups-delay
branch
from
947b1e4 to
659dfbc
Compare
Codecov Report
@@ Coverage Diff @@
## main #147 +/- ##
============================================
+ Coverage 75.70% 75.80% +0.10%
- Complexity 1921 1934 +13
============================================
Files 260 260
Lines 11016 11041 +25
Branches 1724 1735 +11
============================================
+ Hits 8340 8370 +30
+ Misses 1721 1713 -8
- Partials 955 958 +3
Continue to review full report at Codecov.
|
Signed-off-by: Clay Downs <[email protected]>
thalurur
approved these changes
Oct 11, 2021
downsrob
added a commit
to downsrob/index-management
that referenced
this pull request
Oct 16, 2021
…project#147) * Adds delay implementation for rollup jobs * Removes non-continuous delay implementation * Adds additional rollup delay tests Signed-off-by: Clay Downs <[email protected]>
1 task
downsrob
added a commit
that referenced
this pull request
Oct 25, 2021
* Adds implementation for the delay feature in rollup jobs (BACKPORT) (#147) Signed-off-by: Clay Downs <[email protected]>
downsrob
added a commit
that referenced
this pull request
Nov 5, 2021
* Add integTest script to the repo (#94) Signed-off-by: Peter Zhu <[email protected]> * Removing Usages of Action Get Call and using listeners (#100) Signed-off-by: Aditya Jindal <[email protected]> * Enhance ISM template (#105) Signed-off-by: bowenlan-amzn <[email protected]> * Explain response still use old opendistro policy id (#109) * Explain response still use old opendistro policy id * Use hardcoded policyid setting in tests for explain response * Trying to fix flaky tests * Storing user information as part of the job when security plugin is installed (#113) Signed-off-by: Ravi Thaluru <[email protected]> * ISM/Notification channel support (#117) * Updates NAME of transport actions * Upgrades Kotlin version, updates dependencies on other OS plugins, adds notification plugin as a test resource and includes it in test clusters * Adds support for Channels in error notifications and notification actions * Adds support for sending notifications to channels * Adds support for publishing notifications to the legacy destinations through the Notification plugin and some cleanup * Removes notification alerting jar dependency * Adds compile only dep on commons codec for digest utils sha1 method in ism rollup * Updates Error Notification to make channel/destination nullable, and adds helper methods for publish calls * Constructs URL for legacy custom webhook Signed-off-by: Drew Baugher <[email protected]> * Fixes Feature enum and dep Signed-off-by: Drew Baugher <[email protected]> * Trying something else Signed-off-by: Drew Baugher <[email protected]> * Addresses comments Signed-off-by: Drew Baugher <[email protected]> * Storing user object in all APIs and enabling filter of response based on user (#115) Signed-off-by: Ravi Thaluru <[email protected]> * Upgrade dependencies to 1.1 and build snapshot by default. (#121) Signed-off-by: dblock <[email protected]> * Security improvements (#126) Signed-off-by: Ravi Thaluru <[email protected]> * Removes support for notification plugin (#136) Signed-off-by: Drew Baugher <[email protected]> * Updating security filtering logic (#137) Signed-off-by: Ravi Thaluru <[email protected]> * Release note for 1.1.0.0 release. (#139) * Release note for 1.1.0.0 release. Signed-off-by: bowenlan-amzn <[email protected]> * Correct copyright notices * Uses published daily snapshot dependencies (#141) Signed-off-by: Drew Baugher <[email protected]> * License header check (#142) * Provide default copyright header using IDE feature Signed-off-by: bowenlan-amzn <[email protected]> * Address #103 history write index is rolled over even if the history indices are disabled * Removed integtest.sh. (#148) Signed-off-by: dblock <[email protected]> * Adds mavenLocal back to repositories (#158) Signed-off-by: Drew Baugher <[email protected]> * Making snapshot name to scripted input in template (#77) Signed-off-by: Ravi Thaluru <[email protected]> * Fix issues with security changes in rollup runnner (#161) * Updates index management version to 1.2 (#157) * Updates index management version to 1.2 * Updates job scheduler snapshot to 1.2 in test resources Signed-off-by: Robert Downs <[email protected]> * Adds setting to search all rollup jobs on a target index (#165) * Adds cluster setting to search all rollup jobs Signed-off-by: Clay Downs <[email protected]> * Adds implementation for the delay feature in rollup jobs (#147) * Adds delay implementation for rollup jobs * Removes non-continuous delay implementation * Adds additional rollup delay tests Signed-off-by: Clay Downs <[email protected]> * Updates testCompile mockito version, adds AwaitsFix annotation to MetadataRegressionIT tests (#168) * Updates testCompile mockito version to match OpenSearch changes * AwaitsFix on the failing MetadataRegressionIT tests Signed-off-by: Robert Downs <[email protected]> * Adds cluster setting to configure index state management jitter (#153) * Adds jitter cluster setting, sets jitter to 0 for ISM tests Signed-off-by: Clay Downs <[email protected]> * Allows out of band rollovers on an index without causing ISM to fail (#180) * Allows out of band rollovers on an index without causing ISM to fail Signed-off-by: Drew Baugher <[email protected]> * Fixes detekt issue Signed-off-by: Drew Baugher <[email protected]> * Remove policy API on read only indices (#182) * In explain API not showing the total count to all users (#185) Co-authored-by: Peter Zhu <[email protected]> Co-authored-by: Aditya Jindal <[email protected]> Co-authored-by: Bowen Lan <[email protected]> Co-authored-by: Ravi <[email protected]> Co-authored-by: Drew Baugher <[email protected]> Co-authored-by: Daniel Doubrovkine (dB.) <[email protected]>
wuychn
pushed a commit
to ochprince/index-management
that referenced
this pull request
Mar 16, 2023
…project#147) * Adds delay implementation for rollup jobs * Removes non-continuous delay implementation * Adds additional rollup delay tests Signed-off-by: Clay Downs <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Adds implementation for the delay parameter in rollup jobs. Previously, the delay parameter was included in the api and stored with the job, but implementation was never added to use this delay when searching for windows of data to be used in the rollup job. In this PR, the rollup job excludes data from the delay time period, and the delay is passed to the job scheduler to handle delaying the actual execution.
There were job scheduler changes required as a prerequisite to this PR. This Job Scheduler PR. included changes which enabled the delay to be passed to the job scheduler to actually delay execution.
CheckList:
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.