-
Notifications
You must be signed in to change notification settings - Fork 342
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add index curation (close/delete indexes based on age or size)
Merge branch 'topic/curator'
- Loading branch information
Showing
12 changed files
with
537 additions
and
59 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,61 @@ | ||
| FROM debian:buster-slim | ||
|
|
||
| # Copyright (c) 2019 Battelle Energy Alliance, LLC. All rights reserved. | ||
| LABEL maintainer="[email protected]" | ||
|
|
||
| ARG ES_HOST=elasticsearch | ||
| ARG ES_PORT=9200 | ||
| ARG CURATOR_TIMEOUT=120 | ||
| ARG CURATOR_MASTER_ONLY=False | ||
| ARG CURATOR_LOGLEVEL=INFO | ||
| ARG CURATOR_LOGFORMAT=default | ||
|
|
||
| ARG CURATOR_CLOSE_UNITS=years | ||
| ARG CURATOR_CLOSE_COUNT=10 | ||
| ARG CURATOR_DELETE_UNITS=years | ||
| ARG CURATOR_DELETE_COUNT=99 | ||
| ARG CURATOR_DELETE_GIGS=1000000 | ||
|
|
||
| ENV ES_HOST $ES_HOST | ||
| ENV ES_PORT $ES_PORT | ||
| ENV CURATOR_TIMEOUT $CURATOR_TIMEOUT | ||
| ENV CURATOR_MASTER_ONLY $CURATOR_MASTER_ONLY | ||
| ENV CURATOR_LOGLEVEL $CURATOR_LOGLEVEL | ||
| ENV CURATOR_LOGFORMAT $CURATOR_LOGFORMAT | ||
|
|
||
| ENV CURATOR_CLOSE_UNITS $CURATOR_CLOSE_UNITS | ||
| ENV CURATOR_CLOSE_COUNT $CURATOR_CLOSE_COUNT | ||
| ENV CURATOR_DELETE_UNITS $CURATOR_DELETE_UNITS | ||
| ENV CURATOR_DELETE_COUNT $CURATOR_DELETE_COUNT | ||
| ENV CURATOR_DELETE_GIGS $CURATOR_DELETE_GIGS | ||
|
|
||
| ENV DEBIAN_FRONTEND noninteractive | ||
| ENV CURATOR_VERSION "5.7.6" | ||
| ENV CRON "15 */6 * * *" | ||
| ENV CONFIG_FILE "/config/config_file.yml" | ||
| ENV ACTION_FILE "/config/action_file.yml" | ||
| ENV CURATOR_USER "curator" | ||
|
|
||
| RUN sed -i "s/buster main/buster main contrib non-free/g" /etc/apt/sources.list && \ | ||
| apt-get update && \ | ||
| apt-get -y -q install \ | ||
| build-essential \ | ||
| cron \ | ||
| procps \ | ||
| psmisc \ | ||
| python3 \ | ||
| python3-dev \ | ||
| python3-pip && \ | ||
| pip3 install elasticsearch-curator==${CURATOR_VERSION} && \ | ||
| groupadd --gid 1000 ${CURATOR_USER} && \ | ||
| useradd -M --uid 1000 --gid 1000 ${CURATOR_USER} && \ | ||
| apt-get -q -y --purge remove python3-dev build-essential && \ | ||
| apt-get -q -y autoremove && \ | ||
| apt-get clean && \ | ||
| rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* && \ | ||
| bash -c 'echo "${CRON} su -c \"/usr/local/bin/curator --config ${CONFIG_FILE} ${ACTION_FILE}\" ${CURATOR_USER} >/proc/1/fd/1 2>/proc/1/fd/2" | crontab -' | ||
|
|
||
| ADD curator/scripts /usr/local/bin/ | ||
| ADD curator/config /config/ | ||
|
|
||
| CMD ["/usr/local/bin/cron_env_deb.sh"] |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,4 +1,4 @@ | ||
| FROM debian:buster-slim AS build | ||
| FROM debian:buster-slim | ||
|
|
||
| # Copyright (c) 2019 Battelle Energy Alliance, LLC. All rights reserved. | ||
| LABEL maintainer="[email protected]" | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,72 @@ | ||
| actions: | ||
| 1: | ||
| action: delete_indices | ||
| description: >- | ||
| Delete the oldest indices with sessions2- prefixes when their cumulative | ||
| disk consumtion is greater than ${CURATOR_DELETE_GIGS:1000000} gigabytes. Ignore | ||
| the error if the filter does not result in an actionable list of indices | ||
| (ignore_empty_list) and exit cleanly. | ||
| options: | ||
| ignore_empty_list: True | ||
| timeout_override: | ||
| continue_if_exception: True | ||
| disable_action: False | ||
| filters: | ||
| - filtertype: pattern | ||
| kind: prefix | ||
| value: sessions2- | ||
| exclude: | ||
| - filtertype: space | ||
| disk_space: ${CURATOR_DELETE_GIGS:1000000} | ||
| use_age: True | ||
| source: name | ||
| timestring: '%y%m%d' | ||
| exclude: | ||
| 2: | ||
| action: delete_indices | ||
| description: >- | ||
| Delete indices older than ${CURATOR_DELETE_COUNT:99} ${CURATOR_DELETE_UNITS:years} | ||
| (based on index name), for sessions2-prefixed indices. Ignore the error if the | ||
| filter does not result in an actionable list of indices (ignore_empty_list) and | ||
| exit cleanly. | ||
| options: | ||
| ignore_empty_list: True | ||
| timeout_override: | ||
| continue_if_exception: True | ||
| disable_action: False | ||
| filters: | ||
| - filtertype: pattern | ||
| kind: prefix | ||
| value: sessions2- | ||
| exclude: | ||
| - filtertype: age | ||
| source: name | ||
| direction: older | ||
| timestring: '%y%m%d' | ||
| unit: ${CURATOR_DELETE_UNITS:years} | ||
| unit_count: ${CURATOR_DELETE_COUNT:99} | ||
| exclude: | ||
| 3: | ||
| action: close | ||
| description: >- | ||
| Close indices older than ${CURATOR_CLOSE_COUNT:10} ${CURATOR_CLOSE_UNITS:years} | ||
| (based on index name), for sessions2-prefixed indices, to conserve memory. | ||
| Ignore the error if the filter does not result in an actionable list of indices | ||
| (ignore_empty_list) and exit cleanly. | ||
| options: | ||
| ignore_empty_list: True | ||
| timeout_override: | ||
| continue_if_exception: True | ||
| disable_action: False | ||
| filters: | ||
| - filtertype: pattern | ||
| kind: prefix | ||
| value: sessions2- | ||
| exclude: | ||
| - filtertype: age | ||
| source: name | ||
| direction: older | ||
| timestring: '%y%m%d' | ||
| unit: ${CURATOR_CLOSE_UNITS:years} | ||
| unit_count: ${CURATOR_CLOSE_COUNT:10} | ||
| exclude: |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,18 @@ | ||
| client: | ||
| hosts: | ||
| - ${ES_HOST:elasticsearch} | ||
| port: ${ES_PORT:9200} | ||
| url_prefix: | ||
| use_ssl: False | ||
| certificate: | ||
| client_cert: | ||
| client_key: | ||
| ssl_no_validate: True | ||
| http_auth: | ||
| timeout: ${CURATOR_TIMEOUT:120} | ||
| master_only: ${CURATOR_MASTER_ONLY:False} | ||
| logging: | ||
| loglevel: ${CURATOR_LOGLEVEL:INFO} | ||
| logfile: | ||
| logformat: ${CURATOR_LOGFORMAT:default} | ||
| #blacklist: ['elasticsearch', 'urllib3'] |
Oops, something went wrong.