Skip to content

Commit

Permalink
[8.8] [Fleet] Rollover data streams when package w/ TSDB setting chan…
Browse files Browse the repository at this point in the history 
…ged is installed (#157869) (#157916)

# Backport

This will backport the following commits from `main` to `8.8`:
- [[Fleet] Rollover data streams when package w/ TSDB setting changed is
installed (#157869)](#157869)

<!--- Backport version: 8.9.7 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Kyle
Pollich","email":"[email protected]"},"sourceCommit":{"committedDate":"2023-05-16T18:16:14Z","message":"[Fleet]
Rollover data streams when package w/ TSDB setting changed is installed
(#157869)\n\n## Summary\r\n\r\nFixes
https://github.com/elastic/kibana/issues/157345\r\n\r\nWhen a package
with a changed `index.mode` or `source.mode` setting is\r\ninstalled,
Fleet will now automatically perform a rollover to ensure the\r\ncorrect
setting is present on the resulting backing index.\r\n\r\nThere is an
issue with Elasticsearch wherein toggling these settings\r\nback and
forth will incur a backing index range overlap error.
See\r\nhttps://github.com/elastic/elasticsearch/issues/96163.\r\n\r\nTo
test\r\n1. Install the `system` integration at version `1.28.0`\r\n2.
Create an integration policy for the `system` integration (a
standard\r\ndefault agent policy will do)\r\n3. Enroll an agent in this
policy, and allow it to ingest some data\r\n4. Confirm that there are
documents present in the\r\n`metrics-system.cpu-default` data stream,
and note its backing index via\r\nStack Management\r\n5. Create a new
`1.28.1` version of the `system` integration
where\r\n`elasticsearch.index_mode: time_series` is set and install it
via\r\n`elastic-package install --zip`\r\n6. Confirm that a rollover
occurs and the backing index for the\r\n`metrics-system.cpu-default`
data stream has been updated\r\n\r\n### Checklist\r\n\r\nDelete any
items that are not applicable to this PR.\r\n\r\n- [x] [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common
scenarios\r\n\r\n---------\r\n\r\nCo-authored-by: Kibana Machine
<[email protected]>","sha":"22e38472f6f05f9e72d97e74ff8328565da4d53b","branchLabelMapping":{"^v8.9.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","Team:Fleet","backport:prev-minor","v8.9.0"],"number":157869,"url":"https://github.com/elastic/kibana/pull/157869","mergeCommit":{"message":"[Fleet]
Rollover data streams when package w/ TSDB setting changed is installed
(#157869)\n\n## Summary\r\n\r\nFixes
https://github.com/elastic/kibana/issues/157345\r\n\r\nWhen a package
with a changed `index.mode` or `source.mode` setting is\r\ninstalled,
Fleet will now automatically perform a rollover to ensure the\r\ncorrect
setting is present on the resulting backing index.\r\n\r\nThere is an
issue with Elasticsearch wherein toggling these settings\r\nback and
forth will incur a backing index range overlap error.
See\r\nhttps://github.com/elastic/elasticsearch/issues/96163.\r\n\r\nTo
test\r\n1. Install the `system` integration at version `1.28.0`\r\n2.
Create an integration policy for the `system` integration (a
standard\r\ndefault agent policy will do)\r\n3. Enroll an agent in this
policy, and allow it to ingest some data\r\n4. Confirm that there are
documents present in the\r\n`metrics-system.cpu-default` data stream,
and note its backing index via\r\nStack Management\r\n5. Create a new
`1.28.1` version of the `system` integration
where\r\n`elasticsearch.index_mode: time_series` is set and install it
via\r\n`elastic-package install --zip`\r\n6. Confirm that a rollover
occurs and the backing index for the\r\n`metrics-system.cpu-default`
data stream has been updated\r\n\r\n### Checklist\r\n\r\nDelete any
items that are not applicable to this PR.\r\n\r\n- [x] [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common
scenarios\r\n\r\n---------\r\n\r\nCo-authored-by: Kibana Machine
<[email protected]>","sha":"22e38472f6f05f9e72d97e74ff8328565da4d53b"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v8.9.0","labelRegex":"^v8.9.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/157869","number":157869,"mergeCommit":{"message":"[Fleet]
Rollover data streams when package w/ TSDB setting changed is installed
(#157869)\n\n## Summary\r\n\r\nFixes
https://github.com/elastic/kibana/issues/157345\r\n\r\nWhen a package
with a changed `index.mode` or `source.mode` setting is\r\ninstalled,
Fleet will now automatically perform a rollover to ensure the\r\ncorrect
setting is present on the resulting backing index.\r\n\r\nThere is an
issue with Elasticsearch wherein toggling these settings\r\nback and
forth will incur a backing index range overlap error.
See\r\nhttps://github.com/elastic/elasticsearch/issues/96163.\r\n\r\nTo
test\r\n1. Install the `system` integration at version `1.28.0`\r\n2.
Create an integration policy for the `system` integration (a
standard\r\ndefault agent policy will do)\r\n3. Enroll an agent in this
policy, and allow it to ingest some data\r\n4. Confirm that there are
documents present in the\r\n`metrics-system.cpu-default` data stream,
and note its backing index via\r\nStack Management\r\n5. Create a new
`1.28.1` version of the `system` integration
where\r\n`elasticsearch.index_mode: time_series` is set and install it
via\r\n`elastic-package install --zip`\r\n6. Confirm that a rollover
occurs and the backing index for the\r\n`metrics-system.cpu-default`
data stream has been updated\r\n\r\n### Checklist\r\n\r\nDelete any
items that are not applicable to this PR.\r\n\r\n- [x] [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common
scenarios\r\n\r\n---------\r\n\r\nCo-authored-by: Kibana Machine
<[email protected]>","sha":"22e38472f6f05f9e72d97e74ff8328565da4d53b"}}]}]
BACKPORT-->

Co-authored-by: Kyle Pollich <[email protected]>
  • Loading branch information
@kibanamachine @kpollich
kibanamachine and kpollich authored May 16, 2023
1 parent 0c16782 commit 592c180
Show file tree
Hide file tree
Showing 10 changed files with 413 additions and 205 deletions.
49 changes: 45 additions & 4 deletions x-pack/plugins/fleet/server/services/epm/elasticsearch/template/template.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,10 @@
*/

import type { ElasticsearchClient, Logger } from '@kbn/core/server';
import type { IndicesIndexSettings } from '@elastic/elasticsearch/lib/api/typesWithBodyKey';
import type {
IndicesIndexSettings,
MappingTypeMapping,
} from '@elastic/elasticsearch/lib/api/typesWithBodyKey';

import type { Field, Fields } from '../../fields/field';
import type {
Expand Down Expand Up @@ -642,7 +645,20 @@ const updateExistingDataStream = async ({
esClient: ElasticsearchClient;
logger: Logger;
}) => {
const existingDs = await esClient.indices.get({
index: dataStreamName,
});

const existingDsConfig = Object.values(existingDs);
const currentBackingIndexConfig = existingDsConfig.at(-1);

const currentIndexMode = currentBackingIndexConfig?.settings?.index?.mode;
// @ts-expect-error Property 'mode' does not exist on type 'MappingSourceField'
const currentSourceType = currentBackingIndexConfig.mappings?._source?.mode;

let settings: IndicesIndexSettings;
let mappings: MappingTypeMapping;

try {
const simulateResult = await retryTransientEsErrors(() =>
esClient.indices.simulateTemplate({
Expand All @@ -651,14 +667,17 @@ const updateExistingDataStream = async ({
);

settings = simulateResult.template.settings;
const mappings = simulateResult.template.mappings;
mappings = simulateResult.template.mappings;

// for now, remove from object so as not to update stream or data stream properties of the index until type and name
// are added in https://github.com/elastic/kibana/issues/66551. namespace value we will continue
// to skip updating and assume the value in the index mapping is correct
if (mappings && mappings.properties) {
delete mappings.properties.stream;
delete mappings.properties.data_stream;
}

logger.debug(`Updating mappings for ${dataStreamName}`);
await retryTransientEsErrors(
() =>
esClient.indices.putMapping({
Expand All @@ -668,16 +687,38 @@ const updateExistingDataStream = async ({
}),
{ logger }
);
// if update fails, rollover data stream

// if update fails, rollover data stream and bail out
} catch (err) {
logger.error(`Mappings update for ${dataStreamName} failed`);
logger.error(err);

await rolloverDataStream(dataStreamName, esClient);
return;
}

// Trigger a rollover if the index mode or source type has changed
if (
currentIndexMode !== settings?.index?.mode ||
// @ts-expect-error Property 'mode' does not exist on type 'MappingSourceField'
currentSourceType !== mappings?._source?.mode
) {
logger.info(
`Index mode or source type has changed for ${dataStreamName}, triggering a rollover`
);
await rolloverDataStream(dataStreamName, esClient);
}

// update settings after mappings was successful to ensure
// pointing to the new pipeline is safe
// for now, only update the pipeline
if (!settings?.index?.default_pipeline) return;
if (!settings?.index?.default_pipeline) {
return;
}

try {
logger.debug(`Updating settings for ${dataStreamName}`);

await retryTransientEsErrors(
() =>
esClient.indices.putSettings({
Expand Down
Loading

0 comments on commit 592c180

Please sign in to comment.