Initialize migration tool #1

kvch · 2019-01-29T20:03:28Z

THIS IS WORK IN PROGRESS

This tool migrates .management-beats index the following way:

Backups the existing .management-beats index as .management-beats-backup. The backup is then aliased as .management-beats.
A new temporary index is created named .management-beats-new. The new index mapping of 6.7 is PUT to ES.
Reindexes documents from .management-beats-backup which does not require any transformation: beat, enrollment_token.
Queries all tags from .management-beats-backup. These are transformed into new tag documents. Also, configuration_block documents are created for each config element of a tag.
Finalizes the migration by deleting the alias .management-beats. .management-beats-new is reindexed, and the backup is deleted.

If something goes wrong along the process, everything is rolled back.

TODO

fix broken rollback functionality
add missing Aliases to elasticsearch.Client
add more documentation Add more detailed README #4
add missing unit testing vendored lib

Possible enhancements

index new documents in bulk

migration.go

ph · 2019-01-29T21:12:58Z

@kvch Can you vendor libbeat and make a PR with just that? That will make this PR a lot smaller.

central_management_migrate.go

kvch · 2019-01-30T12:31:58Z

@ph This commit includes all files which are not "vendored": 951ee23

urso · 2019-01-30T20:13:54Z

migration.go

+	_, _, err := client.Reindex(managementIndexName, backupManagementIndexName, nil)
+	if err != nil {
+		return fmt.Errorf("error while backup up old .management-beats index: %+v", err)
+	}


When reindexing we create a very new index. Are there any index settings to apply before/when we create the backup index? E.g. set number of shards to 1?

E.g.

load index mapping from existing .management-beats index

create target backup index with index mapping (and other settings?)

reindex into backup index

As it is a temporary index, I found it unnecessary to setup such things. Why do you think it is necessary?

migration.go

urso · 2019-01-30T21:02:24Z

migration.go

+	}
+	log.Println("Alias .management-beats is removed")
+
+	_, _, err = client.Reindex(newManagementIndexName, managementIndexName, nil)


I think managementIndexName does not exist anymore when we are here. We should create the index and establish the mapping before reindexing into the index to be created.

Indeed, I need to put the same mapping and settings as it is done for newManagmentIndexName.

urso · 2019-01-30T21:03:25Z

migration.go

+	return nil
+}
+
+func rollback(client *elasticsearch.Client) error {


what if rollback itself fails. Can we continue from the last ok step if the user runs the script again?

That's my plan, but the rollback functionality is a bit fragile right now. But the architecture you've proposed above will be able to achieve this. I could add a flag to set where to start the migration when running the script.

mattapperson · 2019-02-21T02:38:44Z

Functionally LGTM. Worth noting that when migrating a very large deployment it did hang for a while. But I doubt anyone will be doing that (10k beats, 6 tags per beat, 500 configs per tag)

kvch · 2019-02-21T15:41:38Z

@mattapperson I have done some refactoring to use the Bulk API of Elasticsearch to index new documents. Do you mind running another stress test to see if the script still hangs?

urso · 2019-02-22T12:52:12Z

step.go

+
+func (s steps) Undo(client *elasticsearch.Client) error {
+	i := len(s)
+	for i <= 0 {


for i:=len(s)-1; i >= 0; i-- { ... }

🤦‍♂️

urso · 2019-02-22T12:59:37Z

migration.go

+	for i := step; i < uint(len(migrationSteps)); i++ {
+		err = migrationSteps[i].Do(client)
+		if err != nil {
+			undoErr := stepsToUndo.Undo(client)


Where do you add a step to the undo list?

Nowhere, but fixed.

urso · 2019-02-22T14:30:39Z

step.go

+	if err != nil {
+		return fmt.Errorf("error while backup up old .management-beats index: %+v", err)
+	}
+	log.Println("Old .management-beats index is reindexed into .management-beats-backup")


Better use log.Printf with predefine index name variables.

step.go

kvch · 2019-02-25T17:30:04Z

I still need to do a bit more testing so I can remove the ugly "WIP" "sign" from the PR.

ph · 2019-02-27T19:09:32Z

migration.go

+	"log"
+	"time"
+
+	//"gopkg.in/yaml.v2"


not needed?

ph · 2019-02-27T19:14:29Z

@kvch code look good, have you tested it with CM 6.x to 6.7 with real data?

kvch · 2019-03-04T12:38:32Z

Yes, before the refactoring. Now I have tested it again, and added a few fixes. But I am facing a weird bug now, so I need a bit more time to find the problem.

kvch · 2019-03-08T09:15:31Z

@ph @mattapperson I have found the issue in the code. Now it is ready to be tested.

kvch · 2019-03-25T17:40:14Z

Merging to prepare first release.

kvch added in progress review labels Jan 29, 2019

kvch commented Jan 29, 2019

View reviewed changes

migration.go Outdated Show resolved Hide resolved

ph requested review from ph and mattapperson January 29, 2019 21:10

ph reviewed Jan 29, 2019

View reviewed changes

central_management_migrate.go Show resolved Hide resolved

kvch force-pushed the initialize-migration-tool branch from 952afc2 to 951ee23 Compare January 30, 2019 12:25

urso reviewed Jan 30, 2019

View reviewed changes

kvch force-pushed the initialize-migration-tool branch 2 times, most recently from 15043ac to ca0495b Compare February 7, 2019 12:11

kvch added 4 commits February 7, 2019 21:10

initialize migration script

c296b1b

add TLS to ES connection && rm flags

43735f2

add new index mapping to bin

b05a056

rm version checking

7560f2c

kvch force-pushed the initialize-migration-tool branch from 07a4df6 to 7560f2c Compare February 7, 2019 20:10

kvch added 3 commits February 8, 2019 10:26

vendor yaml.v2

095b6b5

implement rollback

c91116d

do not rely on Hits.Total.Value

030ab7b

use bulk api to speed up indexing new documents

12a9867

urso reviewed Feb 22, 2019

View reviewed changes

kvch added 4 commits February 25, 2019 13:15

refactoring of for loop

ee3422b

substitute index names in logging

0bcb22a

rm unused constant

33792fe

add missing done step

1587566

urso reviewed Feb 25, 2019

View reviewed changes

step.go Outdated Show resolved Hide resolved

kvch added 2 commits February 25, 2019 17:18

more fixes to rollback

e2f5421

more rollback fixes

0911757

kvch force-pushed the initialize-migration-tool branch from 8a751d0 to 0911757 Compare February 25, 2019 16:21

ph reviewed Feb 27, 2019

View reviewed changes

migration.go

"log"

"time"

//"gopkg.in/yaml.v2"

Copy link

ph Feb 27, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not needed?

add missing fixes

89788c4

create index before reindexing && refresh

92dd275

kvch force-pushed the initialize-migration-tool branch from 8a90d95 to 92dd275 Compare March 8, 2019 09:13

kvch removed the in progress label Mar 25, 2019

kvch merged commit 4e23a04 into master Mar 25, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Initialize migration tool #1

Initialize migration tool #1

kvch commented Jan 29, 2019 •

edited

Loading

ph commented Jan 29, 2019

kvch commented Jan 30, 2019 •

edited

Loading

urso Jan 30, 2019

kvch Feb 8, 2019

urso Jan 30, 2019

kvch Feb 8, 2019

urso Jan 30, 2019

kvch Feb 8, 2019

mattapperson commented Feb 21, 2019

kvch commented Feb 21, 2019

urso Feb 22, 2019

kvch Feb 22, 2019

urso Feb 22, 2019

kvch Feb 25, 2019

urso Feb 22, 2019

kvch commented Feb 25, 2019

ph Feb 27, 2019

ph commented Feb 27, 2019

kvch commented Mar 4, 2019

kvch commented Mar 8, 2019

kvch commented Mar 25, 2019

Initialize migration tool #1

Initialize migration tool #1

Conversation

kvch commented Jan 29, 2019 • edited Loading

THIS IS WORK IN PROGRESS

ph commented Jan 29, 2019

kvch commented Jan 30, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mattapperson commented Feb 21, 2019

kvch commented Feb 21, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kvch commented Feb 25, 2019

Choose a reason for hiding this comment

ph commented Feb 27, 2019

kvch commented Mar 4, 2019

kvch commented Mar 8, 2019

kvch commented Mar 25, 2019

kvch commented Jan 29, 2019 •

edited

Loading

kvch commented Jan 30, 2019 •

edited

Loading