Migrate Tool: Update to work with boltdb-shipper, improve logging out…

…put. (#6237)

* update migrate tool

Signed-off-by: Edward Welch <[email protected]>

* improve logging
don't use any caches

Signed-off-by: Edward Welch <[email protected]>

* comment out some more log lines

Signed-off-by: Edward Welch <[email protected]>

* change stat types

Signed-off-by: Edward Welch <[email protected]>

* update docs and comments as well as set some lower times on a few boltdb-shipper timers

Signed-off-by: Edward Welch <[email protected]>

* remove changes to the Makefile which weren't meant to make it to the PR

Signed-off-by: Edward Welch <[email protected]>

* remove some commented code

Signed-off-by: Edward Welch <[email protected]>

* fix tests

Signed-off-by: Edward Welch <[email protected]>

* lint

Signed-off-by: Edward Welch <[email protected]>

cmd/migrate/README.md

@@ -8,8 +8,6 @@ Two stores are created, a source store and destination (abbreviated dest) store.
 
 Chunks are queried from the source store and written to the dest store, new index entries are created in the dest store as well.
 
-This _should_ handle schema changes and different schemas on both the source and dest store.
-
 You should be able to:
 
 * Migrate between clusters
@@ -22,7 +20,7 @@ This is simple and because it uses the storage interfaces, should be complete an
 
 There is however some parallelism built in and there are a few flags to tune this, `migrate -help` for more info
 
-This does not remove any source data, it only reads existing source data and writes to the destination.
+This does not remove or modify any source data, it only reads existing source data and writes to the destination.
 
 ## Usage
 
@@ -54,3 +52,56 @@ Migrate tenant ID within a cluster
 ```
 migrate -source.config.file=/etc/loki-us-west1/config/config.yaml -dest.config.file=/etc/loki-us-west1/config/config.yaml -source.tenant=fake -dest.tenant=1 -from=2020-06-16T14:00:00-00:00 -to=2020-07-01T00:00:00-00:00
 ```
+
+### Stopping and restarting
+
+It's ok to process the same data multiple times, chunks are uniquely addressable, they will just replace each other.
+
+For boltdb-shipper you will end up with multiple index files which contain duplicate entries,
+Loki will handle this without issue and the compactor will reduce the number of files if there are more than 3
+(TODO we should make a compactor mode which forces cleanup to a single file)
+
+You can use the output of the processed sync ranges to help in restarting from a point of already processed data,
+however be aware that because of parallel processing, you need to find the last finished time for *ALL* the threads
+to determine where processing finished, because of the parallel dispatching of sync ranges the order of messages
+will not always be sorted chronologically.
+
+Also be aware of special considerations for a boltdb-shipper destination outlined below.
+
+### batchLen, shardBy, and parallel flags
+
+The defaults here are probably ok for normal sized computers.
+
+If sending data from something like a Raspberry Pi, you probably want to run something like `-batchLen=10 -parallel=4` or risk running out of memory.
+
+The transfer works by breaking up the time range into `shardBy` windows, a window is called a sync range, 
+each sync range is then dispatched to one of up to `parallel` worker threads.
+
+For each sync range, the source index is queried for all the chunks in the sync range, then the list of chunks is processed `batchLen` at a time from the source, 
+re-encoded if necessary (such as changing tenant ID), and send to the destination store. You need enough memory to handle having `batchlen` chunks in memory
+times the number of `parallel` threads.
+
+If you have a really huge amount of chunks, many tens or hundreds of thousands per day, you might want to decrease `shardBy` to a smaller window. 
+If you don't have many chunks and `shardBy` is too small you will process the same chunks from multiple sync ranges.
+
+`parallel` can likely be increased up to a point until you saturate your CPU or exhaust memory.
+
+If you have a lot of really tiny chunks it may make sense to increase `batchLen`, but I'm not sure how much changing this affects the performance. 
+
+There is not an exact science to tuning these params yet, 
+the output window gives information on throughput and you can play around with values to maximize throughput for your data.
+
+
+### boltdb-shipper
+
+When the destination index type is boltdb shipper, it's important to make sure index files are uploaded. 
+This happens automatically in the background with some timers built into the boltdb-shipper code.
+It also happens explicitly when all the sync ranges have been processed and the store is shutdown.
+
+However, if the process crashes while processing, there may be index files which were not uploaded.
+
+If restarting after a crash, it's best to overlap the start time with previously processed sync ranges. 
+Exactly how much to overlap is hard to say, you could look for the most recently uploaded index file in 
+the destination store which is the number of days since the unix epoch, and convert it to seconds to see what day it is.
+
+e.g. index_18262: 18262 * (60 * 60 * 24) = 1577836800 which is `Wednesday, January 1, 2020 12:00:00 AM`