save my work

add notes save work save work fix unit test remove tidbMgr in RestoreController remove some comments remove some comments change logger in SQLWithRetry revert replace log.Logger to *zap.Logger dep: update uuid dependency to latest google/uuid (pingcap#452) * dep: update satori/go.uuid to latest * fix tests * change to google/uuid * fix build * try fix test * get familiar with google/uuid * address comment tidb-lightning-ctl: change default of -d to 'noop://' (pingcap#453) also add noop:// to supported storage types (to represent an empty store) replace tab to space try another port to fix CI remove some comment *: more glue restore: fix the bug that gc life time ttl does not take effect (pingcap#448) * fix gc ttl loop * resolve comment and add tests fix CI report info to host TiDB config: filter out all system schemas by default (pingcap#459) backend: fix auto random default value for primary key (pingcap#457) * fix auto generate auto random primary key column * fix default for auto random primary key * fix test * use prev row id for auto random and add a test * replace chunck with session opt * fix * fix mydumper: fix parquet data parser (pingcap#435) * fix parquet * reorder imports * fix test * use empty collation * fix a error and add more test cases * add pointer type tests * resolve comments Co-authored-by: kennytm <[email protected]> address comment backend/local: use range properties to optimize region range estimate (pingcap#422) * use range propreties to estimate region range * post-restore: add optional level for post-restore operations (pingcap#421) * add optional level for opst-restore operations * trim leading and suffix '" * use UnmarshalTOML to unmarshal post restore op level * resolve comments and fix unit test * backend/local: do not retry epochNotMatch error when ingest sst (pingcap#419) * do not retry epochNotMatch error when ingest sst * add retry ingest for 'Raft raft: proposal dropped' error in ingest * change some retryable error log level from Error to Warn * fix nextKey * add a comment for nextKey * fix comment and add a unit test * wrap time.Sleep in select Co-authored-by: kennytm <[email protected]> * update * use range properties to optimze region range estimate * update pebble * change the default value for batch-size * add unit tests and reslove comments * add a comment to range properties test * add a comment * add a test for range property with pebble * rename const variable Co-authored-by: kennytm <[email protected]> fix pd service id is empty (pingcap#460) fix s3 parquet reader (pingcap#461) Co-authored-by: Neil Shen <[email protected]> fix service gc ttl again (pingcap#465) address comment mydumper: verify file routing config (pingcap#470) * fix file routing * remove useless line * remove redundant if check rename a method in interface save work try fix CI could work change ctx usage try fix CI try fix CI refine function interface refine some fucntion interface debug CI address comment config: allow four byte-size config to be specified using human-readable units ("100 GiB") (pingcap#471) * Makefile: add `make finish-prepare` action * config: accept human-readable size for most byte-related config e.g. allow `region-split-size = '96M'` in additional to `= 100663296` (known issue: these values' precisions will be truncated to 53 bits instead of supporting all 63 bits) * restore: reduce chance of spurious errors from TestGcTTLManagerSingle Co-authored-by: glorv <[email protected]> remove debug log test: change double type syntax (pingcap#474) address comment checkpoint: add glue checkpoint resolve cycle import expose Retry refine change interface to cope with TiDB fix SQL string fix SQL adjust interface to embedded in TiDB could import now reduce TLS restore: add `glue.Glue` interface and other function (pingcap#456) * save my work * add notes * save work * save work * fix unit test * remove tidbMgr in RestoreController * remove some comments * remove some comments * change logger in SQLWithRetry * revert replace log.Logger to *zap.Logger * replace tab to space * try another port to fix CI * remove some comment * *: more glue * report info to host TiDB * fix CI * address comment * address comment * rename a method in interface * save work * try fix CI * could work * change ctx usage * try fix CI * try fix CI * refine function interface * refine some fucntion interface * debug CI * address comment * remove debug log * address comment modify code add comment refine some code
lance6716 · Nov 17, 2020 · b1c3a97 · b1c3a97
1 parent eb609ec
commit b1c3a97
Show file tree

Hide file tree

Showing 57 changed files with 2,726 additions and 630 deletions.
diff --git a/Makefile b/Makefile
@@ -48,6 +48,9 @@ default: clean lightning lightning-ctl checksuccess
 prepare:
 	$(PREPARE_MOD)
 
+finish-prepare:
+	$(FINISH_MOD)
+
 clean:
 	rm -f $(LIGHTNING_BIN) $(LIGHTNING_CTRL_BIN) $(FAILPOINT_CTL_BIN) $(REVIVE_BIN) $(VFSGENDEV_BIN) go.mod go.sum
 
@@ -132,6 +135,11 @@ update:
 	GO111MODULE=on go mod tidy
 	$(FINISH_MOD)
 
+manual_update:
+	GO111MODULE=on go mod verify
+	GO111MODULE=on go mod tidy
+	$(FINISH_MOD)
+
 $(FAILPOINT_CTL_BIN):
 	cd tools && $(GOBUILD) -o ../$(FAILPOINT_CTL_BIN) github.com/pingcap/failpoint/failpoint-ctl
 

diff --git a/cmd/tidb-lightning-ctl/main.go b/cmd/tidb-lightning-ctl/main.go
@@ -22,11 +22,13 @@ import (
 	"strconv"
 	"strings"
 
+	"github.com/google/uuid"
 	"github.com/pingcap/errors"
 	"github.com/pingcap/kvproto/pkg/import_sstpb"
-	uuid "github.com/satori/go.uuid"
+	"github.com/pingcap/tidb-lightning/lightning/checkpoints"
 
 	kv "github.com/pingcap/tidb-lightning/lightning/backend"
+	"github.com/pingcap/tidb-lightning/lightning/checkpoints"
 	"github.com/pingcap/tidb-lightning/lightning/common"
 	"github.com/pingcap/tidb-lightning/lightning/config"
 	"github.com/pingcap/tidb-lightning/lightning/restore"
@@ -49,6 +51,13 @@ func run() error {
 	)
 
 	globalCfg := config.Must(config.LoadGlobalConfig(os.Args[1:], func(fs *flag.FlagSet) {
+		// change the default of `-d` from empty to 'noop://'.
+		// there is a check if `-d` points to a valid storage, and '' is not.
+		// since tidb-lightning-ctl does not need `-d` we change the default to a valid but harmless value.
+		dFlag := fs.Lookup("d")
+		dFlag.Value.Set("noop://")
+		dFlag.DefValue = "noop://"
+
 		compact = fs.Bool("compact", false, "do manual compaction on the target cluster")
 		mode = fs.String("switch-mode", "", "switch tikv into import mode or normal mode, values can be ['import', 'normal']")
 		flagFetchMode = fs.Bool("fetch-mode", false, "obtain the current mode of every tikv in the cluster")
@@ -165,7 +174,7 @@ func fetchMode(ctx context.Context, cfg *config.Config, tls *common.TLS) error {
 }
 
 func checkpointRemove(ctx context.Context, cfg *config.Config, tableName string) error {
-	cpdb, err := restore.OpenCheckpointsDB(ctx, cfg)
+	cpdb, err := checkpoints.OpenCheckpointsDB(ctx, cfg)
 	if err != nil {
 		return errors.Trace(err)
 	}
@@ -175,7 +184,7 @@ func checkpointRemove(ctx context.Context, cfg *config.Config, tableName string)
 }
 
 func checkpointErrorIgnore(ctx context.Context, cfg *config.Config, tableName string) error {
-	cpdb, err := restore.OpenCheckpointsDB(ctx, cfg)
+	cpdb, err := checkpoints.OpenCheckpointsDB(ctx, cfg)
 	if err != nil {
 		return errors.Trace(err)
 	}
@@ -185,7 +194,7 @@ func checkpointErrorIgnore(ctx context.Context, cfg *config.Config, tableName st
 }
 
 func checkpointErrorDestroy(ctx context.Context, cfg *config.Config, tls *common.TLS, tableName string) error {
-	cpdb, err := restore.OpenCheckpointsDB(ctx, cfg)
+	cpdb, err := checkpoints.OpenCheckpointsDB(ctx, cfg)
 	if err != nil {
 		return errors.Trace(err)
 	}
@@ -257,7 +266,7 @@ func checkpointErrorDestroy(ctx context.Context, cfg *config.Config, tls *common
 }
 
 func checkpointDump(ctx context.Context, cfg *config.Config, dumpFolder string) error {
-	cpdb, err := restore.OpenCheckpointsDB(ctx, cfg)
+	cpdb, err := checkpoints.OpenCheckpointsDB(ctx, cfg)
 	if err != nil {
 		return errors.Trace(err)
 	}
@@ -311,7 +320,7 @@ func unsafeCloseEngine(ctx context.Context, importer kv.Backend, engine string)
 		return ce, errors.Trace(err)
 	}
 
-	engineUUID, err := uuid.FromString(engine)
+	engineUUID, err := uuid.Parse(engine)
 	if err != nil {
 		return nil, errors.Trace(err)
 	}

diff --git a/cmd/tidb-lightning/main.go b/cmd/tidb-lightning/main.go
@@ -14,6 +14,7 @@
 package main
 
 import (
+	"context"
 	"fmt"
 	"os"
 	"os/signal"
@@ -28,10 +29,10 @@ import (
 )
 
 func main() {
-	cfg := config.Must(config.LoadGlobalConfig(os.Args[1:], nil))
-	fmt.Fprintf(os.Stdout, "Verbose debug logs will be written to %s\n\n", cfg.App.Config.File)
+	globalCfg := config.Must(config.LoadGlobalConfig(os.Args[1:], nil))
+	fmt.Fprintf(os.Stdout, "Verbose debug logs will be written to %s\n\n", globalCfg.App.Config.File)
 
-	app := lightning.New(cfg)
+	app := lightning.New(globalCfg)
 
 	sc := make(chan os.Signal, 1)
 	signal.Notify(sc,
@@ -59,7 +60,7 @@ func main() {
 	//
 	// Local mode need much more memory than importer/tidb mode, if the gc percentage is too high,
 	// lightning memory usage will also be high.
-	if cfg.TikvImporter.Backend != config.BackendLocal {
+	if globalCfg.TikvImporter.Backend != config.BackendLocal {
 		gogc := os.Getenv("GOGC")
 		if gogc == "" {
 			old := debug.SetGCPercent(500)
@@ -74,11 +75,18 @@ func main() {
 		return
 	}
 
-	if cfg.App.ServerMode {
-		err = app.RunServer()
-	} else {
-		err = app.RunOnce()
-	}
+	err = func() error {
+		if globalCfg.App.ServerMode {
+			return app.RunServer()
+		} else {
+			cfg := config.NewConfig()
+			if err := cfg.LoadFromGlobal(globalCfg); err != nil {
+				return err
+			}
+			return app.RunOnce(context.Background(), cfg, nil, nil)
+		}
+	}()
+
 	if err != nil {
 		logger.Error("tidb lightning encountered error stack info", zap.Error(err))
 		logger.Error("tidb lightning encountered error", log.ShortError(err))
@@ -89,7 +97,7 @@ func main() {
 	}
 
 	// call Sync() with log to stdout may return error in some case, so just skip it
-	if cfg.App.File != "" {
+	if globalCfg.App.File != "" {
 		syncErr := logger.Sync()
 		if syncErr != nil {
 			fmt.Fprintln(os.Stderr, "sync log failed", syncErr)

diff --git a/go.mod1 b/go.mod1
@@ -7,14 +7,17 @@ require (
 	github.com/DATA-DOG/go-sqlmock v1.4.1
 	github.com/aws/aws-sdk-go v1.35.2 // indirect
 	github.com/carlmjohnson/flagext v0.0.11
-	github.com/cockroachdb/pebble v0.0.0-20200617141519-3b241b76ed3b
+	github.com/cockroachdb/pebble v0.0.0-20201023120638-f1224da22976
 	github.com/coreos/go-semver v0.3.0
 	github.com/dgraph-io/ristretto v0.0.2-0.20200115201040-8f368f2f2ab3 // indirect
+	github.com/docker/go-units v0.4.0
 	github.com/fsouza/fake-gcs-server v1.19.0 // indirect
 	github.com/go-sql-driver/mysql v1.5.0
 	github.com/gogo/protobuf v1.3.1
 	github.com/golang/mock v1.4.4
+	github.com/google/btree v1.0.0
 	github.com/google/go-cmp v0.5.0 // indirect
+	github.com/google/uuid v1.1.1
 	github.com/joho/sqltocsv v0.0.0-20190824231449-5650f27fd5b6
 	github.com/juju/loggo v0.0.0-20180524022052-584905176618 // indirect
 	github.com/onsi/ginkgo v1.13.0 // indirect
@@ -29,16 +32,15 @@ require (
 	github.com/pingcap/tidb-tools v4.0.5-0.20200820092506-34ea90c93237+incompatible
 	github.com/prometheus/client_golang v1.5.1
 	github.com/prometheus/client_model v0.2.0
-	github.com/satori/go.uuid v1.2.0
 	github.com/shurcooL/httpgzip v0.0.0-20190720172056-320755c1c1b0
 	github.com/spaolacci/murmur3 v1.1.0 // indirect
 	github.com/tikv/pd v1.1.0-beta.0.20200910042021-254d1345be09
-	github.com/xitongsys/parquet-go v1.5.4-0.20201010004835-f51647f24120
+	github.com/xitongsys/parquet-go v1.5.5-0.20201110004701-b09c49d6d457
 	github.com/xitongsys/parquet-go-source v0.0.0-20200817004010-026bad9b25d0
 	go.etcd.io/etcd v0.5.0-alpha.5.0.20200824191128-ae9734ed278b // indirect
 	go.uber.org/zap v1.16.0
 	golang.org/x/net v0.0.0-20200904194848-62affa334b73
-	golang.org/x/sync v0.0.0-20200625203802-6e8e738ad208
+	golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9
 	golang.org/x/sys v0.0.0-20200905004654-be1d3432aa8f // indirect
 	golang.org/x/text v0.3.3
 	golang.org/x/tools v0.0.0-20200904185747-39188db58858 // indirect