Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feat: auto migration badger/badger数据自动迁移 #210

Merged
merged 4 commits into from
Oct 17, 2022

Conversation

zl03jsj
Copy link
Contributor

@zl03jsj zl03jsj commented Oct 14, 2022

关联的Issues (Related Issues)

closes filecoin-project/venus#5283

改动 (Proposed Changes)

  1. 增加badger数据自动迁移的架构实现.
  2. 自动badger实现自动从v2.2.0迁移到v2.3.0

附注 (Additional Info)

自查清单 (Checklist)

如果为master分支中commit为如下区间的升级595369d 2022-10-14 ... 9a21e00 2022-09-09

日志中包含迁移错误信息是正常现象, 因为这几个版本中的数据已经是新的格式了,但在这几个版本中还未加入自动迁移功能.所以会有这样的错误日志:

2022-10-17T13:12:24.613+0800	INFO	badger-migration	migrate/migrate.go:225	migrate: PayChanMsgDs from  to 1
2022-10-17T13:12:24.615+0800	INFO	badger-migration	migrate/migrate.go:225	migrate: StorageAskDs from  to 1
2022-10-17T13:12:24.616+0800	WARN	badger-migration	migrate/migrate.go:284	migrate:StorageAskDs failed:running up migration: decoding state for key '/t039853': cbor input should be of type map
2022-10-17T13:12:24.616+0800	INFO	badger-migration	migrate/migrate.go:225	migrate: RetrievalAskDs from  to 1
2022-10-17T13:12:24.616+0800	INFO	badger-migration	migrate/migrate.go:225	migrate: CidInfoDs from  to 1
2022-10-17T13:12:24.617+0800	INFO	badger-migration	migrate/migrate.go:225	migrate: RetrievalDealsDS from  to 1
2022-10-17T13:12:24.617+0800	INFO	badger-migration	migrate/migrate.go:225	migrate: FundedAddrStateDs from  to 1
2022-10-17T13:12:24.618+0800	WARN	badger-migration	migrate/migrate.go:284	migrate:FundedAddrStateDs failed:running up migration: decoding state for key '/Addr/t039853': cbor input had wrong number of fields
2022-10-17T13:12:24.618+0800	INFO	badger-migration	migrate/migrate.go:225	migrate: StorageDealDs from  to 1
2022-10-17T13:12:24.619+0800	WARN	badger-migration	migrate/migrate.go:284	migrate:StorageDealDs failed:running up migration: decoding state for key '/bafyreiapzj45cmmgfhltqv7chrvspbodmcdew2yjg37soywbcsgfy4dcve': cbor input had wrong number of fields; decoding state for key '/bafyreibwmrmn7njetylbog5d6ok4wwswzlbhzjrgcba5uwzfbcihjnrzqy': cbor input had wrong number of fields; decoding state for key '/bafyreif3y52x4nxjz72ra5agqdjr47ngplg6gt4wbdgmasp6tw5jtvl2le': cbor input had wrong number of fields
2022-10-17T13:12:24.620+0800	INFO	badger-migration	migrate/migrate.go:225	migrate: PayChanInfoDs from  to 1

如果为9a21e00 2022-09-09之前的,数据则为老版本, 能看到正常迁移的日志, 不会出现任何迁移失败的错误日志:

2022-10-17T13:30:29.090+0800	INFO	badger-migration	migrate/migrate.go:225	migrate: CidInfoDs from  to 1
2022-10-17T13:30:29.092+0800	INFO	badger-migration	migrate/migrate.go:225	migrate: RetrievalDealsDS from  to 1
2022-10-17T13:30:29.094+0800	INFO	badger-migration	migrate/migrate.go:225	migrate: FundedAddrStateDs from  to 1
2022-10-17T13:30:29.095+0800	INFO	badger-migration	migrate/migrate.go:225	migrate: StorageDealDs from  to 1
2022-10-17T13:30:29.096+0800	INFO	badger-migration	migrate/migrate.go:225	migrate: PayChanInfoDs from  to 1
2022-10-17T13:30:29.097+0800	INFO	badger-migration	migrate/migrate.go:225	migrate: PayChanMsgDs from  to 1
2022-10-17T13:30:29.098+0800	INFO	badger-migration	migrate/migrate.go:225	migrate: StorageAskDs from  to 1
2022-10-17T13:30:29.099+0800	INFO	badger-migration	migrate/migrate.go:225	migrate: RetrievalAskDs from  to 1

@zl03jsj zl03jsj self-assigned this Oct 14, 2022
@zl03jsj zl03jsj changed the title Feat: auto migratoin badger from v2.2.0 Feat: auto migration badger/badger数据自动迁移 Oct 14, 2022
@zl03jsj zl03jsj force-pushed the feat/auto-migratoin-badger-from-v2.2.0 branch from bf8a845 to f921dbc Compare October 14, 2022 10:17
@codecov-commenter
Copy link

codecov-commenter commented Oct 14, 2022

Codecov Report

Merging #210 (be3fc20) into master (595369d) will increase coverage by 1.25%.
The diff coverage is 42.78%.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #210      +/-   ##
==========================================
+ Coverage   24.06%   25.32%   +1.25%     
==========================================
  Files         178      181       +3     
  Lines       19337    20753    +1416     
==========================================
+ Hits         4654     5256     +602     
- Misses      14117    14676     +559     
- Partials      566      821     +255     
Flag Coverage Δ
unittests 25.32% <42.78%> (+1.25%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
models/module.go 0.00% <ø> (ø)
models/mysql/db.go 64.58% <0.00%> (+10.51%) ⬆️
models/badger/migrate/v2.2.0/types.go 31.48% <31.48%> (ø)
models/badger/utils.go 49.29% <66.66%> (+2.52%) ⬆️
models/badger/migrate/migrate.go 74.12% <74.12%> (ø)
models/badger/testing.go 80.00% <75.00%> (-6.37%) ⬇️
models/badger/db.go 59.82% <80.00%> (+9.25%) ⬆️
models/badger/storage_deal.go 70.11% <92.30%> (ø)
models/badger/migrate/v2.2.0/testing/testing.go 100.00% <100.00%> (ø)
models/badger/retrieval_ask.go 65.11% <100.00%> (ø)
... and 6 more

@zl03jsj zl03jsj force-pushed the feat/auto-migratoin-badger-from-v2.2.0 branch from 305f318 to be3fc20 Compare October 17, 2022 05:38
models/badger/migrate/migrate.go Outdated Show resolved Hide resolved
models/badger/migrate/migrate.go Outdated Show resolved Hide resolved
models/badger/migrate/migrate.go Show resolved Hide resolved
models/badger/migrate/migrate.go Outdated Show resolved Hide resolved
models/badger/db.go Show resolved Hide resolved
Copy link
Contributor

@hunjixin hunjixin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

1. remove empty lines
2. minor refacror on travelCborAbleDs function.
@zl03jsj zl03jsj force-pushed the feat/auto-migratoin-badger-from-v2.2.0 branch from d1e7826 to dc1b684 Compare October 17, 2022 07:11
@hunjixin hunjixin merged commit 036bca3 into master Oct 17, 2022
@hunjixin hunjixin deleted the feat/auto-migratoin-badger-from-v2.2.0 branch October 17, 2022 07:24
)

const (
DsNameFundedAddrState = "FundedAddrStateDs"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

下面6个命名是***Ds,开始两个没有


if !exist {
dss[name] = ds
log.Warnf("no migration sechedules for : %s", name)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sechedules 拼写错误

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

是写错了, 应该是'schedule'

}
var versionedDs datastore.Batching
versionedDs, err = migrateOne(ctx, name, mfs, ds)
// todo: version为空同时, 有同时存在两个版本的类型的可能性, 为了兼容, 这里暂时不返回错误.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

“version为空同时” 读不通顺

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[doge]原因是, 中间有几个提交, 类型已经升级了, 但是还没有引入自动迁移的功能.为了兼容, 所以这里暂时不返回错误, 等到下个版本之后, 这里就可以直接返回错误了.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

是说这句话不通顺,什么是“为空同时”

Copy link
Contributor Author

@zl03jsj zl03jsj Oct 17, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

是说这句话不通顺,什么是“为空同时”

为空时, hand slipped..

log.Infof("migrate: %s from %s to %s", name, oldVersion, string(targetVersion))
mfs = mfs.subScheduleFrom(oldVersion)
if len(mfs) == 0 {
return nil, fmt.Errorf("migrate:%s failed, can't find schedule from:%s", name, oldVersion)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

没有可执行的migrate要返回错误吗

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这里是开发者需要保证对应的迁移必须存在, 所以, 这里正常情况是肯定存在的, 如果不存在,说明是代码写错了.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[venus-market]: 对v2.2.0 badger已有数据不兼容
4 participants