Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pd leader balance radio 100% and region balance ratio 100% #1378

Closed
laizhebujum opened this issue Dec 13, 2018 · 6 comments
Closed

pd leader balance radio 100% and region balance ratio 100% #1378

laizhebujum opened this issue Dec 13, 2018 · 6 comments
Labels
type/bug The issue is confirmed as a bug.

Comments

@laizhebujum
Copy link

laizhebujum commented Dec 13, 2018

请大神答疑,为了更好的描述问题直接中文啦~

overview页和pd页都显示平衡比例100%(图1),但是查看tikv监控已经达到平衡(图2)

image

image
通过pd-ctl查看tikv store的leader和region信息也是正确的,已经达到平衡。(图3)

image

说一下最近做过的操作:
1、2018-12-10 tidb版本是2.0.8,本想做滚动升级2.1,却遇到点小插曲:
升级过程中报错FAILED! => {"changed": false, "elapsed": 300, "msg": "Timeout when waiting for search string 200 OK in xxxxxxxxx:2379"},以为是这个pd有问题,就缩容去掉了这个pd,后来发现是inventory.ini deploy_dir参数配置有误,然后又添加上这个pd了,(添加删除启动过程都是用的2.0.8版本的tidb-ansible),集群启动正常,版本是2.0.8

2、2018-12-11日 参照官方文档做了滚动升级至2.1,升级成功了,集群正常启动 下载tidb-ansible日期是2018-12-11,就出现了balance radio 100%的问题了。

@disksing disksing added the type/bug The issue is confirmed as a bug. label Dec 13, 2018
@disksing
Copy link
Contributor

Hi @laizhebujum, does your cluster have tombstone store(s)? It looks the balance ratio could be incorrect when there are tombstone store(s).

@laizhebujum
Copy link
Author

Hi @laizhebujum, does your cluster have tombstone store(s)? It looks the balance ratio could be incorrect when there are tombstone store(s).

Thank you for your reply.
Yes, it does exist. It already exists in version 2.0.8, but there was no problem.
How can we clean up the tombstone store(s)?
In addition, after upgrading to 2.1.0,There is such information in the log every minute:
2018/12/13 17:27:29.933 cluster.go:472: [info] buried store id:1 address:"X.X.X.X:20160" state:Tombstone

@disksing
Copy link
Contributor

Because the statistical adjustments made the previous calculations unsuitable, we are planning to replace the original ones with better calculation formulas. At the moment you may have to choose to ignore the balance ratios, sorry about that.
The buried store log message should be a bug (disturbing but not critical). Thanks for your report!
/cc @rleungx

@laizhebujum
Copy link
Author

Thanks

@rleungx
Copy link
Member

rleungx commented Dec 24, 2018

The disturbing log has been removed in #1382.

@rleungx
Copy link
Member

rleungx commented Sep 5, 2019

We remove this metrics in release 3.0 and provide an API to clean the tombstone store in release 2.1 via #1705. Closes this for now.

@rleungx rleungx closed this as completed Sep 5, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type/bug The issue is confirmed as a bug.
Projects
None yet
Development

No branches or pull requests

3 participants