Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

executor, session: refine insert unsigned bigint autoIncreID #8181

Merged
merged 24 commits into from
Dec 18, 2018
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
a7df24d
executor, session: refine insert unsigned bigint autoIncreID
XuHuaiyu Nov 5, 2018
9d21807
address comment
XuHuaiyu Nov 8, 2018
2db694e
update parser
XuHuaiyu Nov 12, 2018
1eb9325
Merge branch 'master' of https://github.com/pingcap/tidb into unsigne…
XuHuaiyu Nov 12, 2018
53fd0a6
address comment
XuHuaiyu Nov 12, 2018
758fd31
address comment
XuHuaiyu Nov 13, 2018
a68a08d
make golint happy
XuHuaiyu Nov 13, 2018
c1e1ff6
address commen
XuHuaiyu Nov 13, 2018
9104f74
Merge branch 'master' of https://github.com/pingcap/tidb into unsigne…
XuHuaiyu Nov 16, 2018
fc03231
Merge branch 'master' of https://github.com/pingcap/tidb into unsigne…
XuHuaiyu Dec 10, 2018
82b12d5
address comment
XuHuaiyu Dec 10, 2018
2e8b425
remove go.sum
XuHuaiyu Dec 10, 2018
ce0e447
merge master
XuHuaiyu Dec 10, 2018
874ff0b
address comment
XuHuaiyu Dec 13, 2018
d0616ac
Merge branch 'master' into unsigned_autoid
jackysp Dec 17, 2018
6792306
add unit tests for Allocator
XuHuaiyu Dec 18, 2018
46d20a4
Merge branch 'master' of https://github.com/pingcap/tidb into unsigne…
XuHuaiyu Dec 18, 2018
548ea41
Merge branch 'unsigned_autoid' of https://github.com/XuHuaiyu/tidb in…
XuHuaiyu Dec 18, 2018
61bfec6
update go.mod
XuHuaiyu Dec 18, 2018
6c4419c
address comment
XuHuaiyu Dec 18, 2018
3f1fcb4
make tidy
XuHuaiyu Dec 18, 2018
71ea318
go mod tidy
XuHuaiyu Dec 18, 2018
18c62fe
Merge branch 'master' into unsigned_autoid
XuHuaiyu Dec 18, 2018
d688346
Merge branch 'master' into unsigned_autoid
XuHuaiyu Dec 18, 2018
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ require (
github.com/pingcap/check v0.0.0-20171206051426-1c287c953996
github.com/pingcap/errors v0.11.0
github.com/pingcap/goleveldb v0.0.0-20171020122428-b9ff6c35079e
github.com/pingcap/kvproto v0.0.0-20181028030329-855d2192cdc7
github.com/pingcap/kvproto v0.0.0-20181105061835-1b5d69cd1d26
github.com/pingcap/parser v0.0.0-20181112055336-6cb6cf66c6df
github.com/pingcap/pd v2.1.0-rc.4+incompatible
github.com/pingcap/tidb-tools v0.0.0-20181101090416-cfac1096162e
Expand Down
269 changes: 0 additions & 269 deletions go.sum

Large diffs are not rendered by default.

37 changes: 28 additions & 9 deletions meta/autoid/autoid.go
Original file line number Diff line number Diff line change
Expand Up @@ -137,13 +137,12 @@ func (alloc *allocator) Rebase(tableID, requiredBase int64, allocIDs bool) error
if allocIDs {
if alloc.isUnsigned {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You want to prevent overflow?
Should we

if uNewBase > (math.Uint64 - step) {
   panic
}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes,
but why we panic here? @tiancaiamao

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, maybe error or warning...
I mean, if the ID overflow, it should not be changed to math.Uint64

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This check is used to prevent the alloc.end overflow.
alloc.base is checked here.

It may be reasonable to change alloc.end to math.Uint64 if it will overflow?
@tiancaiamao

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If alloc.end overflow, what will happen?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tiancaiamao
The result of TiDB master:

tidb> create table t(a bigint signed not null auto_increment, unique key(a));
Query OK, 0 rows affected (0.02 sec)

tidb> insert into t value(9223372036854775806);
Query OK, 1 row affected (0.00 sec)

tidb> insert into t value();
Query OK, 1 row affected (0.00 sec)

tidb> insert into t value();
Query OK, 1 row affected (0.00 sec)

tidb> insert into t value();
Query OK, 1 row affected (0.00 sec)

tidb> insert into t value();
Query OK, 1 row affected (0.00 sec)

tidb> select * from t;
+----------------------+
| a                    |
+----------------------+
| -9223372036854775808 |
| -9223372036854775806 |
| -9223372036854775804 |
| -9223372036854775802 |
|  9223372036854775806 |
+----------------------+
5 rows in set (0.00 sec)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overwrite the old row id is dangerous for data consistency.
Is it acceptable to change the behavior to throw error?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As a tradeoff, #8181 (review) may be better?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think throw error is simple, easier, and more robust.

uNewBase = mathutil.MaxUint64(uCurrentEnd, uRequiredBase)
uNewEnd = uNewBase + uint64(step)
uNewEnd = mathutil.MinUint64(math.MaxUint64-uint64(step), uNewBase) + uint64(step)
} else {
newBase = mathutil.MaxInt64(currentEnd, requiredBase)
newEnd = newBase + step
newEnd = mathutil.MinInt64(math.MaxInt64-step, newBase) + step
}
} else {

if !alloc.isUnsigned && currentEnd >= requiredBase {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

!alloc.isUnsigned will always be true here?

newBase = currentEnd
newEnd = currentEnd
Expand All @@ -164,7 +163,6 @@ func (alloc *allocator) Rebase(tableID, requiredBase int64, allocIDs bool) error
uNewBase = uRequiredBase
uNewEnd = uRequiredBase
}

}
if !alloc.isUnsigned {
_, err1 = m.GenAutoTableID(alloc.dbID, tableID, newEnd-currentEnd)
Expand All @@ -177,7 +175,11 @@ func (alloc *allocator) Rebase(tableID, requiredBase int64, allocIDs bool) error
if err != nil {
return errors.Trace(err)
}
alloc.base, alloc.end = newBase, newEnd
if !alloc.isUnsigned {
alloc.base, alloc.end = newBase, newEnd
} else {
alloc.base, alloc.end = int64(uNewBase), int64(uNewEnd)
}
return nil
}

Expand All @@ -198,21 +200,38 @@ func (alloc *allocator) Alloc(tableID int64) (int64, error) {
if err1 != nil {
return errors.Trace(err1)
}
newEnd, err1 = m.GenAutoTableID(alloc.dbID, tableID, step)
tmpStep := step
if alloc.isUnsigned {
tmpStep = int64(mathutil.MinUint64(math.MaxUint64-uint64(newBase), uint64(step)))
} else {
tmpStep = mathutil.MinInt64(math.MaxInt64-newBase, step)
}
newEnd, err1 = m.GenAutoTableID(alloc.dbID, tableID, tmpStep)
if err1 != nil {
return errors.Trace(err1)
}
return nil
})
metrics.AutoIDHistogram.WithLabelValues(metrics.TableAutoIDAlloc, metrics.RetLabel(err)).Observe(time.Since(startTime).Seconds())
if err != nil {
return 0, errors.Trace(err)
return 0, err
}
alloc.base, alloc.end = newBase, newEnd
}
log.Warning(uint64(alloc.base), uint64(alloc.end))
crazycs520 marked this conversation as resolved.
Show resolved Hide resolved
crazycs520 marked this conversation as resolved.
Show resolved Hide resolved

alloc.base++
log.Debugf("[kv] Alloc id %d, table ID:%d, from %p, database ID:%d", alloc.base, tableID, alloc, alloc.dbID)
if alloc.isUnsigned {
if uint64(alloc.base) == math.MaxUint64 {
Copy link
Contributor

@crazycs520 crazycs520 Nov 12, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

!alloc.isUnsigned need this kind of check( if alloc.base == math.MaxInt64 ) too.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we put this check to line204? Then we needn't do check each time Alloc is called.

return 0, ErrAutoincReadFailed
}
alloc.base = int64(uint64(alloc.base) + 1)
log.Debugf("[kv] Alloc id %d, table ID:%d, from %p, database ID:%d", uint64(alloc.base), tableID, alloc, alloc.dbID)
} else {
if alloc.base < alloc.end {
alloc.base++
}
log.Debugf("[kv] Alloc id %d, table ID:%d, from %p, database ID:%d", alloc.base, tableID, alloc, alloc.dbID)
}
return alloc.base, nil
}

Expand Down
32 changes: 32 additions & 0 deletions meta/autoid/errors.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
// Copyright 2018 PingCAP, Inc.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// See the License for the specific language governing permissions and
// limitations under the License.

package autoid

import (
"github.com/pingcap/parser/mysql"
"github.com/pingcap/parser/terror"
)

// Error instances.
var (
ErrAutoincReadFailed = terror.ClassAutoid.New(mysql.ErrAutoincReadFailed, mysql.MySQLErrName[mysql.ErrAutoincReadFailed])
)

func init() {
// Map error codes to mysql error codes.
tableMySQLErrCodes := map[terror.ErrCode]uint16{
mysql.ErrAutoincReadFailed: mysql.ErrAutoincReadFailed,
}
terror.ErrClassToMySQLCodes[terror.ClassAutoid] = tableMySQLErrCodes
}
2 changes: 1 addition & 1 deletion meta/meta.go
Original file line number Diff line number Diff line change
Expand Up @@ -154,7 +154,7 @@ func (m *Meta) parseTableID(key string) (int64, error) {
return n, errors.Trace(err)
}

// GenAutoTableIDIDKeyValue generate meta key by dbID, tableID and coresponding value by autoID.
// GenAutoTableIDIDKeyValue generate meta key by dbID, tableID and corresponding value by autoID.
func (m *Meta) GenAutoTableIDIDKeyValue(dbID, tableID, autoID int64) (key, value []byte) {
dbKey := m.dbKey(dbID)
autoTableIDKey := m.autoTableIDKey(tableID)
Expand Down
17 changes: 17 additions & 0 deletions session/session_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ import (
"github.com/pingcap/tidb/domain"
"github.com/pingcap/tidb/executor"
"github.com/pingcap/tidb/kv"
"github.com/pingcap/tidb/meta/autoid"
plannercore "github.com/pingcap/tidb/planner/core"
"github.com/pingcap/tidb/privilege/privileges"
"github.com/pingcap/tidb/session"
Expand Down Expand Up @@ -705,12 +706,28 @@ func (s *testSessionSuite) TestAutoIncrementID(c *C) {
tk.MustQuery("select last_insert_id(20)").Check(testkit.Rows(fmt.Sprint(20)))
tk.MustQuery("select last_insert_id()").Check(testkit.Rows(fmt.Sprint(20)))

// Corner cases for unsigned bigint auto_increment Columns.
tk.MustExec("drop table if exists autoid")
tk.MustExec("create table autoid(`auto_inc_id` bigint(20) UNSIGNED NOT NULL AUTO_INCREMENT,UNIQUE KEY `auto_inc_id` (`auto_inc_id`))")
tk.MustExec("insert into autoid values(9223372036854775808);")
tk.MustExec("insert into autoid values();")
tk.MustExec("insert into autoid values();")
tk.MustQuery("select * from autoid").Check(testkit.Rows("9223372036854775808", "9223372036854775810", "9223372036854775812"))
tk.MustExec("insert into autoid values(18446744073709551614);")
_, err := tk.Exec("insert into autoid values()")
c.Assert(terror.ErrorEqual(err, autoid.ErrAutoincReadFailed), IsTrue)
// FixMe: MySQL works fine with the this sql.
_, err = tk.Exec("insert into autoid values(18446744073709551615)")
c.Assert(terror.ErrorEqual(err, autoid.ErrAutoincReadFailed), IsTrue)

// Corner cases for signed bigint auto_increment Columns.
tk.MustExec("drop table if exists autoid")
tk.MustExec("create table autoid(`auto_inc_id` bigint(20) NOT NULL AUTO_INCREMENT,UNIQUE KEY `auto_inc_id` (`auto_inc_id`))")
tk.MustExec("insert into autoid values(9223372036854775806);")
tk.MustExec("insert into autoid values();")
Copy link
Contributor

@crazycs520 crazycs520 Nov 12, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the test has a problem:

mysql root@127.0.0.1:test> create table autoid(`auto_inc_id` bigint(20) NOT NULL AUTO_INCREMENT,UNIQUE KEY `auto_inc_id` (`auto_inc_id`));
Query OK, 0 rows affected
mysql root@127.0.0.1:test> insert into autoid values(9223372036854775806);
Query OK, 1 row affected
mysql root@127.0.0.1:test> select auto_inc_id,_tidb_rowid from autoid
+---------------------+---------------------+
| auto_inc_id         | _tidb_rowid         |
+---------------------+---------------------+
| 9223372036854775806 | 9223372036854775807 |
+---------------------+---------------------+
1 row in set
mysql root@127.0.0.1:test> insert into autoid values(); # auto_id = rowid= 9223372036854775807, cause below admin check error.
Query OK, 1 row affected
mysql root@127.0.0.1:test> select auto_inc_id,_tidb_rowid from autoid
+---------------------+---------------------+
| auto_inc_id         | _tidb_rowid         |
+---------------------+---------------------+
| 9223372036854775807 | 9223372036854775807 |
+---------------------+---------------------+
mysql root@127.0.0.1:test> admin check table autoid;
(8003, u'autoid err:[admin:1]index:&admin.RecordData{Handle:9223372036854775807, Values:[]types.Datum{types.Datum{k:0x1, collation:0x0, decimal:0x0, length:0x0, i:9223372036854775806, b:[]uint8(nil), x:interface {}(nil)}}} != record:&admin.RecordData{Handle:9223372036854775807, Values:[]types.Datum{types.Datum{k:0x1, collation:0x0, decimal:0x0, length:0x0, i:9223372036854775807, b:[]uint8(nil), x:interface {}(nil)}}}')
mysql root@127.0.0.1:test> select count(*) from autoid;
+----------+
| count(*) |
+----------+
| 1        |
+----------+
1 row in set
mysql root@127.0.0.1:test> select count(*) from autoid use index(auto_inc_id);
+----------+
| count(*) |
+----------+
| 2        |
+----------+
1 row in set

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please address this comment @XuHuaiyu.

tk.MustQuery("select * from autoid").Check(testkit.Rows("9223372036854775806", "9223372036854775807"))
_, err = tk.Exec("insert into autoid values();")
c.Assert(terror.ErrorEqual(err, kv.ErrKeyExists), IsTrue)
}

func (s *testSessionSuite) TestAutoIncrementWithRetry(c *C) {
Expand Down