Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

panic: cannot free page 0 or 1: 0 #3771

Open
hehailong5 opened this issue Dec 29, 2017 · 8 comments
Open

panic: cannot free page 0 or 1: 0 #3771

hehailong5 opened this issue Dec 29, 2017 · 8 comments
Labels
type/bug Feature does not function as expected type/crash The issue description contains a golang panic and stack trace

Comments

@hehailong5
Copy link

version: 0.8.4
scenario: disaster recovery with peers.json

==> Starting Consul agent...
panic: cannot free page 0 or 1: 0

goroutine 1 [running]:
github.com/hashicorp/consul/vendor/github.com/boltdb/bolt.(*freelist).free(0xc4203afe90, 0x455, 0x7f36c702a000)
/gopath/src/github.com/hashicorp/consul/vendor/github.com/boltdb/bolt/freelist.go:109 +0x3f0
github.com/hashicorp/consul/vendor/github.com/boltdb/bolt.(*Tx).Commit(0xc4201abdc0, 0x18df07c, 0x4)
/gopath/src/github.com/hashicorp/consul/vendor/github.com/boltdb/bolt/tx.go:176 +0x205
github.com/hashicorp/consul/vendor/github.com/hashicorp/raft-boltdb.(*BoltStore).initialize(0xc4203ad760, 0x0, 0x0)
/gopath/src/github.com/hashicorp/consul/vendor/github.com/hashicorp/raft-boltdb/bolt_store.go:75 +0x139
github.com/hashicorp/consul/vendor/github.com/hashicorp/raft-boltdb.NewBoltStore(0xc4203b2800, 0x33, 0x2, 0xc4203b2800, 0x33)
/gopath/src/github.com/hashicorp/consul/vendor/github.com/hashicorp/raft-boltdb/bolt_store.go:51 +0xc4
github.com/hashicorp/consul/consul.(*Server).setupRaft(0xc420358780, 0x0, 0x0)
/gopath/src/github.com/hashicorp/consul/consul/server.go:492 +0xabc
github.com/hashicorp/consul/consul.NewServerLogger(0xc420358500, 0xc42023ee60, 0x0, 0x0, 0x0)
/gopath/src/github.com/hashicorp/consul/consul/server.go:320 +0xc1f
github.com/hashicorp/consul/command/agent.(*Agent).makeServer(0xc4201a06c0, 0x1, 0xc420153c80, 0x0)
/gopath/src/github.com/hashicorp/consul/command/agent/agent.go:860 +0x112
github.com/hashicorp/consul/command/agent.(*Agent).Start(0xc4201a06c0, 0xc4201a06c0, 0x0)
/gopath/src/github.com/hashicorp/consul/command/agent/agent.go:238 +0x4f6
github.com/hashicorp/consul/command/agent.(*Command).run(0xc4201ac0f0, 0xc42000e140, 0xf, 0x10, 0x0)
/gopath/src/github.com/hashicorp/consul/command/agent/command.go:720 +0x4db
github.com/hashicorp/consul/command/agent.(*Command).Run(0xc4201ac0f0, 0xc42000e140, 0xf, 0x10, 0xc420290140)
/gopath/src/github.com/hashicorp/consul/command/agent/command.go:669 +0x56
github.com/hashicorp/consul/vendor/github.com/mitchellh/cli.(*CLI).Run(0xc420170840, 0xc420170840, 0x40, 0xc42028e240)
/gopath/src/github.com/hashicorp/consul/vendor/github.com/mitchellh/cli/cli.go:160 +0x1cc
main.realMain(0xc4200001a0)
/gopath/src/github.com/hashicorp/consul/main.go:54 +0x40d
main.main()
/gopath/src/github.com/hashicorp/consul/main.go:18 +0x22
@slackpad slackpad added type/bug Feature does not function as expected type/crash The issue description contains a golang panic and stack trace labels Jan 3, 2018
@slackpad slackpad added this to the 1.0.3 milestone Jan 3, 2018
@slackpad
Copy link
Contributor

slackpad commented Jan 3, 2018

We did update Bolt for 1.0, so need to see if any changes there look related.

@slackpad
Copy link
Contributor

Doesn't look like it's possibly fixed via any of the BoltDB updates that came in Consul 1.0.0, needs further investigation.

@slackpad slackpad modified the milestones: 1.0.3, Next Jan 20, 2018
@hehailong5
Copy link
Author

hehailong5 commented Mar 28, 2018 •
edited
Loading

Hi,

Any progress on this issue?
We encounter a similar one recently in 0.8.4, here goes the crash log:

==> WARNING: Bootstrap mode enabled! Do not enable unless necessary
==> Starting Consul agent...
panic: invalid page type: 215: 10

goroutine 1 [running]:
github.com/hashicorp/consul/vendor/github.com/boltdb/bolt.(*Cursor).search(0xc4200a4ca8, 0xc4200a4d10, 0x8, 0x8, 0xd7)
/gopath/src/github.com/hashicorp/consul/vendor/github.com/boltdb/bolt/cursor.go:256 +0x405
github.com/hashicorp/consul/vendor/github.com/boltdb/bolt.(*Cursor).searchPage(0xc4200a4ca8, 0xc4200a4d10, 0x8, 0x8, 0x7faa6027e000)
/gopath/src/github.com/hashicorp/consul/vendor/github.com/boltdb/bolt/cursor.go:314 +0x13e
github.com/hashicorp/consul/vendor/github.com/boltdb/bolt.(*Cursor).search(0xc4200a4ca8, 0xc4200a4d10, 0x8, 0x8, 0x15a)
/gopath/src/github.com/hashicorp/consul/vendor/github.com/boltdb/bolt/cursor.go:271 +0x1bb
github.com/hashicorp/consul/vendor/github.com/boltdb/bolt.(*Cursor).searchPage(0xc4200a4ca8, 0xc4200a4d10, 0x8, 0x8, 0x7faa602a4000)
/gopath/src/github.com/hashicorp/consul/vendor/github.com/boltdb/bolt/cursor.go:314 +0x13e
github.com/hashicorp/consul/vendor/github.com/boltdb/bolt.(*Cursor).search(0xc4200a4ca8, 0xc4200a4d10, 0x8, 0x8, 0x180)
/gopath/src/github.com/hashicorp/consul/vendor/github.com/boltdb/bolt/cursor.go:271 +0x1bb
github.com/hashicorp/consul/vendor/github.com/boltdb/bolt.(*Cursor).seek(0xc4200a4ca8, 0xc4200a4d10, 0x8, 0x8, 0x0, 0x0, 0x968053, 0xc4203f9cf8, 0xc4203fc880, 0x1, ...)
/gopath/src/github.com/hashicorp/consul/vendor/github.com/boltdb/bolt/cursor.go:159 +0xb1
github.com/hashicorp/consul/vendor/github.com/boltdb/bolt.(*Bucket).Get(0xc42058bbc0, 0xc4200a4d10, 0x8, 0x8, 0xc42058bbc0, 0x0, 0x7faa60564960)
/gopath/src/github.com/hashicorp/consul/vendor/github.com/boltdb/bolt/bucket.go:260 +0xef
github.com/hashicorp/consul/vendor/github.com/hashicorp/raft-boltdb.(*BoltStore).GetLog(0xc42041c7e0, 0x946, 0xc420558060, 0x0, 0x0)
/gopath/src/github.com/hashicorp/consul/vendor/github.com/hashicorp/raft-boltdb/bolt_store.go:124 +0xfd
github.com/hashicorp/consul/vendor/github.com/hashicorp/raft.(*LogCache).GetLog(0xc420419e00, 0x946, 0xc420558060, 0x0, 0x0)
/gopath/src/github.com/hashicorp/consul/vendor/github.com/hashicorp/raft/log_cache.go:46 +0x110
github.com/hashicorp/consul/vendor/github.com/hashicorp/raft.NewRaft(0xc42027c240, 0x18eac80, 0xc4202f19a0, 0x18f1040, 0xc420419e00, 0x18ed1c0, 0xc42041c7e0, 0x18eb380, 0xc42041c920, 0x18f2360, ...)
/gopath/src/github.com/hashicorp/consul/vendor/github.com/hashicorp/raft/api.go:489 +0x9ae
github.com/hashicorp/consul/consul.(*Server).setupRaft(0xc4203ea500, 0x0, 0x0)
/gopath/src/github.com/hashicorp/consul/consul/server.go:595 +0x5e9
github.com/hashicorp/consul/consul.NewServerLogger(0xc4203ea280, 0xc4202f0be0, 0x0, 0x0, 0x0)
/gopath/src/github.com/hashicorp/consul/consul/server.go:320 +0xc1f
github.com/hashicorp/consul/command/agent.(*Agent).makeServer(0xc42024e6c0, 0x1, 0xc4201f9d40, 0x0)
/gopath/src/github.com/hashicorp/consul/command/agent/agent.go:860 +0x112
github.com/hashicorp/consul/command/agent.(*Agent).Start(0xc42024e6c0, 0xc42024e6c0, 0x0)
/gopath/src/github.com/hashicorp/consul/command/agent/agent.go:238 +0x4f6
github.com/hashicorp/consul/command/agent.(*Command).run(0xc420298000, 0xc42000e110, 0xd, 0xd, 0x0)
/gopath/src/github.com/hashicorp/consul/command/agent/command.go:720 +0x4db
github.com/hashicorp/consul/command/agent.(*Command).Run(0xc420298000, 0xc42000e110, 0xd, 0xd, 0xc420286440)
/gopath/src/github.com/hashicorp/consul/command/agent/command.go:669 +0x56
github.com/hashicorp/consul/vendor/github.com/mitchellh/cli.(*CLI).Run(0xc420248240, 0xc420248240, 0x40, 0xc420233dc0)
/gopath/src/github.com/hashicorp/consul/vendor/github.com/mitchellh/cli/cli.go:160 +0x1cc
main.realMain(0xc4200001a0)
/gopath/src/github.com/hashicorp/consul/main.go:54 +0x40d
main.main()
/gopath/src/github.com/hashicorp/consul/main.go:18 +0x22

@EugenMayer
Copy link

EugenMayer commented Jul 1, 2018 •
edited
Loading

very similar issue on our side, 1.0.7

goroutine 1 [running]:
github.com/hashicorp/consul/vendor/github.com/boltdb/bolt.(*node).spill(0xc42015e7e0, 0x10, 0x10)
        /gopath/src/github.com/hashicorp/consul/vendor/github.com/boltdb/bolt/node.go:375 +0x677
github.com/hashicorp/consul/vendor/github.com/boltdb/bolt.(*Bucket).spill(0xc4200650b8, 0xa1bed90, 0x1ea8880)
        /gopath/src/github.com/hashicorp/consul/vendor/github.com/boltdb/bolt/bucket.go:570 +0x4b8
github.com/hashicorp/consul/vendor/github.com/boltdb/bolt.(*Tx).Commit(0xc4200650a0, 0x1d4509c, 0x4)
        /gopath/src/github.com/hashicorp/consul/vendor/github.com/boltdb/bolt/tx.go:163 +0x11f
github.com/hashicorp/consul/vendor/github.com/hashicorp/raft-boltdb.(*BoltStore).initialize(0xc42015b160, 0x0, 0x0)
        /gopath/src/github.com/hashicorp/consul/vendor/github.com/hashicorp/raft-boltdb/bolt_store.go:75 +0x12a
github.com/hashicorp/consul/vendor/github.com/hashicorp/raft-boltdb.NewBoltStore(0xc420338740, 0x19, 0x2, 0xc420338740, 0x19)
        /gopath/src/github.com/hashicorp/consul/vendor/github.com/hashicorp/raft-boltdb/bolt_store.go:51 +0xc4
github.com/hashicorp/consul/agent/consul.(*Server).setupRaft(0xc420598000, 0x0, 0x0)
        /gopath/src/github.com/hashicorp/consul/agent/consul/server.go:495 +0x902
github.com/hashicorp/consul/agent/consul.NewServerLogger(0xc42017a280, 0xc4203a6be0, 0xc4204b9f20, 0x0, 0xc4203a6be0, 0xc4202e6a80)
        /gopath/src/github.com/hashicorp/consul/agent/consul/server.go:336 +0xbae
github.com/hashicorp/consul/agent.(*Agent).Start(0xc420528000, 0xc420528000, 0x0)
        /gopath/src/github.com/hashicorp/consul/agent/agent.go:303 +0x33b
github.com/hashicorp/consul/command/agent.(*cmd).run(0xc420250800, 0xc42002e0a0, 0x2, 0x2, 0x0)
        /gopath/src/github.com/hashicorp/consul/command/agent/agent.go:337 +0x3e5
github.com/hashicorp/consul/command/agent.(*cmd).Run(0xc420250800, 0xc42002e0a0, 0x2, 0x2, 0xc420253ba0)
        /gopath/src/github.com/hashicorp/consul/command/agent/agent.go:77 +0x50
github.com/hashicorp/consul/vendor/github.com/mitchellh/cli.(*CLI).Run(0xc4201c5560, 0xc4201c5560, 0x40, 0xc420253e80)
        /gopath/src/github.com/hashicorp/consul/vendor/github.com/mitchellh/cli/cli.go:242 +0x1eb
main.realMain(0xc42007a058)
        /gopath/src/github.com/hashicorp/consul/main.go:52 +0x3ee
main.main()
        /gopath/src/github.com/hashicorp/consul/main.go:19 +0x22
consul --version
Consul v1.0.7
Protocol 2 spoken by default, understands 2 to 3 (agent will automatically use protocol >2 when speaking to compatible agents)

We are using the official consul docker image

FROM consul:1.0.7

We have a setup with

  • acls
  • gossip
  • tls
  • we are using "bootstrap:1"

if that helps in any regard

@EugenMayer
Copy link

For us, this entirely blocks us. That is our server_config without the gossip/acl/tls part:

{
  "datacenter": "stable",
  "data_dir": "/consul/data",
  "ui": true,
  "dns_config": {
    "allow_stale": false
  },
  "node_name": "dwconsul",
  "client_addr": "0.0.0.0",
  "server": true,
  "bootstrap_expect": 1,
  "check_update_interval": "0s",
  "disable_update_check": true
}

@Oneiroi
Copy link

Oneiroi commented Aug 23, 2018

Consul version 1.2.2 running test setup with
CLI: sudo -u consul consul agent -server -bind 127.0.0.1 -data-dir /tmp/consul/ -bootstrap

This was for testing purposes only;' consul throws the error:

bootstrap = true: do not enable unless necessary
==> Starting Consul agent...
panic: invalid page type: 0: 4

goroutine 1 [running]:
github.com/hashicorp/consul/vendor/github.com/boltdb/bolt.(*Cursor).search(0xc420026be8, 0x2be30b0, 0x4, 0x4, 0x4)
        /go/src/github.com/hashicorp/consul/vendor/github.com/boltdb/bolt/cursor.go:256 +0x388
github.com/hashicorp/consul/vendor/github.com/boltdb/bolt.(*Cursor).seek(0xc420026be8, 0x2be30b0, 0x4, 0x4, 0x0, 0x0, 0xc420480560, 0xc420026c10, 0xa97166, 0xc420480560, ...)
        /go/src/github.com/hashicorp/consul/vendor/github.com/boltdb/bolt/cursor.go:159 +0xa5
github.com/hashicorp/consul/vendor/github.com/boltdb/bolt.(*Bucket).Bucket(0xc4204c4398, 0x2be30b0, 0x4, 0x4, 0xc420026c58)
        /go/src/github.com/hashicorp/consul/vendor/github.com/boltdb/bolt/bucket.go:112 +0xde
github.com/hashicorp/consul/vendor/github.com/boltdb/bolt.(*Tx).Bucket(0xc4204c4380, 0x2be30b0, 0x4, 0x4, 0x0)
        /go/src/github.com/hashicorp/consul/vendor/github.com/boltdb/bolt/tx.go:101 +0x4f
github.com/hashicorp/consul/vendor/github.com/hashicorp/raft-boltdb.(*BoltStore).Get(0xc420141fa0, 0x2be3628, 0xb, 0xb, 0x0, 0x0, 0x0, 0x0, 0x0)
        /go/src/github.com/hashicorp/consul/vendor/github.com/hashicorp/raft-boltdb/bolt_store.go:210 +0xc5
github.com/hashicorp/consul/vendor/github.com/hashicorp/raft-boltdb.(*BoltStore).GetUint64(0xc420141fa0, 0x2be3628, 0xb, 0xb, 0x1de6f80, 0x2be3558, 0xc42020ef70)
        /go/src/github.com/hashicorp/consul/vendor/github.com/hashicorp/raft-boltdb/bolt_store.go:226 +0x4d
github.com/hashicorp/consul/vendor/github.com/hashicorp/raft.HasExistingState(0x1df9e00, 0xc4201b2ec0, 0x1df4560, 0xc420141fa0, 0x1df1be0, 0xc420322ec0, 0x0, 0x0, 0xc42006cc00)
        /go/src/github.com/hashicorp/consul/vendor/github.com/hashicorp/raft/api.go:354 +0x62
github.com/hashicorp/consul/agent/consul.(*Server).setupRaft(0xc4200de6c0, 0x0, 0x0)
        /go/src/github.com/hashicorp/consul/agent/consul/server.go:602 +0x44e
github.com/hashicorp/consul/agent/consul.NewServerLogger(0xc4200f2280, 0xc4200c2190, 0xc420222120, 0x0, 0xc4200c2190, 0xc420228080)
        /go/src/github.com/hashicorp/consul/agent/consul/server.go:362 +0xbe1
github.com/hashicorp/consul/agent.(*Agent).Start(0xc4204801e0, 0xc4204801e0, 0x0)
        /go/src/github.com/hashicorp/consul/agent/agent.go:330 +0x36b
github.com/hashicorp/consul/command/agent.(*cmd).run(0xc420261800, 0xc4200c4020, 0x6, 0x6, 0x0)
        /go/src/github.com/hashicorp/consul/command/agent/agent.go:219 +0x454
github.com/hashicorp/consul/command/agent.(*cmd).Run(0xc420261800, 0xc4200c4020, 0x6, 0x6, 0xc420322da0)
        /go/src/github.com/hashicorp/consul/command/agent/agent.go:74 +0x50
github.com/hashicorp/consul/vendor/github.com/mitchellh/cli.(*CLI).Run(0xc420394120, 0xc420394120, 0x40, 0xc420322e40)
        /go/src/github.com/hashicorp/consul/vendor/github.com/mitchellh/cli/cli.go:242 +0x1eb
main.realMain(0xc4200b0058)
        /go/src/github.com/hashicorp/consul/main.go:53 +0x3ee
main.main()
        /go/src/github.com/hashicorp/consul/main.go:20 +0x22

Other projects are having issues with boltdb too see:

odeke-em/drive#828
influxdata/influxdb#6165

I'm unable to proceed with the test case at this time.

@pearkes pearkes mentioned this issue Oct 26, 2018
Closed
@pearkes pearkes modified the milestones: Upcoming, 1.4.3 Jan 30, 2019
@mkeeler mkeeler modified the milestones: 1.4.3, 1.4.4 Mar 4, 2019
@hanshasselberg hanshasselberg modified the milestones: 1.4.4, 1.5 Mar 19, 2019
@pearkes pearkes modified the milestones: 1.5.0, Upcoming Apr 16, 2019
@mkeeler mkeeler mentioned this issue Apr 18, 2019
Closed
@pierresouchay
Copy link
Contributor

FYI We had this issue when having incorrect ACLs

@mkeeler mkeeler mentioned this issue Aug 1, 2019
Closed
@dnephin dnephin mentioned this issue Mar 20, 2020
Open
@jsosulska jsosulska mentioned this issue Aug 5, 2020
Closed
@jsosulska
Copy link
Contributor

Hello all!

To update this thread - I have created a top level issue to track upgrading BoltDB to bbolt here. Please follow that work as a precursor to the issues mentioned here.

Thank you all for your patience!

@mikemorris mikemorris removed this from the Upcoming milestone Nov 10, 2020
@kzys kzys mentioned this issue Mar 12, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type/bug Feature does not function as expected type/crash The issue description contains a golang panic and stack trace
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
10 participants
@hanshasselberg @EugenMayer @Oneiroi @slackpad @pearkes @mikemorris @mkeeler @pierresouchay @hehailong5 @jsosulska