Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pull fails with status 404 #839

Closed
gfalken opened this issue Jan 10, 2017 · 21 comments
Closed

Pull fails with status 404 #839

gfalken opened this issue Jan 10, 2017 · 21 comments

Comments

@gfalken
Copy link

gfalken commented Jan 10, 2017

After many months of working properly, I just noticed than recently pulled files have 0 file sizes. I ran a pull on a single directory and every file looked like this:

pull: /webs/TCHealth/assets/docs/TuolumneCountyHealth.org_map.pdf err: download: failed for url "https://googledrive.com/host/0B4H8hQ8EKW3IQWFLbTAxcmQ0c2M". StatusCode: 404

Any ideas on troubleshooting this?

@odeke-em
Copy link
Owner

Hello there @gfalken, thanks for reporting this issue and welcome back to drive!

For starters, please report what your drive version says.

$ drive version

Thanks.

@gfalken
Copy link
Author

gfalken commented Jan 10, 2017 via email

@odeke-em
Copy link
Owner

Gotcha. So your version is a little behind, the current version is 0.3.9.
Google Drive shut down hosting files on https://googledrive.com/host/fileId in August 2016 and @dermesser reported that bug in #734 (comment) and also fixed that bug in September 2016 with #735.

Please get the latest drive and this problem shouldn't exist anymore

$ go get -u -v github.com/odeke-em/drive/drive-gen && drive-gen

@gfalken
Copy link
Author

gfalken commented Jan 10, 2017 via email

@odeke-em
Copy link
Owner

Yes, on pulling again, you'll have a manifest of files for deletion, then addition ie replacements. But beware that some files might not be properly synchronized. If you just want to deal with the zero byte files, I'd recommend doing this

$ find . -type f -size 0 -exec rm -rf {} \; # Remove the zero byte files locally
$ # And then finally you can do the drive pull
$ drive pull

@odeke-em
Copy link
Owner

I'll close this, please reopen if persists. Also let us know if the problem is fixed on the new pull tomorrow.

@gfalken
Copy link
Author

gfalken commented Jan 10, 2017

I updated Drive and now show version 0.3.9. However, when I tried a pull on one affected directory, I got the following:

pi@raspberrypi:/mnt/usbstorage/gdrive/webs/TCHealth $ drive pull
Resolving...
panic: invalid page type: 13239: 10

goroutine 1 [running]:
github.com/boltdb/bolt.(*Cursor).search(0x10bbda48, 0x10bbda78, 0x1c, 0x20, 0x33b7, 0x0)
/home/pi/go/bin/src/github.com/boltdb/bolt/cursor.go:256 +0x1c0
github.com/boltdb/bolt.(*Cursor).searchPage(0x10bbda48, 0x10bbda78, 0x1c, 0x20, 0x6e2ce000)
/home/pi/go/bin/src/github.com/boltdb/bolt/cursor.go:314 +0x15c
github.com/boltdb/bolt.(*Cursor).search(0x10bbda48, 0x10bbda78, 0x1c, 0x20, 0x33ce, 0x0)
/home/pi/go/bin/src/github.com/boltdb/bolt/cursor.go:271 +0x358
github.com/boltdb/bolt.(*Cursor).seek(0x10bbda48, 0x10bbda78, 0x1c, 0x20, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/home/pi/go/bin/src/github.com/boltdb/bolt/cursor.go:159 +0xd0
github.com/boltdb/bolt.(*Bucket).Get(0x10d90de0, 0x10bbda78, 0x1c, 0x20, 0x0, 0x0, 0x0)
/home/pi/go/bin/src/github.com/boltdb/bolt/bucket.go:267 +0xc8
github.com/odeke-em/drive/config.(*Context).DeserializeIndex.func1(0x10aa7180, 0x0, 0x0)
/home/pi/go/bin/src/github.com/odeke-em/drive/config/config.go:134 +0x150
github.com/boltdb/bolt.(*DB).View(0x10dd4240, 0x10bbdb88, 0x0, 0x0)
/home/pi/go/bin/src/github.com/boltdb/bolt/db.go:629 +0xb0
github.com/odeke-em/drive/config.(*Context).DeserializeIndex(0x10b79080, 0x10e78700, 0x1c, 0x0, 0x0, 0x0)
/home/pi/go/bin/src/github.com/odeke-em/drive/config/config.go:140 +0x124
github.com/odeke-em/drive/src.(*Commands).deserializeIndex(0x10b79aa0, 0x10e78700, 0x1c, 0x20)
/home/pi/go/bin/src/github.com/odeke-em/drive/src/push.go:243 +0x34
github.com/odeke-em/drive/src.(*Commands).(github.com/odeke-em/drive/src.deserializeIndex)-fm(0x10e78700, 0x1c, 0x0)
/home/pi/go/bin/src/github.com/odeke-em/drive/src/changes.go:719 +0x34
github.com/odeke-em/drive/src.resolveConflicts(0x10da4a10, 0x3, 0x4, 0x10b36100, 0x10bbdc78, 0x10ef4750, 0x1, 0x2, 0x0, 0x0, ...)
/home/pi/go/bin/src/github.com/odeke-em/drive/src/changes.go:700 +0x11c
github.com/odeke-em/drive/src.(*Commands).resolveConflicts(0x10b79aa0, 0x10f6f4a0, 0x3c, 0x3c, 0x0, 0x0, 0x0)
/home/pi/go/bin/src/github.com/odeke-em/drive/src/changes.go:719 +0x164
github.com/odeke-em/drive/src.pull(0x10b79aa0, 0x2, 0x0, 0x0)
/home/pi/go/bin/src/github.com/odeke-em/drive/src/pull.go:116 +0x350
github.com/odeke-em/drive/src.(*Commands).Pull(0x10b79aa0, 0x0, 0x0)
/home/pi/go/bin/src/github.com/odeke-em/drive/src/pull.go:71 +0x30
main.(*pullCmd).Run(0x10aa6380, 0x10a7a058, 0x0, 0x0, 0x10b78f20)
/home/pi/go/bin/src/github.com/odeke-em/drive/cmd/drive/main.go:789 +0x8ac
github.com/odeke-em/command.Run()
/home/pi/go/bin/src/github.com/odeke-em/command/command.go:202 +0xac
github.com/odeke-em/command.ParseAndRun()
/home/pi/go/bin/src/github.com/odeke-em/command/command.go:209 +0x18
main.main()
/home/pi/go/bin/src/github.com/odeke-em/drive/cmd/drive/main.go:126 +0x23ec

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
/home/pi/go/src/runtime/asm_arm.s:1036 +0x4

goroutine 19 [syscall]:
os/signal.loop()
/home/pi/go/src/os/signal/signal_unix.go:22 +0x14
created by os/signal.init.1
/home/pi/go/src/os/signal/signal_unix.go:28 +0x30

goroutine 25 [select]:
net/http.(*persistConn).writeLoop(0x10b8e000)
/home/pi/go/src/net/http/transport.go:1009 +0x314
created by net/http.(*Transport).dialConn
/home/pi/go/src/net/http/transport.go:686 +0xad0

goroutine 21 [chan receive]:
github.com/odeke-em/cli-spinner.(*Spinner).spin.func1(0x10b7b8c0)
/home/pi/go/bin/src/github.com/odeke-em/cli-spinner/spinner.go:100 +0x2f4
created by github.com/odeke-em/cli-spinner.(*Spinner).spin
/home/pi/go/bin/src/github.com/odeke-em/cli-spinner/spinner.go:103 +0xc4

goroutine 24 [IO wait]:
net.runtime_pollWait(0x75b57000, 0x72, 0x10a7a000)
/home/pi/go/src/runtime/netpoll.go:157 +0x60
net.(*pollDesc).Wait(0x10a0e6f8, 0x72, 0x0, 0x0)
/home/pi/go/src/net/fd_poll_runtime.go:73 +0x34
net.(*pollDesc).WaitRead(0x10a0e6f8, 0x0, 0x0)
/home/pi/go/src/net/fd_poll_runtime.go:78 +0x30
net.(*netFD).Read(0x10a0e6c0, 0x10ab5000, 0x1000, 0x1000, 0x0, 0x75b16018, 0x10a7a000)
/home/pi/go/src/net/fd_unix.go:232 +0x1c4
net.(*conn).Read(0x10a6d518, 0x10ab5000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/home/pi/go/src/net/net.go:172 +0xc8
crypto/tls.(*block).readFromUntil(0x10b79cc0, 0x75b1ad08, 0x10a6d518, 0x5, 0x0, 0x0)
/home/pi/go/src/crypto/tls/conn.go:455 +0xc8
crypto/tls.(*Conn).readRecord(0x10aa0340, 0x662317, 0x0, 0x0)
/home/pi/go/src/crypto/tls/conn.go:540 +0x240
crypto/tls.(*Conn).Read(0x10aa0340, 0x10bb3000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/home/pi/go/src/crypto/tls/conn.go:901 +0x13c
net/http.noteEOFReader.Read(0x75b1f180, 0x10aa0340, 0x10b8e02c, 0x10bb3000, 0x1000, 0x1000, 0x24a478, 0x0, 0x0)
/home/pi/go/src/net/http/transport.go:1370 +0x5c
net/http.(*noteEOFReader).Read(0x10dc8cf0, 0x10bb3000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
:126 +0xd0
bufio.(*Reader).fill(0x10dde0c0)
/home/pi/go/src/bufio/bufio.go:97 +0x1c4
bufio.(*Reader).Peek(0x10dde0c0, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
/home/pi/go/src/bufio/bufio.go:132 +0xcc
net/http.(*persistConn).readLoop(0x10b8e000)
/home/pi/go/src/net/http/transport.go:876 +0xe4
created by net/http.(*Transport).dialConn
/home/pi/go/src/net/http/transport.go:685 +0xab4

goroutine 41 [IO wait]:
net.runtime_pollWait(0x75b57078, 0x72, 0x10a7a000)
/home/pi/go/src/runtime/netpoll.go:157 +0x60
net.(*pollDesc).Wait(0x10b170f8, 0x72, 0x0, 0x0)
/home/pi/go/src/net/fd_poll_runtime.go:73 +0x34
net.(*pollDesc).WaitRead(0x10b170f8, 0x0, 0x0)
/home/pi/go/src/net/fd_poll_runtime.go:78 +0x30
net.(*netFD).Read(0x10b170c0, 0x10e4e000, 0x1000, 0x1000, 0x0, 0x75b16018, 0x10a7a000)
/home/pi/go/src/net/fd_unix.go:232 +0x1c4
net.(*conn).Read(0x10b12c58, 0x10e4e000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/home/pi/go/src/net/net.go:172 +0xc8
crypto/tls.(*block).readFromUntil(0x10b39de0, 0x75b1ad08, 0x10b12c58, 0x5, 0x0, 0x0)
/home/pi/go/src/crypto/tls/conn.go:455 +0xc8
crypto/tls.(*Conn).readRecord(0x10b321a0, 0x662317, 0x0, 0x0)
/home/pi/go/src/crypto/tls/conn.go:540 +0x240
crypto/tls.(*Conn).Read(0x10b321a0, 0x10e03000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/home/pi/go/src/crypto/tls/conn.go:901 +0x13c
net/http.noteEOFReader.Read(0x75b1f180, 0x10b321a0, 0x10ddc2cc, 0x10e03000, 0x1000, 0x1000, 0x24a478, 0x0, 0x0)
/home/pi/go/src/net/http/transport.go:1370 +0x5c
net/http.(*noteEOFReader).Read(0x10e22a10, 0x10e03000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
:126 +0xd0
bufio.(*Reader).fill(0x10e20db0)
/home/pi/go/src/bufio/bufio.go:97 +0x1c4
bufio.(*Reader).Peek(0x10e20db0, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
/home/pi/go/src/bufio/bufio.go:132 +0xcc
net/http.(*persistConn).readLoop(0x10ddc2a0)
/home/pi/go/src/net/http/transport.go:876 +0xe4
created by net/http.(*Transport).dialConn
/home/pi/go/src/net/http/transport.go:685 +0xab4

goroutine 42 [select]:
net/http.(*persistConn).writeLoop(0x10ddc2a0)
/home/pi/go/src/net/http/transport.go:1009 +0x314
created by net/http.(*Transport).dialConn
/home/pi/go/src/net/http/transport.go:686 +0xad0
pi@raspberrypi:/mnt/usbstorage/gdrive/webs/TCHealth $

@odeke-em
Copy link
Owner

Yap, boltdb trips noticed in #824 and #828. For starters, my boltdb is at https://github.com/boltdb/bolt/tree/852d3024fa8d89dcc9a715bab6f4dcd7d59577dd, so until we've sorted out boltdb, please use that boltdb commit so that we'll be at par. You can do so like this:

$ cd $GOPATH/src/github.com/boltdb/bolt && git reset --hard 852d3024fa8d89dcc9a715bab6f4dcd7d59577dd
$ drive-gen

@gfalken
Copy link
Author

gfalken commented Jan 10, 2017

OK, that change has been made. Same error with a new pull.

@odeke-em
Copy link
Owner

Ah crap, okay so how about your Go version?

@odeke-em
Copy link
Owner

Actually drive version?

@gfalken
Copy link
Author

gfalken commented Jan 10, 2017

drive version: 0.3.9
Commit Hash: '5278f2bbc4a68f116d9ce95ba5a7bc61b1e322ab'
Go Version: go1.5
OS: linux/arm
BuildTime: 2017-01-10 13:06:19.630125416 -0800 PST

And there were a lot more errors when I did the pull. Screen dump attached.
screendump.txt

@odeke-em
Copy link
Owner

Okay, for starters I'd advise getting the latest Go. Debugging Go1.5 wouldn't be feasible. I can't reproduce it here.

In case you can't get an updated Go, I made a few binaries here https://github.com/odeke-em/drive/releases/tag/v0.3.9 under downloads, checkout the arm versions, and please try with one of those.

@gfalken
Copy link
Author

gfalken commented Jan 10, 2017

My system now recognizes go1.6.4 but Drive still reports go1.5. Any ideas?

@nelaaro
Copy link

nelaaro commented Jan 11, 2017

Hi Odeke

Still causing the same issues

$ go version
go version go1.6.3 linux/amd64

$ drive version
drive version: 0.3.9
Commit Hash: '5278f2bbc4a68f116d9ce95ba5a7bc61b1e322ab'
Go Version: go1.6.3
OS: linux/amd64
BuildTime: 2017-01-11 08:27:35.159070749 +0200 SAST

aaron@aaron-ubunix:~/lib/golang/src/github.com/boltdb/bolt
$ git log -1
commit 852d3024fa8d89dcc9a715bab6f4dcd7d59577dd
Author: Luke Champine [email protected]
Date: Thu Nov 5 22:39:15 2015 -0700

$ drive pull
Resolving...
panic: pgid (68719476737) above high water mark (16)

goroutine 1 [running]:
panic(0x56245ac87c40, 0xc820205f90)
/usr/lib/go-1.6/src/runtime/panic.go:481 +0x3e6
github.com/boltdb/bolt.(*node).spill(0xc820b883f0, 0x0, 0x0)
/home/aaron/lib/golang/src/github.com/boltdb/bolt/node.go:370 +0x548
github.com/boltdb/bolt.(*node).spill(0xc820b88380, 0x0, 0x0)
/home/aaron/lib/golang/src/github.com/boltdb/bolt/node.go:345 +0x12c
github.com/boltdb/bolt.(*Bucket).spill(0xc8208a85c8, 0x0, 0x0)
/home/aaron/lib/golang/src/github.com/boltdb/bolt/bucket.go:541 +0x1cf
github.com/boltdb/bolt.(*Tx).Commit(0xc8208a85b0, 0x0, 0x0)
/home/aaron/lib/golang/src/github.com/boltdb/bolt/tx.go:154 +0x1d4
github.com/boltdb/bolt.(*DB).Update(0xc8207d11e0, 0xc820155648, 0x0, 0x0)
/home/aaron/lib/golang/src/github.com/boltdb/bolt/db.go:554 +0x11d
github.com/odeke-em/drive/config.(*Context).CreateIndicesBucket(0xc82014b340, 0x0, 0x0)
/home/aaron/lib/golang/src/github.com/odeke-em/drive/config/config.go:255 +0xb5
github.com/odeke-em/drive/config.(*Context).DeserializeIndex(0xc82014b340, 0xc82043f560, 0x1c, 0x0, 0x0, 0x0)
/home/aaron/lib/golang/src/github.com/odeke-em/drive/config/config.go:115 +0x4e
github.com/odeke-em/drive/src.(*Commands).deserializeIndex(0xc82014b540, 0xc82043f560, 0x1c, 0xc82002e800)
/home/aaron/lib/golang/src/github.com/odeke-em/drive/src/push.go:243 +0x38
github.com/odeke-em/drive/src.(*Commands).(github.com/odeke-em/drive/src.deserializeIndex)-fm(0xc82043f560, 0x1c, 0x2)
/home/aaron/lib/golang/src/github.com/odeke-em/drive/src/changes.go:719 +0x34
github.com/odeke-em/drive/src.resolveConflicts(0xc820282700, 0x7, 0x8, 0xc820c12a00, 0xc8201558f0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/home/aaron/lib/golang/src/github.com/odeke-em/drive/src/changes.go:700 +0x135
github.com/odeke-em/drive/src.(*Commands).resolveConflicts(0xc82014b540, 0xc820c0e500, 0x84b, 0x8a0, 0x800, 0x0, 0x0)
/home/aaron/lib/golang/src/github.com/odeke-em/drive/src/changes.go:719 +0x1df
github.com/odeke-em/drive/src.pull(0xc82014b540, 0x2, 0x0, 0x0)
/home/aaron/lib/golang/src/github.com/odeke-em/drive/src/pull.go:116 +0x41b
github.com/odeke-em/drive/src.(*Commands).Pull(0xc82014b540, 0x0, 0x0)
/home/aaron/lib/golang/src/github.com/odeke-em/drive/src/pull.go:71 +0x36
main.(*pullCmd).Run(0xc82020c000, 0xc820096070, 0x0, 0x0, 0xc8201f1e90)
/home/aaron/lib/golang/src/github.com/odeke-em/drive/cmd/drive/main.go:789 +0xae3
github.com/odeke-em/command.Run()
/home/aaron/lib/golang/src/github.com/odeke-em/command/command.go:202 +0xaa
github.com/odeke-em/command.ParseAndRun()
/home/aaron/lib/golang/src/github.com/odeke-em/command/command.go:209 +0x19
main.main()
/home/aaron/lib/golang/src/github.com/odeke-em/drive/cmd/drive/main.go:126 +0x22d7

@odeke-em
Copy link
Owner

@gfalken, it could be that your $GOPATH isn't properly setup. @nelaaro and @gfalken would y'all mind trying the binaries that I built with go1.8? If those work that can help give some hints whether it might be related to the Go version and if the problem is reproducible with the latest Go. I ask because for now I don't have a clue what's going on, but am hoping to eliminate some suspicions so that we can solve this issue. Thank you.

@odeke-em
Copy link
Owner

Also the initial reported bug was already solved, let's move the conversation to the bug tracking the boltdb panics at #828.

@gfalken
Copy link
Author

gfalken commented Jan 11, 2017

I'm sorry for the extra work but can you give me some quick instructions for downloading and installing the binary? If I have to do it by trial and error, it's going to take me a long time. Thanks.

@nelaaro
Copy link

nelaaro commented Jan 13, 2017

Hi Odeke

I downloaded the following binary
https://github.com/odeke-em/drive/releases/download/v0.3.9/drive_linux

$ ~/Downloads/drive_linux pull
Resolving...
panic: pgid (68719476737) above high water mark (16)

goroutine 1 [running]:
github.com/boltdb/bolt.(*node).spill(0xc4209622a0, 0xc42033a4e0, 0x929780)
/Users/emmanuelodeke/go/src/github.com/boltdb/bolt/node.go:370 +0x631
github.com/boltdb/bolt.(*node).spill(0xc4209621c0, 0x10, 0x10)
/Users/emmanuelodeke/go/src/github.com/boltdb/bolt/node.go:345 +0xb4
github.com/boltdb/bolt.(*Bucket).spill(0xc4202b21d8, 0x328f348a, 0x955b80)
/Users/emmanuelodeke/go/src/github.com/boltdb/bolt/bucket.go:541 +0x17b
github.com/boltdb/bolt.(*Tx).Commit(0xc4202b21c0, 0x0, 0x0)
/Users/emmanuelodeke/go/src/github.com/boltdb/bolt/tx.go:163 +0x125
github.com/boltdb/bolt.(*DB).Update(0xc420146000, 0x7c8580, 0x0, 0x0)
/Users/emmanuelodeke/go/src/github.com/boltdb/bolt/db.go:602 +0xf7
github.com/odeke-em/drive/config.(*Context).CreateIndicesBucket(0xc420016f80, 0x0, 0x0)
/Users/emmanuelodeke/go/src/github.com/odeke-em/drive/config/config.go:255 +0x95
github.com/odeke-em/drive/config.(*Context).DeserializeIndex(0xc420016f80, 0xc4202263e0, 0x1c, 0x0, 0x0, 0x0)
/Users/emmanuelodeke/go/src/github.com/odeke-em/drive/config/config.go:115 +0x64
github.com/odeke-em/drive/src.(*Commands).deserializeIndex(0xc420017100, 0xc4202263e0, 0x1c, 0x40e890)
/Users/emmanuelodeke/go/src/github.com/odeke-em/drive/src/push.go:242 +0x42
github.com/odeke-em/drive/src.(*Commands).(github.com/odeke-em/drive/src.deserializeIndex)-fm(0xc4202263e0, 0x1c, 0xc4200a1948)
/Users/emmanuelodeke/go/src/github.com/odeke-em/drive/src/changes.go:719 +0x3e
github.com/odeke-em/drive/src.resolveConflicts(0xc4207e2000, 0x7, 0x8, 0xc42096e000, 0xc4200a1a20, 0x900, 0xc4207e2000, 0x7, 0x8, 0x84b, ...)
/Users/emmanuelodeke/go/src/github.com/odeke-em/drive/src/changes.go:700 +0x104
github.com/odeke-em/drive/src.(*Commands).resolveConflicts(0xc420017100, 0xc420782800, 0x84b, 0x900, 0x900, 0x0, 0x0)
/Users/emmanuelodeke/go/src/github.com/odeke-em/drive/src/changes.go:719 +0x1c0
github.com/odeke-em/drive/src.pull(0xc420017100, 0x2, 0x1, 0xc4200da600)
/Users/emmanuelodeke/go/src/github.com/odeke-em/drive/src/pull.go:116 +0x294
github.com/odeke-em/drive/src.(*Commands).Pull(0xc420017100, 0xc4200da600, 0xc420017100)
/Users/emmanuelodeke/go/src/github.com/odeke-em/drive/src/pull.go:71 +0x34
main.(*pullCmd).Run(0xc420112000, 0xc42000a2f0, 0x0, 0x0, 0xc4200f7710)
/Users/emmanuelodeke/go/src/github.com/odeke-em/drive/cmd/drive/main.go:789 +0x98c
github.com/odeke-em/command.Run()
/Users/emmanuelodeke/go/src/github.com/odeke-em/command/command.go:202 +0x7a
github.com/odeke-em/command.ParseAndRun()
/Users/emmanuelodeke/go/src/github.com/odeke-em/command/command.go:209 +0x25
main.main()
/Users/emmanuelodeke/go/src/github.com/odeke-em/drive/cmd/drive/main.go:126 +0x1737

aaron@aaron-ubunix:~/Documents/gdrive/nelaaro
$ ~/Downloads/drive_linux version
drive version: 0.3.9
Commit Hash: <CURRENT_COMMIT>
Go Version: <GO_VERSION>
OS: <OS_INFO>
BuildTime:

I am going to try and rebuild on go 1.7.

Will let you know what I find then.

@nelaaro
Copy link

nelaaro commented Jan 13, 2017

Hi Odeke

As per the post at #828
He solved this by 'drive init' on a new directory after resetting bolt

"I tried first on the old drive directory and it failed with the original error. Only by "drive init" on a fresh directory did it work so the issue is related with the disk cache. Hope this info might help other people."

go get -u github.com/odeke-em/drive/cmd/drive
go get github.com/odeke-em/drive/drive-gen && drive-gen
cd $GOPATH/src/github.com/boltdb/bolt && git reset --hard 852d3024fa8d89dcc9a715bab6f4dcd7d59577dd
drive-gen
cd ~; mkdir new_drive_dir; cd new_drive_dir
drive init

This did work for me and I was able to successfully pull a working drive directory.

@odeke-em, what can be done to ensure that new setups don't encounter these same problems.

I have updated to the go 1.7 and rebuilt, everything after removing the src/pkg directories in my $GOPATH

This will cause the same issues for people pulling for the first time or on a new directory.

go get -u github.com/odeke-em/drive/cmd/drive
go get github.com/odeke-em/drive/drive-gen && drive-gen
without doing a bolt src reset.

There must be some changes to the bolt src that is causing the regressions below.
I will see what I can find this weekend.

$ drive version
drive version: 0.3.9
Commit Hash: '5278f2bbc4a68f116d9ce95ba5a7bc61b1e322ab'
Go Version: go1.7.1
OS: linux/amd64
BuildTime: 2017-01-13 13:10:40.962379884 +0200 SAST

aaron@aaron-ubunix:~/Documents/gdrive/nelaaro
$ drive pull
Resolving...
....
Addition count 2223 src: 1.58GB
Proceed with the changes? [Y/n]:y
718170861 / 1694634395 [========================================================================>-------------------------------------------------------------------------------------------------] 42.38% 4m27s

panic: invalid page type: 53: 10

goroutine 1822 [running]:
panic(0x5639e668f4e0, 0xc420ad4800)
/usr/lib/go-1.7/src/runtime/panic.go:500 +0x1a1
github.com/boltdb/bolt.(*Cursor).search(0xc4209bb968, 0xc420ad47e0, 0x7, 0x8, 0x35)
/home/aaron/lib/golang/src/github.com/boltdb/bolt/cursor.go:256 +0x429
github.com/boltdb/bolt.(*Cursor).seek(0xc4209bb968, 0xc420ad47e0, 0x7, 0x8, 0x0, 0x0, 0xc420c33240, 0x18, 0x18, 0xc4209bb958, ...)
/home/aaron/lib/golang/src/github.com/boltdb/bolt/cursor.go:159 +0xb1
github.com/boltdb/bolt.(*Bucket).CreateBucket(0xc42030c718, 0xc420ad47e0, 0x7, 0x8, 0xc420ad47e7, 0x1, 0x0)
/home/aaron/lib/golang/src/github.com/boltdb/bolt/bucket.go:172 +0x124
github.com/boltdb/bolt.(*Bucket).CreateBucketIfNotExists(0xc42030c718, 0xc420ad47e0, 0x7, 0x8, 0x7, 0x8, 0xc420ad47e0)
/home/aaron/lib/golang/src/github.com/boltdb/bolt/bucket.go:206 +0x4d
github.com/boltdb/bolt.(*Tx).CreateBucketIfNotExists(0xc42030c700, 0xc420ad47e0, 0x7, 0x8, 0x7, 0x8, 0xc4209bbb00)
/home/aaron/lib/golang/src/github.com/boltdb/bolt/tx.go:115 +0x4f
github.com/odeke-em/drive/config.(*Context).SerializeIndex.func1(0xc42030c700, 0x5639e679ea38, 0xc42030c700)
/home/aaron/lib/golang/src/github.com/odeke-em/drive/config/config.go:271 +0x86
github.com/boltdb/bolt.(*DB).Update(0xc4201a25a0, 0xc4209bbb98, 0x0, 0x0)
/home/aaron/lib/golang/src/github.com/boltdb/bolt/db.go:598 +0xb5
github.com/odeke-em/drive/config.(*Context).SerializeIndex(0xc420151900, 0xc4209f7f80, 0x0, 0x0)
/home/aaron/lib/golang/src/github.com/odeke-em/drive/config/config.go:279 +0x1a8
github.com/odeke-em/drive/src.(*Commands).createIndex(0xc420151b00, 0xc4205ce6e0, 0x5639e679edd0, 0xc4209bbc80)
/home/aaron/lib/golang/src/github.com/odeke-em/drive/src/fetch.go:314 +0x138
github.com/odeke-em/drive/src.(*Commands).localAdd.func1(0xc4209bbd38, 0xc42056c400, 0xc420151b00)
/home/aaron/lib/golang/src/github.com/odeke-em/drive/src/pull.go:574 +0x63
github.com/odeke-em/drive/src.(*Commands).localAdd(0xc420151b00, 0xc42056c400, 0x0, 0x0, 0x0, 0x0, 0x0)
/home/aaron/lib/golang/src/github.com/odeke-em/drive/src/pull.go:605 +0x241
github.com/odeke-em/drive/src.(*Commands).(github.com/odeke-em/drive/src.localAdd)-fm(0xc42056c400, 0x0, 0x0, 0x0, 0x2, 0xc42024c400)
/home/aaron/lib/golang/src/github.com/odeke-em/drive/src/misc.go:1066 +0x52
github.com/odeke-em/drive/src.(*Commands).playPullChanges.func2.1(0xc42056c400, 0x5639e679fbf0, 0xc420031000)
/home/aaron/lib/golang/src/github.com/odeke-em/drive/src/pull.go:477 +0x53
github.com/odeke-em/drive/src.(*changeJobSt).changeJober.func1(0xc420d896c0, 0x5639e61f1885, 0x5639e668a860, 0xc4206ac180)
/home/aaron/lib/golang/src/github.com/odeke-em/drive/src/misc.go:120 +0xb1
github.com/odeke-em/drive/src.jobSt.Do(0x210, 0xc420b74120, 0x5639e63d819f, 0x5639e668a860, 0xc4206ac180, 0x0)
/home/aaron/lib/golang/src/github.com/odeke-em/drive/src/misc.go:100 +0x27
github.com/odeke-em/drive/src.(*jobSt).Do(0xc420b8fc50, 0x0, 0x0, 0x0, 0x0)
:6 +0x5b
github.com/odeke-em/semalim.Run.func4.1(0xc420891630, 0xc42084bf80, 0xc420891640, 0xc420891650, 0x5639e6ba98c0, 0xc420b8fc50)
/home/aaron/lib/golang/src/github.com/odeke-em/semalim/semalim.go:71 +0x56
created by github.com/odeke-em/semalim.Run.func4
/home/aaron/lib/golang/src/github.com/odeke-em/semalim/semalim.go:76 +0x141
aaron@aaron-ubunix:~/Documents/gdrive/nelaaro

I try to pull again with the same issues.
aaron@aaron-ubunix:~/Documents/gdrive/nelaaro
$ drive pull
Resolving...
panic: invalid page type: 53: 10

goroutine 1 [running]:
panic(0x559a94a0f4e0, 0xc4209a6090)
/usr/lib/go-1.7/src/runtime/panic.go:500 +0x1a1
github.com/boltdb/bolt.(*Cursor).search(0xc4200f5630, 0xc4209a6068, 0x7, 0x8, 0x35)
/home/aaron/lib/golang/src/github.com/boltdb/bolt/cursor.go:256 +0x429
github.com/boltdb/bolt.(*Cursor).seek(0xc4200f5630, 0xc4209a6068, 0x7, 0x8, 0x0, 0x0, 0xc4209a4160, 0x0, 0x0, 0x0, ...)
/home/aaron/lib/golang/src/github.com/boltdb/bolt/cursor.go:159 +0xb1
github.com/boltdb/bolt.(*Bucket).CreateBucket(0xc42036cd38, 0xc4209a6068, 0x7, 0x8, 0xc4209a606f, 0x1, 0x559a946ac000)
/home/aaron/lib/golang/src/github.com/boltdb/bolt/bucket.go:172 +0x124
github.com/boltdb/bolt.(*Bucket).CreateBucketIfNotExists(0xc42036cd38, 0xc4209a6068, 0x7, 0x8, 0x7, 0x8, 0xc4209a6068)
/home/aaron/lib/golang/src/github.com/boltdb/bolt/bucket.go:206 +0x4d
github.com/boltdb/bolt.(*Tx).CreateBucketIfNotExists(0xc42036cd20, 0xc4209a6068, 0x7, 0x8, 0x7, 0x8, 0xc4200f5798)
/home/aaron/lib/golang/src/github.com/boltdb/bolt/tx.go:115 +0x4f
github.com/odeke-em/drive/config.(*Context).CreateIndicesBucket.func1(0xc42036cd20, 0x559a94b1ea38, 0xc42036cd20)
/home/aaron/lib/golang/src/github.com/odeke-em/drive/config/config.go:247 +0x6f
github.com/boltdb/bolt.(*DB).Update(0xc4201941e0, 0x559a94b1ed80, 0x0, 0x0)
/home/aaron/lib/golang/src/github.com/boltdb/bolt/db.go:598 +0xb5
github.com/odeke-em/drive/config.(*Context).CreateIndicesBucket(0xc42018cfc0, 0x0, 0x0)
/home/aaron/lib/golang/src/github.com/odeke-em/drive/config/config.go:255 +0xaf
github.com/odeke-em/drive/config.(*Context).DeserializeIndex(0xc42018cfc0, 0xc4203f3960, 0x1c, 0x0, 0x0, 0x0)
/home/aaron/lib/golang/src/github.com/odeke-em/drive/config/config.go:115 +0x64
github.com/odeke-em/drive/src.(*Commands).deserializeIndex(0xc42018d1c0, 0xc4203f3960, 0x1c, 0xc4200f59a0)
/home/aaron/lib/golang/src/github.com/odeke-em/drive/src/push.go:243 +0x42
github.com/odeke-em/drive/src.(*Commands).(github.com/odeke-em/drive/src.deserializeIndex)-fm(0xc4203f3960, 0x1c, 0x559a94a02360)
/home/aaron/lib/golang/src/github.com/odeke-em/drive/src/changes.go:719 +0x3e
github.com/odeke-em/drive/src.resolveConflicts(0xc4209a4080, 0x3, 0x4, 0xc4204dea00, 0xc4200f5ab8, 0x6a0, 0xc4209a4080, 0x3, 0x4, 0x559a946470a5, ...)
/home/aaron/lib/golang/src/github.com/odeke-em/drive/src/changes.go:700 +0x104
github.com/odeke-em/drive/src.(*Commands).resolveConflicts(0xc42018d1c0, 0xc4204db500, 0x6a0, 0x6a0, 0x600, 0x0, 0x0)
/home/aaron/lib/golang/src/github.com/odeke-em/drive/src/changes.go:719 +0x1c0
github.com/odeke-em/drive/src.pull(0xc42018d1c0, 0x2, 0x0, 0x0)
/home/aaron/lib/golang/src/github.com/odeke-em/drive/src/pull.go:116 +0x2bb
github.com/odeke-em/drive/src.(*Commands).Pull(0xc42018d1c0, 0xc420190400, 0xc42018d1c0)
/home/aaron/lib/golang/src/github.com/odeke-em/drive/src/pull.go:71 +0x34
main.(*pullCmd).Run(0xc4202021e0, 0xc42000c2d0, 0x0, 0x0, 0xc420217530)
/home/aaron/lib/golang/src/github.com/odeke-em/drive/cmd/drive/main.go:789 +0x97d
github.com/odeke-em/command.Run()
/home/aaron/lib/golang/src/github.com/odeke-em/command/command.go:202 +0x81
github.com/odeke-em/command.ParseAndRun()
/home/aaron/lib/golang/src/github.com/odeke-em/command/command.go:209 +0x19
main.main()
/home/aaron/lib/golang/src/github.com/odeke-em/drive/cmd/drive/main.go:126 +0x1976

USING the bindary from
https://github.com/odeke-em/drive/releases/download/v0.3.9/drive_linux

aaron@aaron-ubunix:~/Documents/gdrive/nelaaro
$ ~/Downloads/drive_linux version
drive version: 0.3.9
Commit Hash: <CURRENT_COMMIT>
Go Version: <GO_VERSION>
OS: <OS_INFO>
BuildTime:

aaron@aaron-ubunix:~/Documents/gdrive/nelaaro
$ ~/Downloads/drive_linux pull
Resolving...
panic: invalid page type: 53: 10

goroutine 1 [running]:
github.com/boltdb/bolt.(*Cursor).search(0xc42008b558, 0xc4202452c0, 0x7, 0x8, 0x35)
/Users/emmanuelodeke/go/src/github.com/boltdb/bolt/cursor.go:256 +0x405
github.com/boltdb/bolt.(*Cursor).seek(0xc42008b558, 0xc4202452c0, 0x7, 0x8, 0x0, 0x0, 0xc4203e4b60, 0x0, 0x0, 0x0, ...)
/Users/emmanuelodeke/go/src/github.com/boltdb/bolt/cursor.go:159 +0xb1
github.com/boltdb/bolt.(*Bucket).CreateBucket(0xc4202b80f8, 0xc4202452c0, 0x7, 0x8, 0xc4202452c7, 0x1, 0x427900)
/Users/emmanuelodeke/go/src/github.com/boltdb/bolt/bucket.go:164 +0x124
github.com/boltdb/bolt.(*Bucket).CreateBucketIfNotExists(0xc4202b80f8, 0xc4202452c0, 0x7, 0x8, 0x7, 0x8, 0xc4202452c0)
/Users/emmanuelodeke/go/src/github.com/boltdb/bolt/bucket.go:199 +0x4d
github.com/boltdb/bolt.(*Tx).CreateBucketIfNotExists(0xc4202b80e0, 0xc4202452c0, 0x7, 0x8, 0x7, 0x8, 0xc42012e280)
/Users/emmanuelodeke/go/src/github.com/boltdb/bolt/tx.go:115 +0x4f
github.com/odeke-em/drive/config.(*Context).CreateIndicesBucket.func1(0xc4202b80e0, 0x7c8548, 0xc4202b80e0)
/Users/emmanuelodeke/go/src/github.com/odeke-em/drive/config/config.go:247 +0x6f
github.com/boltdb/bolt.(*DB).Update(0xc42014e000, 0x7c8580, 0x0, 0x0)
/Users/emmanuelodeke/go/src/github.com/boltdb/bolt/db.go:595 +0x9f
github.com/odeke-em/drive/config.(*Context).CreateIndicesBucket(0xc420016f80, 0x0, 0x0)
/Users/emmanuelodeke/go/src/github.com/odeke-em/drive/config/config.go:255 +0x95
github.com/odeke-em/drive/config.(*Context).DeserializeIndex(0xc420016f80, 0xc4202293c0, 0x1c, 0x0, 0x0, 0x0)
/Users/emmanuelodeke/go/src/github.com/odeke-em/drive/config/config.go:115 +0x64
github.com/odeke-em/drive/src.(*Commands).deserializeIndex(0xc420017100, 0xc4202293c0, 0x1c, 0x40e890)
/Users/emmanuelodeke/go/src/github.com/odeke-em/drive/src/push.go:242 +0x42
github.com/odeke-em/drive/src.(*Commands).(github.com/odeke-em/drive/src.deserializeIndex)-fm(0xc4202293c0, 0x1c, 0xc42008b948)
/Users/emmanuelodeke/go/src/github.com/odeke-em/drive/src/changes.go:719 +0x3e
github.com/odeke-em/drive/src.resolveConflicts(0xc4203e4a20, 0x3, 0x4, 0xc4204e4a00, 0xc42008ba20, 0x6a0, 0xc4203e4a20, 0x3, 0x4, 0x6a0, ...)
/Users/emmanuelodeke/go/src/github.com/odeke-em/drive/src/changes.go:700 +0x104
github.com/odeke-em/drive/src.(*Commands).resolveConflicts(0xc420017100, 0xc4204e1500, 0x6a0, 0x6a0, 0x600, 0x0, 0x0)
/Users/emmanuelodeke/go/src/github.com/odeke-em/drive/src/changes.go:719 +0x1c0
github.com/odeke-em/drive/src.pull(0xc420017100, 0x2, 0x1, 0xc4200c4600)
/Users/emmanuelodeke/go/src/github.com/odeke-em/drive/src/pull.go:116 +0x294
github.com/odeke-em/drive/src.(*Commands).Pull(0xc420017100, 0xc4200c4600, 0xc420017100)
/Users/emmanuelodeke/go/src/github.com/odeke-em/drive/src/pull.go:71 +0x34
main.(*pullCmd).Run(0xc4200fe000, 0xc42000a2f0, 0x0, 0x0, 0xc4200e3710)
/Users/emmanuelodeke/go/src/github.com/odeke-em/drive/cmd/drive/main.go:789 +0x98c
github.com/odeke-em/command.Run()
/Users/emmanuelodeke/go/src/github.com/odeke-em/command/command.go:202 +0x7a
github.com/odeke-em/command.ParseAndRun()
/Users/emmanuelodeke/go/src/github.com/odeke-em/command/command.go:209 +0x25
main.main()
/Users/emmanuelodeke/go/src/github.com/odeke-em/drive/cmd/drive/main.go:126 +0x1737

@nelaaro
Copy link

nelaaro commented Jan 16, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants