P-Tree mutatation operations must hand in ValueReader (if any) to sequenceChunker.Done()

[ahl]

This is reproducible but requires this dataset http://demo.noms.io/ahl::bug2213 and my fuse branch https://github.com/ahl/noms/tree/nomsfs as well as fuse (e.g. osxfuse).

removing b from the root of that filesystem causes the following panic:

panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x20 pc=0xecc52]

goroutine 90 [running]:
panic(0x5ba720, 0xc82000e0d0)
    /opt/local/lib/go/src/runtime/panic.go:481 +0x3e6
github.com/attic-labs/noms/go/types.Ref.TargetValue(0xf34774abb3558a72, 0x9165eca6384a2e92, 0xff3545c6, 0x5, 0xc8281b24c0, 0xc82829d100, 0x0, 0x0, 0x0, 0x0)
    /Users/ahl/gosrc/src/github.com/attic-labs/noms/go/types/ref.go:50 +0x42
github.com/attic-labs/noms/go/types.metaTuple.getChildSequence(0xf34774abb3558a72, 0x9165eca6384a2e92, 0xff3545c6, 0x5, 0xc8281b24c0, 0xc82829d100, 0x1, 0xe8ad00, 0xc82000e3d0, 0x0, ...)
    /Users/ahl/gosrc/src/github.com/attic-labs/noms/go/types/meta_sequence.go:45 +0xb2
github.com/attic-labs/noms/go/types.(*sequenceChunker).Done(0xc82006aa10, 0x0, 0x0, 0x0, 0x0)
    /Users/ahl/gosrc/src/github.com/attic-labs/noms/go/types/sequence_chunker.go:242 +0x360
github.com/attic-labs/noms/go/types.(*sequenceChunker).Done(0xc82006a9a0, 0x0, 0x0, 0x0, 0x0)
    /Users/ahl/gosrc/src/github.com/attic-labs/noms/go/types/sequence_chunker.go:223 +0x150
github.com/attic-labs/noms/go/types.Map.splice(0xe8b980, 0xc8281b2840, 0xc82829d160, 0xc8281d4a60, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /Users/ahl/gosrc/src/github.com/attic-labs/noms/go/types/map.go:177 +0x2b7
github.com/attic-labs/noms/go/types.Map.Remove(0xe8b980, 0xc8281b2840, 0xc82829d160, 0xe8ad00, 0xc828294800, 0x0, 0x0, 0x0)
    /Users/ahl/gosrc/src/github.com/attic-labs/noms/go/types/map.go:162 +0xba
main.(*nomsFS).removeCommon(0xc82010cc40, 0xc82829452b, 0x1, 0x792218, 0x0)
    /Users/ahl/src/noms/samples/go/nomsfs/nomsfs.go:379 +0x391
main.(*nomsFS).Unlink(0xc82010cc40, 0xc82829452b, 0x1, 0xc8200f01c8, 0x1)
    /Users/ahl/src/noms/samples/go/nomsfs/nomsfs.go:355 +0x41
github.com/attic-labs/noms/vendor/github.com/attic-labs/go-fuse/fuse/pathfs.(*pathInode).Unlink(0xc82829e060, 0xc82829452b, 0x1, 0xc8200f01c8, 0xc8200b6ac8)
    /Users/ahl/gosrc/src/github.com/attic-labs/noms/vendor/github.com/attic-labs/go-fuse/fuse/pathfs/pathfs.go:436 +0x13e
github.com/attic-labs/noms/vendor/github.com/attic-labs/go-fuse/fuse/nodefs.(*rawBridge).Unlink(0xc82829e0f0, 0xc8200f01b0, 0xc82829452b, 0x1, 0x0)
    /Users/ahl/gosrc/src/github.com/attic-labs/noms/vendor/github.com/attic-labs/go-fuse/fuse/nodefs/fsops.go:300 +0x89
github.com/attic-labs/noms/vendor/github.com/attic-labs/go-fuse/fuse.doUnlink(0xc8200aa000, 0xc8200f0000)
    /Users/ahl/gosrc/src/github.com/attic-labs/noms/vendor/github.com/attic-labs/go-fuse/fuse/opcode.go:295 +0x86
github.com/attic-labs/noms/vendor/github.com/attic-labs/go-fuse/fuse.(*Server).handleRequest(0xc8200aa000, 0xc8200f0000, 0xc8200aa0c8)
    /Users/ahl/gosrc/src/github.com/attic-labs/noms/vendor/github.com/attic-labs/go-fuse/fuse/server.go:394 +0x2ff
created by github.com/attic-labs/noms/vendor/github.com/attic-labs/go-fuse/fuse.(*Server).loop
    /Users/ahl/gosrc/src/github.com/attic-labs/noms/vendor/github.com/attic-labs/go-fuse/fuse/server.go:371 +0xc9
exit status 2

The relevant call chain is Map.Remove() -> Map.splice() -> sequenceChunker.Done() and the vr ValueReader parameter to Done() is nil. We crash in the recursive call to sequenceChunker.Done() at line 241 since vr is nil.

[ahl]

@rafael-atticlabs you might want to take a look

[ghost]

looking now.

[ghost]

So far I'm failing to get nomsfs working. I get "Mount failed: no FUSE devices found". I've tried installing the osxfuse installer (https://osxfuse.github.io/) (both 2.8.3 & 3.4.1) as well as homebrew osxfuse-beta. I must be missing some simple step. Any idea, ahl?

[ahl]

I'll be in tomorrow morning. Let's look at it then? Miraculously this fuse stuff has basically just worked for me to this point.

On Aug 1, 2016, at 5:11 PM, Rafael Weinstein [email protected] wrote:

So far I'm failing to get nomsfs working. I get "Mount failed: no FUSE devices found". I've tried installing the osxfuse installer (https://osxfuse.github.io/) (both 2.8.3 & 3.4.1) as well as homebrew osxfuse-beta. I must be missing some simple step. Any idea, ahl?

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.

[ghost]

sgtm.

[ghost]

The issue here is that the chunking code was errorenously only handling two cases: (1) when a p-tree is constructed and eagerly written to a ValueStore and (2) when constructed in memory.

The third case of being mutated and possibly not entirely loaded into memory isn't correctly handled.

In List/Set/Map/Blob, the sequenceChunker which is created must be handed any existing ValueReader available on the root sequence into Done(). Not that the following assert is wrong and should be removed: https://github.com/attic-labs/noms/blob/master/go/types/sequence_chunker.go#L205

I haven't tried it but I'm fairly sure that the following test will currently fail and doing the above should fix:

1. Create a "reasonably sized" prolly tree
2. Write it to a MemoryStore
3. Read the root of the collection back out via it's Ref
4. Iteratively remove the "last" value in the collection until it has nothing left

[ghost]

@mikegray, @kalman are you either of you guys interested in taking this one? go+js.