Continuous fuzzing on Fuzzit #432

bookmoons · 2019-08-29T07:10:11Z

Integrates with Fuzzit for automated bug discovery. Initially discussed in #429.

I've tried to fuzz everything that does some parsing. The main target AttackerHTTP decodes fuzz into a valid HTTP response. It stubs the targeter and communicates through a socket file.

Other targets are:

AttackerTCP - Fuzz response handling with unstructured byte stream.
HTTPTargeter - Fuzz decoding of a target list in HTTP format.
JSONTargeter - Fuzz decoding of a target list in JSON format.
ResultsFormatDetection - Fuzz result list format detection.
GobDecoder - Fuzz decoding of a result list in gob format.
CSVDecoder - Fuzz decoding of a result list in CSV format.
JSONDecoder - Fuzz decoding of a result list in JSON format.

The format detection fuzzer caught one crash in a local run. I'm attaching the data here. The unsuffixed file contains the crashing input. Seems to have run out of memory, maybe a memory leak somewhere.

crashers.FormatDetection.zip

There's a vegeta org on Fuzzit all preconfigured and ready to go. If you sign in to set up an account I can ask them to add you to the org. There's a little setup to do:

In Fuzzit settings grab an API Key.
In repo settings in Travis paste it to envvar FUZZIT_API_KEY.

Including the "fuzz target parser" issue in the close list because I think this patch fulfills it.

Closes #429. Closes #140.

Background

Enables automated fuzzing. Proposed in #429.

Checklist

Git commit messages conform to community standards.
Each Git commit represents meaningful milestones or atomic units of work.
Changed or added code is covered by appropriate tests.

bookmoons · 2019-08-29T22:54:55Z

Just pushed an improvement to this. I got it decoding the fuzz into a more meaningful HTTP response, including headers. I think it will give better coverage.

Also split out raw byte stream fuzzing into AttackerTCP. Most app code must be behind HTTP parsing but if it does crash on a malformed response you probably want to know that.

tsenart

Great work, thank you for this 🙇

The format detection fuzzer caught one crash in a local run. I'm attaching the data here.

Will look into this. Is the best way to reproduce this to just run the fuzz tests again?

If you sign in to set up an account I can ask them to add you to the org.

I have signed up. Will wait to be added to the org before doing the rest of the setup.

lib/attack_fuzz.go

lib/targets_fuzz.go

tsenart · 2019-09-01T14:02:46Z

I am not able to reproduce the crash with this code.

package main

import (
	"os"

	vegeta "github.com/tsenart/vegeta/lib"
)

func main() {
	f, err := os.Open("2d8aaa305cd7b972396ef7c69da1f59204b0a255")
	if err != nil {
		panic(err)
	}
	defer f.Close()
	vegeta.DecoderFor(f)
}

lib/results_fuzz.go

bookmoons · 2019-09-01T19:39:08Z

Thanks for looking at it! Will go over all of this soon.

bookmoons · 2019-09-02T22:45:47Z

I think you should be added to the org.

bookmoons · 2019-09-02T23:12:40Z

I am not able to reproduce the crash with this code.

Hm. It crashes for me. Maybe it just takes more memory than my system has.

The same input crashes if I read it with a gob decoder. Same error. It doesn't crash until reading. No crash with CSV or JSON.

package main

import (
	"io"
	"os"

	vegeta "github.com/tsenart/vegeta/lib"
)

func main() {
	f, err := os.Open("2d8aaa305cd7b972396ef7c69da1f59204b0a255")
	if err != nil {
		panic(err)
	}
	defer f.Close()
	decoder := vegeta.NewDecoder(f)
	readAllResults(decoder)
}

func readAllResults(decoder vegeta.Decoder) (ok bool) {
	for {
		result := &vegeta.Result{}
		err := decoder.Decode(result)
		if err == io.EOF {
			return true
		} else if err != nil {
			return false
		}
	}
}

fatal error: runtime: out of memory

bookmoons · 2019-09-02T23:18:46Z

There's the hex of that crashing input.

fce4303030

Enables automated fuzzing on continuous fuzzing platform Fuzzit. Fuzz regression tests run every build and PR. Full length fuzzing runs every push to master. Targets everything that does some parsing. Defined fuzzing targets are: * AttackerHTTP - Delivers fuzz as an HTTP response. Uses socket file. * AttackerTCP - Delivers fuzz as a TCP byte stream. Uses socket file. * HTTPTargeter - Fuzz decoding of a target list in HTTP format. * JSONTargeter - Fuzz decoding of a target list in JSON format. * ResultsFormatDetection - Fuzz result list format detection. * GobDecoder - Fuzz decoder of a result list in gob format. * CSVDecoder - Fuzz decoding of a result list in CSV format. * JSONDecoder - Fuzz decoding a result list in JSON format.

bookmoons · 2019-09-03T00:26:03Z

I think everything here is addressed.

tsenart · 2019-09-03T07:12:35Z

Thanks for addressing all the feedback!

Hm. It crashes for me. Maybe it just takes more memory than my system has.
The same input crashes if I read it with a gob decoder. Same error. It doesn't crash until reading. No crash with CSV or JSON.

I find this very strange. I just compiled and ran the code you posted above and the maximum resident set size is 131 MB. I'd like to solve this mystery before merging this in :-)

time -lp ./test
real         0.06
user         0.03
sys          0.02
 131784704  maximum resident set size
         0  average shared memory size
         0  average unshared data size
         0  average unshared stack size
     32220  page reclaims
         0  page faults
         0  swaps
         0  block input operations
         0  block output operations
         0  messages sent
         0  messages received
         0  signals received
         0  voluntary context switches
       249  involuntary context switches

bookmoons · 2019-09-04T04:46:54Z

I'll do some investigation. There's what I get running with time. Seems like it's ~6MB memory usage before it crashes.

[user@bookmoons vegeta]$ /usr/bin/time -vp ./test
fatal error: runtime: out of memory

runtime stack:
runtime.throw(0x70267c, 0x16)
	/usr/share/go/src/runtime/panic.go:617 +0x72
runtime.sysMap(0xc004000000, 0xe8000000, 0x9702f8)
	/usr/share/go/src/runtime/mem_linux.go:170 +0xc7
runtime.(*mheap).sysAlloc(0x957e80, 0xe4304000, 0x957e90, 0x72182)
	/usr/share/go/src/runtime/malloc.go:633 +0x1cd
runtime.(*mheap).grow(0x957e80, 0x72182, 0x0)
	/usr/share/go/src/runtime/mheap.go:1222 +0x42
runtime.(*mheap).allocSpanLocked(0x957e80, 0x72182, 0x970308, 0x957e88)
	/usr/share/go/src/runtime/mheap.go:1150 +0x37f
runtime.(*mheap).alloc_m(0x957e80, 0x72182, 0x746abed50101, 0x746abed5fd98)
	/usr/share/go/src/runtime/mheap.go:977 +0xc2
runtime.(*mheap).alloc.func1()
	/usr/share/go/src/runtime/mheap.go:1048 +0x4c
runtime.(*mheap).alloc(0x957e80, 0x72182, 0x10101, 0x746abed5fd08)
	/usr/share/go/src/runtime/mheap.go:1047 +0x8a
runtime.largeAlloc(0xe4303030, 0x970101, 0x746abed5fd08)
	/usr/share/go/src/runtime/malloc.go:1055 +0x99
runtime.mallocgc.func1()
	/usr/share/go/src/runtime/malloc.go:950 +0x46
runtime.systemstack(0x4567d9)
	/usr/share/go/src/runtime/asm_amd64.s:351 +0x66
runtime.mstart()
	/usr/share/go/src/runtime/proc.go:1153

goroutine 1 [running]:
runtime.systemstack_switch()
	/usr/share/go/src/runtime/asm_amd64.s:311 fp=0xc0000b9c88 sp=0xc0000b9c80 pc=0x4568d0
runtime.mallocgc(0xe4303030, 0x694420, 0xc0000b9d01, 0x638363)
	/usr/share/go/src/runtime/malloc.go:949 +0x872 fp=0xc0000b9d28 sp=0xc0000b9c88 pc=0x40d3e2
runtime.makeslice(0x694420, 0xe4303030, 0xe4303030, 0x4)
	/usr/share/go/src/runtime/slice.go:49 +0x6c fp=0xc0000b9d58 sp=0xc0000b9d28 pc=0x44228c
encoding/gob.(*decBuffer).Size(...)
	/usr/share/go/src/encoding/gob/decode.go:65
encoding/gob.(*Decoder).readMessage(0xc0000fe080, 0xe4303030)
	/usr/share/go/src/encoding/gob/decoder.go:101 +0x16e fp=0xc0000b9db8 sp=0xc0000b9d58 pc=0x6417ce
encoding/gob.(*Decoder).recvMessage(0xc0000fe080, 0xc0000b9e38)
	/usr/share/go/src/encoding/gob/decoder.go:90 +0xfe fp=0xc0000b9e10 sp=0xc0000b9db8 pc=0x64162e
encoding/gob.(*Decoder).decodeTypeSequence(0xc0000fe080, 0x712e00, 0xc0000fe080)
	/usr/share/go/src/encoding/gob/decoder.go:143 +0x12c fp=0xc0000b9e30 sp=0xc0000b9e10 pc=0x641afc
encoding/gob.(*Decoder).DecodeValue(0xc0000fe080, 0x6ad380, 0xc0000fe100, 0x16, 0x0, 0x0)
	/usr/share/go/src/encoding/gob/decoder.go:211 +0xf6 fp=0xc0000b9e70 sp=0xc0000b9e30 pc=0x641e46
encoding/gob.(*Decoder).Decode(0xc0000fe080, 0x6ad380, 0xc0000fe100, 0x6b8a01, 0xc0000fe100)
	/usr/share/go/src/encoding/gob/decoder.go:188 +0x191 fp=0xc0000b9ed8 sp=0xc0000b9e70 pc=0x641cb1
github.com/tsenart/vegeta/lib.NewDecoder.func1(0xc0000fe100, 0xc0000fe100, 0xc00007de50)
	/home/user/go/src/github.com/tsenart/vegeta/lib/results.go:116 +0x40 fp=0xc0000b9f10 sp=0xc0000b9ed8 pc=0x664bd0
github.com/tsenart/vegeta/lib.Decoder.Decode(...)
	/home/user/go/src/github.com/tsenart/vegeta/lib/results.go:121
main.readAllResults(0xc00007de50, 0xc000088528)
	/home/user/go/src/github.com/tsenart/vegeta/test.go:23 +0x45 fp=0xc0000b9f48 sp=0xc0000b9f10 pc=0x665025
main.main()
	/home/user/go/src/github.com/tsenart/vegeta/test.go:17 +0xae fp=0xc0000b9f98 sp=0xc0000b9f48 pc=0x664f9e
runtime.main()
	/usr/share/go/src/runtime/proc.go:200 +0x20c fp=0xc0000b9fe0 sp=0xc0000b9f98 pc=0x42eb2c
runtime.goexit()
	/usr/share/go/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc0000b9fe8 sp=0xc0000b9fe0 pc=0x4589a1
Command exited with non-zero status 2
	Command being timed: "./test"
	User time (seconds): 0.00
	System time (seconds): 0.00
	Percent of CPU this job got: 76%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.01
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 6008
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 0
	Minor (reclaiming a frame) page faults: 376
	Voluntary context switches: 50
	Involuntary context switches: 66
	Swaps: 0
	File system inputs: 0
	File system outputs: 0
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 2

bookmoons · 2019-09-04T04:47:46Z

There's a reduced test case without the file read. Same crash.

package main

import (
	"bytes"
	"io"

	vegeta "github.com/tsenart/vegeta/lib"
)

func main() {
	input := []byte{0xfc, 0xe4, 0x30, 0x30, 0x30}
	decoder := vegeta.NewDecoder(bytes.NewReader(input))
	readAllResults(decoder)
}

func readAllResults(decoder vegeta.Decoder) (ok bool) {
	for {
		result := &vegeta.Result{}
		err := decoder.Decode(result)
		if err == io.EOF {
			return true
		} else if err != nil {
			return false
		}
	}
}

bookmoons · 2019-09-04T04:53:04Z

What's your environment like? I'm running Fedora and Go 1.12.

My Fedora version is old. Maybe the system has a bug.

[user@bookmoons vegeta]$ cat /etc/fedora-release
Fedora release 28 (Twenty Eight)
[user@bookmoons vegeta]$ go version
go version go1.12.5 linux/amd64

bookmoons · 2019-09-04T05:54:43Z

Tried this with Go 1.12.9 and 1.13. Same crash on both.

bookmoons · 2019-09-04T06:01:56Z

Removing anything from the input seems to prevent the crash. Adding to the beginning also prevents it. Adding to the end preserves.

Crashing inputs:

fce4303030
fce430303030
fce43030303030

Noncrashing inputs:

fce43030
e4303030
fc303030
30fce4303030

bookmoons · 2019-09-04T07:05:22Z

I think what's happening here is the gob decoder allocates a large amount then doesn't use it, so it never ends up in that resident set size. Your machine probably has enough to handle the allocation so it's accepted.

encoding/gob starts reading a message:

runtime.makeslice(0x694420, 0xe4303030, 0xe4303030, 0x4)
	/usr/share/go/src/runtime/slice.go:49 +0x6c fp=0xc0000b9d58 sp=0xc0000b9d28 pc=0x44228c
encoding/gob.(*decBuffer).Size(...)
	/usr/share/go/src/encoding/gob/decode.go:65
encoding/gob.(*Decoder).readMessage(0xc0000fe080, 0xe4303030)
	/usr/share/go/src/encoding/gob/decoder.go:101 +0x16e fp=0xc0000b9db8 sp=0xc0000b9d58 pc=0x6417ce

I see in the gob encoding a byte slice is sent as length followed by bytes. The length is prefixed with a byte count, encoded as a negated single byte. Ours is 0xfc which decodes to 4.

makeslice() is called at the top of this stack trace. The 2nd argument is slice length, number of elements. The code is here:
https://golang.org/src/runtime/slice.go

It shows up as 0xe4303030, right from our fuzz. That comes to 3828363312 decimal, 3.8 billion items. For 1 byte elements that's ~3.5GB memory.

That's the end of the fuzz so it just allocates then releases. But I guess if my system doesn't have that available it refuses to allocate.

tsenart · 2019-09-04T08:38:27Z

Thanks for the investigation!. I think there's nothing left to do here.

yevgenypats · 2019-09-04T08:49:56Z

Hooray:) feel free to RT https://twitter.com/fuzzitdev/status/1169170601623379968

bookmoons · 2019-09-04T10:10:37Z

Thanks a lot @tsenart. I think you can remove me from both orgs in Settings, then they're all yours.

tsenart · 2019-10-27T13:33:16Z

@bookmoons: I'm migrating to Github Actions for CI (away from Travis). Hitting this issue with our Fuzzit steps: https://github.com/tsenart/vegeta/pull/456/checks?check_run_id=276595658

Any clue how to fix this?

bookmoons requested review from tsenart and xla as code owners August 29, 2019 07:10

bookmoons force-pushed the master branch 6 times, most recently from 85c6651 to 1769825 Compare August 29, 2019 22:42

tsenart previously approved these changes Sep 1, 2019

View reviewed changes

lib/attack_fuzz.go Show resolved Hide resolved

lib/targets_fuzz.go Outdated Show resolved Hide resolved

tsenart reviewed Sep 1, 2019

View reviewed changes

lib/results_fuzz.go Show resolved Hide resolved

tsenart reviewed Sep 1, 2019

View reviewed changes

lib/results_fuzz.go Outdated Show resolved Hide resolved

bookmoons dismissed tsenart’s stale review via e5e7d57 September 2, 2019 22:52

bookmoons force-pushed the master branch from e5e7d57 to f01f8cf Compare September 2, 2019 23:02

bookmoons force-pushed the master branch from f01f8cf to 524fc75 Compare September 2, 2019 23:13

bookmoons force-pushed the master branch 2 times, most recently from 2c0371a to dfd8adc Compare September 3, 2019 00:01

bookmoons force-pushed the master branch from dfd8adc to 13242e1 Compare September 3, 2019 00:11

tsenart approved these changes Sep 3, 2019

View reviewed changes

tsenart merged commit 0dfc10f into tsenart:master Sep 4, 2019

bookmoons mentioned this pull request Sep 7, 2019

Suggestion: Continuous Fuzzing mna/pigeon#89

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Continuous fuzzing on Fuzzit #432

Continuous fuzzing on Fuzzit #432

bookmoons commented Aug 29, 2019 •

edited

Loading

bookmoons commented Aug 29, 2019

tsenart left a comment

tsenart commented Sep 1, 2019

bookmoons commented Sep 1, 2019

bookmoons commented Sep 2, 2019

bookmoons commented Sep 2, 2019

bookmoons commented Sep 2, 2019

bookmoons commented Sep 3, 2019

tsenart commented Sep 3, 2019

bookmoons commented Sep 4, 2019

bookmoons commented Sep 4, 2019

bookmoons commented Sep 4, 2019

bookmoons commented Sep 4, 2019

bookmoons commented Sep 4, 2019

bookmoons commented Sep 4, 2019 •

edited

Loading

tsenart commented Sep 4, 2019

yevgenypats commented Sep 4, 2019

bookmoons commented Sep 4, 2019

tsenart commented Oct 27, 2019

Continuous fuzzing on Fuzzit #432

Continuous fuzzing on Fuzzit #432

Conversation

bookmoons commented Aug 29, 2019 • edited Loading

Background

Checklist

bookmoons commented Aug 29, 2019

tsenart left a comment

Choose a reason for hiding this comment

tsenart commented Sep 1, 2019

bookmoons commented Sep 1, 2019

bookmoons commented Sep 2, 2019

bookmoons commented Sep 2, 2019

bookmoons commented Sep 2, 2019

bookmoons commented Sep 3, 2019

tsenart commented Sep 3, 2019

bookmoons commented Sep 4, 2019

bookmoons commented Sep 4, 2019

bookmoons commented Sep 4, 2019

bookmoons commented Sep 4, 2019

bookmoons commented Sep 4, 2019

bookmoons commented Sep 4, 2019 • edited Loading

tsenart commented Sep 4, 2019

yevgenypats commented Sep 4, 2019

bookmoons commented Sep 4, 2019

tsenart commented Oct 27, 2019

bookmoons commented Aug 29, 2019 •

edited

Loading

bookmoons commented Sep 4, 2019 •

edited

Loading