Skip to content

Commit

Permalink
cmd/benchstat: new version of benchstat
Browse files Browse the repository at this point in the history
This is a complete rewrite of benchstat. Basic usage remains the same,
as does the core idea of showing statistical benchmark summaries and
A/B comparisons in a table, but there are several major improvements.

The statistics is now more robust. Previously, benchstat used
IQR-based outlier rejection, showed the mean of the reduced sample,
its range, and did a non-parametric difference-of-distribution test on
reduced samples. Any form of outlier rejection must start with
distributional assumptions, in this case assuming normality, which is
generally not sound for benchmark data. Hence, now benchstat does not
do any outlier rejection. As a result, it must use robust summary
statistics as well, so benchstat now uses the median and confidence
interval of the median as summary statistics. Benchstat continues to
use the same Mann-Whitney U-test for the delta, but now does it on the
full samples since the U-test is already non-parametric, hence
increasing the power of this test.

As part of these statistical improvements, benchstat now detects and
warns about several common mistakes, such as having too few samples
for meaningful statistical results, or having incomparable geomeans.

The output format is more consistent. Previously, benchstat
transformed units like "ns/op" into a metric like "time/op", which it
used as a column header; and a numerator like "sec", which it used to
label each measurement. This was easy enough for the standard units
used by the testing framework, but was basically impossible to
generalize to custom units. Now, benchstat does unit scaling, but
otherwise leaves units alone. The full (scaled) unit is used as a
column header and each measurement is simply a scaled value shown with
an SI prefix. This also means that the text and CSV formats can be
much more similar while still allowing the CSV format to be usefully
machine-readable.

Benchstat will also now do A/B comparisons even if there are more than
two inputs. It shows a comparison to the base in the second and all
subsequent columns. This approach is consistent for any number of
inputs.

Benchstat now supports the full Go benchmark format, including
sophisticated control over exactly how it structures the results into
rows, columns, and tables. This makes it easy to do meaningful
comparisons across benchmark data that isn't simply structured into
two input files, and gives significantly more control over how results
are sorted. The default behavior is still to turn each input file into
a column and each benchmark into a row.

Fixes golang/go#19565 by showing all results, even if the benchmark
sets don't match across columns, and warning when geomean sets are
incompatible.

Fixes golang/go#19634 by no longer doing outlier rejection and clearly
reporting when there are not enough samples to do a meaningful
difference test.

Updates golang/go#23471 by providing more through command
documentation. I'm not sure it quite fixes this issue, but it's much
better than it was.

Fixes golang/go#30368 because benchstat now supports filter
expressions, which can also filter down units.

Fixes golang/go#33169 because benchstat now always shows file
configuration labels.

Updates golang/go#43744 by integrating unit metadata to control
statistical assumptions into the main tool that implements those
assumptions.

Fixes golang/go#48380 by introducing a way to override labels from the
command line rather than always using file names.

Change-Id: Ie2c5a12024e84b4918e483df2223eb1f10413a4f
Reviewed-on: https://go-review.googlesource.com/c/perf/+/309969
Run-TryBot: Austin Clements <[email protected]>
Reviewed-by: Michael Pratt <[email protected]>
TryBot-Result: Gopher Robot <[email protected]>
  • Loading branch information
aclements committed Jan 13, 2023
1 parent 91a0461 commit 02c5517
Show file tree
Hide file tree
Showing 64 changed files with 3,236 additions and 3,158 deletions.
13 changes: 13 additions & 0 deletions benchstat/doc.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
// Copyright 2021 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.

// Package benchstat is deprecated.
//
// This package contains the underlying implementation of an old
// version of the benchstat command.
//
// Deprecated: The latest version of benchstat can be found at
// golang.org/x/perf/cmd/benchstat. To work with benchmark data, see
// golang.org/x/perf/benchproc and golang.org/x/perf/benchmath.
package benchstat
91 changes: 0 additions & 91 deletions cmd/benchstat/README.md

This file was deleted.

106 changes: 106 additions & 0 deletions cmd/benchstat/doc_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,106 @@
// Copyright 2021 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.

package main

import (
"bytes"
"go/ast"
"go/doc"
"go/parser"
"go/token"
"os"
"regexp"
"strings"
"testing"
)

// Test that the examples in the command documentation do what they
// say.
func TestDoc(t *testing.T) {
// Read the package documentation.
fset := token.NewFileSet()
f, err := parser.ParseFile(fset, "main.go", nil, parser.ParseComments)
if err != nil {
t.Fatal(err)
}
p, err := doc.NewFromFiles(fset, []*ast.File{f}, "p")
if err != nil {
t.Fatal(err)
}
tests := parseDocTests(p.Doc)
if len(tests) == 0 {
t.Fatal("failed to parse doc tests: found 0 tests")
}

// Run the tests.
if err := os.Chdir("testdata"); err != nil {
t.Fatal(err)
}
defer os.Chdir("..")
for _, test := range tests {
var got, gotErr bytes.Buffer
t.Logf("benchstat %s", strings.Join(test.args, " "))
if err := benchstat(&got, &gotErr, test.args); err != nil {
t.Fatalf("unexpected error: %s", err)
}

// None of the doc tests should have error output.
if gotErr.Len() != 0 {
t.Errorf("unexpected stderr output:\n%s", gotErr.String())
continue
}

// Compare the output
diff(t, []byte(test.want), got.Bytes())
}
}

type docTest struct {
args []string
want string
}

var docTestRe = regexp.MustCompile(`(?m)^[ \t]+\$ benchstat (.*)\n((?:\t.*\n|\n)+)`)

func parseDocTests(doc string) []*docTest {
var tests []*docTest
for _, m := range docTestRe.FindAllStringSubmatch(doc, -1) {
want := m[2]
// Strip extra trailing newlines
want = strings.TrimRight(want, "\n") + "\n"
// Strip \t at the beginning of each line
want = strings.Replace(want[1:], "\n\t", "\n", -1)
tests = append(tests, &docTest{
args: parseArgs(m[1]),
want: want,
})
}
return tests
}

// parseArgs is a very basic parser for shell-like word lists.
func parseArgs(x string) []string {
// TODO: Use strings.Cut
var out []string
x = strings.Trim(x, " ")
for len(x) > 0 {
if x[0] == '"' {
x = x[1:]
i := strings.Index(x, "\"")
if i < 0 {
panic("missing \"")
}
out = append(out, x[:i])
x = strings.TrimLeft(x[i+1:], " ")
} else if i := strings.Index(x, " "); i < 0 {
out = append(out, x)
x = ""
} else {
out = append(out, x[:i])
x = strings.TrimLeft(x[i+1:], " ")
}
}
return out
}
Loading

0 comments on commit 02c5517

Please sign in to comment.