net/mail: mail.ReadMessage in 1.20 cannot parse mbox headers

[nferch]

What version of Go are you using (go version)?

1.20.3

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""    
GOARCH="amd64"                                                                                                                     
GOBIN=""
GOCACHE="/home/nf/.cache/go-build"
GOENV="/home/nf/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/nf/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/nf/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/lib/go-1.20"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/go-1.20/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.20.3"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/nf/Projects/mutt-save-hook/go.mod"
GOWORK=""
CGO_CFLAGS="-O2 -g"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-O2 -g"
CGO_FFLAGS="-O2 -g"
CGO_LDFLAGS="-O2 -g"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build3946607393=/tmp/go-build -gno-record-gcc-switches"

What did you do?

A change introduced in Go 1.20 prevents mail.ReadMessage from being able to handle messages preceded by a mbox header. A mbox header consists of a From header (no colon) preceding the message content https://www.loc.gov/preservation/digital/formats/fdd/fdd000383.shtml

This prevents mail.ReadMessage from being able to read single-message mbox files, or messages piped by MUAs like Mutt.

To reproduce: https://go.dev/play/p/lgqlLBsWBX8?v=goprev

What did you expect to see?

I did not expect my code to behave differently with a newer version of Go.

What did you see instead?

Due to a change in Go (presumably https://cs.opensource.google/go/go/+/42137704fca158f14c4aebff61b54ebeae788a4a), my Go program started reporting an error given identical input.

I believe that the prior behavior of ignoring mbox headers, even if unintended, should be restored in order to prevent programs that depended on it from breaking, to maintain its compatibility with previous versions, and to try and conform to the Robustness principle by handling a common malformed input.

[seankhliao]

I believe this is from the stricter textproto parser introduced in #53188
related? #58862
https://go.dev/play/p/qNVg2bZCk0K

(side note, the robustness principle is a poor argument)

cc @neild

[Ikke]

I believe this is from the stricter textproto parser introduced in #53188

I can confirm. Debugging shows that due to the lack of a ":", the entire line is interpreted as a header field name, and so fails because it contains invalid header field characters.

go/src/net/textproto/reader.go

Line 727 in dbf9bf2

func canonicalMIMEHeaderKey(a []byte) (_ string, ok bool) {

[ianlancetaylor]

@gopherbot Please open backport to 1.20.

This stopped working due to the changes in #53188, but those changes don't apply to net/mail.

[gopherbot]

Backport issue(s) opened: #60874 (for 1.20).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://go.dev/wiki/MinorReleases.

[gopherbot]

Change https://go.dev/cl/504416 mentions this issue: net/mail: permit more characters in mail headers

[gopherbot]

Change https://go.dev/cl/504881 mentions this issue: [release-branch.go1.20] net/mail: permit more characters in mail headers