net/mail: characters allowed in RFC 5322 are invalid while parsing email header

[iredmail]

What version of Go are you using (go version)?

$ go version
go version go1.20.1 darwin/amd64

Does this issue reproduce with the latest release?

Yes.

What did you do?

Create an empty directory and save code below in main.go, run go mod tidy && go run ..
Note: it contains header Custom/Header: v.

package main

import (
	"log"
	"net/mail"
	"strings"
)

func main() {
	msg := `Date: Mon, 23 Jun 2015 11:40:36 -0400
From: Gopher <[email protected]>
To: Another Gopher <[email protected]>
Subject: Gophers at Gophercon
Custom/Header: v

Message body
`

	r := strings.NewReader(msg)
	m, err := mail.ReadMessage(r)
	if err != nil {
		log.Fatal(err)
	}

	println(m.Header.Get("Custom/Header"))
}

What did you expect to see?

Print value of email header Custom/Header: v

What did you see instead?

2023/03/04 20:57:52 malformed MIME header line: Custom/Header: v
exit status 1

Function mail.ReadMessage() calls textproto.ReadMIMEHeader(), it calls internal function validHeaderValueByte() (in net/textproto) to validate characters in email header field, / in header field is considered as invalid.

The problem is, validHeaderValueByte() is designed to validate characters in HTTP header, not email header.

According to RFC 5322 "Internet Message Format", section "2.2 Header fields":

   A field name MUST be composed of printable US-ASCII characters (i.e.,
   characters that have values between 33 and 126, inclusive), except
   colon.

Here's full list of printable US-ASCII characters:
https://www.ascii-code.com/characters/printable-characters

For example, characters /, *, [, ] are allowed in email header field according to RFC doc, but it won't pass current validHeaderFieldByte() because it's disallowed in http header field. This causes parse error as shown in above sample code. I believe there're some similar issues caused by validHeaderValueByte() too.

We may need a new exported function like ReadMIMEHeader but for email (maybe name it as ReadEmailMIMEHeader()), and allow characters as defined in RFC 5322, but other code in net/textproto may need some changes too.

[bcmills]

https://dev.golang.org/owners lists @bradfitz as the maybe-owner of net/mail?

[mengzhuo]

Duplicate #39591

[iredmail]

Duplicate #39591

Not a duplicate. This issue is for email header field, but #39591 is for header value.

[180909]

It's working on version 1.19

[iredmail]

It's working on version 1.19

Yes it works in Go v1.19, but changed in v1.20 due to the fix in this issue: #53188

Again, net/textproto is (mostly) designed for http, not for email.

[seankhliao]

cc @neild

[mediumdaver]

The problem code in net/mail/message.go is:

    53  func ReadMessage(r io.Reader) (msg *Message, err error) {
    54  	tp := textproto.NewReader(bufio.NewReader(r))
    55  
    56  	hdr, err := tp.ReadMIMEHeader()
    57  	if err != nil {
    58  		return nil, err
    59  	}
    60  
    61  	return &Message{
    62  		Header: Header(hdr),
    63  		Body:   tp.R,
    64  	}, nil
    65  }

Using textproto.ReadMIMEHeader() is incorrect, as a valid header in textproto is, per the comment in that code:

   655  // validHeaderFieldByte reports whether c is a valid byte in a header
   656  // field name. RFC 7230 says:
   657  //
   658  //	header-field   = field-name ":" OWS field-value OWS
   659  //	field-name     = token
   660  //	tchar = "!" / "#" / "$" / "%" / "&" / "'" / "*" / "+" / "-" / "." /
   661  //	        "^" / "_" / "`" / "|" / "~" / DIGIT / ALPHA
   662  //	token = 1*tchar

which is incorrect for a mail header, RFC 5322 says a valid header name is:

Header fields are lines beginning with a field name, followed by a
colon (":"), followed by a field body, and terminated by CRLF.  A
field name MUST be composed of printable US-ASCII characters (i.e.,
characters that have values between 33 and 126, inclusive), except
colon.

Can someone please fix it? ....please....

[ianlancetaylor]

@gopherbot Please open backport to 1.20.

This stopped working due to the changes in #53188, but those changes don't apply to net/mail.

[gopherbot]

Change https://go.dev/cl/504416 mentions this issue: net/mail: permit more characters in mail headers

[gopherbot]

Backport issue(s) opened: #60875 (for 1.20).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://go.dev/wiki/MinorReleases.

[gopherbot]

Change https://go.dev/cl/504881 mentions this issue: [release-branch.go1.20] net/mail: permit more characters in mail headers