Serialize using mapstructure conversion instead of JSON marshalling.

[slewsys]

Merge of Alex Castle's (acastle) PR #1009 against HEAD of branch master.

To make concrete the errors that this PR fixes, in a Linux terminal, install age:

go install filippo.io/age/cmd/age@latest
go install filippo.io/age/cmd/age-keygen@latest

Generate an age key:

install -d -m 0700 $HOME/.config/sops/age
export SOPS_AGE_KEY_FILE=$HOME/.config/sops/age/keys.txt
age-keygen -o  "$SOPS_AGE_KEY_FILE"
export SOPS_AGE_RECIPIENTS=$(age-keygen -y "$SOPS_AGE_KEY_FILE")

Generate an age-encrypted secret:

echo 'secret: abc' >o.yml
sops -e o.yml >o.enc.yml

Use the secret in a script:

sops exec-env o.enc.yml 'echo $secret'
# => abc

So far, so good. Now, try a numeric secret:

echo 'secret: 123' >o.yml
sops -e o.yml >o.enc.yml
sops -d o.enc.yml
# => secret: 123

Decryption still works, but not in a script context:

sops exec-env o.enc.yml 'echo $secret'

panic: interface conversion: interface {} is int, not string

goroutine 1 [running]:
go.mozilla.org/sops/v3/stores/dotenv.(*Store).EmitPlainFile(0xc0004c4000, {0xc0001234d0, 0x1, 0x1})
${GOPATH}/pkg/mod/go.mozilla.org/sops/[email protected]/stores/dotenv/store.go:122 +0x305
main.decrypt({{0x102bc30, 0xc000422db0}, {0x7ffb607bca10, 0x165b608}, {0x1038c98, 0x165b608}, {0x7ffff43fc8aa, 0x9}, 0x0, {0x0, ...}, ...})
${GOPATH}/pkg/mod/go.mozilla.org/sops/[email protected]/cmd/sops/decrypt.go:47 +0x23f
main.main.func1(0xc00018cb00)
${GOPATH}/pkg/mod/go.mozilla.org/sops/[email protected]/cmd/sops/main.go:156 +0x258
gopkg.in/urfave/cli%2ev1.HandleAction({0xd0d640, 0xea99a8}, 0x8)
${GOPATH}/pkg/mod/gopkg.in/urfave/[email protected]/app.go:490 +0x5a
gopkg.in/urfave/cli%2ev1.Command.Run({{0xe5d7f8, 0x8}, {0x0, 0x0}, {0x0, 0x0, 0x0}, {0xe94cd7, 0x45}, {0x0, ...}, ...}, ...)
${GOPATH}/pkg/mod/gopkg.in/urfave/[email protected]/command.go:210 +0x8f8
gopkg.in/urfave/cli%2ev1.(*App).Run(0xc0003cd040, {0xc00012e000, 0x4, 0x4})
${GOPATH}/pkg/mod/gopkg.in/urfave/[email protected]/app.go:255 +0x6ac
main.main()
${GOPATH}/pkg/mod/go.mozilla.org/sops/[email protected]/cmd/sops/main.go:987 +0x3699

This error occurs for both go v1.17 and v1.18.

Per the original PR:

These changes remove the json encoding/decoding steps that are
performed during the serialization of ini and dotenv files. This
roundtrip loses type information during the transformation which
causes values to be incorrectly converted to the json marshaller
defaults (int becomes float64, bool becomes string, etc, etc). In
place of this json encoding the mapstructure library allows for a
direct conversion between the Metadata struct and
map[string]interface{} needed to leverage the stores.Flatten and
stores.Unflatten functions.

In addition this adds mapstructure tags to the metadata structures to
allow backwards compatibility with the json encoding.

Resolves #879 & #857

[felixfontein]

develop will be merged into master on release. There's no need to create PRs to add individual commits from develop to master.

[slewsys]

Sorry about the noise. Just wanted to wave my hands and give this PR a thumbs up :)