Vault CLI support for HCL input throughout by converting to JSON

[jgiles]

Feature Request: Vault CLI support for HCL input throughout by converting to JSON

Environment:

• Vault Version: Vault v0.9.1 ('87b6919dea55da61d7cd444b2442cabb8ede8ab1')
• Operating System/Architecture: OSX

Vault Config File:
Running -dev with default config.

Startup Log Output:

$ VAULT_DEV_ROOT_TOKEN_ID=test-root-token vault server -dev
==> Vault server configuration:

                     Cgo: disabled
         Cluster Address: https://127.0.0.1:8201
              Listener 1: tcp (addr: "127.0.0.1:8200", cluster address: "127.0.0.1:8201", tls: "disabled")
               Log Level:
                   Mlock: supported: false, enabled: false
        Redirect Address: http://127.0.0.1:8200
                 Storage: inmem
                 Version: Vault v0.9.1
             Version Sha: 87b6919dea55da61d7cd444b2442cabb8ede8ab1

==> WARNING: Dev mode is enabled!

In this mode, Vault is completely in-memory and unsealed.
Vault is configured to only have a single unseal key. The root
token has already been authenticated with the CLI, so you can
immediately begin using the Vault CLI.

The only step you need to take is to set the following
environment variables:

    export VAULT_ADDR='http://127.0.0.1:8200'

The unseal key and root token are reproduced below in case you
want to seal/unseal the Vault or play with authentication.

Unseal Key: jb7/sRFph/4l3OfeFzyirqq3xGwcGBGibokzbvXtyTo=
Root Token: test-root-token

==> Vault server started! Log data will stream in below:

2018/01/15 09:25:17.773080 [INFO ] core: security barrier not initialized
2018/01/15 09:25:17.773308 [INFO ] core: security barrier initialized: shares=1 threshold=1
2018/01/15 09:25:17.773487 [INFO ] core: post-unseal setup starting
2018/01/15 09:25:17.786857 [INFO ] core: loaded wrapping token key
2018/01/15 09:25:17.786872 [INFO ] core: successfully setup plugin catalog: plugin-directory=
2018/01/15 09:25:17.789758 [INFO ] core: successfully mounted backend: type=kv path=secret/
2018/01/15 09:25:17.790010 [INFO ] core: successfully mounted backend: type=cubbyhole path=cubbyhole/
2018/01/15 09:25:17.790362 [INFO ] core: successfully mounted backend: type=system path=sys/
2018/01/15 09:25:17.790899 [INFO ] core: successfully mounted backend: type=identity path=identity/
2018/01/15 09:25:17.793150 [INFO ] expiration: restoring leases
2018/01/15 09:25:17.793175 [INFO ] rollback: starting rollback manager
2018/01/15 09:25:17.793410 [INFO ] expiration: lease restore complete
2018/01/15 09:25:17.794486 [INFO ] identity: entities restored
2018/01/15 09:25:17.794559 [INFO ] identity: groups restored
2018/01/15 09:25:17.794587 [INFO ] core: post-unseal setup complete
2018/01/15 09:25:17.794863 [INFO ] core: root token generated
2018/01/15 09:25:17.794868 [INFO ] core: pre-seal teardown starting
2018/01/15 09:25:17.794873 [INFO ] core: cluster listeners not running
2018/01/15 09:25:17.794884 [INFO ] rollback: stopping rollback manager
2018/01/15 09:25:17.794932 [INFO ] core: pre-seal teardown complete
2018/01/15 09:25:17.795071 [INFO ] core: vault is unsealed
2018/01/15 09:25:17.795094 [INFO ] core: post-unseal setup starting
2018/01/15 09:25:17.795150 [INFO ] core: loaded wrapping token key
2018/01/15 09:25:17.795154 [INFO ] core: successfully setup plugin catalog: plugin-directory=
2018/01/15 09:25:17.795381 [INFO ] core: successfully mounted backend: type=kv path=secret/
2018/01/15 09:25:17.795537 [INFO ] core: successfully mounted backend: type=system path=sys/
2018/01/15 09:25:17.795704 [INFO ] core: successfully mounted backend: type=identity path=identity/
2018/01/15 09:25:17.795716 [INFO ] core: successfully mounted backend: type=cubbyhole path=cubbyhole/
2018/01/15 09:25:17.796976 [INFO ] expiration: restoring leases
2018/01/15 09:25:17.797181 [INFO ] rollback: starting rollback manager
2018/01/15 09:25:17.797346 [INFO ] identity: entities restored
2018/01/15 09:25:17.797442 [INFO ] identity: groups restored
2018/01/15 09:25:17.797559 [INFO ] expiration: lease restore complete
2018/01/15 09:25:17.797618 [INFO ] core: post-unseal setup complete

Expected Behavior:

Similar to policy value .hcl files, I would like to encode other Vault config files using HCL and write them using the Vault CLI. This provides friendlier syntax (e.g. comments, no trailing-comma fragility).

Actual Behavior:

Policy values are "special" with HCL handling on the server side. Other configs need to be written using JSON.

Steps to Reproduce:

# We have some Vault config file in HCL format.
$ cat my-secret.hcl
# Explanatory comment.
foo = "fooVal"
bar = "barVal"
# Let's try writing it.
$ vault write secret/my-secret @my-secret.hcl
Error loading data: Invalid key/value pair '@my-secret.hcl': invalid character '#' looking for beginning of value
# If we convert it to JSON, things work.
$ cat my-secret.hcl | json2hcl --reverse | vault write secret/my-secret -
Success! Data written to: secret/my-secret

References:

https://www.hashicorp.com/blog/codifying-vault-policies-and-configuration.html is the inspiration for our desired workflow.
#3795 discusses general difficulty with posting complex values to the API (including properly encoding policies).
I realize I can do the above conversion operation myself across the board, but it would be nice if the Vault client did HCL handling and communicated to the server using JSON. This would make policies less "special", simplify the broad use of HCL in codified Vault configs, and generally encourage API consistency of the sort discussed in #3795

[vishalnayak]

@jgiles The values stored in secret/my-secret can be any generic secret and need not necessarily be a configuration. It is a KV store and the values in there are never parsed. If you want to store .hcl files in the KV store, you can do so by treating .hcl payload as a string value.

[jefferai]

Vault's entire API is JSON. The K/V stores JSON; for ease of general use when using vault write it converts each key/value pair into a separate key in a JSON dict, but the input/ouptut of the K/V store is JSON. As a result if you're going to provide input inside a file it must be a properly-formatted JSON file, just like it would be if using curl. As Vishal said you can absolutely store HCL in it, you just need to string-encode it.

[jgiles]

The secret example was perhaps ill-chosen; it was just the simplest example of "I want to put an object in Vault". I'm not actually trying to write secrets in HCL and store them.

What I'm actually trying to do is write down auth and secret backend configuration (e.g. Okta config, Duo config, users, groups, SSH CA config, SSH roles, Postgres roles, MSSQL roles, etc) in the style of the Codifying Vault Policies and Configuration blog post.

Since Vault's entire API is JSON, I can write all this configuration down as JSON files. That is what we do currently, but JSON is sub-optimal as a configuration language in many ways - I can't leave comments, and the syntax is verbose and unforgiving (don't forget that comma! no, you can't break that huge SQL statement string up into multiple lines!).

HCL is a minimal extension of JSON and works better as a configuration language. Vault already seems to recognize the difficulties of using JSON as a configuration language: It supports specifying policies as HCL (in addition to JSON, since all JSON is valid HCL).

The suggestion here is to stop treating policies as a special case: Keep the entire Vault API as JSON. Support humans writing Vault data in HCL by converting it to JSON on the client side before POSTing it to the server.

Note that it might even make sense to do this for policies themselves (thus truly honoring "Vault's entire API is JSON"). The main thing you would lose would be the particular format of the input HCL file, but it's not clear that you really want to store and return functionally-equivalent policies as different strings. You could always store the raw HCL input on the side for a human-format response. On the other hand, a straight JSON version of the policy would be much more easily interoperable with tools that might grab policies from the API.

[jgiles]

@jefferai any thoughts on the above clarification of the feature request?

[jefferai]

Support humans writing Vault data in HCL by converting it to JSON on the client side before POSTing it to the server.

The problem with this is that it requires some heuristics in the CLI that I would prefer we don't codify in for everybody, especially since there are tools out there that can convert HCL to JSON, at which point you can easily pipe the output of such a tool into the input of the CLI.

I'm not disagreeing with the issues around JSON, at all, but I don't think the solution in the general case is to support HCL for any given input, especially given that there are few API calls that generally take more than a few parameters, and for policies HCL can be provided as-is since that's codified as the official input.

[sean-]

@jgiles I've run into this a few times and is why I wrote cfgt. Hopefully this can ease your frustration with JSON and be integrated into your tooling. JSON is an unreasonably awful configuration language, so I sympathize fully with the desire here. Good luck.