Using json policies - example json policy needed

[timgent]

I am using the Ruby client for Vault and hence need to use json policies, as it won't allow me to use HCL directly. I haven't been able to get it working and suggest some examples are needed in the docs.

I have tried using the hcl2json tool here: hashicorp/hcl#24

Starting hcl policy:

path "secret/*" {
  policy = "write"
}

Converted json policy:

{
  "path": [
    {
      "secret/*": [
        {
          "policy": "write"
        }
      ]
    }
  ]
}

Just using the command line tool I get:

$ vault policy-write secret test/resources/policies/writer.json
==> Error: Error making API request.

URL: PUT http://localhost:8200/v1/sys/policy/secret
Code: 400. Errors:

* Invalid path policy: &vault.PathPolicy{Prefix:"", Policy:"", Glob:false}

Possible the json is wrong, though I haven't been able to manually put in anything that works. I think it would be helpful for the docs to have an example of a json policy that definitely works.

[jefferai]

As other commenters pointed out in that pull request, it adds arrays where they shouldn't be. Here is equivalent JSON:

{
  "path": {
      "secret/*": {
          "policy": "write"
        }
    }
}

[aburnett]

How does this translate for use via the REST API? Seems like that wants an object containing a single "rules" attribute with a string value.

{ "rules": ??? }

[jefferai]

That's correct. It's simply the raw text of an equivalent policy file set as the value of the "rules" key. All of the actual parsing happens within the Vault server.

[aburnett]

Ok, thanks. It would be convenient if this could be the actual json document describing the rules
e.g.

[
{
  "path": {
      "secret/*": {
          "policy": "write"
        }
    }
},
{ ... }
] 

[timgent]

@jefferai the example you gave above is helpful, though how would you add multiple policies in a single file with json?

[jefferai]

@timgent The API acts on a single policy at a time, so (just like when using the CLI) you have to use separate API calls to handle separate policies. Sorry!

[timgent]

@jefferai sorry I probably phrased this badly. I mean having a single policy that defines access rights for multiple paths

[jefferai]

@timgent Oh, sorry. Here's an example with multiple paths:

{
  "name": "dev",
  "path": {
    "*": {
      "policy": "deny"
    },
    "stage/*": {
      "policy": "sudo"
    }
  }
}

[ethicalaakash]

@jefferai when Using Vault API to create a policy.An empty policy is created.
though when creating from CLI using the same json it works.
Here's the Json I'm using:

{
  "path": {
    "sys/policy/*": {
      "policy": "read"
    },
    "auth/userpass/users/": {
      "policy": "write"
    }
  }
}

[jefferai]

@aakashsinghal1994 Without information about the API calls you're making I can't really help!

[ethicalaakash]

@jefferai Here's the API call I'm making.

curl \
    --header "X-Vault-Token: *******" \
    --request PUT \
    --data @add_user.json \
    https://xyz.com/v1/sys/policy/add_user

[jefferai]

@aakashsinghal1994 Take a look at the docs (https://www.vaultproject.io/api/system/policy.html) -- you need to specify the serialized HCL or JSON string as the "policy" argument to the call.

[osvacaneljr]

@jefferai, is this is expected to work?

policies.json
{
"path" : {
"auth/approle/role/" : {
"policy" : "write"
},
"secret/" : {
"policy" : "write"
}
}
}

curl command

curl
-X PUT
-H "X-Vault-Token:******"
--data @policies.json
http://xx.xx.xx.xx:8200/v1/sys/policy/test

after this command runs with not errors, then run this command

curl \
-X GET
-H "X-Vault-Token:*****"
http://xx.xx.xx.xx:8200/v1/sys/policy/test

this is the response

{"name":"test","rules":"","request_id":"c2d1ecbd-0eaf-658d-21e7-e0188a356c02","lease_id":"","renewable":false,"lease_duration":0,"data":{"name":"test","rules":""},"wrap_info":null,"warnings":null,"auth":null}

[chrishoffman]

@osvacaneljr The reason your rules are blank is because you not putting your policy into the rules parameter.

Your policies.json should look something like this:

{"rules":"{\"path\":{\"auth\/approle\/role\/\":{\"policy\":\"write\"},\"secret\/\":{\"policy\":\"write\"}}}"}

Check out the documentation for this endpoint here https://www.vaultproject.io/api/system/policy.html#create-update-policy.

[tburt11]

@chrishoffman Thank you for the working example. I have been looking all day for this.
w/rt the endpoint documentation... The example there is limited to:
{ "rules": "path \"secret/foo\" {..." }
Which is quite different from your example:
{ "rules":"{\"path\":{\"auth\/approle\/role\/\":{\"policy\":\"write\"},\"secret\/\":{\"policy\":\"write\"}}}" }

note the additional curly brackets in your example and the escape before the "/" in the path.
Perhaps someone could update the docs with a working example?

Yes, your example is for included data and the doc example is for a data file, but the missing curly brackets are still problem, no?

Thanks again!

[Lechucico]

I'm having the following error when I try to install a new policy:

{"errors":["1 error occurred:\n\n* unsupported path"]}

I try the following http request:

conn = Faraday.new(:url => 'http://127.0.0.1:8200')
res = conn.put do |req|
	req.url '/v1/sys/policies/acl/elastic-read'
	req.headers['X-Vault-Token'] = root_token
	req.body = '{"path": {"secret/elastic": {"policy": "read"}}}'
end

I'm using version 0.8.3 on a CentOS
node.default['hashicorp-vault']['version'] = '0.8.3'

[patrick-motard]

I don't normally chime in on these sort of things but I spent alot of time working through this issue as well. Thank you to @tburt11 in this thread and Jeff Mitchell over on a google forum thread for helping me "see the light" with the "rules" typo. I was following this article on "codifying vault" to script out provisioning a vault server. In the article they show the policy json files to have contents like this:

{"rules":"path \"postgresql/creds/readonly\" {\n capabilities = [\"read\"]\n}"}

In this case they have a json object who's "rules" parameter value is a string representation of a policy written in hcl. However, in both the documentation on policies and this google thread it's mentioned (or shown offhandedly in slightly inconsistent code) that "rules" must be a string but the policy within the string can be in .hcl or .json format. In either case I didn't like that it was stringified because it's very easy to make mistakes when editing and reading.

I modified the provision script provided in the Codifying article to allow the contents of each policy file to be json with no stringifying, and instead doing the stringifying in the script where necessary. This way you can modify your policies as json in a json file, and trust the the script will place your policy into an object like so:

mypolicy.json:

{
    "path": {
        "secret/*": {
            "capabilities": [
                "read",
                "list"
            ]
        }
    }
}

what the modified provision script hands to --data in the curl:

{"rules":"{\"path\":{\"secret/*\":{\"capabilities\":[\"read\",\"list\"]}}}"}

Below is the modified script:

#!/usr/bin/env bash
VAULT_TOKEN=root
VAULT_ADDR=http://0.0.0.0:8200

set -e

shopt -s nullglob

function provision() {
    set +e
    pushd "$1" > /dev/null
    for f in $(ls "$1"/*.json); do
      p="$1/${f%.json}"

      if [[ $1 =~ ^sys/policy.* ]]
      then
          string="$(echo "$(cat ${f})" | jq '@json')"
          payload='{"rules":'"${string}"'}'
      else
          payload="$(cat ${f})"
      fi

      curl \
        --location \
        --header "X-Vault-Token: ${VAULT_TOKEN}" \
        --data "$(echo "${payload}")" \
        --fail \
        --silent \
        "${VAULT_ADDR}/v1/${p}" \
        | jq .
    done
    popd > /dev/null
    set -e
}

vault status > /dev/null

pushd vault-policy >/dev/null
provision sys/auth
provision sys/mounts
provision sys/policy
provision postgresql/config
provision postgresql/roles
provision auth/userpass/users
popd > /dev/null

[jefferai]

Closing -- at this point there are numerous examples and I believe the OP's request is solved.

[zippycup]

I spend alot of time on this. Hashicorp should fix the api to be json compatible. In the meantime, I spend some to automate this process as https://www.hashicorp.com/blog/codifying-vault-policies-and-configuration process really sucks.
https://github.com/zippycup/hashicorp_vault_config

[jefferai]

Totally unclear what you mean here. The API is 100% JSON compatible. JSON is all that Vault speaks.

[zippycup]

Perhaps the wrong choice of words. Here's an example of what i am referring to:

When I enable approle /v1/sys/auth/approle

{
  "type": "approle",
    "description": "Login with approle"
}

This works fine via the api without escaping the double quotes

What i want to do something like this hash for /v1/sys/policies/acl/system_admins:

{
  "policy": {
  " path": {
      "policy/*": { "capabilities": [ "create","read","list","update" ] }
      "secret/*": { "capabilities": [ "create","read","list","update" ] }
      "mount/*": { "capabilities": [ "create","read","list","update" ] }
      "auth/*": { "capabilities": [ "create","read","list","update" ] }
    }
  }
}

It seems that you can pass the json but only as one key value / pair as oppose to the entire hash. So the above hash looks like this json to be passed via the api

{"policy": "{\"path\":{\"secret/*\":{\"capabilities\":[\"create\",\"read\",\"list\",\"update\"]}}}, {\"path\":{\"policy/*\":{\"capabilities\":[\"create\",\"read\",\"list\",\"update\"]}}}, {\"path\":{\"mount/*\":{\"capabilities\":[\"create\",\"read\",\"list\",\"update\"]}}}, {\"path\":{\"auth/*\":{\"capabilities\":[\"create\",\"read\",\"list\",\"update\"]}}}"}

Why is the behavior different ?

[calvn]

@zippycup please use our mailing list in the future for any questions to avoid spamming everyone in GH threads.

To answer your question, the policy field accepts a string as stated in the docs, so you will either need to provide a string-escaped or base64-encoded JSON object.

[alexandruast]

I am using json policy files, and just parse the policy string using jq and sed to match what vault expects in --data curl arg:

policy_string=$(cat vault/policies/admin.json | jq -c . | sed 's/"/\\\"/g')
  curl -Ssf -X PUT \
    -H "X-Vault-Token:${VAULT_ROOT_TOKEN}" \
    -d "{\"policy\":\"${policy_string}\"}" ${VAULT_ADDR}/v1/sys/policy/admin

example admin.json

{
  "path": {
    "auth/*": {
      "capabilities": ["create", "read", "update", "delete", "list", "sudo"]
    },
    "sys/auth/*": {
      "capabilities": ["create", "read", "update", "delete", "sudo"]
    },
    "sys/policy": {
      "capabilities": ["read"]
    },
    "sys/policy/*": {
      "capabilities": ["create", "read", "update", "delete", "list", "sudo"]
    },
    "secret/*": {
      "capabilities": ["create", "read", "update", "delete", "list", "sudo"]
    },
    "sys/mounts/*": {
      "capabilities": ["create", "read", "update", "delete", "list", "sudo"]
    },
    "sys/health": {
      "capabilities": ["read", "sudo"]
    }
  }
}

[geek65535]

I think it's pretty obvious from the sheer number of articles, blog posts, mailing list threads, and so on: vault has a serious problem when it comes to configuration. People are getting confused, there are odd workarounds being used, there are inconsistencies with the way vault handles config data.
Part of the problem is vault and its API. The other part is the documentation. Not everything is fully documented, there are not nearly enough examples, and a lot of times, you have to jump from the regular docs to the API docs and back just to figure something out.

HashiCorp, for a product you charge $120,000/yr for, vault is frustrating as hell to use sometimes...

[jefferai]

Hi Paul,

It's unclear from your post exactly what is frustrating you, but I would encourage you to open issues (or even better, PRs) if you are coming across specific documentation problems. Unfotunately, statements like "there aren't enough examples" aren't particularly helpful, because we don't know what problems you're having, what pieces of Vault you're using, what examples you're looking for, or whether it's just a discovery issue (e.g. there may be examples in our guides but not our API sections).

Since you are a paying customer, if you hare having difficulties you can (and should) go through our support team as unlike the mailing list or GitHub that's governed by SLAs and they escalate to Vault devs as needed.

[geek65535]

Fair enough, Jeff. I'll do my best to document and report problems when I come to them.
But besides venting a bit, I was trying to provide a sort of call to arms, in which you (HashiCorp) step back and look at where users are having issues, and where people are complaining, and focus some engineering efforts on those areas.

BTW, I definitely hate the HCL vs JSON issue. If I can't use HCL everywhere, I'd rather not bother using it. Consistency is more important to me, and I don't mind working with JSON. But as JSON does have the major, major, WTF-were-they-thinking, major flaw of not allowing comments, I'll make a suggestion for improving usability for folks like me that just use JSON: allow a "comment" field anywhere that it wouldn't be interpreted as data.
Example:

{
    "path": {
        "secret/*": {
            "comment": "this allows anyone to read and list; will assign to new users",
            "capabilities": [
                "read",
                "list"
            ]
        }
    }
}

I suppose it might even be possible to convert actual comments in HCL into this format, although simply allowing the "comment" field would be a big improvement.

[jefferai]

@geek65535 The problem of course is that there are many issue and complaints and not enough capacity to address them all in any particular time frame. The engineering team has scaled up significantly over the last few years, but so has our user base. What you've described -- "look at where users are having issues, and where people are complaining, and focus some engineering efforts on those areas" -- is pretty much our entire workflow outside of new feature development. I'd encourage you to look at the items listed under IMPROVEMENTS and BUG FIXES in our changelog -- these are pretty much driven directly by user requests, and they don't include updates to our website, including docs, API info, and our guides section which has been expanding significantly. The problem isn't that we don't listen to users, it's that there just aren't enough hours in the day.

In 0.11 we've whitelisted "comment" as a key in policies. For future suggestions like this, please open a new ticket.

[geek65535]

@jefferai Thanks for putting in the "comment" as a key.
I should apologize for the tone of my comments. I had spent the entire day with vault, trying to figure several things out, and pouring through docs does get frustrating after a while, no matter what the product.

[jefferai]

No problem. Glad to help!

[alexwwang]

interesting, I've spent more than 2 weeks on vault by now to figure out how to put it into production.

[jefferai]

@alexwwang If you're having troubles I suggest asking questions on the mailing list. We also have various guides (https://www.vaultproject.io/guides/index.html) that may be of use.

[alexwwang]

@jefferai one week ago I posted an issue both here (#110) and in the mailing list for help but by now no further reply received.
Not mention with the various guides, I've similar experience with @geek65535 on it, frankly.
Sorry for off-topic, I don't mean it. I believe vault is amazing and a powerful secrets tool so I do have some expectation on it.

[Xtigyro]

@alexandruast Thank you!!!

Team (@jefferai),
I still believe the examples in the docs are quite insufficient.
Could you please add more examples.

[jmls]

man, I am so having issues with policies, json and the rest api .

why on earth can't we just pass in json, like most of the other api's .. and saying that policies: somestring is json doesn't really cut it , that's a copout.

I'm trying to write a policy from another node app, calling the rest api. In this other app, my json is

const fooPolicy = {
path: {
    'auth/*': {
        capabilities: ['create', 'read', 'update', 'delete', 'list', 'sudo']
    },

    'sys/auth/*': {
        capabilities: ['create', 'read', 'update', 'delete', 'sudo']
    }}};

Would some kind soul let me know how I should manipulate this object into a string that works with the policies key in the api ? I have tried JSON.stringify, base64 etc etc . it writes ok, but is obviously a duff policy as I always get permission denied when trying to use a token with the fooPolicy policy applied

[jefferai]

Seems to work fine for me.

Two notes:

1. I'm using the CLI but it just takes the input and puts it in the policies key and does normal JSON marshaling

2. I touched up your quoting. I'm not sure about Node and its conversion of objects but what's in your code isn't actually proper JSON in a pure JSON sense.

11:56 $ cat testpol.json
{
  "path": {
    "auth/*": {
      "capabilities": [
        "create",
        "read",
        "update",
        "delete",
        "list",
        "sudo"
      ]
    },
    "sys/auth/*": {
      "capabilities": [
        "create",
        "read",
        "update",
        "delete",
        "sudo"
      ]
    }
  }
}

11:57 $ vault policy write mypol testpol.json
Success! Uploaded policy: mypol

11:57 $ vault token create -policy=mypol
Key                  Value
---                  -----
token                s.PE0exn5JRdRCr6k783zS0DLE
token_accessor       93zNjZzTkJ9gDsITmYuqc1VR
token_duration       768h
token_renewable      true
token_policies       ["default" "mypol"]
identity_policies    []
policies             ["default" "mypol"]

11:58 $ VAULT_TOKEN=s.PE0exn5JRdRCr6k783zS0DLE vault auth enable ldap
Success! Enabled ldap auth method at: ldap/

[jmls]

thanks - yes, the cli seems to work just fine - but it's the rest api that I have problems with. Internally the cli must be doing something to the json to get it into a format suitable for the rest api. IOW - what do I need to do to convert the json in testpol.json to a format suitable for the policy key in the rest api ?

[jefferai]

It's simply treating the value as a string and calling Go's json.Marshal function. IOW it's JSON-escaping the string.

[jmls]

FWIW, I've found that this works for node. Seems awfully clumsy though ...

const foo = {
    path: {
        'auth/*': {
            capabilities: ['create', 'read', 'update', 'delete', 'list', 'sudo']
        },

        'sys/auth/*': {
            capabilities: ['create', 'read', 'update', 'delete', 'sudo']
        }
    }
};

const y = JSON.stringify({ policy: JSON.stringify(foo) });
console.log(y);

if y is now stored in foo.json, the resultant data can then be passed on the command line

curl -H "Content-Type: application/json" -H "x-vault-token: <token>" -X POST -d @foo.json http://localhost:8200/v1/sys/policies/acl/admin

[seanw2020]

Here's a quick hack if you want to keep the policy in HCL--i.e., not rewrite it in JSON:

# Write a new JSON policy file: https://github.com/ned1313/Getting-Started-Vault/blob/master/m4/devpol.json
tee foo.json <<\EOF
{
  "policy": "path "foo/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
  }

  path "foo/appId*" {
    capabilities = ["create", "read", "update", "delete", "list"]

    allowed_parameters = {
      "api-key" = []
      "environment" = ["dev","qa","staging","production"]
      "description" = []
    }
  }

  path "secret/data/{{identity.entity.id}}/*" {
    capabilities = ["create", "update", "read", "delete"]
  }

  path "secret/metadata/{{identity.entity.id}}/*" {
    capabilities = ["list"]
  }"
}
EOF

# Reformat. The trick, like `jefferai` wrote above, is to see that "policy" is actually an escaped single value: "path..." (with any quotes inside that being escaped) in HCL format
cat foo.json | tr '\n' ' ' | sed 's/"/\\\"/g; s/\\"policy\\": \\"/"policy": "/; s/}\\"/}"/' | tee tmp; mv tmp foo.json


# Create a new policy
curl -s --header "X-Vault-Token: $VAULT_TOKEN" --request PUT --data @foo.json $VAULT_ADDR/v1/sys/policies/acl/dev-clone | jq

Output:

{   "policy": "path \"foo/*\" {   capabilities = [\"create\", \"read\", \"update\", \"delete\", \"list\"]   }    path \"foo/appId*\" {     capabilities = [\"create\", \"read\", \"update\", \"delete\", \"list\"]      allowed_parameters = {       \"api-key\" = []       \"environment\" = [\"dev\",\"qa\",\"staging\",\"production\"]       \"description\" = []     }   }    path \"secret/data/{{identity.entity.id}}/*\" {     capabilities = [\"create\", \"update\", \"read\", \"delete\"]   }    path \"secret/metadata/{{identity.entity.id}}/*\" {     capabilities = [\"list\"]   }" } 

That matches the Github contents of the Pluralsight course I'm following.

Results:

$ curl -s --header "X-Vault-Token: $VAULT_TOKEN" --request PUT --data @foo.json $VAULT_ADDR/v1/sys/policies/acl/dev-clone
$ echo $?
$ 0

After accounting for spacing changes, putting the outer object brackets on one line, and replacing devkv with foo, I get these results:

$ diff /tmp/a /tmp/b
$

Confirming:

curl -s --header "X-Vault-Token: $VAULT_TOKEN" --request GET $VAULT_ADDR/v1/sys/policy/dev-clone | jq '.'
{
  "rules": "path \"foo/*\" {   capabilities = [\"create\", \"read\", \"update\", \"delete\", \"list\"]   }    path \"foo/appId*\" {     capabilities = [\"create\", \"read\", \"update\", \"delete\", \"list\"]      allowed_parameters = {       \"api-key\" = []       \"environment\" = [\"dev\",\"qa\",\"staging\",\"production\"]       \"description\" = []     }   }    path \"secret/data/{{identity.entity.id}}/*\" {     capabilities = [\"create\", \"update\", \"read\", \"delete\"]   }    path \"secret/metadata/{{identity.entity.id}}/*\" {     capabilities = [\"list\"]   }",
  "name": "dev-clone",
  "request_id": "21fc6547-c66b-daa7-6cc2-09ac73d3dd5f",
  "lease_id": "",
  "renewable": false,
  "lease_duration": 0,
  "data": {
    "name": "dev-clone",
    "rules": "path \"foo/*\" {   capabilities = [\"create\", \"read\", \"update\", \"delete\", \"list\"]   }    path \"foo/appId*\" {     capabilities = [\"create\", \"read\", \"update\", \"delete\", \"list\"]      allowed_parameters = {       \"api-key\" = []       \"environment\" = [\"dev\",\"qa\",\"staging\",\"production\"]       \"description\" = []     }   }    path \"secret/data/{{identity.entity.id}}/*\" {     capabilities = [\"create\", \"update\", \"read\", \"delete\"]   }    path \"secret/metadata/{{identity.entity.id}}/*\" {     capabilities = [\"list\"]   }"
  },
  "wrap_info": null,
  "warnings": null,
  "auth": null
}

Long story short, it's close enough for me--at least, until there's a better way. What a mess...

[fabi-alstom]

I know this thread is old, but... even Vault is an awesome product, this thread makes sense. Today I've spent the whole morning making this work. I've followed the official tutorial, and I've read the docs. However, I had to come here and use the @alexandruast solution. Now diffing, I see stupid mistakes on my part (we're running all the time), that a proper example could avoid doing.

Some ideas:

• Place HCL samples plus their JSON equivalent in the tutorial. Line by line if possible.
• Same as before for the documentation. Plus, there's nothing bad in providing a whole sample JSON file. A small one is enough.
• Or provide a script to ease conversion from HCL to JSON.

Maybe these exist, but I expected it at first sight.

However, good product, thanks and keep rocking!

[gregkeys]

Trying to figure out hcl to json policies is extremely frustrating, ive spent several days on this already, there are a lot of inconsistencies in the documentation and the tools, even now in 2020

e.g. take any old hcl use https://www.hcl2json.com/ and try to apply the resulting json as a policy. combine that with generic responses about what it should look like and it's a recipe for pain...

[jmls]

@gregkeys what works for me is

a post to "/sys/policies/acl/<policy_name>"

with a body of

{ policy: JSON.stringify({ path: aclPaths }) }

where aclPaths is

{
    'path1: { capabilities: ['read','update'] },
    'path2: { capabilities: ['delete',list'] },
  }

etc

hth. It took a while to figure out ;)

I use the -output-curl-string on the cli commands to see what the equivalent curl call is.