Can't pass a dictionary through included task to aws_secret

[apogrebnyak]

Summary

When I wrap aws_secret into the task file with parameters and passing a dictionary with or without to_nice_json filter always get the warning

[WARNING]: The value "{'foo': 'bar'}" (type dict) was converted to ""{'foo': 'bar'}"" (type string). If this does not look like what you expect, quote the entire value to ensure it does not change.

Needless to say this is not a JSON that I want for my secret.

Here is the contents of the upload-secret.yml

---

- name: create upload_secret_temp_dir
  ansible.builtin.tempfile:
    prefix: upload_secret.
    state: directory
  register:
    upload_secret_temp_dir

- name: 'Execute tasks using upload_secret_temp_dir <{{ upload_secret_temp_dir.path }}>'
  block:

  # This causes validation of all dependent variables in secret_content during CI build
  - name: 'Copy secret to <{{ upload_secret_temp_dir.path}}/secret-content.out>'
    copy:
      content: '{{ (secret_content }}'
      dest: '{{ upload_secret_temp_dir.path }}/secret-content.out'
      mode: 0600

  - name: 'Secret content type'
    debug:
      msg: "{{ (secret_content is mapping) | ternary('is dictionary', 'is NOT dictionary') }}"

  - name: 'Upload {{ secret_description }} to <{{ account.region }}:{{ secret_name }}> secret'
    community.aws.aws_secret:
      description:  '{{ secret_description }}'
      name:         '{{ secret_name }}'
      region:       '{{ account.region }}'
      secret:       '{{ (secret_content }}'
      secret_type:  string
      state:        present
    tags:
        - non-ci-build

  always:
  - name: 'delete upload_secret_temp_dir <{{ upload_secret_temp_dir.path }}>'
    file:
      path: '{{ upload_secret_temp_dir.path }}'
      state: absent
...

Here is the call site

- name: My secret upload
  import_tasks: upload-secret.yml
  vars:
    secret_description: My Secret
    secret_name: /my/secret/path
    secret_content: '{{ my_var_dict | to_nice_json }}'

vars/main.yml

my_var_dict:
  foo: bar

Passing the same dictionary to community.aws.aws_secret task does not produce the warning and results in correct JSON in the secret

Issue Type

Bug Report

Component Name

aws_secret

Ansible Version

$ ansible --version

ansible-playbook 2.10.10
config file = /redacted/ansible/ansible.cfg
configured module search path = ['redacted']
ansible python module location = /redacted/.local/lib/python3.8/site-packages/ansible
executable location = /redacted/.local/bin/ansible-playbook
python version = 3.8.5 (default, Feb 18 2021, 02:43:09) [GCC 7.2.1 20170915 (Red Hat 7.2.1-2)]

Collection Versions

$ ansible-galaxy collection list

AWS SDK versions

$ pip show boto boto3 botocore

Configuration

$ ansible-config dump --only-changed

OS / Environment

No response

Steps to Reproduce

 See the description

Expected Results

See the description

Actual Results

Code of Conduct

• I agree to follow the Ansible Code of Conduct

[ansibullbot]

Files identified in the description:

• plugins/modules/aws_secret.py

If these files are inaccurate, please update the component name section of the description or use the !component bot command.

click here for bot help

[ansibullbot]

cc @jillr @markuman @rrey @s-hertel @tremble @wimnat
click here for bot help

[tremble]

@apogrebnyak,

Thanks for taking the time to open this issue. I'm trying to understand what you're actually expecting to see happen here. The APIs support two types of data "string" and "binary", there is no option to pass JSON / dictionaries into the secrets.

The closest we can approximate a dictionary to is the string {'foo': 'bar'} which is what it's telling you it's doing. Your example also looks to be invalid syntax (a mismatched opening bracket), so I'm guessing you've tried to tweak from your original code, but what you've tweaked it to doesn't run.

The warning is a side effect of Ansible trying to be clever and converting the string from "to_json" back into a dictionary (not something we can control in this collection). The easiest way to get rid of the warning would be

  content: '{{ my_var_dict | to_nice_json | string }}'

Which prevents the Ansible side of things trying to be clever, but that's still feeding the same thing only with some additional whitespace into the module. Is that what you're trying to achieve?

Unfortunately in the expected results you're rather vague, what exactly (preferably with a example) are you expecting to see. If you're referring to the use of single quotes rather than double quotes, then the | string should work around the issue.

[nick-zh]

@tremble i think i might have run into the same, since i am new, i probably am just using the syntax wrong.
So if you want to add a key / value pair to aws_secret, the following syntax works:

secret: '{ "username": "someUsername"}'

when i retrieve the value, it is formatted as key and value properly (the value is a json string with double quotes).

The following syntax doesn't seem to work:

secret: '{ "username": "{{ usernameVaultVar }}"}'

it will result in the following warning: the value (type: dict) was conveted to (type: string)
in AWS it will result in a string with singel quotes instead of double quotes:

{'username': 'someUsername'}

In the vault i just used

usernameVaultVar: "someUsername"

EDIT: I noticed using an inline vault, like this works as well:

secret: !vault |
      $ANSIBLE_VAULT;1.1;AES256
      62313365396662343061393464336163383764373764613633653634306231386433626436623361
      6134333665353966363534333632666535333761666131620a663537646436643839616531643561
      63396265333966386166373632626539326166353965363262633030333630313338646335303630
      3438626666666137650a353638643435666633633964366338633066623234616432373231333331
      6564

[LawrenceWarren]

Hi @tremble I am having the same issue as @nick-zh, it appears the module does not parse strings as key/value pairs when using variables.

secret: '{ "password": "secret_text_here" }' gives a key/value pair in AWS:

secret: '{ "password": "{{ my_secret }}" }' gives an improperly formatted plaintext string.

[tremble]

@LawrenceWarren @nick-zh,

Thanks for the extra info. Unfortunately this is primarily a limitation of Ansible itself. What the devs said over on IRC:

tremble : Anyone know if there's a way for a module to tell Core not to convert templated strings over to dicts, the current behaviour is really confusing: https://github.com/ansible-collections/community.aws/issues/656#issuecomment-939943026
jtanner : '{ "password": "{{ my_secret }}" }' ... if only part of that is expected to be templated, i think you need to split it out into separate vars to have specific intent applied to each. the {{}} is going to trigger is_template and safe_eval is going to think { } is a dict
bcoca : leading space 'fixes' safe eval or |to_json at the end if that is what you want

In theory we can hack around this by converting things back to JSON if they're passed to us as a dict/list, but it's not a brilliant solution. Unfortunately the original implementation used secret for both the binary and string input, which means we can't just tell Ansible that it's a string we expect.

What you can try is secret: ' { "password": "{{ my_secret }}" }' (note the extra whitespace between ' and {. Otherwise you'll need to template the whole thing:

aws_secret:
  secret: '{{ _secret | to_nice_json | string }}'
vars:
  _secret: '{ "password": "{{ my_secret }}" }'

[nick-zh]

@tremble thx a lot for the elaboration and the follow up 🙏
I can confirm, that an additional space at the beginning of the secret JSON string solves this problem 🎉 , i agree it's not the nicest / most intuitive way, but it does the job 😄

[rrey]

@tremble I wonder if adding a json_string type that would make sure the string remains a valid json would not allow to fix this.

wdyt ? I could do the PR if that seems a good solution.