gpg renderer includes new line when decrypting data #31709

asyncmind0 · 2016-03-07T07:42:16Z

Description of Issue/Question

When rendering gpg data in pillars a new line is included in the pillar data

Setup

#!yaml|gpg
testdata: |
    -----BEGIN PGP MESSAGE-----
    Version: GnuPG v2

    hQEMA/Vz8pLFN8VCAQf/fICBeA4s05X6MRPajqxKn8+PrSLMCF57xcE9//zhuAre
    vgBZCuHGzUK4TqoF7nrRPqwkJNFYdbXzIduekE5WzLnWgcidlQAnMrkdsp6reTTA
    afxoU5Q17v2mdQNe0aR30aKJJpf/lex03y8clCu9FKFCedbP6C6woqhzPJEInJVA
    fmQK8M7yfkM0U8kPHwKkOA2r5uo2NcIeVGRXVAeC/cxs2HwGgopTbvcrY3viPccy
    +00eRAkiBbYmjDm02FdJffXUO1P1vBNfbn3l6xZ86Bocwz6QIWWxnyj1fxs+nqkU
    WR5icGB6Su05Hh4p4qWXgCMAaPdFd1eRhQrUqNi9nYUBDANQxr5cKZY/aAEIAJnS
    scRsbpZXluCFY5oJv/4QDIGbzNtJYKlh/SZAvNLXlJE2GXDXva7jkxv19fUbnr3S
    Kss6YXNBPGzDhljH3tGmJFToIX3vlOpFUQMxDQASqIuUsfIIs92exIenu3nLvH9H
    fEfQygHhGoG9nutgrvObsXSQuwnv/E7rkfQpD8vhr1ES3ovgXHF8XkYXP800Pbpz
    XQJute1gIKI3sEK/319XTSXTpJrXTEGbCgmrrBSAB+qzZ850NEjFQsFPK3H7uYn5
    sADQ3oEXJpov3hNdlVR6zrboaH45wy/uRmSvbtUkIKhQjtiwHDMOKED7SisCkbaH
    tz0wykplVynqzqTRK6aFAQwDyq8jBy8IDsgBB/4lxptOxIO6LH6ifscmQMxDcSpT
    8CNHYgHZH/u0nJPHT5PL+GAaXu7h7WI7k14Gx9rHGA6x3E32XrF87BDBiJMV6gz5
    yRno+oTeFYKIxeBwiC9yT+/gYL/Fttrj/M+ugseucSp9BtA3ZGg6IPwRn7RbCZ7P
    0gRkhu/u+IYJb8FnAVBMfhigmi+a9RUVp/oa4Y4uBQFu6Qg2eR6+zDL769nhY3mk
    3t45Mc5bNelmMbFmL5Gqwy9ag1Lbey1qgtCbBOw3tI1Ul7zp6sJ7q/NqR5tNnWO2
    TV1BVtGBEHE76EYiDZR31AMYb6EEvikzjpDGbQUKlBqBfwHxocM0yXLcjF9f0kQB
    ibUSh0zIVNrNTshIbJHwn0bsfJFPSsu8ZWbuq9lJ92SlfhPJEzYVjgo+jvLLVdTG
    a0+twD7xErdX4iC7MJ7/2os02w==
    =fdB2
    -----END PGP MESSAGE-----

Steps to Reproduce Issue

salt-call --local --output=yaml -l debug  pillar.get testdata

local: 'testdata

'

Include debug logs if possible and relevant.

Versions Report

Salt Version:
Salt: 2016.3.0rc1

Dependency Versions:
Jinja2: 2.7.3
M2Crypto: 0.21.1
Mako: 1.0.3
PyYAML: 3.11
PyZMQ: 15.2.0
Python: 2.7.11 (default, Dec 6 2015, 15:43:46)
RAET: 0.6.5
Tornado: 4.3
ZMQ: 4.1.3
cffi: 1.5.2
cherrypy: 4.0.0
dateutil: 2.5.0
gitdb: Not Installed
gitpython: 0.1.7
ioflo: 1.5.0
libgit2: Not Installed
libnacl: 1.4.4
msgpack-pure: Not Installed
msgpack-python: 0.4.7
mysql-python: Not Installed
pycparser: 2.14
pycrypto: 2.6.1
pygit2: Not Installed
python-gnupg: 0.3.8
smmap: Not Installed
timelib: Not Installed

System Versions:
dist:
machine: x86_64
release: 4.1.18-1-lts

Provided by running salt --versions-report

The text was updated successfully, but these errors were encountered:

asyncmind0 · 2016-03-07T08:20:12Z

/salt/renderers/gpg.py:Line 273

return str(decrypted_data).strip()

fixes the issue, but im not sure if thats right. looks like Popen call is returning with a new line at the end

asyncmind0 · 2016-03-07T22:09:52Z

~:» gpg --version                                                                                                                                                                                                     [130] [master**] 09:07:56
gpg (GnuPG) 2.1.11
libgcrypt 1.6.5
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Home: ~/.gnupg
Supported algorithms:
Pubkey: RSA, ELG, DSA, ECDH, ECDSA, EDDSA
Cipher: IDEA, 3DES, CAST5, BLOWFISH, AES, AES192, AES256, TWOFISH,
        CAMELLIA128, CAMELLIA192, CAMELLIA256
Hash: SHA1, RIPEMD160, SHA256, SHA384, SHA512, SHA224
Compression: Uncompressed, ZIP, ZLIB, BZIP2

jfindlay · 2016-03-07T22:52:22Z

@jagguli, thanks for reporting.

QuinnyPig · 2016-04-11T20:32:28Z

I can replicate this behavior. I'm unclear as to why this is only impacting one particular Pillar value rather than across the board, however,.

vectorsigma · 2016-04-12T23:24:45Z

fwiw, the last time we experienced a bug like this, it's because someone forgot the -n flag to echo when creating gpg blobs for pillar.

e.g.: echo "this is my rifle" | gpg --recipient [email protected] --armor --output supersecret.txt --encrypt

gpg dutifully encrypted it as-is, line-break intact. Adding back -n resolved the issue. (Granted, this was in 2015.5.3) ymmv.

QuinnyPig · 2016-04-12T23:32:14Z

You are correct. In this case, 'cat' also had a newline appended. Crisis averted!

vectorsigma · 2016-04-13T00:15:40Z

Wondering if @jagguli had the same situation?

eliasp · 2016-05-15T14:48:59Z

This once again shows that a utility to handle the encryption of values in Pillar SLS would be quite handy, e.g. start with a plain Pillar SLS like this one in your Pillar repo:

foo/secret.sls:

foo:
  bar: some super secret

Run a command like: salt-dev pillar.encrypt_gpg key=0x0A52AC sls=foo.secret pillar=foo:bar
…and the SLS will be transformed to this:

#!yaml|gpg
foo:
  bar: |
    -----BEGIN PGP MESSAGE-----
    Version: GnuPG v2

    hQEOA0p+wNBgxEAxEAP/ZBYQUbTN2XyEXQPoiPlXJNcHKmas7xcjVKlAK5H/b0A+
    FICekhhM5RbW8iDjwRuVi6s/yQdF0wJjSaqG3o6JYywXUo3MGtvt2VDIswRu7mdK
    TVuj8VijrwnNVspPafw9icf16nSHqtlCh8zhkDrJzmueml7mR3wCQHw2fbXqXYQE
    AImaqcVQ5JTKGir3U/JxsrvKLTjC3ixmG7DewosFae8vP/Kr3nEr9fwJmJRzmJao
    Dg1LX3dYOr6W1UtSTw80EKJAP1IL62tqTIsA+/c1RzEMFKaOnU4KD5fkPofRQ0cc
    ye9QjjNwj/WruXBvteZc+Jn3KRX4wt17QU2NZCxI4FFGhQIMAzfltYjGxBdxAQ/+
    NmePQc2pjHxdMRuyxnT00vJq8S14QgO2CR+eZY40acnkGX2tf6lFMgeZIn801iif
    rFYumtKzcZ/B+ngtpv7dsDe5qfEbmrOx8YpRCC6gPzXvvNnwXlTAi8ktxs3VpRbS
    Bu95/RnRbKIRh6IlYkGqs7ItAkDWZZDBLCmovEP5DJyGZilXVNNwVX9qayVq7fMD
    snzzmhbpumlqztssto89RdqQg9CYxD2vKSlKaeNPwPvagQU65t/Umomut59XxwRF
    X9E8BnoKUAKR66xuJTm/BhhjUD1RK/lSt2ZNQC5TXpEJcDR25HiurAKUGVh6D1V1
    bztw6HRMlEr+29900USN8fB0baA0SaYRIRlcIyAeL2jL+9o5whnH0G7Mmuwlk5tz
    AGKKq/Co0k5gm1KnqYhjhijjM2cZWOugTCMssz6dbk4XtI4GGFJfi1w//LxhZsJc
    mSRf7DD8K3W3VAQPUWIo+XAYN34igi9JtMKaiiUlROM/DkxEB/hHdtzjAyfKSZCs
    WRetKgdBytaGHS9IcauJBWIkffVvwmQAFAfhbBvjutilsFYhYExvG6sBPb0VJq8e+m29K
    /gVPwQthQdp5GSkaDF4jmf82I5oQY6Vmegl/tkFhrrMSylNOnBrlBePzyBjj/a46
    0MMIcGmncdGH8wB8fUBO6SOM9jNReQyX3UczxYnrtgXSRgFMGVlRgjfOQtD2YbXl
    WgRekz+3eIXPELRJKh7z0U2k/SxvtVTeyRx1eNzDlREYu6RVniHe52LlibYTu6Zt
    qxBmeg6XjoA=
    =MV/8
    -----END PGP MESSAGE-----

DanyC97 · 2016-05-17T21:34:33Z

@techhat @cachedout fyi @eliasp suggestion which sounds great imho.

asyncmind0 · 2016-06-09T03:50:03Z

Nope echo -n is not enough just tested against

salt/srv:» salt --versions-report                                                                                                                                                                                        [2] [master*] 01:49:34
/usr/local/lib/python2.7/dist-packages/cffi/model.py:526: UserWarning: 'git_checkout_notify_t' has no values explicitly defined; next version will refuse to guess which integer type it is meant to be (unsigned/signed, int/long)
  % self._get_c_name())
/usr/local/lib/python2.7/dist-packages/cffi/model.py:526: UserWarning: 'git_merge_flag_t' has no values explicitly defined; next version will refuse to guess which integer type it is meant to be (unsigned/signed, int/long)
  % self._get_c_name())
Salt Version:
           Salt: 2016.3.0

Dependency Versions:
           cffi: 1.6.0
       cherrypy: 6.0.1
       dateutil: 2.2
          gitdb: 0.5.4
      gitpython: 0.3.2 RC1
          ioflo: Not Installed
         Jinja2: 2.8
        libgit2: 0.24.0
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.7
   mysql-python: 1.2.3
      pycparser: 2.14
       pycrypto: 2.6.1
         pygit2: 0.24.0
         Python: 2.7.9 (default, Mar  1 2015, 12:57:24)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.2.0
           RAET: Not Installed
          smmap: 0.8.2
        timelib: Not Installed
        Tornado: 4.3
            ZMQ: 4.1.2

System Versions:
           dist: debian 8.5
        machine: x86_64
        release: 3.16.0-4-amd64
         system: Linux
        version: debian 8.5

have an outdated fix here StreetHawkInc@86c9217

WillPlatnick · 2016-06-09T12:20:37Z

@jagguli I spent a while working on encrypted pillars with newlines last night as well. I haven't thoroughly tested this for all situations, but I was able to encrypt multiline and single line successfully with this script:

https://gist.github.com/WillPlatnick/1dbe859f552390b778f9453098dfd081

vitalyisaev2 · 2016-10-04T15:20:21Z

I faced the same issue but it turns out that I misused echo command. Thanks everyone for shared workarounds.

jbouse · 2016-10-04T16:20:03Z

When using echo I run it as echo -n so that it doesn't add the new line... GPG simply encrypts what it is given and echo with the -n option will add a LF to the output.

stale · 2018-07-06T19:03:21Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

vectorsigma · 2018-11-18T18:38:10Z

In case anyone ever finds this issue, this utility would likely help a lot. It will take a fully unencrypted yaml pillar file, and gpgify it for you.

defanator · 2019-09-23T07:04:58Z

It seems like 2018.3.4 still has the same (?) issue.

State example:

Slack Client Secret:
  file.managed:
    - name: /etc/xxx/oauth.slack.client.secret
    - user: xxx
    - group: xxx
    - mode: 600
    - contents_pillar: xxx:slack:client_secret
    - contents_newline: False

Pillar is stored in GPG format, generated by echo -n mysecret | gpg -ae (no newline in the end of string, checked by cat mysecret.asc | gpg | hexdump -C).

Applying the state leads to the change when 0x0a is being added at the end of existing file:

          ID: Slack Client Secret
    Function: file.managed
        Name: /etc/xxx/oauth.slack.client.secret
      Result: None
     Comment: The file /etc/xxx/oauth.slack.client.secret is set to be changed
     Started: 06:59:02.427698
    Duration: 1.616 ms
     Changes:   
              ----------
              diff:
                  --- 
                  +++ 
                  @@ -1 +1 @@
                  -mysecretcode+mysecretcode

Versions:

# salt --versions
Salt Version:
           Salt: 2018.3.4
 
Dependency Versions:
           cffi: Not Installed
       cherrypy: 3.5.0
       dateutil: 2.6.1
      docker-py: Not Installed
          gitdb: 0.6.4
      gitpython: 2.0.8
          ioflo: Not Installed
         Jinja2: 2.8
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: 1.0.3
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.12 (default, Nov 12 2018, 14:36:49)
   python-gnupg: 0.3.8
         PyYAML: 3.11
          PyZMQ: 15.2.0
           RAET: Not Installed
          smmap: 0.9.0
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4
 
System Versions:
           dist: Ubuntu 16.04 xenial
         locale: UTF-8
        machine: x86_64
        release: 4.4.0-1090-aws
         system: Linux
        version: Ubuntu 16.04 xenial

It seems like contents_newline does not have any effect in the above case. Trying to get more details now, appreciate any input.

defanator · 2019-09-23T07:36:07Z

Seems like the actual issue is here: #54177

Fix from #54364 seems to help.

jfindlay added this to the Approved milestone Mar 7, 2016

jfindlay added P2 Priority 2 and removed P3 Priority 3 labels May 16, 2016

jfindlay added P1 Priority 1 and removed P2 Priority 2 labels Jun 9, 2016

stale bot added the stale label Jul 6, 2018

stale bot closed this as completed Jul 13, 2018

defanator mentioned this issue Sep 23, 2019

contents_newline in file.managed is effectively ignored #54177

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

gpg renderer includes new line when decrypting data #31709

gpg renderer includes new line when decrypting data #31709

asyncmind0 commented Mar 7, 2016

asyncmind0 commented Mar 7, 2016

asyncmind0 commented Mar 7, 2016

jfindlay commented Mar 7, 2016

QuinnyPig commented Apr 11, 2016

vectorsigma commented Apr 12, 2016

QuinnyPig commented Apr 12, 2016

vectorsigma commented Apr 13, 2016

eliasp commented May 15, 2016

DanyC97 commented May 17, 2016

asyncmind0 commented Jun 9, 2016 •

edited

Loading

WillPlatnick commented Jun 9, 2016

vitalyisaev2 commented Oct 4, 2016 •

edited

Loading

jbouse commented Oct 4, 2016

stale bot commented Jul 6, 2018

vectorsigma commented Nov 18, 2018 •

edited

Loading

defanator commented Sep 23, 2019

defanator commented Sep 23, 2019

gpg renderer includes new line when decrypting data #31709

gpg renderer includes new line when decrypting data #31709

Comments

asyncmind0 commented Mar 7, 2016

Description of Issue/Question

Setup

Steps to Reproduce Issue

Versions Report

asyncmind0 commented Mar 7, 2016

asyncmind0 commented Mar 7, 2016

jfindlay commented Mar 7, 2016

QuinnyPig commented Apr 11, 2016

vectorsigma commented Apr 12, 2016

QuinnyPig commented Apr 12, 2016

vectorsigma commented Apr 13, 2016

eliasp commented May 15, 2016

DanyC97 commented May 17, 2016

asyncmind0 commented Jun 9, 2016 • edited Loading

WillPlatnick commented Jun 9, 2016

vitalyisaev2 commented Oct 4, 2016 • edited Loading

jbouse commented Oct 4, 2016

stale bot commented Jul 6, 2018

vectorsigma commented Nov 18, 2018 • edited Loading

defanator commented Sep 23, 2019

defanator commented Sep 23, 2019

asyncmind0 commented Jun 9, 2016 •

edited

Loading

vitalyisaev2 commented Oct 4, 2016 •

edited

Loading

vectorsigma commented Nov 18, 2018 •

edited

Loading