yaml file sequence items spaces after hyphen #514

lukeab · 2019-09-03T10:55:51Z

Ubuntu 18.04 and archlinux
sops v3.3.1

Have weird yaml thing, encrypted a file with lines that looks like

  ENV:
  - name: "AWS_KEY"
    value: "BLAHBLAHBLAH"
  - name: "AWS_REGION"
    value: "eu-west-1"
... etc..

SO i do `sops --encrypt file.yaml --output > file-sops.yaml
then do sops --decrypt file-sops.yaml > file-dec.yaml
the decrypted file now has those lines looking like

    ENV:
    -   name: AWS_KEY
        value: BLAHBLAHBLAH
    -   name: AWS_REGION
        value: eu-west-1

So, it's changed the yaml indent to 4 spaces, that's fine, it's in the spec,
But it's added 2 spaces to the 1 space that was between the key and the - for each sequence item key
yamlint blows up on this. though it doesn't cause issues in the python script that i then pass this config yaml to in my usecase, the CI issues yamllint causes are the problem.

The resulting yamllint output on my actual file in my environment, shows up like:

~$> yamllint myfile-dec.yaml 
myfile-dec.yaml
  1:1       warning  missing document start "---"  (document-start)
  10:8      error    too many spaces after hyphen  (hyphens)
  10:5      error    wrong indentation: expected 8 but found 4  (indentation)
  12:8      error    too many spaces after hyphen  (hyphens)
  14:8      error    too many spaces after hyphen  (hyphens)
  16:8      error    too many spaces after hyphen  (hyphens)
...etc...

The text was updated successfully, but these errors were encountered:

lukeab · 2019-09-03T11:09:40Z

reproduced the issue with a standalone file

example

~$> cat myfile.yaml

ENV:
- name: AWS_KEY
  value: BLAHBLAHBLAH
- name: AWS_REGION
  value: eu-west-1

~$> SOPS_PGP_FP=<<mykey>> sops --encrypt myfile.yaml --output > myfile.sops.yaml
~$> cat myfile.sops.yaml

ENV:
-   name: ENC[AES256_GCM,data:An20+bmHvg==,iv:ea0DqrSFpb9T6wlAClBqRbpg5UD5/lvW/doKVhFsbpc=,tag:bNMPWBuIWg71okuBz2atvw==,type:str]
    value: ENC[AES256_GCM,data:iJzPHPrKZ5N62P2A,iv:7XDXT9FWmIjSQ4W+cPk6jQFp+fiMC72Jt2fEkt3Zdys=,tag:Zk4Dp7T1i/IMCxXRNxGWjg==,type:str]
-   name: ENC[AES256_GCM,data:ivyP16RZQlJXdw==,iv:1wTRa1AyoTYpeOUCHobEr4s2WtVc2wzThUmEQjz3AB8=,tag:vjBVjXgI8U9xoC4K5iiuSA==,type:str]
    value: ENC[AES256_GCM,data:LMmCluV9PU+j,iv:Z7cKzjBqpzFsh6wmMdh8xpHIKYniUXeh5qB4wAAKlM4=,tag:8DIWkROViYsIDvALiNlSGg==,type:str]
sops:
    kms: []
    gcp_kms: []
    azure_kv: []
    lastmodified: '2019-09-03T11:05:59Z'
    mac: ENC[AES256_GCM,data:1BDKFlREqTIuoa+sfC+swZlJxPcOpXY7qEn8I+NY9zUsPUmH5OShRunbbh15ZCf2Gr0/bnXzfxZVwUZm7ZYeVu1ls3EQkkf2sPY+/7DVr+1VR0U5zRPJuLZdbFNJ295RDqNicQDmgBjUbsbXTEs+zdoNY5yHHAHhymjIc+jbC6g=,iv:x2whgFJktAxNSJjgsJFLnqx4wc6VhEcO/ZCpaLuo5W4=,tag:LagzO+RSj0ATcxfg0vQd/w==,type:str]
    pgp:
    -   created_at: '2019-09-03T11:05:59Z'
        enc: |
            -----BEGIN PGP MESSAGE-----

            hQIMA6u00URofj69ARAAhbAHTmXEEMvoUip76RX3EklIkJSiSe8eDV1NkSEx1b8v
            2IDF/6HI2DuOaiR53SbfIA1dJKaWIiF/QQNZTxGwAQXFnCdKEByAk1qomSfLHYhv
            siJn/solf94F/YMGmWW+rCxTip3HZP7ZSk62HUyXI/byfEEnmddHmkQiU8HnGX2N
            EcwYZXgsp35Yync0YNrYnavuTvrwtpcnsYDrYVC9IjtNRXGQeZhvITYH8X1eMiIW
            dfam72MoiigBBIIVCqNlvchWNwBrehfsr5IotKM2uOXLU4z3XBrSMNwMbhev0lS0
            3KRW2EMYNcGXkMgZmCBFrlbxotREyCEU3OR7eOS4D5uHcIUAI3NFC6XGSZCnVJIx
            PQrBqnerOZSC1V8+H4zbfmnViwQ8bGtUHGPtg53fPC1BZltx3RsKJQALEHz4a3bN
            3KEAo4iOx9u7tpZWeLZZD0YPxEvqLke1wND6exOUMW0Y0GBxgG5uqusk6tFLiejl
            HXq6envDqAU1rEpcp2QBKJQKodcU6RaWfKOQewfrstt5IgFWctNz69Ig/Fpu85p8
            IP8ZAY7W2ZMwjVkJp1FIxxYU7+sDwhhodUJ0dIF/Gggb0KJsF85aOiI26DcMTtd0
            VPDSKg9B5OZxDgPj4y3PLzWblkDTASylj7jlNe1Ke2hWQBuRxvxWwR6gjcl8WnvS
            XAHbWStg12rkDeRM6OJD54sJYhVQyerCMOwm7yCX2uEuFE9ddUfWMLyGPcdFAB7R
            iTZNDXu9erC0LG7zc/aCipnUXdbHDoNVa/MgRtMyXpP6xs9cZzT1E2qMOzTw
            =Gkhz
            -----END PGP MESSAGE-----
        fp: 83BFC84EFA98B9284D3939220954AFBD84AD4B85
    unencrypted_suffix: _unencrypted
    version: 3.3.1

~$> SOPS_PGP_FP=<<mykey>> sops --decryptt myfile.sops.yaml --output > myfile.dec.yaml
~$> cat myfile.dec.yaml

ENV:
-   name: AWS_KEY
    value: BLAHBLAHBLAH
-   name: AWS_REGION
    value: eu-west-1

~$> yamllint myfile.dec.yaml

myfile.dec.yaml
  1:1       warning  missing document start "---"  (document-start)
  2:4       error    too many spaces after hyphen  (hyphens)
  2:1       error    wrong indentation: expected 2 but found 0  (indentation)
  4:4       error    too many spaces after hyphen  (hyphens)

lukeab · 2019-09-03T11:10:27Z

just a note, origional bug was encountered while use aws kms for encryption, so doesn't seem to be related to encryption type anyway.

autrilla · 2019-09-04T07:16:02Z

It seems to me that the output is spec compliant. We don’t really make any guarantees on the styling of the YAML, and we especially do not guarantee the output style will be consistent with the input style. The yaml library we use has now been updated such that we could theoretically support it, but nobody has put in the time yet.

…

On Tue 3. Sep 2019 at 13:10, Luke Ashe-Browne ***@***.***> wrote: just a note, origional bug was encountered while use aws kms for encryption, so doesn't seem to be related to encryption type anywya. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#514>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AARH4V3APIYBCGOXNZ4O5RLQHZA2LANCNFSM4ITE3SGQ> .

lukeab · 2019-09-04T09:48:22Z

hmm, seems there is a yamllint flag to control the number of spaces allowed after a hyphen
https://yamllint.readthedocs.io/en/stable/rules.html#module-yamllint.rules.hyphens

I assumed since, by default, yamllint threw an error, it was actually a violation of the spec, not just a style concern, but now you have cast doubt on my assumption. Is there any rules in the spec about spaces after sequence item hyphens?

I thought i might try fix it, but you say there is a new library in use, i'd give it a shot, but i haven't been able to find where in the code this fix could be affected easily yet. A pointer in the right direction would be motivating for me to try again.

lukeab · 2019-09-04T09:56:49Z

Answering my own uncertainty: had a look at the spec
https://yaml.org/spec/1.2/spec.html#id2759963
and
https://yaml.org/spec/1.2/spec.html#id2797382

Seems it just says

The “-” indicator must be separated from the node by white space.

So there's no guarantee in the spec that it should be only one space character.

kunickiaj · 2019-09-06T22:20:11Z

Ran into this as well, formatting inconsistency from the rest of our files is a bit inconvenient. Would also be interesting in understanding where we could add support for control of the formatting.

The more important issue I encountered, is that since keys are reordered, block comments no longer appear next to the keys they're intended for.

valueB: 1234
# comment for valueA
valueA: abcdef

yields something like

# comment for valueA
valueB: 1234
valueA: abcdef

autrilla · 2019-09-07T12:05:34Z

@kunickiaj we would need to upgrade to https://github.com/go-yaml/yaml/tree/v3, parsing into https://godoc.org/gopkg.in/yaml.v3#Node.

FWIW, SOPS does not reorder keys. #300 (comment) has more information on why the comment behavior you see happens.

rbabyuk · 2020-11-19T12:41:17Z

hi all,

I am hitting this issue as well,
sops version is 3.6.1

onedr0p · 2021-02-03T20:52:19Z

I ran into this issue too...

sops:
    kms: []
    gcp_kms: []
    azure_kv: []
    hc_vault: []
    lastmodified: '2021-02-03T20:45:32Z'
    mac: ENC[AES256_GCM,data:sqFu3XksD/IKE4b6WFW/hJ39ba//PD4YikgmENv2K4rNcR0nZCh5e/sEUPKSDIkUM+fyZeNlwwxodhh5oVMsY+0bQ5ISWbH+dxi+h3j1GrMgfms+EYImik49bjSYbcYBj9BeUWtnzz8Ab/PHUqMrbIMNVaHWAfbuClbooYmyKeQ=,iv:tbDF2y3+86qrKEDv8vJS2zIBB/PzJUe20nLX513+5s8=,tag:3GeY6ZqqSdFJqvzVMzcmhA==,type:str]
    pgp:
    -   created_at: '2021-01-27T18:14:44Z'
        enc: |
            -----BEGIN PGP MESSAGE-----

            hQEMAyUpShfNkFB/AQf/RxF3NLkoonJP4wQqELkfPtJl5Oar1UH7OBYwzep3uEPE
            MCPFeI/Fdr/8WxT0x6ieUC7odZX1J9tNzUeStyaWCS3CX6+zmEb0uavV7RlewdFc
            noIPyM7Q4R6/L6QyGWx6aPwkUq8qvH+8hk6VdqtAdipY8uBVogO/+BHd502nE+5E
            WqzxRFxFr7/6mjFHrPYcab5QvSoVc3Lrhh08hKXJYpquPrjDI4VYkJhTxmbaZK5w
            e676rq+eob8tQ7Nz2nnw5W51Oy/i2Yt6Q/wHj5dSPc4g+LMVUPwdx72+OB829uZR
            3FNKH2EJ6BCVVOe7JUueeoHjz1/UCooAx9vwAlJ8MNJeAbrEJPL1OUEJfkqJuoQk
            v4LGyJD4Gi+58FqLcOpx1DGqNsMZ8ElGKbBUKC3Resksd4K4uTfMyy0HwDRpvUdG
            M47W5K9K28MePouf7T/lsemwoYB59GSoeqIviyeteQ==
            =DSca
            -----END PGP MESSAGE-----
        fp: 3D16CEE4A27381B4
    unencrypted_suffix: _unencrypted
    version: 3.6.1

and yamllint output:

yamllint secret.enc.yaml
my-secrets.enc.yaml
  1:1       warning  missing document start "---"  (document-start)
  9:8      error    too many spaces after hyphen  (hyphens)
  9:5      error    wrong indentation: expected 8 but found 4  (indentation)

The only workaround I found is to have yamllint ignore filenames matching a certain regex, which works until you need to embed the sops data into an existing yaml file.

felixfontein · 2021-03-03T07:43:50Z

This should be fixed by the change to yaml.v3 in the current develop branch.

onedr0p · 2021-03-30T12:39:17Z

I can confirm formatting is still the same, even with the new release.

felixfontein · 2021-03-30T13:20:00Z

The latest version uses a different YAML library, and lists are definitely serialized differently. So formatting is definitely not the same. If yamllint likes the new style better is another question :-)

onedr0p · 2021-03-30T13:29:17Z

You are correct @felixfontein apologies, I did not test the right binary :(

Before...

    pgp:
    -   created_at: '2021-01-27T18:14:44Z'
        enc: |

After...

    pgp:
        - created_at: "2021-03-25T00:36:44Z"
          enc: |

almereyda · 2022-07-27T23:32:37Z

Would also be interesting in understanding where we could add support for control of the formatting.

It appears support for https://editorconfig.org/ would also allow us to make some linters happy.

Ph0tonic · 2023-11-25T20:34:02Z

Hi,
I think that this issue can be closed as the formatting is now different and the indentation can now be configured.

lukeab changed the title ~~yaml file sequence items~~ yaml file sequence items invalid spaces Sep 3, 2019

lukeab changed the title ~~yaml file sequence items invalid spaces~~ yaml file sequence items spaces after hyphen Sep 5, 2019

onedr0p mentioned this issue Feb 3, 2021

Investigate using another secret method onedr0p/home-ops#692

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

yaml file sequence items spaces after hyphen #514

yaml file sequence items spaces after hyphen #514

lukeab commented Sep 3, 2019 •

edited

Loading

lukeab commented Sep 3, 2019 •

edited

Loading

lukeab commented Sep 3, 2019 •

edited

Loading

autrilla commented Sep 4, 2019 via email

lukeab commented Sep 4, 2019

lukeab commented Sep 4, 2019

kunickiaj commented Sep 6, 2019 •

edited

Loading

autrilla commented Sep 7, 2019

rbabyuk commented Nov 19, 2020

onedr0p commented Feb 3, 2021 •

edited

Loading

felixfontein commented Mar 3, 2021

onedr0p commented Mar 30, 2021

felixfontein commented Mar 30, 2021

onedr0p commented Mar 30, 2021

almereyda commented Jul 27, 2022

Ph0tonic commented Nov 25, 2023

yaml file sequence items spaces after hyphen #514

yaml file sequence items spaces after hyphen #514

Comments

lukeab commented Sep 3, 2019 • edited Loading

lukeab commented Sep 3, 2019 • edited Loading

example

lukeab commented Sep 3, 2019 • edited Loading

autrilla commented Sep 4, 2019 via email

lukeab commented Sep 4, 2019

lukeab commented Sep 4, 2019

kunickiaj commented Sep 6, 2019 • edited Loading

autrilla commented Sep 7, 2019

rbabyuk commented Nov 19, 2020

onedr0p commented Feb 3, 2021 • edited Loading

felixfontein commented Mar 3, 2021

onedr0p commented Mar 30, 2021

felixfontein commented Mar 30, 2021

onedr0p commented Mar 30, 2021

almereyda commented Jul 27, 2022

Ph0tonic commented Nov 25, 2023

lukeab commented Sep 3, 2019 •

edited

Loading

lukeab commented Sep 3, 2019 •

edited

Loading

lukeab commented Sep 3, 2019 •

edited

Loading

kunickiaj commented Sep 6, 2019 •

edited

Loading

onedr0p commented Feb 3, 2021 •

edited

Loading