Decripting failure: yaml comment next to inline array

[slavniyteo]

OS: Ubuntu 20.04
Sops version: 3.6.1 (latest)
GnuPG version: 2.2.19
Libgcrypt version: 1.8.5

Description: failure appears when try to decrypt yaml file with inline arrays followed with comments. Multiline arrays followed with comments are not affected.

Given a plain yaml file:

root:
  - a: [ "one" ]
  # comment
  - b: three
  - c: 
    - one
  # comment
  - d: three

Encrypt given file with sops -e plain.yaml > enc.yaml:

root:
-   a:
    - ENC[AES256_GCM,data:+BtY,iv:k+68b9m6cQiMT65fziaNyCGivg63kYKHfVxucpvNwM8=,tag:7ZNLc5zR/7Kyhe2jFB/QHg==,type:str]
    #ENC[AES256_GCM,data:lLqrh4ddZeQ=,iv:L2zKeYMiQJfrXzOBI8+lJjWn1ML4HcjCmMiaEAgUlvk=,tag:MPvUtZ2XKzAyBqdAgVA81w==,type:comment]
-   b: ENC[AES256_GCM,data:Z4ZjVMo=,iv:jyMM5r654ZjSxxX57VLExSjfE6vgrAGiSpApKGFhiKM=,tag:I0VF32T4ba7edljOr4U3cA==,type:str]
-   c:
    - ENC[AES256_GCM,data:37iY,iv:OTGpZzwsMSCOdygwyrOBOWOGUgf1Nl5Y6jtlJiJLHpI=,tag:b7QBj6uHn83jED1Nh2EEnQ==,type:str]
    - ENC[AES256_GCM,data:F+GvuUUR0KU=,iv:hVvjalB+oVcBbEW0hrfjsKk+yLmqvfXZ+XqYLVTvXs8=,tag:NKW2yAuw1CWGmpBlB1cKqQ==,type:comment]
-   d: ENC[AES256_GCM,data:JdGhqyk=,iv:XP3pq9zJzgPBTD1+U4I1KUmLMdy3pDxwpUOOsqlgeZA=,tag:2fa8xsBD5VeRXdkCc9Sx4Q==,type:str]
sops:
    kms: []
    gcp_kms: []
    azure_kv: []
    hc_vault: []
    lastmodified: '2021-02-01T08:50:35Z'
    mac: ENC[AES256_GCM,data:++FZYW1soXD4lktuuIVDlqKOJiD4wiFIl8/0LQRkRrzgQ+e4OuWwZiDsbH7gY44JpYfu8r3YW1vHeCXvo/l/6mglqnuq9BRA3/9rkFOckygXr88e/nnjzBLq+0KlM7DRyGzyslfUyY9qpaF81t7zknrhmWlxh351u9q/K5HhyFw=,iv:QSXe1BnLImXn95lX02KSs5Xx/t3z+aZkxXKVNfti878=,tag:Wu7XBO9fbH+6KTUFCrW+3A==,type:str]
    pgp:
    -   created_at: '2021-02-01T08:50:34Z'
        enc: |
            -----BEGIN PGP MESSAGE-----

            hQIMA5GvttmdEBizAQ/9H4Znr9UCgyYYM8ro3uoK5kI1m5ZLid6KVEaeiagXqaXc
            TAqSIPnWwnxr2X0ryLz/gV4iPkMwDpf/dcDY9kMq0uMovhNq+SciFK55naPC88S7
            aAE3RTAtSVyYGiLbTeCZi8GiRsRxIq9XhhTxlnS1shj3mucd2fnWZfkdSaSO5q46
            NTXBwacInNjtSvhZok6NGCiYplnhFi9o9ddaBP4IHALzibGMVCaqtyqFCsg4KL1b
            pN5zjW8e6OikkYqYBm/Xbh0FBLwlKT6isVhrBpxczJ4ox2eT4Qi5GlKwI2LFDc89
            kIxGC6yZ1OuUDFPBDUiMgWHpdBVXOazJ0R/KhYCHb1owzS7U19CuCPUbWk27HeCp
            tmwsWe8saaK+fv0MGXTKKC/lPpEunQu7HHzXnljiCmL2psH7Dr0yvE1HeSAfrLLV
            uVRoMcDB0x+/oN0usrlDI92r2o84ThW74HnfyDIJrPx8o8JNPR0XtNaIxqU305ij
            r8/6f1gdU59cFPCJb+NBB0/2AO89GQVJT5xG4EQRw75WVOALLmCGFIxkh7m5tdLe
            cso4ryIo7UW+7z05PiboeK7i3TjYLueup+QMDn+cYAujpNhRuXBXD1MRITj5I1TS
            cjItoFCSKcd+a1CB7I9aGhG6W4kxGu49CUYiDdPRGM16YH+UbfBVr4HAkkccTZ7S
            XAH1k5015JRQZmICjbnCdjU3pFLHpvS14oIxI9NpGkWidcEbwJZCDJv7nK1pu60Z
            cwFIA2LEekY3O+YXOM5wikhwHQzrUKe8JT8gIbreGiLuxNl8qbU+p4xnaEKJ
            =9y1v
            -----END PGP MESSAGE-----
        fp: A680501DE8BAA63EDC0BDE22656B9B08A3CEC525
    -   created_at: '2021-02-01T08:50:34Z'
        enc: |
            -----BEGIN PGP MESSAGE-----

            hQEMA/Wwuogfa25UAQgAkWkt1yLVmmj3pVeOSvSRIiaFWgpt8CfESzmqW7go9Fb7
            UykGrudpiYtxGKT0nT4Y0usFc26/LjsIxfdGGE+BWKNFwntQxSvd2uq0mLeuCCxd
            FrLkh7E5vqSnCXkamfEy1zH3KRadAH3HqMbO9qqkdKmcnJeetmojetizr+S1ejfy
            6mjgyckhupsI7j5tEWnkEJchKV9sRW7Y5GGz4Hb5mGyDWjikNlnX+DZIZke/gGzP
            DLjdtRq3CnhXJWeNkmTkt6KkIlXCOpSPaTF3turiOiGkNCWXR9wUWOihxeLBMSIv
            QYJbmCWwlJ6Ko1ymRuYTK/Iy4iqzuARz0LxhzWwvCtJcAePRLdFOw8f9fyl5166p
            XtAjwAzec7QcHJR4NJrdpxn1QyuxHY/n5Y+n80lV1dp5HSsx2xgesGXCOx4CVNSe
            zFaLok0ps2tOB4WDopKWPhYwwkmTseVpRkfyVCQ=
            =0qCb
            -----END PGP MESSAGE-----
        fp: E4E6E8662CF0B8EF683C39A991C35762F87A0EE1
    unencrypted_suffix: _unencrypted
    version: 3.6.1

Decrypting enc.yaml results in warning message and partially decrypted original file (except one of the comments):

$ sops -d enc.yaml
[SOPS]   WARN[0000] Found possibly unencrypted comment in file. This is to be expected if the file being decrypted was created with an older version of SOPS.  comment="ENC[AES256_GCM,data:lLqrh4ddZeQ=,iv:L2zKeYMiQJfrXzOBI8+lJjWn1ML4HcjCmMiaEAgUlvk=,tag:MPvUtZ2XKzAyBqdAgVA81w==,type:comment]"

root:
-   a:
    - one
    #ENC[AES256_GCM,data:lLqrh4ddZeQ=,iv:L2zKeYMiQJfrXzOBI8+lJjWn1ML4HcjCmMiaEAgUlvk=,tag:MPvUtZ2XKzAyBqdAgVA81w==,type:comment]
-   b: three
-   c:
    - one
    # comment
-   d: three

[autrilla]

This unfortunately happens because SOPS moves your comment during the round-trip, and that breaks decryption since we use the 'path' of the item in the tree as the additional data for AES-GCM.

Basically, in your original document, SOPS believes your comment is under root, but on the encrypted document, it thinks the comment is under root > a.

This is hard to fix.

[slavniyteo]

Thank you for your reply.

It is sad to hear that this bag is hard to fix due to some internal reasons. Sometimes it is impossible to change the plain-text yaml file (e.g. when it is generated by 3rd-party software).

Worth it documenting this exceptional case in the Important information of types of the README.md alongside yaml anchors and yaml/json root array problems?

[felixfontein]

This might be fixed by #791, though probably only after re-encoding the file. yaml.v3 hopefully has less problems (once go-yaml/yaml#690 is fixed), and receives more fixes, at least over time.