Argument handling in shell scripts broken by ANSI escape sequences in colorized output

[brianshumate]

Argument handling from shell scripts (i.e. stdin not tty 🤣) seems to be broken in the new CLI. In this case, vault operator unseal is not accepting a key as an arg from within the included simple shell script, but accepts it as expected from a shell in a terminal.

Environment:

CLI: Vault v0.9.3 ('5acd6a21d5a69ab49d0f7c0bf540123a9b2c696d')
Server: 0.9.3
Linux amd64

Vault Config File:

N/A

Startup Log Output:

N/A

Expected Behavior:

vault operator unseal executed within my shell script should accept a variable and use it as an argument to the command.

Actual Behavior:

Error unsealing: Error making API request.

URL: PUT http://localhost:8200/v1/sys/unseal
Code: 400. Errors:

* 'key' must be a valid hex or base64 string

Steps to Reproduce:

Use this shell script:

#!/bin/sh

vault version

vault operator init > vault.tmp

cat vault.tmp

vault status

K1=$(sed '2q;d' vault.tmp | awk '{print $NF}')

echo "Got unseal key $K1"

vault operator unseal "$K1"

Initially, I suspected quoting, but this script worked before upgrading to 0.9.2; it makes no difference whether "$K1" is quoted or not in the vault operator unseal command.

Important Factoids:

N/A

References:

N/A

[tniswong]

Actually, the CLI is working correctly. What's really going on is that this latest update added ansi color markup to cli output.... with no way of disabling it. Your unseal key actually contains hidden ansi color markup.

You can see it by doing echo "$K1" | cat -v

This problem affects me in many scenarios, and I've had to work around it by doing something like the following:

K1=$(sed '2q;d' vault.tmp | awk '{print $NF}' | sed 's/\x1b\[[0-9;]*[a-zA-Z]//g')

I found this solution here: https://superuser.com/questions/380772/removing-ansi-color-codes-from-text-stream.

I really hope the vault team will add some sort of flag to disable color output for any command, or even better an environment variable to disable color output globally.

[brianshumate]

That is indeed the issue @tniswong. Perhaps it would make sense to leave ANSI escapes off of some values in output to avoid these issues.

[tniswong]

If I may request a secondary solve, at least for my case, would be allowing the -format flag for operator commands, as this would eliminate my need to grep/awk/sed for values from the shell.

[norman-abramovitz]

Ran into this issue even trying to process vault version command since it contains ansi control sequences.
Issue also exists in 1.9.3 but not in 1.9.1

vault version >x-1.9.2
od -hc x-1.9.2
0000000 5b1b 3b30 6d30 6156 6c75 2074 3076 392e
033 [ 0 ; 0 m V a u l t v 0 . 9
0000020 322e 2820 3327 6662 3738 3333 6463 3936
. 2 ( ' 3 b f 8 7 3 3 c d 6 9
0000040 6262 6130 3163 6434 3961 6161 3661 3331
b b 0 a c 1 4 d a 9 a a a 6 1 3
0000060 6235 6263 6637 3137 6330 3563 2766 1b29
5 b c b 7 f 7 1 0 c c 5 f ' ) 033
0000100 305b 0a6d
[ 0 m \n

[jhunt]

In particular, with the vault version use case, the ANSI escape codes are [0;0m, bracketing the output, to turn off the coloring. Would it be possible to just omit the reset sequence when no coloring is involved?

[srix55]

I was going mad trying to figure this out. I thought I had a problem with groovy's execute. Was frantically updating groovy / java.. since the commands run fine from console & script files.

[norman-abramovitz]

This is definitely a breaking change when they do not provide a way to override the color coding or at least follow conventions that only terminal output gets color-coded and not the data being redirected. There way too many examples of how to do this right to get this wrong.

[jefferai]

@norman-abramovitz We will be fixing that particular behavior for 0.9.4, but it's important to note that we have never made any sorts of claims or promises that output from the CLI would remain consistent, as it's designed for human interactive use, and generally do not take into consideration scripts when deciding whether to change the main output format. The main commands (read/write/etc.) all support a format flag that allows for structured output (JSON or YAML) that is suitable for using in scripts, and to make things easier we will be extending this to all commands for 0.9.4. Additionally, the CLI is purely an HTTP API client, so simple curl commands can be used in place of any CLI command and the output is always structured since it's a JSON API.

[luke-clifton]

FYI, Vault v1.0.3 seems to be printing escape sequences even when not going to a tty again.

[todd-dsm]

Confirmed, it's back; can we reopen this one?

$ cat -vte /tmp/jelly.out 
^[[0mUnseal Key 1: S4a1OJRx+2AB5hRmDxkeLy50OZfDmeOw1We794DjeBs=^[[0m^M$
^[[0m^[[0m^M$
^[[0mInitial Root Token: s.uYWlPlLj3vXC2aJjmhZTLwW1^[[0m^M$
^[[0m^[[0m^M$
...

Attempted to fix by dropping this line into values.yaml

  # extraEnvironmentVars is a list of extra enviroment variables...
  # used to include variables required for auto-unseal.
  extraEnvironmentVars:
    VAULT_CLI_NO_COLOR: 1

Verified the parameter/value made it to the container:

$ kubectl exec -it vault-0 -- printenv | grep VAULT_CLI_NO_COLOR
VAULT_CLI_NO_COLOR=1

This had no effect on the output. The garbage text is still in there.

• Please remove this stuff from the output, or
• Provide an option to pass as an argument (kubectl exec -it vault-0 -- vault operator init -no-color), or
• set the environmental variable in the container.

[zulrang]

Nothing like getting errors due to obscure stuff like this when just trying to follow the basic tutorials.

[grove-mountain]

It might be helpful to know that Vault also allows formatting output for init. e.g.:

vault operator init -format=json

{
"unseal_keys_b64": [
"tmH6BPxdmNsrpcbQ2CjbD476RdxZBxAS1X/XRrb5pLYX",
"xsu56VFliTL8Jbi/vBNVO/O4dZ8gNITwh6ukpFOsmEgr",
"/WBobe4b74VhuL/WhHjsjDZpjoHcdFk93HMdCZf8UHvW",
"HTQvh/BqzVgALEjzSV5kTm9oBjR+awM3hH+SL7q/6YIf",
"SQkJ1DSyluVzSkcaEdaFla4+91AIp2DS76KXl9u0W8Ov"
],
"unseal_keys_hex": [
"b661fa04fc5d98db2ba5c6d0d828db0f8efa45dc59071012d57fd746b6f9a4b617",
"c6cbb9e951658932fc25b8bfbc13553bf3b8759f203484f087aba4a453ac98482b",
"fd60686dee1bef8561b8bfd68478ec8c36698e81dc74593ddc731d0997fc507bd6",
"1d342f87f06acd58002c48f3495e644e6f6806347e6b0337847f922fbabfe9821f",
"490909d434b296e5734a471a11d68595ae3ef75008a760d2efa29797dbb45bc3af"
],
"unseal_shares": 5,
"unseal_threshold": 3,
"recovery_keys_b64": [],
"recovery_keys_hex": [],
"recovery_keys_shares": 5,
"recovery_keys_threshold": 3,
"root_token": "s.mWHTzcVCIFjp5ObE6TzVgUsy"
}

[todd-dsm]

have time for a quick call?

…

On Mon, Apr 27, 2020 at 8:16 PM Jake Lundberg ***@***.***> wrote: It might be helpful to know that Vault also allows formatting output for init. e.g.: vault operator init -format=json { "unseal_keys_b64": [ "tmH6BPxdmNsrpcbQ2CjbD476RdxZBxAS1X/XRrb5pLYX", "xsu56VFliTL8Jbi/vBNVO/O4dZ8gNITwh6ukpFOsmEgr", "/WBobe4b74VhuL/WhHjsjDZpjoHcdFk93HMdCZf8UHvW", "HTQvh/BqzVgALEjzSV5kTm9oBjR+awM3hH+SL7q/6YIf", "SQkJ1DSyluVzSkcaEdaFla4+91AIp2DS76KXl9u0W8Ov" ], "unseal_keys_hex": [ "b661fa04fc5d98db2ba5c6d0d828db0f8efa45dc59071012d57fd746b6f9a4b617", "c6cbb9e951658932fc25b8bfbc13553bf3b8759f203484f087aba4a453ac98482b", "fd60686dee1bef8561b8bfd68478ec8c36698e81dc74593ddc731d0997fc507bd6", "1d342f87f06acd58002c48f3495e644e6f6806347e6b0337847f922fbabfe9821f", "490909d434b296e5734a471a11d68595ae3ef75008a760d2efa29797dbb45bc3af" ], "unseal_shares": 5, "unseal_threshold": 3, "recovery_keys_b64": [], "recovery_keys_hex": [], "recovery_keys_shares": 5, "recovery_keys_threshold": 3, "root_token": "s.mWHTzcVCIFjp5ObE6TzVgUsy" } — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#3869 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AASZRP4UEPT4MUQJIIDUST3ROZDCNANCNFSM4EOK54IA> .