docker-compose file doesn't handle YAML in flow mode (where tabs are valid)

[Curtis-Fletcher]

According to the YAML spec, when in flow style, leading whitespace (tabs or spaces) is "neither content nor indentation." whereas in a docker compose file specified as *.json or *.yml a trivial case of valid flow style cosmetically indented with tabs

{
	"version": "3",
	"services": {
		"test": {}
	}
}

results in a YAML parse error:

ERROR: yaml.scanner.ScannerError: while scanning for the next token
found character '\t' that cannot start any token

Which is incorrect for tokens in flow mode.

This is for docker-compose version 1.18.0, build 8dd22a9 on MacOS

[shin-]

I believe you are misreading the spec - The only occurrence of the "neither content nor indentation" phrase you mention was related to whitespace appearing before comment lines. That same section also includes the following mention:

Note that indentation must not contain any tab characters

In the section on tabs, the following mention can be found:

Tabs may appear inside comments and quoted or block scalar content. Tabs must not appear elsewhere, such as in indentation and separation spaces.

The section describing flow styles doesn't make any mention of tabs being allowed, so this particular rule doesn't seem to be overridden like you suggest.

[Curtis-Fletcher]

None of that contradict the line quoted, "indentation" which is the determination of block scoping via whitespace does not apply to flow styling. So "indentation" correctly can't contain tabs, but when using flow styling, it is sigils and not "indentation" that determines scope. whitespace (including tabs) prior to keys and other content is not considered indentation.
as per:

http://www.yaml.org/spec/current.html#id2530054

Node content may be presented in either a flow style or a block style. Block content always extends to the end of a line and uses indentation to denote structure, while flow content starts and ends at some non-space character within a line and uses indicators to denote structure.

[shin-]

indentation and separation spaces.

                                ^^^^^^^^^^^^^^^

[Curtis-Fletcher]

Separation spaces are between tokens not prior. The fact is that well behaving YAML parsers correctly handle tabs preceding flow statement lines.

[TomaszChrosniak]

The case presented by @Curtis-Fletcher seems to be a special case where flow style is used in the entirety of the document, so technically there is no indentation going on, just presentation detail.

The spec actually shows a similar example of a flow-style used within a block-style parent element here.
EDIT: (pasting linked example for readability)

··# Leading comment line spaces are
···# neither content nor indentation.
····
Not indented:
·By one space: |
····By four
······spaces
·Flow style: [    # Leading spaces
···By two,        # in flow style
··Also by two,    # are neither
··→Still by two   # content nor
····]             # indentation.

You can see it the second-to-last line that tabs are also ignored in flow context, which would make the example valid.

EDIT:
To bring it closer to the example opening this issue, the below is also a valid flow-style YAML, to which the ignorance towards leading whitespaces can be applied.

{
"adjacent":value,
"readable":·value,
"empty":°
}

[Curtis-Fletcher]

Here is an example of flow style being validated with tabs as non-indent presentation by the YAML reference parser:

http://ben-kiki.org/ypaste/data/272/index.html

both tabs and space are evaluated as "non content whitespace"

Also note that LibYAML as exposed in (for example Perl) has no issue:

perl -MYAML::XS -MData::Dumper -e "print Dump(Load("{\n\t\"test\":\"fish\"}"))"

--- test: fish

[shin-]

Thanks for the links ; looks like I overlooked some of the details.

Unfortunately, neither pyyaml nor ruamel.yaml seem to support tabs in the given examples, and I don't know of any other viable YAML parser in Python that we could substitute. I'm also hesitant to rewrite a YAML scanner just to support this specific edge-case.

[Curtis-Fletcher]

Perfectly reasonable, if I happen to find a simple solution for python I assume I can post it here?

[shin-]

Sure, I'd be happy to look at it ; Moving off pyyaml has been a recurrent consideration given the lack of support it has received in recent years, so if there's a library out there that does it better that'd be an additional motivator.

[Curtis-Fletcher]

Seems like using the libYAML bindings in PyYAML rather than the pure python implimentation parses correctly:

python -c "import yaml;from yaml import load, CLoader as Loader; print yaml.dump(load("{\n\t\"test\":\"fish\"}", Loader=Loader))"

You can even wrap it so that systems without libYAML work:

from yaml import load, dump
try:
    from yaml import CLoader as Loader, CDumper as Dumper
except ImportError:
    from yaml import Loader, Dumper

[shin-]

Hmm, that requires libyaml to be installed, and pyyaml to be installed explicitly with libyaml support. I'm also not sure about Windows support, but maybe we can attempt parsing with libyaml and if absent, default to the python implementation. I'll look into it.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed because it had not recent activity during the stale period.