YAML validation issue

[RoboMagus]

The problem

When writing the following automation and asking HA to validate my config, no warnings or errors show up. Also when reloading the automations, nothing warns me about the incorrect quoting. When reloading, the entire file seems to be just skipped completely, as other automations in the same file will also be missing from the UI.

- id: '1658085239190'
  alias: Config validation test
  description: ''
  trigger:
  - platform: time
    at: 00:02:03
  condition: []
  action:
  - service: script.notify_admin
    data:
      # 3 quotes here are not caught at all!
      title: 'Here's something that does not work...!'
      message: failing
  mode: single

Yes this is user error, but it feels like something that should still be caught by the config validation.

What version of Home Assistant Core has the issue?

2022.7.5

What was the last working version of Home Assistant Core?

No response

What type of installation are you running?

Home Assistant Container

Integration causing the issue

No response

Link to integration documentation on our website

No response

Diagnostics information

Nothing is shows in the logs, also not with automation logging set to debug.

Example YAML snippet

automation:
- id: '1658085239190'
  alias: Config validation test
  description: ''
  trigger:
  - platform: time
    at: 00:02:03
  condition: []
  action:
  - service: script.notify_admin
    data:
      # 3 quotes here are not caught at all!
      title: 'Here's something that does not work...!'
      message: failing
  mode: single

Anything in the logs that might be useful for us?

No response

Additional information

No response

[ghost]

The YAML loader has been switched over a new C based loader in #73337. For those of us who use primarily YAML to write automations and configure sensors the old pure python based loader may be preferable due to the ease of debugging and graceful handling of errors. Is there a way via configuration.yaml to force the use of the python loader?

[RoboMagus]

I was hoping this would be something that could be fixed in Home Assistant, but seeing as this is an external utility that's misbehaving I agree that having the option to select the slower but stable one instead would definitely have my vote.

[ghost]

Looking at the code it appears the intended behavior is to try the C loader first. If it fails, then use the python loader to get a useful debug message to dump into the logs which is a reasonable approach. CSafeLoader is failing in a manner not triggering the exception.

[RoboMagus]

@bdraco, could you weigh in on this?

[ghost]

Other untriaged issues related to new YAML loader #74806, #75006, #75070

[bdraco]

@bdraco, could you weigh in on this?

Reproduction is proving difficult on this one but it might be because the examples are pasted into GitHub and they have been modified in some way via that process.

It would be helpful if someone has an example configuration file that has this behavior hasn't been pasted into GitHub and was uploaded as a .txt attachment

[RoboMagus]

It would be helpful if someone has an example configuration file that has this behavior hasn't been pasted into GitHub and was uploaded as a .txt attachment

Here ya go:
automations.yaml.txt

[bdraco]

Thanks. Making progress now

[bdraco]

Looks like you opened yaml/pyyaml#655 as well

Thats good you can get it to throw. I was worried you wouldn't be able to

[bdraco]

Oh but it looks like the exception is a usage error not a parser error

[bdraco]

yaml.parser.ParserError: while parsing a block mapping
  in "automations.yaml", line 11, column 7
did not find expected key
  in "automations.yaml", line 11, column 20

from yaml import load, dump
from yaml import CLoader as Loader, CDumper as Dumper
#from yaml import Loader, Dumper

with open("automations.yaml", "r") as fh:
    print(load(fh, Loader=Loader))

I get a parser error with that

[bdraco]

>>> from homeassistant.util.yaml import load_yaml
>>> load_yaml("config/automations.yaml")
OrderedDict()

But I get an empty thing in home assistant

[RoboMagus]

But I get an empty thing in home assistant

That's indeed what I've been able to find as well when troubleshooting a bit myself. The fact that it simply does not return anything instead of an error.

Based on the testing done for yaml/pyyaml#655, it seems like what's going on in CSafeLoader is that for malformed inputs, the load function does not even exist for the object created.

[bdraco]

So the object isn't a TextIO in this case its a TextIOWrapper which we don't have in the list of objects to rewind so we can try again

[bdraco]

needs to change to if isinstance(content, (StringIO, TextIO, TextIOWrapper)):

[bdraco]

So the test is mocking out the IO which ends up returning StringIO instead of TextIO

[bdraco]

@RoboMagus fixed in #75510

Thanks for the report

[RoboMagus]

Thanks for the great work @bdraco!
I was also going to suggest adding a test case for catching bad yaml, but looking at your PR you already beat me to it ;)

[bdraco]

Thanks for the great work @bdraco!

I was also going to suggest adding a test case for catching bad yaml, but looking at your PR you already beat me to it ;)

Well we had a test but there was too much mocking so I added one that is more end to end since the problem was the mocked I/O had the different object class