Issue with object references in configuration file

[stefan-k]

Hi. If this isn't the right place to ask such questions please let me know.

I'm trying to set up a basic configuration. The following seems to work (based on the tutorial you gave us this summer):

pipeline:
  - __type__: cobald.controller.linear.LinearController
    low_utilisation: 0.90
    high_allocation: 0.90
    rate: 1
  - __type__: tardis.resources.poolfactory.create_composite_pool
    configuration: 'tardis.yml'

If I understand the documentation correctly, this is discouraged. The preferred format is straightforward for the case of the LinearController:

  - !LinearController
    low_utilization: 0.9
    high_allocation: 0.9
    rate: 1

But it doesn't seem to work, as running cobald cobald_test.yml seems to get stuck at

2019-10-02 14:50:04 (589822) COBalD 0.10.0
2019-10-02 14:50:04 (589822) https://github.com/MatterMiners/cobald
2019-10-02 14:50:04 (589822) CPython 3.7.4 (/home/stefan/.virtualenvs/cobaldtest/bin/python)
2019-10-02 14:50:04 (589822) Using configuration cobald_test.yml
2019-10-02 14:50:05 (589822) Starting daemon services...
2019-10-02 14:50:05 (589822) ServiceRunner starting
2019-10-02 14:50:05 (589822) ServiceRunner adopts cobald.daemon.runners.service:ServiceUnit(...)
2019-10-02 14:50:05 (589822) starting all runners
2019-10-02 14:50:05 (589822) runner started: <cobald.daemon.runners.thread_runner.ThreadRunner object at 0x7f1ff8fa0310>
2019-10-02 14:50:05 (589822) runner started: <cobald.daemon.runners.trio_runner.TrioRunner object at 0x7f1ff8f90e10>
2019-10-02 14:50:05 (589822) ServiceRunner started
2019-10-02 14:50:05 (589822) runner started: <cobald.daemon.runners.asyncio_runner.AsyncioRunner object at 0x7f1ff8f90f50>
^C

Furthermore I'm unsure how to address TARDIS with the preferred format. I've tried:

pipeline:
  - __type__: cobald.controller.linear.LinearController
    low_utilisation: 0.90
    high_allocation: 0.90
    rate: 1
  - !!python/object/apply:tardis.resources.poolfactory.create_composite_pool
    configuration: 'tardis.yml'

which crashes with

2019-10-02 14:55:40 (601034) COBalD 0.10.0
2019-10-02 14:55:40 (601034) https://github.com/MatterMiners/cobald
2019-10-02 14:55:40 (601034) CPython 3.7.4 (/home/stefan/.virtualenvs/cobaldtest/bin/python)
2019-10-02 14:55:40 (601034) Using configuration cobald_test.yml
Traceback (most recent call last):
  File "/home/stefan/.virtualenvs/cobaldtest/bin/cobald", line 11, in <module>
    load_entry_point('cobald', 'console_scripts', 'cobald')()
  File "/home/stefan/coding/tardiscobald/testtest/cobald/src/cobald/daemon/core/main.py", line 40, in cli_run
    short_format=options.log_journal,
  File "/home/stefan/coding/tardiscobald/testtest/cobald/src/cobald/daemon/core/main.py", line 28, in run
    with load(configuration):
  File "/usr/lib64/python3.7/contextlib.py", line 112, in __enter__
    return next(self.gen)
  File "/home/stefan/coding/tardiscobald/testtest/cobald/src/cobald/daemon/core/config.py", line 61, in load
    translator=PipelineTranslator()
  File "/home/stefan/coding/tardiscobald/testtest/cobald/src/cobald/daemon/config/yaml.py", line 15, in load_configuration
    config_data = loader_instance.get_single_data()
  File "/home/stefan/.virtualenvs/cobaldtest/lib/python3.7/site-packages/yaml/constructor.py", line 43, in get_single_data
    return self.construct_document(node)
  File "/home/stefan/.virtualenvs/cobaldtest/lib/python3.7/site-packages/yaml/constructor.py", line 52, in construct_document
    for dummy in generator:
  File "/home/stefan/.virtualenvs/cobaldtest/lib/python3.7/site-packages/yaml/constructor.py", line 399, in construct_yaml_seq
    data.extend(self.construct_sequence(node))
  File "/home/stefan/.virtualenvs/cobaldtest/lib/python3.7/site-packages/yaml/constructor.py", line 122, in construct_sequence
    for child in node.value]
  File "/home/stefan/.virtualenvs/cobaldtest/lib/python3.7/site-packages/yaml/constructor.py", line 122, in <listcomp>
    for child in node.value]
  File "/home/stefan/.virtualenvs/cobaldtest/lib/python3.7/site-packages/yaml/constructor.py", line 92, in construct_object
    data = constructor(self, node)
  File "/home/stefan/.virtualenvs/cobaldtest/lib/python3.7/site-packages/yaml/constructor.py", line 420, in construct_undefined
    node.start_mark)
yaml.constructor.ConstructorError: could not determine a constructor for the tag 'tag:yaml.org,2002:python/object/apply:tardis.resources.poolfactory.create_composite_pool'
  in "cobald_test.yml", line 13, column 5

This could be related to yaml/pyyaml#266 which suggests that yaml.SafeLoader may be the problem.

I'm not sure if this is a bug or if I'm just being stupid again. I'd appreciate any pointers.

I'm running the current master of Cobald and Tardis on an up-to-date Arch Linux. Let me know if you need me to provide more information.

[maxfischer2781]

The !!python/object syntax does no longer work since 0.10.0 because COBalD always uses safe YAML loading. The error message definitely deserves being made more user-friendly.

---

As for the stuck process: Please try the cobald demon and transform the example YAML config gradually. You should be able to run it with a !LinearController:

pipeline:
  - !LinearController
    low_utilisation: 0.9
    high_allocation: 1.1
  - __type__: cobald_demo.draw_line.DrawLineHook
  - __type__: cobald_demo.cpu_pool.CpuPool
    interval: 1

[maxfischer2781]

I just realised that the documentation does not make it clear !!python/object works only for objects marked as safe. TARDIS does not yet mark its objects as such.

[giffels]

I think this is not possible for functions, right? Do we need to move to a factory class instead?

[maxfischer2781]

@giffels For the time being it should be enough to register the key configuration components of TARDIS as cobald YAML plugins.

[stefan-k]

Thank you for the quick replies!

As suspected, I was just stupid: I wrote low_utili**s**ation for the __type__ syntax (correct) and low_utili**z**ation for the !LinearController syntax (incorrect). It would be great if PyYAML would catch typos, but I guess thats not possible because it probably also needs to reflect python's **kwargs.

The !!python/object syntax does no longer work since 0.10.0 because COBalD always uses safe YAML loading.

Should the __type__ syntax be used in such a case?

I think this is not possible for functions, right?

I've used !!python/object/apply: [...] which I think should in principle work for functions (at least thats how I interpret the information in the PyYAML docs).

[maxfischer2781]

I think we can improve the error messages - PyYAML will let it slide when parsing but there should be an error when instantiating.

Should the __type__ syntax be used in such a case?

Yes, please use the old syntax where needed until we have switched everything to the new mechanism.

[stefan-k]

Thanks a lot!