Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixes #35: Adding support for loading configs from stdin #36

Merged
1 commit merged into from
Aug 17, 2022
Merged

Conversation

ghost
Copy link

@ghost ghost commented Aug 12, 2022

Changes introduced with this PR

This PR adds support for loading config files from the stdin by specifying -f - in plugins.


By contributing to this repository, I agree to the contribution guidelines.

@ghost ghost linked an issue Aug 12, 2022 that may be closed by this pull request
@ghost ghost self-requested a review August 12, 2022 09:28
@ghost ghost force-pushed the file-from-stdin branch from df12dad to 1290ac0 Compare August 12, 2022 09:34
@dustinblack
Copy link
Member

Testing locally, I no longer get a file extension error trying to pass stdin to -f -, but the same params file that works when included in the container is not working when being cat'd to the container stdin.

This works as expected w/ the smallfile-example.yaml file included in the container.

$ podman run --rm smallfile-test -f smallfile-example.yaml

But I get a typing error traceback with the stdin aproach.

$ cat smallfile-example.yaml | podman run --rm smallfile-test -f - --debug
Invalid input encountered while executing step 'workload' from file '-':
  Validation failed: Must be a dict, got NoneType

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/arcaflow_plugin_sdk/schema.py", line 1644, in _unserialize_input
    return step.input.unserialize(data)
  File "/usr/local/lib/python3.9/site-packages/arcaflow_plugin_sdk/schema.py", line 1193, in unserialize
    raise ConstraintException(path, "Must be a dict, got {}".format(type(data).__name__))
arcaflow_plugin_sdk.schema.ConstraintException: Validation failed: Must be a dict, got NoneType

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/arcaflow_plugin_sdk/plugin.py", line 648, in _execute_file
    output_id, output_data = s(step_id, data)
  File "/usr/local/lib/python3.9/site-packages/arcaflow_plugin_sdk/schema.py", line 1708, in __call__
    input_param = self._unserialize_input(step, data)
  File "/usr/local/lib/python3.9/site-packages/arcaflow_plugin_sdk/schema.py", line 1646, in _unserialize_input
    raise InvalidInputException(e) from e
arcaflow_plugin_sdk.schema.InvalidInputException: Validation failed: Must be a dict, got NoneType

Params file for reference:

smallfile_params:
  top: "/var/tmp"
  operation: "create"
  threads: 5
  file-size: 64
  files: 100
  auto-pause: 'N'
  fsync: 'N'
  files-per-dir: 100
  dirs-per-dir: 10
  stonewall: 'Y'
  finish: 'Y'
  prefix: 'foo'
  suffix: 'bar'
  hash-into-dirs: 'N'
  same-dir: 'N'
  verify-read: 'Y'
  incompressible: 'N'
  cleanup-delay-usec-per-file: 0

@ghost ghost force-pushed the file-from-stdin branch from 1290ac0 to 5ebf0ed Compare August 16, 2022 18:28
@ghost
Copy link
Author

ghost commented Aug 16, 2022

I've updated the integration to use sys.stdin.buffer instead of sys.stdin, please check if it helps. Containers must be run with -i to use stdin.

@ghost ghost force-pushed the file-from-stdin branch 2 times, most recently from e69bebd to 3773309 Compare August 16, 2022 19:33
@ghost ghost force-pushed the file-from-stdin branch from 3773309 to a5075c5 Compare August 16, 2022 19:35
Copy link
Member

@dustinblack dustinblack left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested with both podman run -i and docker run -i, using both direct echo of json and cat of both yaml and json input files, and all conditions passed.

@ghost ghost merged commit a3851fc into main Aug 17, 2022
@ghost ghost deleted the file-from-stdin branch August 17, 2022 11:01
dustinblack added a commit to dustinblack/arcaflow-plugins that referenced this pull request Aug 17, 2022
dustinblack added a commit to dustinblack/arcaflow-plugins that referenced this pull request Aug 17, 2022
dustinblack added a commit to dustinblack/arcaflow-plugins that referenced this pull request Aug 17, 2022
Updated readme for containerized run

Now possible per arcalot/arcaflow-plugin-sdk-python#36

update plugin directory
dustinblack added a commit to dustinblack/arcaflow-plugins that referenced this pull request Aug 18, 2022
Updated readme for containerized run

Now possible per arcalot/arcaflow-plugin-sdk-python#36

update plugin directory
dustinblack added a commit to dustinblack/arcaflow-plugins that referenced this pull request Aug 18, 2022
Updated readme for containerized run

Now possible per arcalot/arcaflow-plugin-sdk-python#36

update plugin directory
dustinblack added a commit to dustinblack/arcaflow-plugins that referenced this pull request Aug 19, 2022
Updated readme for containerized run

Now possible per arcalot/arcaflow-plugin-sdk-python#36

update plugin directory
dustinblack added a commit to dustinblack/arcaflow-plugins that referenced this pull request Aug 19, 2022
git rid of samples parameter

enable tests in dockerfile

convert operations parameters to enum

enable yes/no enum for compatibility with smallfile boolean params

significantly expanded supported input params

comment cleanup

stagify dockerfile

variable format standardization

autopep8 formatting

remove .gitignore

housekeeping fixes

rsptimes serialization test fix

add input params documentation

Updated readme for containerized run

Now possible per arcalot/arcaflow-plugin-sdk-python#36

update plugin directory

separate schema module for better clarity

document output schema

handle errors for opening rsptimes csv file

cleanup

autopep8

remove per-plugin license file
mfleader pushed a commit to arcalot/arcaflow-plugins-incubator that referenced this pull request Aug 19, 2022
add python plugin for smallfile

git rid of samples parameter

enable tests in dockerfile

convert operations parameters to enum

enable yes/no enum for compatibility with smallfile boolean params

significantly expanded supported input params

comment cleanup

stagify dockerfile

variable format standardization

autopep8 formatting

remove .gitignore

housekeeping fixes

rsptimes serialization test fix

add input params documentation

Updated readme for containerized run

Now possible per arcalot/arcaflow-plugin-sdk-python#36

update plugin directory

separate schema module for better clarity

document output schema

handle errors for opening rsptimes csv file

cleanup

autopep8

remove per-plugin license file
This pull request was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Allow a plugin to accept input parameters from stdin when used with the CLI
1 participant