Build fails because it can't find phenio-test.json

[caufieldjh]

Most recent build failed due to the following:

18:01:08  Converting data/raw/phenio.owl to data/raw/phenio.json...
18:01:08  Complete.
18:01:08  Transforming to KGX TSV with config in transform.yaml...
18:01:08  Traceback (most recent call last):
18:01:08    File "/var/lib/jenkins/workspace/ledge-graph-hub_kg-phenio_master/gitrepo/run.py", line 100, in <module>
18:01:08      cli()
18:01:08    File "/var/lib/jenkins/workspace/ledge-graph-hub_kg-phenio_master/gitrepo/venv/lib/python3.9/site-packages/click/core.py", line 1157, in __call__
18:01:08      return self.main(*args, **kwargs)
18:01:08    File "/var/lib/jenkins/workspace/ledge-graph-hub_kg-phenio_master/gitrepo/venv/lib/python3.9/site-packages/click/core.py", line 1078, in main
18:01:08      rv = self.invoke(ctx)
18:01:08    File "/var/lib/jenkins/workspace/ledge-graph-hub_kg-phenio_master/gitrepo/venv/lib/python3.9/site-packages/click/core.py", line 1688, in invoke
18:01:08      return _process_result(sub_ctx.command.invoke(sub_ctx))
18:01:08    File "/var/lib/jenkins/workspace/ledge-graph-hub_kg-phenio_master/gitrepo/venv/lib/python3.9/site-packages/click/core.py", line 1434, in invoke
18:01:08      return ctx.invoke(self.callback, **ctx.params)
18:01:08    File "/var/lib/jenkins/workspace/ledge-graph-hub_kg-phenio_master/gitrepo/venv/lib/python3.9/site-packages/click/core.py", line 783, in invoke
18:01:08      return __callback(*args, **kwargs)
18:01:08    File "/var/lib/jenkins/workspace/ledge-graph-hub_kg-phenio_master/gitrepo/run.py", line 75, in transform
18:01:08      kg_transform(*args, **kwargs)
18:01:08    File "/var/lib/jenkins/workspace/ledge-graph-hub_kg-phenio_master/gitrepo/kg_phenio/transform.py", line 36, in transform
18:01:08      t.run()
18:01:08    File "/var/lib/jenkins/workspace/ledge-graph-hub_kg-phenio_master/gitrepo/kg_phenio/transform_utils/phenio/phenio_transform.py", line 68, in run
18:01:08      self.parse(k, data_file, k)
18:01:08    File "/var/lib/jenkins/workspace/ledge-graph-hub_kg-phenio_master/gitrepo/kg_phenio/transform_utils/phenio/phenio_transform.py", line 143, in parse
18:01:08      transform(
18:01:08    File "/var/lib/jenkins/workspace/ledge-graph-hub_kg-phenio_master/gitrepo/venv/lib/python3.9/site-packages/kgx/cli/cli_utils.py", line 486, in transform
18:01:08      source_properties["input"]["filename"] = _validate_files(
18:01:08    File "/var/lib/jenkins/workspace/ledge-graph-hub_kg-phenio_master/gitrepo/venv/lib/python3.9/site-packages/kgx/cli/cli_utils.py", line 367, in _validate_files
18:01:08      raise FileNotFoundError(
18:01:08  FileNotFoundError: Filename '/data/raw/phenio-test.json' for source 'phenio-transform-test' does not exist!

On my local system, phenio-test.json is written to data/raw/phenio-test.json as expected, so it's strange that this FileNotFound happens on Jenkins.
Will need to check if there's a relative path issue.

[matentzn]

In my world this error happens most often if data/raw does not exist

[caufieldjh]

Good point @matentzn - in this case the build shouldn't even be able to get to this stage without creating data/raw, but strange things happen in Jenkins space often.

[caufieldjh]

Ah, the reason this happens is because the kgx transform, as per its config, is trying to do all its operations at once (for both phenio and phenio-test) but the previous steps only handle a single file at a time.
So when kgx tries to do the transform for phenio.json it tries to find phenio-test.json too, which doesn't exist yet.