Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Trial name could not be passed to env #1844

Closed
alexeygorobets opened this issue Apr 7, 2022 · 6 comments · Fixed by d2iq-archive/katib#2 or #1848
Closed

Trial name could not be passed to env #1844

alexeygorobets opened this issue Apr 7, 2022 · 6 comments · Fixed by d2iq-archive/katib#2 or #1848
Labels

Comments

@alexeygorobets
Copy link
Contributor

/kind bug

What steps did you take and what happened:
I am trying to get access to the trial name from the trial's pod (I need it to build S3 bucket path for trained model).
I follow this example.
But I've got an error

Error from server: error when creating "trial-metadata-substitution.yaml": admission webhook "validator.experiment.katib.kubeflow.org" denied the request: parameter reference ${trialSpec.Name} does not exist in spec.parameters: [{lr double {0.03 0.01 [] }}]

This example itself does not work.

What did you expect to happen:
Trials running with env TRIAL_NAME resolved as trail's name.

Anything else you would like to add:
This example was working on Katib 0.12.0.
It looks like additional check added in this PR might cause this issue. One of the resolution might be to skip this check here if it's trial metadata (such as name, kind, apiversion etc.).

But I will be happy with any other solution which allows me to pass trial name to pod as environment variable.

Environment:

  • Katib version (check the Katib controller image version): 0.13.0
  • Kubernetes version: 1.22
  • OS : CentOS

Impacted by this bug? Give it a 👍 We prioritize the issues with the most 👍

@wenxinax
Copy link

same problem...

@johnugeorge
Copy link
Member

johnugeorge commented Apr 11, 2022

This needs to be fixed for the next release.

@johnugeorge
Copy link
Member

johnugeorge commented Apr 11, 2022

Validation added in #1726 needs to be extended to substitutions as well

@johnugeorge
Copy link
Member

/cc @henrysecond1

@henrysecond1
Copy link
Contributor

I'm sorry that I didn't consider the parameter substitution case.

As @alexeygorobets said, I think we can skip parameter reference checking when it's a trial metadata. Please let me know if there's anything I can help with.

@johnugeorge
Copy link
Member

Instead of skipping metadata, we can validate the metadata fields.

We need to move the some of the validations from https://github.com/kubeflow/katib/blob/master/pkg/controller.v1beta1/experiment/manifest/generator.go#L123 to verify fields

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
4 participants