Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a V2 builtin rule to go from FilesContent -> Snapshot #7718

Closed
Eric-Arellano opened this issue May 14, 2019 · 6 comments · Fixed by #8226
Closed

Add a V2 builtin rule to go from FilesContent -> Snapshot #7718

Eric-Arellano opened this issue May 14, 2019 · 6 comments · Fixed by #8226

Comments

@Eric-Arellano
Copy link
Contributor

We would like a nice abstraction to create a Snapshot given a set of file paths with their corresponding content (i.e. FilesContent).

This would be far better than the current approach of using touch to do the same thing, as done in #7696. Notably, the current implementation only works for empty files too.

However, going from FilesContent -> Snapshot would likely lead to a cycle so we might need a different name than this. Refer to #7696 (comment).

@illicitonion
Copy link
Contributor

@stuhood Any thoughts on what this API should look like? Ideally in my mind it would be:

files_content = FilesContent((
  FileContent(path="roland", content=b"European Burmese"),
  FileContent(path="susannah", content=b"Not sure actually"),
))
snapshot = yield Get(Snapshot, FilesContent, files_content)

but that would lead to graph ambiguity through Snapshot -> FilesContent -> Snapshot cycles. I guess we could do something like:

files_content = FilesToSnapshot((
  FileContent(path="roland", content=b"European Burmese"),
  FileContent(path="susannah", content=b"Not sure actually"),
))
snapshot = yield Get(Snapshot, FilesToSnapshot, files_content)

but it feels kind of sketchy to introduce a whole new type just to differentiate between "we expect this to be an output" and "we expect this to be an input" when really the two should be interchangeable...

@stuhood
Copy link
Member

stuhood commented May 14, 2019

but that would lead to graph ambiguity through Snapshot -> FilesContent -> Snapshot cycles. I guess we could do something like:

It shouldn't, I don't think? Although I have been surprised before (cf #7710).

@illicitonion
Copy link
Contributor

If I add a rule to go from FilesContent -> Digest, I get a whole host of No rule was available to compute errors.

If I add a rule to go from FilesContent -> Snapshot, I get no errors. But I expect to because there is a cycle in the rule graph.

The fact that these give different results is surprising and concerning...

@stuhood
Copy link
Member

stuhood commented May 15, 2019

The Digest issue is expected due to #7710 (the bug is that there is ambiguity between a Digest already in a rules' params and requesting a Digest via a Get).

If I add a rule to go from FilesContent -> Snapshot, I get no errors. But I expect to because there is a cycle in the rule graph.

Mind attaching a dot file or PDF?

@stuhood
Copy link
Member

stuhood commented May 15, 2019

Oh, and I don't expect FilesContent -> Snapshot to be ambiguous in practice precisely due to the constraint that is described as "mostly working" in #7710: the parameter to a Get must be consumed in a subgraph, so it will be non-ambiguous to Get in that case.

EDIT: except the cycle would still theoretically be usable. Hm. Will think about it.

@stuhood
Copy link
Member

stuhood commented May 16, 2019

Oh, and I don't expect FilesContent -> Snapshot to be ambiguous in practice precisely due to the constraint that is described as "mostly working" in #7710: the parameter to a Get must be consumed in a subgraph, so it will be non-ambiguous to Get in that case.

EDIT: except the cycle would still theoretically be usable. Hm. Will think about it.

Having said that, if it currently works, I'd rather that you take advantage of that, as it will force me to apply constraints in #7710 that continue to allow for the clearly useful case of a bidirectional conversion.

illicitonion added a commit to twitter/pants that referenced this issue Sep 9, 2019
Fixes pantsbuild#7718.

This currently fails because of weird ambiguity errors.
illicitonion added a commit to twitter/pants that referenced this issue Sep 10, 2019
Fixes pantsbuild#7718.

This currently fails because of weird ambiguity errors.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
3 participants