Add relocated_resources target type
#11559
Conversation
# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]
# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]
Eric-Arellano
changed the title
relocated_resources target typerelocated_resources target type
A PEX is a deployable application. You cannot locate files in a deployed application by relying on any build root or any specific CWD for that matter. This however should have 0 bearing on whether we allow folks to package files in a PEX. They can completely legitimately use So I stopped on that paragraph since its not correct. I can review further if that's reasoning that can be left out of the description of this PR and there is a better trumping reason. |
Agreed.
Nack. This is the issue identified in #11551 (comment): if you have source roots, If we can figure out a solution for that inconsistency, I'll all in for allowing |
# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]
|
I'm thoroughly confused. Lets leave aside how things work now for a moment. If I have a pex_binary that uses relocated_files, then pants package and pants run should both package / arrange things the same now matter how its done under the hood. I.E.: If package doesn't strip source roots for relocated files, neither should run. If so, then access via file should work the same in both. Where is that wrong? (and note this focus on pants run and pants package providing consistent layouts has nothing to do with python - we should do this for every language) |
Yes to never stripping relocated files. Those are never stripped in Pants. The inconsistency is how we handle resources and Python files. Originally, we always stripped source roots - I introduced the original sin in V2 land in 2019 with #7696. But Benjy and I slowly chipped away at replacing this, as stripping has several downsides, including tool output not displaying the full path, and issues with namespace packages having conflicting content, like if Now, the sole place we strip is when running I agree it's broken that |
These two examples are both examples of bad not good. At deploy run time conflicting namespace packages will cause a problem, they should do the same at I want to again highlight the non-Pythonness of this issue / decision. The way we go here sets a precedent for all language handling. |
The existing pex-tool/pex#987 would cover this.
This seems broken to me. So, take for example --venv mode. If we package a PEX file that supports source roots via PYTHONPATH ammendments, then creating a venv would look like this:
|
Yea, this. A lot of languages have a "resource" concept -- embedded in my app and loaded from a "path" in a language-specific way. And when things are loaded from a path like that, it makes sense for them to be shaped like path entries (ie, the language's modules). An overview of Java's, for example: https://docs.oracle.com/javase/8/docs/technotes/guides/lang/resources.html The advantage of resources is supposed to be that they dodge the question of "where does the file actually live, and is it actually a file, or is it just a stream of bytes?": and that's handy precisely because it dodges the the kinds of problems that we have with the
I don't have a good answer for the "I'd like to load the same resource from multiple locations" usecase (it's a bit like wanting to load the same Python module from multiple places). In the usecases that I've seen, resources have been loaded from a consistent path by whichever code module they are associated with. It's possible that this usecase is weird enough (unsure) that we should actually recommend using |
As re-found in #11551, we intentionally do not include
files()targets when packaging a PEX because the files cannot be easily accessed via the traditional filesystem APIs. Even when using unzip and venv modes, you must still relativize theopen()to the__file__, rather than using a path relative to the build root. When you have source roots,__file__behaves differently between./pants runand./pants package, as the latter strips source roots.When dealing with
pex_binary, almost always you should either useresourcesor usearchive. But a user needs the flexibility ofrelocated_files- they have a filecommon_assets/common.css, along with several projects in the format<project>/assets/*.css. They needcommon.cssto be in that final path<project>/assets/*.csswhen the PEX is built, but they don't want to copy that file in several places; symlinks work but are not first-class. They want to use the equivalent ofrelocated_files, but for it to work with their PEX.relocated_resourcesare pretty advanced and will likely confuse some users with getting source roots correct. But this isn't necessarily a reason to avoid the feature; for the right user, this advanced feature will be necessary to their workflow. Beyond good docs, we leave it to the user to get the path manipulation working properly. Most users can ignore this.[ci skip-rust]
[ci skip-build-wheels]