update Compat for rename of Pkg to OldPkg

[KristofferC]

WIll be needed for JuliaLang/julia#27095.

[martinholters]

Needs a VERSION dependency. Of course, we don't know the version of merging JuliaLang/julia#27095 yet, but this should be updated once JuliaLang/julia#27095 has landed.

[martinholters]

So the plan is to merge this ASAP after JuliaLang/julia#27095 is in and then probably also tag a release soonish to make it generally available?

[martinholters]

On second thought, this is backwards: Compat.Pkg will give the old Pkg on 0.7. Shouldn't this rather define Compat.OldPkg to give Base.Pkg on 0.6 and OldPkg on 0.7 after JuliaLang/julia#27095?

[KristofferC]

Yeah, that is probably the correct direction.

[martinholters]

We should also figure out what to do with Compat.Pkg. Will the new Pkg (Pkg3) be sufficiently similar API-wise to the old Base.Pkg (Pkg2) that having Compat.Pkg being one of them depending on Julia version is useful?

[KristofferC]

We should also figure out what to do with Compat.Pkg. Will the new Pkg (Pkg3) be sufficiently similar API-wise to the old Base.Pkg (Pkg2) that having Compat.Pkg being one of them depending on Julia version is useful?

Not really. What I thought here was just to keep those packages that are using Compat.Pkg to work a bit longer (in 0.7) until deprecations are removed and Pkg is deleted for 1.0.

[martinholters]

Thinking aloud here, not saying we actually should do this, but the "clean" way would be to

1. Add Compat.OldPkg, deprecate Compat.Pkg to it.
2. Wait for people to update.
3. Add Compat.Pkg as Pkg3, providing as much functionality API-wise as possibly on Julia 0.6. Deprecate Compat.OldPkg, asking to use the corresponding functionality of Compat.Pkg.
4. Wait for people to update.
5. Remove Compat.OldPkg.

And all this has to happen before OldPkg is removed from Julia. (To happen with Julua 1.0, I guess?)

What a nightmare.

[fredrikekre]

Datapoint: Compat.Pkg is only used 11 times (in source files) across all registered packages (and 4 of those are PkgDev).

[KristofferC]

Usage of Pkg in non-interactive environments should be quite minimal. A big change like this will not be super clean and things like:

Add Compat.Pkg as Pkg3, providing as much functionality API-wise as possibly on Julia 0.6. Deprecate Compat.OldPkg, asking to use the corresponding functionality of Compat.Pkg.

is way too much work for little benefit.

Right now, I have tried including enough functionality so existing CI scripts will work with Pkg3 -> Pkg which I think would be the main annoyance.

[martinholters]

Given the premises:

• Compat provides syntax/API of the newest Julia version on older versions.
• Things working without deprecation warnings on 0.7 should work on 1.0.

I'd say providing a way on Compat to access Pkg2 is a non-starter. Too bad we're already doing so, but at least its uses are rare.

How about the following: Let's focus on those functions which are in common use in non-interactive context and for which a clear Pkg3 alternative exists. On Julia 0.6, provide a custom Pkg module mimicking the Pkg3 interface of only those most important functions. On Julia 0.7, let Pkg just be Pkg3. Alternatively, provide a custom module on Julia 0.7, too, and add deprecated Pkg2-style wrappers to avoid breaking the few present users of Compat.Pkg; not sure it's worth the effort.

[martinholters]

Right now, I have tried including enough functionality so existing CI scripts will work with Pkg3 -> Pkg which I think would be the main annoyance.

What would break then if we don't change anything here?

[KristofferC]

I'm not sure what people are using Pkg for inside their packages except trying to find out stuff about the current package environment, which is completely redisigned. So there is no way typically that you could use the same code for these type in 0.6 and 0.7. I'm not sure how much point there is to Compat.Pkg at all after the rename tbh.

[martinholters]

Alright, the most common non-interactive usage of Pkg seems to be

julia -e 'Pkg.add("Documenter"); cd(Pkg.dir("MyPackage")); include(joinpath("docs", "make.jl"))'

(or similar for Coverage). What would be the proper Pkg3 way of writing this? Should this become

julia -e 'import Pkg; Pkg.add("Documenter"); cd(Base.locate_package(Base.identify_package("MyPackage")); include(joinpath("docs", "make.jl"))'

?

Then it seems Compat should enable the following to work on Julia 0.6 or later:

julia -e 'import Compat.Pkg; Pkg.add("Documenter"); cd(Compat.locate_package(Compat.identify_package("MyPackage")); include(joinpath("docs", "make.jl"))'

If that's correct so far, there indeed seems to be no need for this PR; Compat.Pkg can silently become Pkg3. But we would need identify_package and locate_package. Also, from a cursory glance, Pkg.dir seems one of the more common Pkg operations invoked inside packages, so providing a future-proof alternative in Compat sounds quite useful.

[KristofferC]

julia -e 'Pkg.add("Documenter"); cd(Pkg.dir("MyPackage")); include(joinpath("docs", "make.jl"))'

That should work since I've put in this:

https://github.com/JuliaLang/Pkg3.jl/blob/master/src/API.jl#L399-L418

[martinholters]

https://github.com/JuliaLang/Pkg3.jl/blob/b19fbd72555ec0fe3430c4feb02ed205aa1540c3/src/API.jl#L412 left me under the impression this was deprecated...

[KristofferC]

Yeah, but thinking about it, it doesn't have to be. Perhaps being able to get the path to the package being used is useful. The problem is that most packages will not be developed and their source is considered immutable so you shouldn't really go in and do stuff to them, but if it is just for reading things (like what PkgBenchmark does) then it should be fine.

Anway, my goal of those functions has been to just keep CI working while people upgrade their packages.

[martinholters]

Anway, my goal of those functions has been to just keep CI working while people upgrade their packages.

Yes, of course, but in this case, the question is (was): upgrade to what? If Pkg.add and Pkg.dir are there to stay, then I think we can let Compat.Pkg just be whatever Pkg is on 0.7 see what complaints we are getting after that becomes Pkg3 😄

Oh, err, wait, obviously the UUID in Project.toml decides whether we're getting Pkg2 or Pkg3. So maybe we define Pkg2 and Pkg3 in Project.toml and then do a const Pkg = ..., selecting Pkg2 or Pkg3 depending on Julia version?

[KristofferC]

Yes, so compat.Pkg -> Pkg3 on 0.7 and OldPkg on 0.6 and see the complaints? I think that would work ok.

[martinholters]

Um, hang on, in the name = UUID entries in Project.toml, does the name have to match the package identified by the UUID? I was under the impression that no, to allow loading different packages with the same name (but associating different names in Project.toml), but from some experimentation, the answer seems to be yes. Does that mean it will be impossible to have a Compat version that works pre- and post- JuliaLang/julia#27095? Because the Pkg= entry will either be correct before or after?

[KristofferC]

I was under the impression that no, to allow loading different packages with the same name (but associating different names in Project.toml), but from some experimentation, the answer seems to be yes.

You can have a dependency graph for a project / package A as:

A -> B, C
B -> D_1
C -> D_2
D_1 -> ∅
D_2 -> ∅

where -> is dependency and D_1 and D_2 have the same name (but different UUID). You cannot have

A -> B_1, B_2

and you cannot locally "rename" a package by just changing its name in the Project file.

[KristofferC]

Does that mean it will be impossible to have a Compat version that works pre- and post- JuliaLang/julia#27095? Because the Pkg= entry will either be correct before or after?

That looks correct to me, @StefanKarpinski

[martinholters]

Actually, there is a possibility: remove Project.toml 😛 💣 💥

I'll see myself out...

[KristofferC]

Yeah... would perhaps solve it for the short term until nightlies for #27095 are updated and it is possible to tell people to just get a new nightly...