Support pinning packages with an archive url

[gridbugs]

Currently it looks like we support packages pinned to local paths and git urls but not archive urls. The ocaml.org project has a pin ["tailwindcss.dev" "https://github.com/tmattio/opam-tailwindcss/archive/3e60fc32bbcf82525999d83ad0f395e16107026b.tar.gz"]. Currently locking this project (after removing the package stanza from its dune-project file to workaround #10120) produces the error:

File "ocamlorg.opam", line 1, characters 0-0:
Error: Could not determine location of repository
https://github.com/tmattio/opam-tailwindcss/archive/3e60fc32bbcf82525999d83ad0f395e16107026b.tar.gz
Hint: Specify either a file path or git repo via SSH/HTTPS

[rgrinberg]

This is defo a bug, but is it critical? One can just swap the URL to use git and move on.

[gridbugs]

I guess it depends on how we measure the MVP, and whether the release process of the affected package is just making an archive of the repo at the given revision or if there is additional processing before archiving.

[gridbugs]

I checked and in this case the archive is just a snapshot of the repo.

[maiste]

@rgrinberg I have started to investigate this. The case of tailwind is a good example of why this is needed. If you use the git version, you end up downloading the entire git history with all the built executables. Using the archive prevent this by only downloading the required version.

Digging into the project, the changes needed seem to be more important than expected. To make sure I don't go in the wrong direction, I was wondering what the best way to address that is.

My understanding is that I need to alter this function in the `http case to return the URL or the Path in case it is a tar or a tar.gz file:

let local_or_git_only url loc =
  match (url : t).backend with
  | `rsync when is_local url -> `Path (Path.of_string url.path)
  | `git -> `Git
  | `http -> (* Add this change. Check if tar or tar.gz and handle the situation*) `Tar
  | `rsync | `darcs | `hg ->
    User_error.raise
      ~loc
      ~hints:[ Pp.text "Specify either a file path or git repo via SSH/HTTPS" ]
      [ Pp.textf "Could not determine location of repository %s" @@ OpamUrl.to_string url
      ]
;;

The consequence of this is that it requires the Mount.of_opam_url function to extend the t module type and support a new case:

type backend =
  | Path of Path.t
  | Git of Rev_store.At_rev.t
  | Tar of Path.t (* Or url? Or something else? *)

Regarding this, my questions are the following:

• Is it the right place to update the behavior to make sure we are able to use the tar version?
• Is there a module Tar I can use to smartly extract the directory in the cache as it is done for the Git of Rev_store.At_rev.t or is it something that needs to be written?
• I'm not sure to understand when the Git file is effectively downloaded. Is it when Dune creates the Rev_store or is it later, something done by the Fetch module?

[rgrinberg]

Is it the right place to update the behavior to make sure we are able to use the tar version?

It seems like the right place.

Is there a module Tar I can use to smartly extract the directory in the cache as it is done for the Git of Rev_store.At_rev.t or is it something that needs to be written?

I think you can use our existing code untaring things. Doing things directly here without worrying about optimization is fine. There's a few other places where we re-download things anyway.

I'm not sure to understand when the Git file is effectively downloaded. Is it when Dune creates the Rev_store or is it later, something done by the Fetch module?

Creating the rev store does not download anything, but calls that fetch the contents of files might. It's an implementation detail of the rev store when this is done, so you shouldn't have to worry about this. Suffice to say that if the file already exists in the revision store, it will not be redownloaded.