Download GitHub release artifacts

[manics]

Support downloading from GitHub releases. Currently limited to:

• Full release tags only (latest isn't supported, nor are partial tag prefixes such as 5.5)
• Only .zip (matches the current behaviour of omego download, but e.g. this means you won't see the Insight.dmg)

E.g.:

$ omego download --github ome/omero-insight --release 5.5.5
Artifacts available for download. Initial partial matching is supported for all except named-components). Alternatively a full filename can be specified to download any artifact, including those not listed.

omerozips:
  imagej-5.5.5
  importer-5.5.5
  insight-5.5.5
zips:
  OMERO.imagej-5.5.5
  OMERO.importer-5.5.5
  OMERO.insight-5.5.5

$ omego download --github ome/omero-insight --release 5.5.5 insight
2020-03-19 17:57:12,123 [omego.artifa] INFO  Checking https://github.com/ome/omero-insight/releases/download/v5.5.5/OMERO.insight-5.5.5.zip
2020-03-19 17:57:12,124 [omego.fileut] INFO  Downloading https://github.com/ome/omero-insight/releases/download/v5.5.5/OMERO.insight-5.5.5.zip
2020-03-19 17:57:24,708 [omego.artifa] INFO  Unzipping OMERO.insight-5.5.5.zip

[mtbc]

👍 for the helpful messages.

[sbesson]

Functionally tested using the OMERO 5.6.0 artifacts uploaded to ome/openmicroscopy (thanks @joshmoore)

sbesson@ls30630:omego ((ae0dacd...)) $ venv/bin/omego download --github ome/openmicroscopy --release 5.6.0 
Artifacts available for download. Initial partial matching is supported for all except named-components). Alternatively a full filename can be specified to download any artifact, including those not listed.
named-components:
  server
omerozips:
  server-5.6.0-ice36-b136
zips:
  OMERO.server-5.6.0-ice36-b136
sbesson@ls30630:omego ((ae0dacd...)) $  time venv/bin/omego download   --release 5.6.0 server --skipunzip 
2020-03-20 12:18:23,092 [omego.artifa] INFO  Checking https://downloads.openmicroscopy.org/omero/5.6.0/artifacts/OMERO.server-5.6.0-ice36-b136.zip
2020-03-20 12:18:23,092 [omego.fileut] INFO  Downloading https://downloads.openmicroscopy.org/omero/5.6.0/artifacts/OMERO.server-5.6.0-ice36-b136.zip

real	0m44.779s
user	0m1.187s
sys	0m0.746s
sbesson@ls30630:omego ((ae0dacd...)) $ mv OMERO.server-5.6.0-ice36-b136.zip OMERO.server-5.6.0-ice36-b136.zip.2
sbesson@ls30630:omego ((ae0dacd...)) $ time venv/bin/omego download --github ome/openmicroscopy --release 5.6.0 server --skipunzip
2020-03-20 12:24:17,499 [omego.artifa] INFO  Checking https://github.com/ome/openmicroscopy/releases/download/v5.6.0/OMERO.server-5.6.0-ice36-b136.zip
2020-03-20 12:24:17,499 [omego.fileut] INFO  Downloading https://github.com/ome/openmicroscopy/releases/download/v5.6.0/OMERO.server-5.6.0-ice36-b136.zip

real	0m30.740s
user	0m1.095s
sys	0m0.695s
sbesson@ls30630:omego ((ae0dacd...)) $ shasum OMERO.server-5.6.0-ice36-b136.zip*
83fde9edef3818afac3a86b973ab7301c24e3690  OMERO.server-5.6.0-ice36-b136.zip
83fde9edef3818afac3a86b973ab7301c24e3690  OMERO.server-5.6.0-ice36-b136.zip.2

So in terms of functionality, this is working as expected. As per the numbers above it looks like the networking issues which initially drove this investigation have been partly resolved but I see no reason not to get this merged and released as an opt-in mechanism which might become use more and more widely.

[sbesson]

Could be renamed as GitHubArtifacts but not a blocker

[manics]

I was undecided on following the GitHub branding vs CapitalisedsubtypeArtifacts

[sbesson]

Main caveat of this approach is that I found it only worked with release assets of public repositories. TO be reviewed if we want to investigate using it for private repositories

[sbesson]

Adding a few notes about my investigation of using this branch for private repository releases. Individual releases can be accessed by setting a valid GitHub OAuth token in the request header as per:

diff --git a/omego/fileutils.py b/omego/fileutils.py
index fb02653..f794dc8 100644
--- a/omego/fileutils.py
+++ b/omego/fileutils.py
@@ -96,6 +96,11 @@ def open_url(url, httpuser=None, httppassword=None, method=None):
 
     # Override method http://stackoverflow.com/a/4421485
     req = urllib.request.Request(url)
+    if 'GITHUB_TOKEN' in os.environ:
+        req.add_header(
+            'Authorization', 'token %s' % os.environ.get('GITHUB_TOKEN'))
+        log.debug('Setting authorization')
+
     if method:
         req.get_method = lambda: method

With this change GithubArtifacts.read_downloads will list all release assets as expected.
However during the download, I was unable to consume browser_download_url even when also setting the Accept request header as per https://developer.github.com/v3/repos/releases/#get-a-single-release-asset. Likely the code might need to be adapted to consume the release asset endpoint instead. However, this might requires a larger refactoring of this code as the component name cannot be simply extracted from the asset URL.

[manics]

I think I've fixed Travis which looks like it was broken for a while. The remaining failing tests should be the ones that download from openmicroscopy.org

[joshmoore]

A few thoughts:

• I couldn't tell if @sbesson's items had been covered. Not sure if they are follow-ons, etc.
• Code looks sensible. As a new option, I assume there's no blocker to getting it in and iterating (with the caveat that breaking changes should be avoided if possible)
• I'm not overly concerned with waiting until travis passes if this will help us with getting things green again.