Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

build: docker images for latest release and main branch #2116

Merged
merged 1 commit into from
Sep 17, 2022

Conversation

aucampia
Copy link
Member

@aucampia aucampia commented Sep 14, 2022

Summary of changes

This PR adds docker image building to the github actions workflow.

Two images are built:

  • unstable: This is built from the current branch
  • latest: This is built from the version of rdflib in
    docker/latest/requirements.txt, but as this will be updated by
    dependabot it will essentially always be the latest version.

The resulting image references are:

  • ghcr.io/rdflib/rdflib:unstable
  • ghcr.io/rdflib/rdflib:latest with an alias
    ghcr.io/rdflib/rdflib:${rdflib_version}, e.g.
    ghcr.io/rdflib/rdflib:6.2.0.

As dependabot is conifgured for both pyton and docker dependencies, any
change/update to dependencies will result in a new image being built.

To avoid spurious changes in image digests the existing image is used as
the cache source, so if the requirements.txt and Dockerfile for
ghcr.io/rdflib/rdflib:latest has not changed then a new image should
not be published for it even if the build runs.

I have tested this in my forked repo, but there may be some kinks to
still work out.

Checklist

  • Checked that there aren't other open pull requests for
    the same change.
  • Checked that all tests and type checking passes.
  • Considered granting push permissions to the PR branch,
    so maintainers can fix minor issues and keep your PR up to date.

@coveralls
Copy link

coveralls commented Sep 14, 2022

Coverage Status

Coverage remained the same at 90.634% when pulling 3585c67 on aucampia:iwana-20220914T2257-docker_images into bcd05e9 on RDFLib:main.

@aucampia aucampia force-pushed the iwana-20220914T2257-docker_images branch from cdb5f66 to 12b01c9 Compare September 14, 2022 23:03
This PR adds docker image building to the github actions workflow.

Two images are built:
- unstable: This is built from the current branch
- latest: This is built from the version of rdflib in
  `docker/latest/requirements.txt`, but as this will be updated by
  dependabot it will essentially always be the latest version.

The resulting image references are:

- `ghcr.io/rdflib/rdflib:unstable`
- `ghcr.io/rdflib/rdflib:latest` with an alias
  `ghcr.io/rdflib/rdflib:${rdflib_version}`, e.g.
  `ghcr.io/rdflib/rdflib:6.2.0`.

As dependabot is conifgured for both pyton and docker dependencies, any
change/update to dependencies will result in a new image being built.

To avoid spurious changes in image digests the existing image is used as
the cache source, so if the requirements.txt and Dockerfile for
`ghcr.io/rdflib/rdflib:latest` has not changed then a new image should
not be published for it even if the build runs.

I have tested this in my forked repo, but there may be some kinks to
still work out.
@aucampia aucampia force-pushed the iwana-20220914T2257-docker_images branch from 12b01c9 to 3585c67 Compare September 14, 2022 23:05
@aucampia
Copy link
Member Author

@nicholascar this could be somewhat simpler with poetry, at the very least the redudant requirements.txt files could be avoided somewhat, but otherwise it would be the same for the most part.

With poetry though you want to first export your dependencies to requirements.txt, something like:

poetry export --without-hashes --dev --format requirements.txt > requirements.txt

Then you would copy that requirements.txt file into the image.

If that file is not stable (i.e. if it fluctuates randomly for the same poetry.lock) then the caching will not work too well.

@aucampia aucampia marked this pull request as ready for review September 14, 2022 23:10
@aucampia aucampia requested review from nicholascar and a team September 14, 2022 23:11
@aucampia
Copy link
Member Author

aucampia commented Sep 14, 2022

Image can be used as follow:

$ echo '<ex:1> <ex:2> <ex:3> .' | docker run --rm -i ghcr.io/rdflib/rdflib:6.2.0 rdfpipe -i ntriples -o turtle -
@prefix ns1: <ex:> .

ns1:1 ns1:2 ns1:3 .

@aucampia
Copy link
Member Author

Another way to use this:

$ docker run --rm -i ghcr.io/rdflib/rdflib:6.2.0 python -c 'import rdflib; g = rdflib.Graph(); g.add((rdflib.RDF.type, rdflib.RDF.type, rdflib.RDF.type, )); print(g.serialize())'
@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .

rdf:type a rdf:type .

@nicholascar
Copy link
Member

@nicholascar this could be somewhat simpler with poetry, at the very least the redudant requirements.txt files could be avoided somewhat, but otherwise it would be the same for the most part.

With poetry though you want to first export your dependencies to requirements.txt, something like:

poetry export --without-hashes --dev --format requirements.txt > requirements.txt

Then you would copy that requirements.txt file into the image.

If that file is not stable (i.e. if it fluctuates randomly for the same poetry.lock) then the caching will not work too well.

I do think for a long-term stable repo, it's probably always worth having the requirements.txt file built so non-Poetry users can use it with PIP, however we could happily do the more efficient image building with Poetry as you say. That's where we're going across the board (on other projects).

Do you want to make this change?

@nicholascar
Copy link
Member

@jamiefeiss can you please pick up the patterns used here for Prez?

@aucampia
Copy link
Member Author

Do you want to make this change?

I will create a discussion to check with others what they think sometime, I like poetry a lot and use it extensively, but I also don't want to make things harder for others so best to just see the level of support there is.

@aucampia aucampia merged commit 3a41821 into RDFLib:main Sep 17, 2022
@nicholascar
Copy link
Member

@aucampia where do the images from this PR's creation scripts end up and how do people use them? I think that they might be private as I can't pull them from ghcr.io/rdflib/rdflib:latest etc. ("unauthorized").

I think we, or at least I!, need documentation on how to use the resultant images...

@aucampia
Copy link
Member Author

@aucampia where do the images from this PR's creation scripts end up and how do people use them? I think that they might be private as I can't pull them from ghcr.io/rdflib/rdflib:latest etc. ("unauthorized").

The visibility needs to be changed here

image

But I don't think I can do it.

I think we, or at least I!, need documentation on how to use the resultant images...

I should add documentation and will when I can, hopefully this weekend or next week.

@aucampia aucampia deleted the iwana-20220914T2257-docker_images branch April 9, 2023 15:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants