diff --git a/.flake8 b/.flake8
index 460421cf5f..a609f12d9c 100644
--- a/.flake8
+++ b/.flake8
@@ -1,4 +1,4 @@
[flake8]
max-line-length = 88
-select = C,E,F,W,B,B950
+select = C, E, F, W, B, B950
extend-ignore = E203, E501, W503
diff --git a/.github/ISSUE_TEMPLATE/bug_report.yml b/.github/ISSUE_TEMPLATE/bug_report.yml
index 3dc95c4f7f..064028d4ca 100644
--- a/.github/ISSUE_TEMPLATE/bug_report.yml
+++ b/.github/ISSUE_TEMPLATE/bug_report.yml
@@ -1,6 +1,5 @@
name: Bug report
description: Create a report to help us improve
-title: "[BUG] -
"
labels: ["type:Bug"]
body:
@@ -10,7 +9,7 @@ body:
Hi! Thanks for using the Jupyter Docker Stacks and taking some time to contribute to this project.
We'd appreciate it if you could check out the [Troubleshooting common problems](https://jupyter-docker-stacks.readthedocs.io/en/latest/using/troubleshooting.html) section in the documentation,
- as well as [existing issues](https://github.com/jupyter/docker-stacks/issues) prior to submitting an issue to avoid duplication.
+ as well as [existing issues](https://github.com/jupyter/docker-stacks/issues?q=is%3Aissue) prior to submitting an issue to avoid duplication.
Please answer the following sections to help us troubleshoot the problem.
@@ -23,9 +22,11 @@ body:
- all-spark-notebook
- base-notebook
- datascience-notebook
+ - docker-stacks-foundation
- julia-notebook
- minimal-notebook
- pyspark-notebook
+ - pytorch-notebook
- r-notebook
- scipy-notebook
- tensorflow-notebook
@@ -34,10 +35,19 @@ body:
- type: input
attributes:
- label: Host OS system and architecture running docker image
+ label: Host OS system
placeholder: |
Example:
- Ubuntu 22.04 / aarch64
+ Ubuntu 22.04
+ validations:
+ required: true
+
+ - type: dropdown
+ attributes:
+ label: Host architecture
+ options:
+ - x86_64
+ - aarch64
validations:
required: true
@@ -48,7 +58,7 @@ body:
What complete docker command do you run to launch the container (omitting sensitive values)?
placeholder: |
Example:
- `docker run -it --rm -p 8888:8888 jupyter/all-spark-notebook`
+ `docker run -it --rm -p 8888:8888 quay.io/jupyter/base-notebook`
validations:
required: true
@@ -96,7 +106,7 @@ body:
description: |
A clear and concise description of what the bug is.
placeholder: |
- Example: No output is visible in the notebook and the notebook server log contains messages about ...
+ Example: No output is visible in the notebook and the Server log contains messages about ...
validations:
required: true
@@ -115,5 +125,5 @@ body:
label: Latest Docker version
description: You should try to use the latest Docker version
options:
- - label: I've updated my Docker version to the latest available, and the issue still persists
+ - label: I've updated my Docker version to the latest available, and the issue persists
required: true
diff --git a/.github/ISSUE_TEMPLATE/feature_request.yml b/.github/ISSUE_TEMPLATE/feature_request.yml
index 89242a2bb3..f0da40624e 100644
--- a/.github/ISSUE_TEMPLATE/feature_request.yml
+++ b/.github/ISSUE_TEMPLATE/feature_request.yml
@@ -1,6 +1,5 @@
name: Feature request
description: Suggest a new feature for this project
-title: "[ENH] - "
labels: ["type:Enhancement"]
body:
@@ -21,9 +20,11 @@ body:
- all-spark-notebook
- base-notebook
- datascience-notebook
+ - docker-stacks-foundation
- julia-notebook
- minimal-notebook
- pyspark-notebook
+ - pytorch-notebook
- r-notebook
- scipy-notebook
- tensorflow-notebook
@@ -33,7 +34,7 @@ body:
- type: textarea
attributes:
- label: What changes are you proposing?
+ label: What change(s) are you proposing?
description: |
Be concise and feel free to add supporting links or references.
placeholder: |
@@ -52,7 +53,7 @@ body:
Example:
- Altair is a declarative statistical visualization library for Python, based on Vega and Vega-Lite, and the source is available on GitHub.
- With Altair, you can spend more time understanding your data and its meaning.
- - Altair's API is simple, friendly and consistent and built on top of the powerful Vega-Lite visualization grammar.
+ - Altair's API is simple, friendly, and consistent and built on top of the powerful Vega-Lite visualization grammar.
- This elegant simplicity produces beautiful and effective visualizations with a minimal amount of code.
validations:
required: true
diff --git a/.github/actions/create-dev-env/action.yml b/.github/actions/create-dev-env/action.yml
new file mode 100644
index 0000000000..7323494f6c
--- /dev/null
+++ b/.github/actions/create-dev-env/action.yml
@@ -0,0 +1,20 @@
+name: Build environment
+description: Create a build environment
+
+runs:
+ using: composite
+ steps:
+ # actions/setup-python doesn't support Linux aarch64 runners
+ # See: https://github.com/actions/setup-python/issues/108
+ # python3 is manually preinstalled in the aarch64 VM self-hosted runner
+ - name: Set Up Python ๐
+ uses: actions/setup-python@v5
+ with:
+ python-version: 3.x
+ if: runner.arch == 'X64'
+
+ - name: Install Dev Dependencies ๐ฆ
+ run: |
+ pip install --upgrade pip
+ pip install --upgrade -r requirements-dev.txt
+ shell: bash
diff --git a/.github/actions/load-image/action.yml b/.github/actions/load-image/action.yml
new file mode 100644
index 0000000000..cf53e4a6f6
--- /dev/null
+++ b/.github/actions/load-image/action.yml
@@ -0,0 +1,27 @@
+name: Load Docker image
+description: Download the image tar and load it to Docker
+
+inputs:
+ image:
+ description: Image name
+ required: true
+ platform:
+ description: Image platform
+ required: true
+ variant:
+ description: Variant tag prefix
+ required: true
+
+runs:
+ using: composite
+ steps:
+ - name: Download built image ๐ฅ
+ uses: actions/download-artifact@v4
+ with:
+ name: ${{ inputs.image }}-${{ inputs.platform }}-${{ inputs.variant }}
+ path: /tmp/jupyter/images/
+ - name: Load downloaded image to docker ๐ฅ
+ run: |
+ zstd --uncompress --stdout --rm /tmp/jupyter/images/${{ inputs.image }}-${{ inputs.platform }}-${{ inputs.variant }}.tar.zst | docker load
+ docker image ls --all
+ shell: bash
diff --git a/.github/dependabot.yml b/.github/dependabot.yml
index 136832b017..8f17357fbb 100644
--- a/.github/dependabot.yml
+++ b/.github/dependabot.yml
@@ -1,7 +1,7 @@
# To get started with Dependabot version updates, you'll need to specify which
# package ecosystems to update and where the package manifests are located.
# Please see the documentation for all configuration options:
-# https://docs.github.com/en/code-security/supply-chain-security/keeping-your-dependencies-updated-automatically/configuration-options-for-dependency-updates
+# https://docs.github.com/en/code-security/dependabot/dependabot-version-updates/configuration-options-for-the-dependabot.yml-file
version: 2
updates:
@@ -9,3 +9,11 @@ updates:
directory: /
schedule:
interval: weekly
+ - package-ecosystem: github-actions
+ directory: .github/actions/create-dev-env/
+ schedule:
+ interval: weekly
+ - package-ecosystem: github-actions
+ directory: .github/actions/load-image/
+ schedule:
+ interval: weekly
diff --git a/.github/workflows/docker-build-test-upload.yml b/.github/workflows/docker-build-test-upload.yml
index 539e4701ae..1dacad4179 100644
--- a/.github/workflows/docker-build-test-upload.yml
+++ b/.github/workflows/docker-build-test-upload.yml
@@ -1,24 +1,35 @@
-name: Download parent image, build a new one and test it; then upload the image, tags and manifests to GitHub artifacts
+name: Download a parent image, build a new one, and test it; then upload the image, tags, and manifests to GitHub artifacts
env:
+ REGISTRY: quay.io
OWNER: ${{ github.repository_owner }}
on:
workflow_call:
inputs:
- parentImage:
+ parent-image:
description: Parent image name
required: true
type: string
+ parent-variant:
+ description: Parent variant tag prefix
+ required: false
+ type: string
+ default: default
image:
description: Image name
required: true
type: string
- # platform:
- # description: Image platform
- # required: true
- # type: string
- runsOn:
+ variant:
+ description: Variant tag prefix
+ required: false
+ type: string
+ default: default
+ platform:
+ description: Image platform
+ required: true
+ type: string
+ runs-on:
description: GitHub Actions Runner image
required: true
type: string
@@ -34,109 +45,60 @@ on:
jobs:
build-test-upload:
- runs-on: ${{ inputs.runsOn }}
- # This will create a registry that we use for docker images
- # as an intermediary between build and test.
- # services:
- # registry:
- # image: registry:2
- # ports:
- # - 5000:5000
+ runs-on: ${{ inputs.runs-on }}
steps:
- - name: Checkout Repo โก๏ธ
- uses: actions/checkout@v3
-
- - name: Set Up Python ๐
- uses: actions/setup-python@v4
- with:
- python-version: 3.x
-
- - uses: actions/cache@v3
- name: Cache Python
+ # Image with CUDA needs extra disk space
+ - name: Free disk space ๐งน
+ if: contains(inputs.variant, 'cuda') && inputs.platform == 'x86_64'
+ uses: jlumbroso/free-disk-space@54081f138730dfa15788a46383842cd2f914a1be
with:
- path: ${{ env.pythonLocation }}
- key: ${{ env.pythonLocation }}-${{ hashFiles('setup.py') }}-${{ hashFiles('requirements.txt') }}
+ tool-cache: false
+ android: true
+ dotnet: true
+ haskell: true
+ large-packages: false
+ docker-images: false
+ swap-storage: false
- - name: Install Dev Dependencies ๐ฆ
+ - name: Checkout Repo โก๏ธ
+ uses: actions/checkout@v4
+ - name: Create dev environment ๐ฆ
+ uses: ./.github/actions/create-dev-env
+
+ # Self-hosted runners share a state (whole VM) between runs
+ # Also, they might have running or stopped containers,
+ # which are not cleaned up by `docker system prune`
+ - name: Reset docker state and cleanup artifacts ๐๏ธ
+ if: inputs.platform != 'x86_64'
run: |
pip install --upgrade pip
pip install --upgrade -r requirements-dev.txt
shell: bash
- - name: Set up Docker Buildx
- uses: docker/setup-buildx-action@v2
- with:
- driver-opts: |
- network=host
- -
- name: Download artifact ๐ฅ
- if: ${{ inputs.parentImage != '' }}
- uses: actions/download-artifact@v3
- with:
- name: ${{ inputs.parentImage }}
- path: /tmp/
-
- - name: Load parent built image to Docker ๐ณ
- if: ${{ inputs.parentImage != '' }}
- run: |
- docker load --input /tmp/${{ inputs.parentImage }}.tar
-
- - name: Login to Docker Hub ๐
- uses: docker/login-action@f4ef78c080cd8ba55a85445d5b36e214a81df20a # v2.1.0
- with:
- username: ${{ secrets.DOCKERHUB_USERNAME }}
- password: ${{ secrets.DOCKERHUB_TOKEN }}
- logout: true
-
- - name: Login to Containers ๐ซ
- uses: docker/login-action@f4ef78c080cd8ba55a85445d5b36e214a81df20a # v2.1.0
- with:
- registry: containers.renci.org
- username: ${{ secrets.CONTAINERHUB_USERNAME }}
- password: ${{ secrets.CONTAINERHUB_TOKEN }}
- logout: true
-
- - name: Docker meta
- id: meta
- uses: docker/metadata-action@v4
- with:
- # list of Docker images to use as base name for tags
- images: |
- ${{ env.OWNER }}/${{ inputs.image }}
- containers.renci.org/${{ env.OWNER }}/jupyter/docker-stacks/${{ inputs.image }}
- # generate Docker tags based on the following events/attributes
- tags: |
- type=ref,event=branch
- type=sha
-
- - name: Build Containers ๐ ๏ธ
- uses: docker/build-push-action@v4
+ - name: Load parent built image to Docker ๐ฅ
+ if: inputs.parent-image != ''
+ uses: ./.github/actions/load-image
with:
- context: ${{ inputs.image }}/
- push: false
- build-args: |
- NB_GID=0
- ROOT_CONTAINER=nvidia/cuda:12.1.1-cudnn8-runtime-ubuntu22.04
- # Push to renci-registry and dockerhub here.
- tags: |
- ${{ steps.meta.outputs.tags }}
- ${{ env.OWNER }}/${{ inputs.image }}:latest
- ${{ env.OWNER }}/${{ inputs.image }}:cuda
- outputs: type=docker, dest=/tmp/${{ inputs.image }}.tar
- cache-from: type=registry,ref=${{ env.OWNER }}/${{ inputs.image }}:buildcache-${{ inputs.image }}
- cache-to: type=registry,ref=${{ env.OWNER }}/${{ inputs.image }}:buildcache-${{ inputs.image }},mode=max
+ image: ${{ inputs.parent-image }}
+ platform: ${{ inputs.platform }}
+ variant: ${{ inputs.parent-variant }}
- - name: Load image to docker
- run: |
- docker load --input /tmp/${{ inputs.image }}.tar
+ - name: Pull base ubuntu image ๐ฅ
+ if: inputs.parent-image == ''
+ run: docker pull ubuntu:24.04
+ shell: bash
- # - name: Run Test Docker Image
- # run: |
- # docker run --force-rm --rm ${{ env.OWNER }}/${{ inputs.image }}:latest
+ - name: Build image ๐
+ run: docker build --rm --force-rm --tag ${{ env.REGISTRY }}/${{ env.OWNER }}/${{ inputs.image }} images/${{ inputs.image }}/${{ inputs.variant != 'default' && inputs.variant || '.' }}/ --build-arg REGISTRY=${{ env.REGISTRY }} --build-arg OWNER=${{ env.OWNER }}
+ env:
+ DOCKER_BUILDKIT: 1
+ # Full logs for CI build
+ BUILDKIT_PROGRESS: plain
+ shell: bash
- name: Run tests โ
- run: python3 -m tests.run_tests --short-image-name ${{ inputs.image }} --owner ${{ env.OWNER }}
+ run: python3 -m tests.run_tests --short-image-name ${{ inputs.image }} --registry ${{ env.REGISTRY }} --owner ${{ env.OWNER }}
shell: bash
# We do not want to push "latest" tag unless
@@ -146,31 +108,39 @@ jobs:
- name: Remove latest tag if Develop Branch
if: ${{ github.ref != 'refs/heads/main' }}
run: |
- docker rmi ${{ env.OWNER }}/${{ inputs.image }}:latest
-
- - name: Upload artifact ๐ค
- uses: actions/upload-artifact@v3
+ python3 -m tagging.write_tags_file --short-image-name ${{ inputs.image }} --tags-dir /tmp/jupyter/tags/ --registry ${{ env.REGISTRY }} --owner ${{ env.OWNER }} --variant ${{ inputs.variant }}
+ shell: bash
+ - name: Upload tags file ๐พ
+ uses: actions/upload-artifact@v4
with:
- name: ${{ inputs.image }}
- path: /tmp/${{ inputs.image }}.tar
+ name: ${{ inputs.image }}-${{ inputs.platform }}-${{ inputs.variant }}-tags
+ path: /tmp/jupyter/tags/${{ inputs.platform }}-${{ inputs.variant }}-${{ inputs.image }}.txt
+ retention-days: 3
- - name: Push to Docker ๐ ๏ธ
- run: |
- docker push --all-tags ${{ env.OWNER }}/${{ inputs.image }}
-
- # Because Containers often fails, we keep going and display a message.
- - name: Push to Containers ๐ซ
- id: containers
- run: |
- docker push --all-tags containers.renci.org/${{ env.OWNER }}/jupyter/docker-stacks/${{ inputs.image }}
- continue-on-error: true
+ - name: Write manifest and build history file ๐ท
+ run: python3 -m tagging.write_manifest --short-image-name ${{ inputs.image }} --hist-lines-dir /tmp/jupyter/hist_lines/ --manifests-dir /tmp/jupyter/manifests/ --registry ${{ env.REGISTRY }} --owner ${{ env.OWNER }} --variant ${{ inputs.variant }}
+ shell: bash
+ - name: Upload manifest file ๐พ
+ uses: actions/upload-artifact@v4
+ with:
+ name: ${{ inputs.image }}-${{ inputs.platform }}-${{ inputs.variant }}-manifest
+ path: /tmp/jupyter/manifests/${{ inputs.platform }}-${{ inputs.variant }}-${{ inputs.image }}-*.md
+ retention-days: 3
+ - name: Upload build history line ๐พ
+ uses: actions/upload-artifact@v4
+ with:
+ name: ${{ inputs.image }}-${{ inputs.platform }}-${{ inputs.variant }}-history_line
+ path: /tmp/jupyter/hist_lines/${{ inputs.platform }}-${{ inputs.variant }}-${{ inputs.image }}-*.txt
+ retention-days: 3
- # Sometimes containers.recnci.org fails so we try again.
- - name: If Containers Failed
- if: ${{ steps.containers.conclusion == 'failure' }}
+ - name: Save image as a tar for later use ๐พ
run: |
- echo "Last push failed retrying "
- docker push --all-tags containers.renci.org/${{ env.OWNER }}/jupyter/docker-stacks/${{ inputs.image }}
- echo "Completed!"
- continue-on-error: true
-
+ mkdir -p /tmp/jupyter/images/
+ docker save ${{ env.REGISTRY }}/${{ env.OWNER }}/${{ inputs.image }} | zstd > /tmp/jupyter/images/${{ inputs.image }}-${{ inputs.platform }}-${{ inputs.variant }}.tar.zst
+ shell: bash
+ - name: Upload image as artifact ๐พ
+ uses: actions/upload-artifact@v4
+ with:
+ name: ${{ inputs.image }}-${{ inputs.platform }}-${{ inputs.variant }}
+ path: /tmp/jupyter/images/${{ inputs.image }}-${{ inputs.platform }}-${{ inputs.variant }}.tar.zst
+ retention-days: 3
diff --git a/.github/workflows/docker-merge-tags.yml b/.github/workflows/docker-merge-tags.yml
new file mode 100644
index 0000000000..a476879408
--- /dev/null
+++ b/.github/workflows/docker-merge-tags.yml
@@ -0,0 +1,66 @@
+name: Download all tags from GitHub artifacts and create multi-platform manifests
+
+env:
+ OWNER: ${{ github.repository_owner }}
+ PUSH_TO_REGISTRY: ${{ (github.repository_owner == 'jupyter' || github.repository_owner == 'mathbunnyru') && (github.ref == 'refs/heads/main' || github.event_name == 'schedule') }}
+
+on:
+ workflow_call:
+ inputs:
+ variant:
+ description: Variant tag prefix
+ required: true
+ type: string
+ image:
+ description: Image name
+ required: true
+ type: string
+ secrets:
+ REGISTRY_USERNAME:
+ required: true
+ REGISTRY_TOKEN:
+ required: true
+
+jobs:
+ merge-tags:
+ runs-on: ubuntu-latest
+
+ steps:
+ - name: Checkout Repo โก๏ธ
+ uses: actions/checkout@v4
+ - name: Create dev environment ๐ฆ
+ uses: ./.github/actions/create-dev-env
+
+ - name: Download x86_64 tags file ๐ฅ
+ uses: actions/download-artifact@v4
+ with:
+ name: ${{ inputs.image }}-x86_64-${{ inputs.variant }}-tags
+ path: /tmp/jupyter/tags/
+ - name: Download aarch64 tags file ๐ฅ
+ uses: actions/download-artifact@v4
+ with:
+ name: ${{ inputs.image }}-aarch64-${{ inputs.variant }}-tags
+ path: /tmp/jupyter/tags/
+ if: github.repository_owner == 'jupyter' && !contains(inputs.variant, 'cuda')
+
+ # Docker might be stuck when pulling images
+ # https://github.com/docker/for-mac/issues/2083
+ # https://stackoverflow.com/questions/38087027/docker-compose-stuck-downloading-or-pulling-fs-layer
+ - name: Reset docker state ๐๏ธ
+ run: |
+ docker system prune --all --force
+ sudo systemctl restart docker
+ shell: bash
+
+ - name: Login to Registry ๐
+ if: env.PUSH_TO_REGISTRY == 'true'
+ uses: docker/login-action@9780b0c442fbb1117ed29e0efdff1e18412f7567 # v3.3.0
+ with:
+ registry: quay.io
+ username: ${{ secrets.REGISTRY_USERNAME }}
+ password: ${{ secrets.REGISTRY_TOKEN }}
+
+ - name: Merge tags for the images ๐
+ if: env.PUSH_TO_REGISTRY == 'true'
+ run: python3 -m tagging.merge_tags --short-image-name ${{ inputs.image }} --tags-dir /tmp/jupyter/tags/ --variant ${{ inputs.variant }}
+ shell: bash
diff --git a/.github/workflows/docker-tag-push.yml b/.github/workflows/docker-tag-push.yml
new file mode 100644
index 0000000000..887e2d2e01
--- /dev/null
+++ b/.github/workflows/docker-tag-push.yml
@@ -0,0 +1,80 @@
+name: Download a Docker image and its tags from GitHub artifacts, apply them, and push the image to the Registry
+
+env:
+ REGISTRY: quay.io
+ OWNER: ${{ github.repository_owner }}
+ PUSH_TO_REGISTRY: ${{ (github.repository_owner == 'jupyter' || github.repository_owner == 'mathbunnyru') && (github.ref == 'refs/heads/main' || github.event_name == 'schedule') }}
+
+on:
+ workflow_call:
+ inputs:
+ image:
+ description: Image name
+ required: true
+ type: string
+ platform:
+ description: Image platform
+ required: true
+ type: string
+ variant:
+ description: Variant tag prefix
+ required: true
+ type: string
+ secrets:
+ REGISTRY_USERNAME:
+ required: true
+ REGISTRY_TOKEN:
+ required: true
+
+jobs:
+ tag-push:
+ runs-on: ubuntu-latest
+
+ steps:
+ # Image with CUDA needs extra disk space
+ - name: Free disk space ๐งน
+ if: contains(inputs.variant, 'cuda') && inputs.platform == 'x86_64'
+ uses: jlumbroso/free-disk-space@54081f138730dfa15788a46383842cd2f914a1be
+ with:
+ tool-cache: false
+ android: true
+ dotnet: true
+ haskell: true
+ large-packages: false
+ docker-images: false
+ swap-storage: false
+
+ - name: Checkout Repo โก๏ธ
+ uses: actions/checkout@v4
+ - name: Create dev environment ๐ฆ
+ uses: ./.github/actions/create-dev-env
+ - name: Load image to Docker ๐ฅ
+ uses: ./.github/actions/load-image
+ with:
+ image: ${{ inputs.image }}
+ platform: ${{ inputs.platform }}
+ variant: ${{ inputs.variant }}
+
+ - name: Login to Registry ๐
+ if: env.PUSH_TO_REGISTRY == 'true'
+ uses: docker/login-action@9780b0c442fbb1117ed29e0efdff1e18412f7567 # v3.3.0
+ with:
+ registry: quay.io
+ username: ${{ secrets.REGISTRY_USERNAME }}
+ password: ${{ secrets.REGISTRY_TOKEN }}
+
+ - name: Download tags file ๐ฅ
+ uses: actions/download-artifact@v4
+ with:
+ name: ${{ inputs.image }}-${{ inputs.platform }}-${{ inputs.variant }}-tags
+ path: /tmp/jupyter/tags/
+ - name: Apply tags to the loaded image ๐ท
+ run: python3 -m tagging.apply_tags --short-image-name ${{ inputs.image }} --tags-dir /tmp/jupyter/tags/ --platform ${{ inputs.platform }} --variant ${{ inputs.variant }} --registry ${{ env.REGISTRY }} --owner ${{ env.OWNER }}
+ # This step is needed to prevent pushing non-multi-arch "latest" tag
+ - name: Remove the "latest" tag from the image ๐๏ธ
+ run: docker image rmi ${{ env.REGISTRY }}/${{ env.OWNER }}/${{ inputs.image }}:latest
+
+ - name: Push Images to Registry ๐ค
+ if: env.PUSH_TO_REGISTRY == 'true'
+ run: docker push --all-tags ${{ env.REGISTRY }}/${{ env.OWNER }}/${{ inputs.image }}
+ shell: bash
diff --git a/.github/workflows/docker-wiki-update.yml b/.github/workflows/docker-wiki-update.yml
new file mode 100644
index 0000000000..b0abefc513
--- /dev/null
+++ b/.github/workflows/docker-wiki-update.yml
@@ -0,0 +1,48 @@
+name: Download build manifests from GitHub artifacts and push them to GitHub wiki
+# We're doing everything in one workflow on purpose
+# This way we make sure we don't access wiki pages from several jobs simultaneously
+
+env:
+ PUSH_TO_REGISTRY: ${{ (github.repository_owner == 'jupyter' || github.repository_owner == 'mathbunnyru') && (github.ref == 'refs/heads/main' || github.event_name == 'schedule') }}
+
+on:
+ workflow_call:
+
+jobs:
+ wiki-update:
+ runs-on: ubuntu-latest
+
+ steps:
+ - name: Checkout Repo โก๏ธ
+ uses: actions/checkout@v4
+ - name: Create dev environment ๐ฆ
+ uses: ./.github/actions/create-dev-env
+
+ - name: Download all history lines ๐ฅ
+ uses: actions/download-artifact@v4
+ with:
+ pattern: "*-history_line"
+ path: /tmp/jupyter/hist_lines/
+
+ - name: Download all manifests ๐ฅ
+ uses: actions/download-artifact@v4
+ with:
+ pattern: "*-manifest"
+ path: /tmp/jupyter/manifests/
+
+ - name: Checkout Wiki Repo ๐
+ uses: actions/checkout@v4
+ with:
+ repository: ${{ github.repository }}.wiki
+ path: wiki/
+
+ - name: Update wiki ๐ท
+ run: python3 -m tagging.update_wiki --wiki-dir wiki/ --hist-lines-dir /tmp/jupyter/hist_lines/ --manifests-dir /tmp/jupyter/manifests/
+ shell: bash
+
+ - name: Push Wiki to GitHub ๐ค
+ if: env.PUSH_TO_REGISTRY == 'true'
+ uses: stefanzweifel/git-auto-commit-action@8621497c8c39c72f3e2a999a26b4ca1b5058a842 # v5.0.1
+ with:
+ commit_message: "Automated wiki publish for ${{ github.sha }}"
+ repository: wiki/
diff --git a/.github/workflows/docker.yml b/.github/workflows/docker.yml
index 448417b377..a3163abe6b 100644
--- a/.github/workflows/docker.yml
+++ b/.github/workflows/docker.yml
@@ -1,152 +1,508 @@
-name: Build, test and push Docker Images
+name: Build, test, and push Docker Images
+
+# [FAST_BUILD] in the PR title makes this workflow only build
+# the `jupyter/docker-stacks-foundation` and `jupyter/base-notebook` images
+# This allows to run CI faster if a full build is not required
+# This only works for a `pull_request` event and does not affect `push` to the `main` branch
on:
- # schedule:
- # # Weekly, at 03:00 on Monday UTC time
- # - cron: "0 3 * * 1"
+ schedule:
+ # Weekly, at 03:00 on Monday UTC
+ - cron: "0 3 * * 1"
+ pull_request:
+ paths:
+ - ".github/workflows/docker.yml"
+ # We use local reusable workflows to make architecture clean and simple
+ # https://docs.github.com/en/actions/using-workflows/reusing-workflows
+ - ".github/workflows/docker-build-test-upload.yml"
+ - ".github/workflows/docker-merge-tags.yml"
+ - ".github/workflows/docker-tag-push.yml"
+ - ".github/workflows/docker-wiki-update.yml"
+
+ # We use local composite actions to combine multiple workflow steps within one action
+ # https://docs.github.com/en/actions/creating-actions/about-custom-actions#composite-actions
+ - ".github/actions/create-dev-env/action.yml"
+ - ".github/actions/load-image/action.yml"
+
+ - "images/**"
+ - "!images/*/README.md"
+ - "tagging/**"
+ - "!tagging/README.md"
+ - "tests/**"
+ - "!tests/README.md"
+ - "requirements-dev.txt"
push:
branches:
- main
- - develop
- # paths-ignore:
- # - ".github/workflows/*"
- # - ".github/*"
+ paths:
+ - ".github/workflows/docker.yml"
+ - ".github/workflows/docker-build-test-upload.yml"
+ - ".github/workflows/docker-merge-tags.yml"
+ - ".github/workflows/docker-tag-push.yml"
+ - ".github/workflows/docker-wiki-update.yml"
+
+ - ".github/actions/create-dev-env/action.yml"
+ - ".github/actions/load-image/action.yml"
+
+ - "images/**"
+ - "!images/*/README.md"
+ - "tagging/**"
+ - "!tagging/README.md"
+ - "tests/**"
+ - "!tests/README.md"
+ - "requirements-dev.txt"
workflow_dispatch:
# https://docs.github.com/en/actions/using-jobs/using-concurrency
concurrency:
- # only cancel in-progress jobs or runs for the current workflow - matches against branch & tags
+ # Only cancel in-progress jobs or runs for the current workflow - matches against branch & tags
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
jobs:
+ aarch64-foundation:
+ uses: ./.github/workflows/docker-build-test-upload.yml
+ with:
+ parent-image: ""
+ image: docker-stacks-foundation
+ platform: aarch64
+ runs-on: ARM64_FAST
+ if: github.repository_owner == 'jupyter'
foundation:
uses: ./.github/workflows/docker-build-test-upload.yml
with:
- parentImage: ""
+ parent-image: ""
image: docker-stacks-foundation
- runsOn: ubuntu-latest
- secrets:
- DOCKERHUB_USERNAME: ${{ secrets.DOCKERHUB_USERNAME }}
- DOCKERHUB_TOKEN: ${{ secrets.DOCKERHUB_TOKEN }}
- CONTAINERHUB_USERNAME: ${{ secrets.CONTAINERHUB_USERNAME }}
- CONTAINERHUB_TOKEN: ${{ secrets.CONTAINERHUB_TOKEN }}
+ platform: x86_64
+ runs-on: ubuntu-latest
- base:
+ aarch64-base:
uses: ./.github/workflows/docker-build-test-upload.yml
with:
- parentImage: docker-stacks-foundation
+ parent-image: docker-stacks-foundation
image: base-notebook
- runsOn: ubuntu-latest
- secrets:
- DOCKERHUB_USERNAME: ${{ secrets.DOCKERHUB_USERNAME }}
- DOCKERHUB_TOKEN: ${{ secrets.DOCKERHUB_TOKEN }}
- CONTAINERHUB_USERNAME: ${{ secrets.CONTAINERHUB_USERNAME }}
- CONTAINERHUB_TOKEN: ${{ secrets.CONTAINERHUB_TOKEN }}
- needs: [foundation]
+ platform: aarch64
+ runs-on: ARM64_FAST
+ needs: [aarch64-foundation]
+ if: github.repository_owner == 'jupyter'
+
+ x86_64-base:
+ uses: ./.github/workflows/docker-build-test-upload.yml
+ with:
+ parent-image: docker-stacks-foundation
+ image: base-notebook
+ platform: x86_64
+ runs-on: ubuntu-latest
+ needs: [x86_64-foundation]
+
+ aarch64-minimal:
+ uses: ./.github/workflows/docker-build-test-upload.yml
+ with:
+ parent-image: base-notebook
+ image: minimal-notebook
+ platform: aarch64
+ runs-on: ARM64_FAST
+ needs: [aarch64-base]
+ if: github.repository_owner == 'jupyter' && !contains(github.event.pull_request.title, '[FAST_BUILD]')
minimal:
uses: ./.github/workflows/docker-build-test-upload.yml
with:
- parentImage: base-notebook
+ parent-image: base-notebook
image: minimal-notebook
- runsOn: ubuntu-latest
- secrets:
- DOCKERHUB_USERNAME: ${{ secrets.DOCKERHUB_USERNAME }}
- DOCKERHUB_TOKEN: ${{ secrets.DOCKERHUB_TOKEN }}
- CONTAINERHUB_USERNAME: ${{ secrets.CONTAINERHUB_USERNAME }}
- CONTAINERHUB_TOKEN: ${{ secrets.CONTAINERHUB_TOKEN }}
- needs: [base]
+ platform: x86_64
+ runs-on: ubuntu-latest
+ needs: [x86_64-base]
+ if: ${{ !contains(github.event.pull_request.title, '[FAST_BUILD]') }}
+
+ aarch64-scipy:
+ uses: ./.github/workflows/docker-build-test-upload.yml
+ with:
+ parent-image: minimal-notebook
+ image: scipy-notebook
+ platform: aarch64
+ runs-on: ARM64_FAST
+ needs: [aarch64-minimal]
+ if: github.repository_owner == 'jupyter' && !contains(github.event.pull_request.title, '[FAST_BUILD]')
scipy:
uses: ./.github/workflows/docker-build-test-upload.yml
with:
- parentImage: minimal-notebook
+ parent-image: minimal-notebook
image: scipy-notebook
- runsOn: ubuntu-latest
- secrets:
- DOCKERHUB_USERNAME: ${{ secrets.DOCKERHUB_USERNAME }}
- DOCKERHUB_TOKEN: ${{ secrets.DOCKERHUB_TOKEN }}
- CONTAINERHUB_USERNAME: ${{ secrets.CONTAINERHUB_USERNAME }}
- CONTAINERHUB_TOKEN: ${{ secrets.CONTAINERHUB_TOKEN }}
- needs: [minimal]
+ platform: x86_64
+ runs-on: ubuntu-latest
+ needs: [x86_64-minimal]
+ if: ${{ !contains(github.event.pull_request.title, '[FAST_BUILD]') }}
+
+ aarch64-r:
+ uses: ./.github/workflows/docker-build-test-upload.yml
+ with:
+ parent-image: minimal-notebook
+ image: r-notebook
+ platform: aarch64
+ runs-on: ARM64_SLOW
+ needs: [aarch64-minimal]
+ if: github.repository_owner == 'jupyter' && !contains(github.event.pull_request.title, '[FAST_BUILD]')
r:
uses: ./.github/workflows/docker-build-test-upload.yml
with:
- parentImage: minimal-notebook
+ parent-image: minimal-notebook
image: r-notebook
- runsOn: ubuntu-latest
- secrets:
- DOCKERHUB_USERNAME: ${{ secrets.DOCKERHUB_USERNAME }}
- DOCKERHUB_TOKEN: ${{ secrets.DOCKERHUB_TOKEN }}
- CONTAINERHUB_USERNAME: ${{ secrets.CONTAINERHUB_USERNAME }}
- CONTAINERHUB_TOKEN: ${{ secrets.CONTAINERHUB_TOKEN }}
- needs: [minimal]
+ platform: x86_64
+ runs-on: ubuntu-latest
+ needs: [x86_64-minimal]
+ if: ${{ !contains(github.event.pull_request.title, '[FAST_BUILD]') }}
- tensorflow:
+ aarch64-julia:
uses: ./.github/workflows/docker-build-test-upload.yml
with:
- parentImage: scipy-notebook
+ parent-image: minimal-notebook
+ image: julia-notebook
+ platform: aarch64
+ runs-on: ARM64_SLOW
+ needs: [aarch64-minimal]
+ if: github.repository_owner == 'jupyter' && !contains(github.event.pull_request.title, '[FAST_BUILD]')
+
+ x86_64-julia:
+ uses: ./.github/workflows/docker-build-test-upload.yml
+ with:
+ parent-image: minimal-notebook
+ image: julia-notebook
+ platform: x86_64
+ runs-on: ubuntu-latest
+ needs: [x86_64-minimal]
+ if: ${{ !contains(github.event.pull_request.title, '[FAST_BUILD]') }}
+
+ aarch64-tensorflow:
+ uses: ./.github/workflows/docker-build-test-upload.yml
+ with:
+ parent-image: scipy-notebook
image: tensorflow-notebook
- runsOn: ubuntu-latest
- secrets:
- DOCKERHUB_USERNAME: ${{ secrets.DOCKERHUB_USERNAME }}
- DOCKERHUB_TOKEN: ${{ secrets.DOCKERHUB_TOKEN }}
- CONTAINERHUB_USERNAME: ${{ secrets.CONTAINERHUB_USERNAME }}
- CONTAINERHUB_TOKEN: ${{ secrets.CONTAINERHUB_TOKEN }}
- needs: [scipy]
+ platform: aarch64
+ runs-on: ARM64_SLOW
+ needs: [aarch64-scipy]
+ if: github.repository_owner == 'jupyter' && !contains(github.event.pull_request.title, '[FAST_BUILD]')
+
+ x86_64-tensorflow:
+ uses: ./.github/workflows/docker-build-test-upload.yml
+ with:
+ parent-image: scipy-notebook
+ image: tensorflow-notebook
+ platform: x86_64
+ runs-on: ubuntu-latest
+ needs: [x86_64-scipy]
+ if: ${{ !contains(github.event.pull_request.title, '[FAST_BUILD]') }}
+
+ x86_64-tensorflow-cuda:
+ uses: ./.github/workflows/docker-build-test-upload.yml
+ with:
+ parent-image: scipy-notebook
+ image: tensorflow-notebook
+ variant: cuda
+ platform: x86_64
+ runs-on: ubuntu-latest
+ needs: [x86_64-scipy]
+ if: ${{ !contains(github.event.pull_request.title, '[FAST_BUILD]') }}
+
+ aarch64-pytorch:
+ uses: ./.github/workflows/docker-build-test-upload.yml
+ with:
+ parent-image: scipy-notebook
+ image: pytorch-notebook
+ platform: aarch64
+ runs-on: ARM64_SLOW
+ needs: [aarch64-scipy]
+ if: github.repository_owner == 'jupyter' && !contains(github.event.pull_request.title, '[FAST_BUILD]')
+
+ x86_64-pytorch:
+ uses: ./.github/workflows/docker-build-test-upload.yml
+ with:
+ parent-image: scipy-notebook
+ image: pytorch-notebook
+ platform: x86_64
+ runs-on: ubuntu-latest
+ needs: [x86_64-scipy]
+ if: ${{ !contains(github.event.pull_request.title, '[FAST_BUILD]') }}
+
+ x86_64-pytorch-cuda11:
+ uses: ./.github/workflows/docker-build-test-upload.yml
+ with:
+ parent-image: scipy-notebook
+ image: pytorch-notebook
+ variant: cuda11
+ platform: x86_64
+ runs-on: ubuntu-latest
+ needs: [x86_64-scipy]
+ if: ${{ !contains(github.event.pull_request.title, '[FAST_BUILD]') }}
+
+ x86_64-pytorch-cuda12:
+ uses: ./.github/workflows/docker-build-test-upload.yml
+ with:
+ parent-image: scipy-notebook
+ image: pytorch-notebook
+ variant: cuda12
+ platform: x86_64
+ runs-on: ubuntu-latest
+ needs: [x86_64-scipy]
+ if: ${{ !contains(github.event.pull_request.title, '[FAST_BUILD]') }}
datascience:
uses: ./.github/workflows/docker-build-test-upload.yml
with:
- parentImage: scipy-notebook
+ parent-image: scipy-notebook
image: datascience-notebook
- runsOn: ubuntu-latest
- secrets:
- DOCKERHUB_USERNAME: ${{ secrets.DOCKERHUB_USERNAME }}
- DOCKERHUB_TOKEN: ${{ secrets.DOCKERHUB_TOKEN }}
- CONTAINERHUB_USERNAME: ${{ secrets.CONTAINERHUB_USERNAME }}
- CONTAINERHUB_TOKEN: ${{ secrets.CONTAINERHUB_TOKEN }}
- needs: [scipy]
+ platform: aarch64
+ runs-on: ARM64_SLOW
+ needs: [aarch64-scipy]
+ if: github.repository_owner == 'jupyter' && !contains(github.event.pull_request.title, '[FAST_BUILD]')
- julia:
+ x86_64-datascience:
uses: ./.github/workflows/docker-build-test-upload.yml
with:
- parentImage: minimal-notebook
- image: julia-notebook
- runsOn: ubuntu-latest
- secrets:
- DOCKERHUB_USERNAME: ${{ secrets.DOCKERHUB_USERNAME }}
- DOCKERHUB_TOKEN: ${{ secrets.DOCKERHUB_TOKEN }}
- CONTAINERHUB_USERNAME: ${{ secrets.CONTAINERHUB_USERNAME }}
- CONTAINERHUB_TOKEN: ${{ secrets.CONTAINERHUB_TOKEN }}
- needs: [minimal]
+ parent-image: scipy-notebook
+ image: datascience-notebook
+ platform: x86_64
+ runs-on: ubuntu-latest
+ needs: [x86_64-scipy]
+ if: ${{ !contains(github.event.pull_request.title, '[FAST_BUILD]') }}
+
+ aarch64-pyspark:
+ uses: ./.github/workflows/docker-build-test-upload.yml
+ with:
+ parent-image: scipy-notebook
+ image: pyspark-notebook
+ platform: aarch64
+ runs-on: ARM64_FAST
+ needs: [aarch64-scipy]
+ if: github.repository_owner == 'jupyter' && !contains(github.event.pull_request.title, '[FAST_BUILD]')
pyspark:
uses: ./.github/workflows/docker-build-test-upload.yml
with:
- parentImage: scipy-notebook
+ parent-image: scipy-notebook
image: pyspark-notebook
- runsOn: ubuntu-latest
- secrets:
- DOCKERHUB_USERNAME: ${{ secrets.DOCKERHUB_USERNAME }}
- DOCKERHUB_TOKEN: ${{ secrets.DOCKERHUB_TOKEN }}
- CONTAINERHUB_USERNAME: ${{ secrets.CONTAINERHUB_USERNAME }}
- CONTAINERHUB_TOKEN: ${{ secrets.CONTAINERHUB_TOKEN }}
- needs: [scipy]
+ platform: x86_64
+ runs-on: ubuntu-latest
+ needs: [x86_64-scipy]
+ if: ${{ !contains(github.event.pull_request.title, '[FAST_BUILD]') }}
- all-spark:
+ aarch64-all-spark:
uses: ./.github/workflows/docker-build-test-upload.yml
with:
- parentImage: pyspark-notebook
+ parent-image: pyspark-notebook
image: all-spark-notebook
- runsOn: ubuntu-latest
+ platform: aarch64
+ runs-on: ARM64_FAST
+ needs: [aarch64-pyspark]
+ if: github.repository_owner == 'jupyter' && !contains(github.event.pull_request.title, '[FAST_BUILD]')
+
+ x86_64-all-spark:
+ uses: ./.github/workflows/docker-build-test-upload.yml
+ with:
+ parent-image: pyspark-notebook
+ image: all-spark-notebook
+ platform: x86_64
+ runs-on: ubuntu-latest
+ needs: [x86_64-pyspark]
+ if: ${{ !contains(github.event.pull_request.title, '[FAST_BUILD]') }}
+
+ aarch64-images-tag-push:
+ uses: ./.github/workflows/docker-tag-push.yml
+ with:
+ platform: aarch64
+ image: ${{ matrix.image-variant.image }}
+ variant: ${{ matrix.image-variant.variant }}
+ secrets:
+ REGISTRY_USERNAME: ${{ secrets.QUAY_USERNAME }}
+ REGISTRY_TOKEN: ${{ secrets.QUAY_ROBOT_TOKEN }}
+ strategy:
+ matrix:
+ image-variant:
+ [
+ { image: docker-stacks-foundation, variant: default },
+ { image: base-notebook, variant: default },
+ { image: minimal-notebook, variant: default },
+ { image: scipy-notebook, variant: default },
+ { image: r-notebook, variant: default },
+ { image: julia-notebook, variant: default },
+ { image: tensorflow-notebook, variant: default },
+ { image: pytorch-notebook, variant: default },
+ { image: datascience-notebook, variant: default },
+ { image: pyspark-notebook, variant: default },
+ { image: all-spark-notebook, variant: default },
+ ]
+ needs:
+ [
+ aarch64-foundation,
+ aarch64-base,
+ aarch64-minimal,
+ aarch64-scipy,
+ aarch64-r,
+ aarch64-julia,
+ aarch64-tensorflow,
+ aarch64-pytorch,
+ aarch64-datascience,
+ aarch64-pyspark,
+ aarch64-all-spark,
+ ]
+ if: github.repository_owner == 'jupyter' && !contains(github.event.pull_request.title, '[FAST_BUILD]')
+
+ aarch64-images-tag-push-fast:
+ uses: ./.github/workflows/docker-tag-push.yml
+ with:
+ platform: aarch64
+ image: ${{ matrix.image-variant.image }}
+ variant: ${{ matrix.image-variant.variant }}
+ secrets:
+ REGISTRY_USERNAME: ${{ secrets.QUAY_USERNAME }}
+ REGISTRY_TOKEN: ${{ secrets.QUAY_ROBOT_TOKEN }}
+ strategy:
+ matrix:
+ image-variant:
+ [
+ { image: docker-stacks-foundation, variant: default },
+ { image: base-notebook, variant: default },
+ ]
+ needs: [aarch64-foundation, aarch64-base]
+ if: github.repository_owner == 'jupyter' && contains(github.event.pull_request.title, '[FAST_BUILD]')
+
+ x86_64-images-tag-push:
+ uses: ./.github/workflows/docker-tag-push.yml
+ with:
+ platform: x86_64
+ image: ${{ matrix.image-variant.image }}
+ variant: ${{ matrix.image-variant.variant }}
+ secrets:
+ REGISTRY_USERNAME: ${{ secrets.QUAY_USERNAME }}
+ REGISTRY_TOKEN: ${{ secrets.QUAY_ROBOT_TOKEN }}
+ strategy:
+ matrix:
+ image-variant:
+ [
+ { image: docker-stacks-foundation, variant: default },
+ { image: base-notebook, variant: default },
+ { image: minimal-notebook, variant: default },
+ { image: scipy-notebook, variant: default },
+ { image: r-notebook, variant: default },
+ { image: julia-notebook, variant: default },
+ { image: tensorflow-notebook, variant: default },
+ { image: tensorflow-notebook, variant: cuda },
+ { image: pytorch-notebook, variant: default },
+ { image: pytorch-notebook, variant: cuda11 },
+ { image: pytorch-notebook, variant: cuda12 },
+ { image: datascience-notebook, variant: default },
+ { image: pyspark-notebook, variant: default },
+ { image: all-spark-notebook, variant: default },
+ ]
+ needs:
+ [
+ x86_64-foundation,
+ x86_64-base,
+ x86_64-minimal,
+ x86_64-scipy,
+ x86_64-r,
+ x86_64-julia,
+ x86_64-tensorflow,
+ x86_64-pytorch,
+ x86_64-datascience,
+ x86_64-pyspark,
+ x86_64-all-spark,
+ ]
+ if: ${{ !contains(github.event.pull_request.title, '[FAST_BUILD]') }}
+
+ x86_64-images-tag-push-fast:
+ uses: ./.github/workflows/docker-tag-push.yml
+ with:
+ platform: x86_64
+ image: ${{ matrix.image-variant.image }}
+ variant: ${{ matrix.image-variant.variant }}
+ secrets:
+ REGISTRY_USERNAME: ${{ secrets.QUAY_USERNAME }}
+ REGISTRY_TOKEN: ${{ secrets.QUAY_ROBOT_TOKEN }}
+ strategy:
+ matrix:
+ image-variant:
+ [
+ { image: docker-stacks-foundation, variant: default },
+ { image: base-notebook, variant: default },
+ ]
+ needs: [x86_64-foundation, x86_64-base]
+ if: contains(github.event.pull_request.title, '[FAST_BUILD]')
+
+ merge-tags:
+ uses: ./.github/workflows/docker-merge-tags.yml
+ with:
+ image: ${{ matrix.image-variant.image }}
+ variant: ${{ matrix.image-variant.variant }}
secrets:
- DOCKERHUB_USERNAME: ${{ secrets.DOCKERHUB_USERNAME }}
- DOCKERHUB_TOKEN: ${{ secrets.DOCKERHUB_TOKEN }}
- CONTAINERHUB_USERNAME: ${{ secrets.CONTAINERHUB_USERNAME }}
- CONTAINERHUB_TOKEN: ${{ secrets.CONTAINERHUB_TOKEN }}
- needs: [pyspark]
-
\ No newline at end of file
+ REGISTRY_USERNAME: ${{ secrets.QUAY_USERNAME }}
+ REGISTRY_TOKEN: ${{ secrets.QUAY_ROBOT_TOKEN }}
+ strategy:
+ matrix:
+ image-variant:
+ [
+ { image: docker-stacks-foundation, variant: default },
+ { image: base-notebook, variant: default },
+ { image: minimal-notebook, variant: default },
+ { image: scipy-notebook, variant: default },
+ { image: r-notebook, variant: default },
+ { image: julia-notebook, variant: default },
+ { image: tensorflow-notebook, variant: default },
+ { image: tensorflow-notebook, variant: cuda },
+ { image: pytorch-notebook, variant: default },
+ { image: pytorch-notebook, variant: cuda11 },
+ { image: pytorch-notebook, variant: cuda12 },
+ { image: datascience-notebook, variant: default },
+ { image: pyspark-notebook, variant: default },
+ { image: all-spark-notebook, variant: default },
+ ]
+ needs: [aarch64-images-tag-push, x86_64-images-tag-push]
+ if: |
+ always() &&
+ needs.x86_64-images-tag-push.result == 'success' &&
+ (needs.aarch64-images-tag-push.result == 'success' || needs.aarch64-images-tag-push.result == 'skipped') &&
+ !contains(github.event.pull_request.title, '[FAST_BUILD]')
+
+ merge-tags-fast:
+ uses: ./.github/workflows/docker-merge-tags.yml
+ with:
+ image: ${{ matrix.image-variant.image }}
+ variant: ${{ matrix.image-variant.variant }}
+ secrets:
+ REGISTRY_USERNAME: ${{ secrets.QUAY_USERNAME }}
+ REGISTRY_TOKEN: ${{ secrets.QUAY_ROBOT_TOKEN }}
+ strategy:
+ matrix:
+ image-variant:
+ [
+ { image: docker-stacks-foundation, variant: default },
+ { image: base-notebook, variant: default },
+ ]
+ needs: [aarch64-images-tag-push-fast, x86_64-images-tag-push-fast]
+ if: |
+ always() &&
+ needs.x86_64-images-tag-push-fast.result == 'success' &&
+ (needs.aarch64-images-tag-push-fast.result == 'success' || needs.aarch64-images-tag-push-fast.result == 'skipped') &&
+ contains(github.event.pull_request.title, '[FAST_BUILD]')
+
+ wiki-update:
+ uses: ./.github/workflows/docker-wiki-update.yml
+ needs: [aarch64-images-tag-push, x86_64-images-tag-push]
+ if: github.repository_owner == 'jupyter' && !contains(github.event.pull_request.title, '[FAST_BUILD]')
+ permissions:
+ contents: write
+
+ wiki-update-fast:
+ uses: ./.github/workflows/docker-wiki-update.yml
+ needs: [aarch64-images-tag-push-fast, x86_64-images-tag-push-fast]
+ if: github.repository_owner == 'jupyter' && contains(github.event.pull_request.title, '[FAST_BUILD]')
+ permissions:
+ contents: write
+
+ contributed-recipes:
+ uses: ./.github/workflows/contributed-recipes.yml
+ needs: [merge-tags]
+ if: github.repository_owner == 'jupyter' && (github.ref == 'refs/heads/main' || github.event_name == 'schedule')
diff --git a/.github/workflows/helx-build-test-upload.yml b/.github/workflows/helx-build-test-upload.yml
new file mode 100644
index 0000000000..d56693b904
--- /dev/null
+++ b/.github/workflows/helx-build-test-upload.yml
@@ -0,0 +1,165 @@
+name: Download parent image, build a new one and test it; then upload the image, tags and manifests to GitHub artifacts
+
+env:
+ OWNER: ${{ github.repository_owner }}
+
+on:
+ workflow_call:
+ inputs:
+ parentImage:
+ description: Parent image name
+ required: true
+ type: string
+ image:
+ description: Image name
+ required: true
+ type: string
+ # platform:
+ # description: Image platform
+ # required: true
+ # type: string
+ runsOn:
+ description: GitHub Actions Runner image
+ required: true
+ type: string
+ secrets:
+ DOCKERHUB_USERNAME:
+ required: true
+ DOCKERHUB_TOKEN:
+ required: true
+ CONTAINERHUB_USERNAME:
+ required: true
+ CONTAINERHUB_TOKEN:
+ required: true
+
+jobs:
+ build-test-upload:
+ runs-on: ${{ inputs.runsOn }}
+
+ steps:
+ - name: Checkout Repo โก๏ธ
+ uses: actions/checkout@v4
+
+ - name: Set Up Python ๐
+ uses: actions/setup-python@v5
+ with:
+ python-version: 3.x
+
+ - uses: actions/cache@v3
+ name: Cache Python
+ with:
+ path: ${{ env.pythonLocation }}
+ key: ${{ env.pythonLocation }}-${{ hashFiles('setup.py') }}-${{ hashFiles('requirements.txt') }}
+
+ - name: Install Dev Dependencies ๐ฆ
+ run: |
+ pip install --upgrade pip
+ pip install --upgrade -r requirements-dev.txt
+ shell: bash
+
+ - name: Set up Docker Buildx
+ uses: docker/setup-buildx-action@v3
+ with:
+ driver-opts: |
+ network=host
+ -
+ name: Download artifact ๐ฅ
+ if: ${{ inputs.parentImage != '' }}
+ uses: actions/download-artifact@v4
+ with:
+ name: ${{ inputs.parentImage }}
+ path: /tmp/jupyter/images/
+
+ - name: Load parent built image to Docker ๐ณ
+ if: ${{ inputs.parentImage != '' }}
+ run: |
+ docker load --input /tmp/jupyter/images/${{ inputs.parentImage }}.tar
+
+ - name: Login to Docker Hub ๐
+ uses: docker/login-action@v3
+ with:
+ username: ${{ secrets.DOCKERHUB_USERNAME }}
+ password: ${{ secrets.DOCKERHUB_TOKEN }}
+ logout: true
+
+ - name: Login to Containers ๐ซ
+ uses: docker/login-action@v3
+ with:
+ registry: containers.renci.org
+ username: ${{ secrets.CONTAINERHUB_USERNAME }}
+ password: ${{ secrets.CONTAINERHUB_TOKEN }}
+ logout: true
+
+ - name: Docker meta
+ id: meta
+ uses: docker/metadata-action@v5
+ with:
+ # list of Docker images to use as base name for tags
+ images: |
+ ${{ env.OWNER }}/${{ inputs.image }}
+ # containers.renci.org/${{ env.OWNER }}/jupyter/docker-stacks/${{ inputs.image }}
+ # generate Docker tags based on the following events/attributes
+ # using the type=raw,value="" syntax allows for custom tags.
+ tags: |
+ type=ref,event=branch
+ type=sha
+ type=raw,value=cuda
+ type=raw,value=latest
+
+ - name: Build Containers ๐ ๏ธ
+ uses: docker/build-push-action@v6
+ with:
+ context: /images/${{ inputs.image }}/
+ push: false
+ build-args: |
+ NB_GID=0
+ ROOT_CONTAINER=nvidia/cuda:12.2.2-cudnn8-runtime-ubuntu22.04
+ # Push to renci-registry and dockerhub here.
+ tags: |
+ ${{ steps.meta.outputs.tags }}
+ outputs: type=docker, dest=/tmp/jupyter/images/${{ inputs.image }}.tar
+ cache-from: type=registry,ref=${{ env.OWNER }}/${{ inputs.image }}:buildcache-${{ inputs.image }}
+ cache-to: type=registry,ref=${{ env.OWNER }}/${{ inputs.image }}:buildcache-${{ inputs.image }},mode=max
+
+ - name: Load image to docker
+ run: |
+ docker load --input /tmp/jupyter/images/${{ inputs.image }}.tar
+
+ - name: Run tests โ
+ run: python3 -m tests.run_tests --short-image-name ${{ inputs.image }} --owner ${{ env.OWNER }}
+ shell: bash
+
+ # We do not want to push "latest" tag unless
+ # the images being built are in the main branch.
+ # the 'Run tests' step however expects "latest" tag;
+ # hence - we just delete the tag after the image is tested.
+ - name: Remove latest tag if Develop Branch
+ if: ${{ github.ref != 'refs/heads/main' }}
+ run: |
+ docker rmi ${{ env.OWNER }}/${{ inputs.image }}:latest
+
+ - name: Upload artifact ๐ค
+ uses: actions/upload-artifact@v4
+ with:
+ name: ${{ inputs.image }}
+ path: /tmp/jupyter/images/${{ inputs.image }}.tar
+
+ - name: Push to Docker ๐ ๏ธ
+ run: |
+ docker push --all-tags ${{ env.OWNER }}/${{ inputs.image }}
+
+ # Because Containers often fails, we keep going and display a message.
+ - name: Push to Containers ๐ซ
+ id: containers
+ run: |
+ docker push --all-tags containers.renci.org/${{ env.OWNER }}/jupyter/docker-stacks/${{ inputs.image }}
+ continue-on-error: true
+
+ # Sometimes containers.recnci.org fails so we try again.
+ - name: If Containers Failed
+ if: ${{ steps.containers.conclusion == 'failure' }}
+ run: |
+ echo "Last push failed retrying "
+ docker push --all-tags containers.renci.org/${{ env.OWNER }}/jupyter/docker-stacks/${{ inputs.image }}
+ echo "Completed!"
+ continue-on-error: true
\ No newline at end of file
diff --git a/.github/workflows/helx-docker.yml b/.github/workflows/helx-docker.yml
new file mode 100644
index 0000000000..c83eba70ad
--- /dev/null
+++ b/.github/workflows/helx-docker.yml
@@ -0,0 +1,148 @@
+name: Build, test and push Docker Images
+
+on:
+ push:
+ branches:
+ - main
+ - develop
+ # paths-ignore:
+ # - ".github/workflows/*"
+ # - ".github/*"
+ workflow_dispatch:
+
+# https://docs.github.com/en/actions/using-jobs/using-concurrency
+concurrency:
+ # only cancel in-progress jobs or runs for the current workflow - matches against branch & tags
+ group: ${{ github.workflow }}-${{ github.ref }}
+ cancel-in-progress: true
+
+jobs:
+
+ foundation:
+ uses: ./.github/workflows/docker-build-test-upload.yml
+ with:
+ parentImage: ""
+ image: docker-stacks-foundation
+ runsOn: ubuntu-latest
+ secrets:
+ DOCKERHUB_USERNAME: ${{ secrets.DOCKERHUB_USERNAME }}
+ DOCKERHUB_TOKEN: ${{ secrets.DOCKERHUB_TOKEN }}
+ CONTAINERHUB_USERNAME: ${{ secrets.CONTAINERHUB_USERNAME }}
+ CONTAINERHUB_TOKEN: ${{ secrets.CONTAINERHUB_TOKEN }}
+
+ base:
+ uses: ./.github/workflows/docker-build-test-upload.yml
+ with:
+ parentImage: docker-stacks-foundation
+ image: base-notebook
+ runsOn: ubuntu-latest
+ secrets:
+ DOCKERHUB_USERNAME: ${{ secrets.DOCKERHUB_USERNAME }}
+ DOCKERHUB_TOKEN: ${{ secrets.DOCKERHUB_TOKEN }}
+ CONTAINERHUB_USERNAME: ${{ secrets.CONTAINERHUB_USERNAME }}
+ CONTAINERHUB_TOKEN: ${{ secrets.CONTAINERHUB_TOKEN }}
+ needs: [foundation]
+
+ minimal:
+ uses: ./.github/workflows/docker-build-test-upload.yml
+ with:
+ parentImage: base-notebook
+ image: minimal-notebook
+ runsOn: ubuntu-latest
+ secrets:
+ DOCKERHUB_USERNAME: ${{ secrets.DOCKERHUB_USERNAME }}
+ DOCKERHUB_TOKEN: ${{ secrets.DOCKERHUB_TOKEN }}
+ CONTAINERHUB_USERNAME: ${{ secrets.CONTAINERHUB_USERNAME }}
+ CONTAINERHUB_TOKEN: ${{ secrets.CONTAINERHUB_TOKEN }}
+ needs: [base]
+
+ scipy:
+ uses: ./.github/workflows/docker-build-test-upload.yml
+ with:
+ parentImage: minimal-notebook
+ image: scipy-notebook
+ runsOn: ubuntu-latest
+ secrets:
+ DOCKERHUB_USERNAME: ${{ secrets.DOCKERHUB_USERNAME }}
+ DOCKERHUB_TOKEN: ${{ secrets.DOCKERHUB_TOKEN }}
+ CONTAINERHUB_USERNAME: ${{ secrets.CONTAINERHUB_USERNAME }}
+ CONTAINERHUB_TOKEN: ${{ secrets.CONTAINERHUB_TOKEN }}
+ needs: [minimal]
+
+ r:
+ uses: ./.github/workflows/docker-build-test-upload.yml
+ with:
+ parentImage: minimal-notebook
+ image: r-notebook
+ runsOn: ubuntu-latest
+ secrets:
+ DOCKERHUB_USERNAME: ${{ secrets.DOCKERHUB_USERNAME }}
+ DOCKERHUB_TOKEN: ${{ secrets.DOCKERHUB_TOKEN }}
+ CONTAINERHUB_USERNAME: ${{ secrets.CONTAINERHUB_USERNAME }}
+ CONTAINERHUB_TOKEN: ${{ secrets.CONTAINERHUB_TOKEN }}
+ needs: [minimal]
+
+ tensorflow:
+ uses: ./.github/workflows/docker-build-test-upload.yml
+ with:
+ parentImage: scipy-notebook
+ image: tensorflow-notebook
+ runsOn: ubuntu-latest
+ secrets:
+ DOCKERHUB_USERNAME: ${{ secrets.DOCKERHUB_USERNAME }}
+ DOCKERHUB_TOKEN: ${{ secrets.DOCKERHUB_TOKEN }}
+ CONTAINERHUB_USERNAME: ${{ secrets.CONTAINERHUB_USERNAME }}
+ CONTAINERHUB_TOKEN: ${{ secrets.CONTAINERHUB_TOKEN }}
+ needs: [scipy]
+
+ datascience:
+ uses: ./.github/workflows/docker-build-test-upload.yml
+ with:
+ parentImage: scipy-notebook
+ image: datascience-notebook
+ runsOn: ubuntu-latest
+ secrets:
+ DOCKERHUB_USERNAME: ${{ secrets.DOCKERHUB_USERNAME }}
+ DOCKERHUB_TOKEN: ${{ secrets.DOCKERHUB_TOKEN }}
+ CONTAINERHUB_USERNAME: ${{ secrets.CONTAINERHUB_USERNAME }}
+ CONTAINERHUB_TOKEN: ${{ secrets.CONTAINERHUB_TOKEN }}
+ needs: [scipy]
+
+ julia:
+ uses: ./.github/workflows/docker-build-test-upload.yml
+ with:
+ parentImage: minimal-notebook
+ image: julia-notebook
+ runsOn: ubuntu-latest
+ secrets:
+ DOCKERHUB_USERNAME: ${{ secrets.DOCKERHUB_USERNAME }}
+ DOCKERHUB_TOKEN: ${{ secrets.DOCKERHUB_TOKEN }}
+ CONTAINERHUB_USERNAME: ${{ secrets.CONTAINERHUB_USERNAME }}
+ CONTAINERHUB_TOKEN: ${{ secrets.CONTAINERHUB_TOKEN }}
+ needs: [minimal]
+
+ pyspark:
+ uses: ./.github/workflows/docker-build-test-upload.yml
+ with:
+ parentImage: scipy-notebook
+ image: pyspark-notebook
+ runsOn: ubuntu-latest
+ secrets:
+ DOCKERHUB_USERNAME: ${{ secrets.DOCKERHUB_USERNAME }}
+ DOCKERHUB_TOKEN: ${{ secrets.DOCKERHUB_TOKEN }}
+ CONTAINERHUB_USERNAME: ${{ secrets.CONTAINERHUB_USERNAME }}
+ CONTAINERHUB_TOKEN: ${{ secrets.CONTAINERHUB_TOKEN }}
+ needs: [scipy]
+
+ all-spark:
+ uses: ./.github/workflows/docker-build-test-upload.yml
+ with:
+ parentImage: pyspark-notebook
+ image: all-spark-notebook
+ runsOn: ubuntu-latest
+ secrets:
+ DOCKERHUB_USERNAME: ${{ secrets.DOCKERHUB_USERNAME }}
+ DOCKERHUB_TOKEN: ${{ secrets.DOCKERHUB_TOKEN }}
+ CONTAINERHUB_USERNAME: ${{ secrets.CONTAINERHUB_USERNAME }}
+ CONTAINERHUB_TOKEN: ${{ secrets.CONTAINERHUB_TOKEN }}
+ needs: [pyspark]
\ No newline at end of file
diff --git a/.github/workflows/hub-overview.yml b/.github/workflows/hub-overview.yml
index 929367d53e..ce0f1f5c45 100644
--- a/.github/workflows/hub-overview.yml
+++ b/.github/workflows/hub-overview.yml
@@ -65,3 +65,4 @@ jobs:
# description: "Python and Spark Jupyter Notebook Stack from https://github.com/jupyter/docker-stacks"
# - image: all-spark-notebook
# description: "Python, Scala, R and Spark Jupyter Notebook Stack from https://github.com/jupyter/docker-stacks"
+
diff --git a/.github/workflows/pre-commit.yml b/.github/workflows/pre-commit.yml
deleted file mode 100644
index 20bcf94f8f..0000000000
--- a/.github/workflows/pre-commit.yml
+++ /dev/null
@@ -1,33 +0,0 @@
-name: Run pre-commit hooks
-
-on:
- pull_request:
- push:
- branches:
- - main
- workflow_dispatch:
-
-permissions:
- contents: read
-
-jobs:
- run-hooks:
- name: Run pre-commit hooks
- runs-on: ubuntu-latest
-
- steps:
- - name: Checkout Repo โก๏ธ
- uses: actions/checkout@v3
-
- - name: Set Up Python ๐
- uses: actions/setup-python@v4
- with:
- python-version: 3.x
-
- - name: Install pre-commit ๐ฆ
- run: |
- pip install --upgrade pip
- pip install --upgrade pre-commit
-
- - name: Run pre-commit hooks โ
- run: pre-commit run --all-files --hook-stage manual
diff --git a/.hadolint.yaml b/.hadolint.yaml
index 3d6d83e9da..11ee226343 100644
--- a/.hadolint.yaml
+++ b/.hadolint.yaml
@@ -2,3 +2,4 @@
ignored:
- DL3006
- DL3008
+ - DL3013
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
index 34df15e0fc..25658fa145 100644
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@@ -14,51 +14,62 @@
repos:
# Autoupdate: Python code
- repo: https://github.com/asottile/pyupgrade
- rev: v3.6.0
+ rev: v3.17.0
hooks:
- id: pyupgrade
args: [--py39-plus]
# Automatically sort python imports
- - repo: https://github.com/pycqa/isort
- rev: 5.12.0
+ - repo: https://github.com/PyCQA/isort
+ rev: 5.13.2
hooks:
- id: isort
args: [--profile, black]
# Autoformat: Python code
- repo: https://github.com/psf/black
- rev: 23.3.0
+ rev: 24.8.0
hooks:
- id: black
args: [--target-version=py39]
# Check python code static typing
- repo: https://github.com/pre-commit/mirrors-mypy
- rev: v1.3.0
+ rev: v1.11.2
hooks:
- id: mypy
args: [--config, ./mypy.ini]
additional_dependencies:
- ["numpy", "pytest", "requests", "types-requests", "types-tabulate"]
- # Unfortunately, `pre-commit` only runs on changed files
+ [
+ "beautifulsoup4",
+ "numpy",
+ "pytest",
+ "requests",
+ "urllib3",
+ "types-beautifulsoup4",
+ "types-PyYAML",
+ "types-requests",
+ "types-tabulate",
+ "types-urllib3",
+ ]
+ # Unfortunately, `pre-commit` only runs on modified files
# This doesn't work well with `mypy --follow-imports error`
# See: https://github.com/pre-commit/mirrors-mypy/issues/34#issuecomment-1062160321
#
# To work around this we run `mypy` only in manual mode
- # So it won't run as part of `git commit` command
- # But it will still be run as part of `pre-commit` workflow and give expected results
+ # So it won't run as part of `git commit` command,
+ # but it will still be run as part of `pre-commit` workflow and give expected results
stages: [manual]
# Autoformat: YAML, JSON, Markdown, etc.
- repo: https://github.com/pre-commit/mirrors-prettier
- rev: v3.0.0-alpha.9-for-vscode
+ rev: v4.0.0-alpha.8
hooks:
- id: prettier
# `pre-commit sample-config` default hooks
- repo: https://github.com/pre-commit/pre-commit-hooks
- rev: v4.4.0
+ rev: v4.6.0
hooks:
- id: check-added-large-files
- id: end-of-file-fixer
@@ -66,22 +77,33 @@ repos:
- id: trailing-whitespace
# Lint: Dockerfile
- - repo: https://github.com/hadolint/hadolint.git
- rev: v2.12.1-beta
+ - repo: https://github.com/hadolint/hadolint
+ rev: v2.13.0-beta
hooks:
- id: hadolint-docker
entry: hadolint/hadolint:v2.12.1-beta hadolint
+ # Lint: Dockerfile
+ # We're linting .dockerfile files as well
+ - repo: https://github.com/hadolint/hadolint
+ rev: v2.13.0-beta
+ hooks:
+ - id: hadolint-docker
+ name: Lint *.dockerfile Dockerfiles
+ entry: hadolint/hadolint:v2.12.1-beta hadolint
+ types: [file]
+ files: \.dockerfile$
+
# Lint: YAML
- - repo: https://github.com/adrienverge/yamllint.git
- rev: v1.32.0
+ - repo: https://github.com/adrienverge/yamllint
+ rev: v1.35.1
hooks:
- id: yamllint
args: ["-d {extends: relaxed, rules: {line-length: disable}}", "-s"]
files: \.(yaml|yml)$
# Lint: Bash scripts
- - repo: https://github.com/openstack-dev/bashate.git
+ - repo: https://github.com/openstack/bashate
rev: 2.1.1
hooks:
- id: bashate
@@ -89,34 +111,34 @@ repos:
# Lint: Shell scripts
- repo: https://github.com/shellcheck-py/shellcheck-py
- rev: v0.9.0.5
+ rev: v0.10.0.1
hooks:
- id: shellcheck
args: ["-x"]
# Lint: Python
- repo: https://github.com/PyCQA/flake8
- rev: 6.0.0
+ rev: 7.1.1
hooks:
- id: flake8
# Lint: Markdown
- repo: https://github.com/igorshubovych/markdownlint-cli
- rev: v0.34.0
+ rev: v0.41.0
hooks:
- id: markdownlint
args: ["--fix"]
# Strip output from Jupyter notebooks
- repo: https://github.com/kynan/nbstripout
- rev: 0.6.1
+ rev: 0.7.1
hooks:
- id: nbstripout
# nbQA provides tools from the Python ecosystem like
# pyupgrade, isort, black, and flake8, adjusted for notebooks.
- repo: https://github.com/nbQA-dev/nbQA
- rev: 1.7.0
+ rev: 1.8.7
hooks:
- id: nbqa-pyupgrade
args: [--py39-plus]
@@ -126,13 +148,13 @@ repos:
- id: nbqa-flake8
# Run black on python code blocks in documentation files.
- - repo: https://github.com/asottile/blacken-docs
- rev: 1.13.0
+ - repo: https://github.com/adamchainz/blacken-docs
+ rev: 1.18.0
hooks:
- id: blacken-docs
# --skip-errors is added to allow us to have python syntax highlighting even if
- # the python code blocks includes jupyter specific additions such as % or !
- # See https://github.com/asottile/blacken-docs/issues/127 for an upstream
+ # the python code blocks include jupyter-specific additions such as % or !
+ # See https://github.com/adamchainz/blacken-docs/issues/127 for an upstream
# feature request about this.
args: [--target-version=py39, --skip-errors]
diff --git a/.readthedocs.yaml b/.readthedocs.yaml
index df3d6c324b..a1a47a1313 100644
--- a/.readthedocs.yaml
+++ b/.readthedocs.yaml
@@ -1,30 +1,38 @@
----
-# .readthedocs.yaml
-# Read the Docs configuration file
+# Read the Docs configuration file for Sphinx projects
# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
# Required
version: 2
-# Set the version of Python and other tools you might need
+# Set the OS, Python version and other tools you might need
build:
os: ubuntu-22.04
tools:
- python: "3.11"
+ python: "3.12"
# You can also specify other tool versions:
- # nodejs: "19"
- # rust: "1.64"
- # golang: "1.19"
+ # nodejs: "20"
+ # rust: "1.70"
+ # golang: "1.20"
+ jobs:
+ post_checkout:
+ - git fetch --unshallow || true
-# Build documentation in the docs/ directory with Sphinx
+# Build documentation in the "docs/" directory with Sphinx
sphinx:
configuration: docs/conf.py
+ # You can configure Sphinx to use a different builder, for instance use the dirhtml builder for simpler URLs
+ # builder: "dirhtml"
+ # Fail on all warnings to avoid broken references
+ # fail_on_warning: true
-# If using Sphinx, optionally build your docs in additional formats such as PDF
+# Optionally build your docs in additional formats such as PDF and ePub
# formats:
-# - pdf
+# - pdf
+# - epub
-# Optionally declare the Python requirements required to build your docs
+# Optional but recommended, declare the Python requirements required
+# to build your documentation
+# See https://docs.readthedocs.io/en/stable/guides/reproducible-builds.html
python:
install:
- requirements: docs/requirements.txt
diff --git a/CHANGELOG.md b/CHANGELOG.md
new file mode 100644
index 0000000000..bbf2f7aeb5
--- /dev/null
+++ b/CHANGELOG.md
@@ -0,0 +1,9 @@
+# Changelog
+
+## 2024-10-09
+
+### Changed
+
+_This change might only breaking if you build your custom set of images._
+
+- **Breaking:** Rename: `ROOT_CONTAINER`->`ROOT_IMAGE`, `BASE_CONTAINER`->`BASE_IMAGE` ([#2154](https://github.com/jupyter/docker-stacks/pull/2154), [#2155](https://github.com/jupyter/docker-stacks/pull/2155))
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index 0685e10505..c14c69809d 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -5,6 +5,6 @@ for information about how to contribute
[features](https://jupyter-docker-stacks.readthedocs.io/en/latest/contributing/features.html),
[recipes](https://jupyter-docker-stacks.readthedocs.io/en/latest/contributing/recipes.html),
[tests](https://jupyter-docker-stacks.readthedocs.io/en/latest/contributing/tests.html),
-[community-maintained stacks](https://jupyter-docker-stacks.readthedocs.io/en/latest/contributing/stacks.html).
+and [community-maintained stacks](https://jupyter-docker-stacks.readthedocs.io/en/latest/contributing/stacks.html).
diff --git a/Makefile b/Makefile
index c1a33edb0d..9222d6777d 100644
--- a/Makefile
+++ b/Makefile
@@ -2,12 +2,14 @@
# Distributed under the terms of the Modified BSD License.
.PHONY: docs help test
-# Use bash for inline if-statements in arch_patch target
SHELL:=bash
+REGISTRY?=quay.io
OWNER?=jupyter
-# Need to list the images in build dependency order
-# All of the images
+# Enable BuildKit for Docker build
+export DOCKER_BUILDKIT:=1
+
+# All the images listed in the build dependency order
ALL_IMAGES:= \
docker-stacks-foundation \
base-notebook \
@@ -16,13 +18,11 @@ ALL_IMAGES:= \
julia-notebook \
scipy-notebook \
tensorflow-notebook \
+ pytorch-notebook \
datascience-notebook \
pyspark-notebook \
all-spark-notebook
-# Enable BuildKit for Docker build
-export DOCKER_BUILDKIT:=1
-
# https://marmelab.com/blog/2016/02/29/auto-documented-makefile.html
@@ -36,84 +36,78 @@ help:
build/%: DOCKER_BUILD_ARGS?=
+build/%: ROOT_IMAGE?=ubuntu:24.04
build/%: ## build the latest image for a stack using the system's architecture
- docker build $(DOCKER_BUILD_ARGS) --rm --force-rm --tag $(OWNER)/$(notdir $@):latest ./$(notdir $@) --build-arg OWNER=$(OWNER)
+ docker build $(DOCKER_BUILD_ARGS) --rm --force-rm --tag "$(REGISTRY)/$(OWNER)/$(notdir $@):latest" "./images/$(notdir $@)" --build-arg REGISTRY="$(REGISTRY)" --build-arg OWNER="$(OWNER)" --build-arg ROOT_IMAGE="$(ROOT_IMAGE)"
@echo -n "Built image size: "
- @docker images $(OWNER)/$(notdir $@):latest --format "{{.Size}}"
+ @docker images "$(REGISTRY)/$(OWNER)/$(notdir $@):latest" --format "{{.Size}}"
build-all: $(foreach I, $(ALL_IMAGES), build/$(I)) ## build all stacks
-check-outdated/%: ## check the outdated mamba/conda packages in a stack and produce a report (experimental)
- @TEST_IMAGE="$(OWNER)/$(notdir $@)" pytest tests/base-notebook/test_outdated.py
+
+check-outdated/%: ## check the outdated mamba/conda packages in a stack and produce a report
+ @TEST_IMAGE="$(REGISTRY)/$(OWNER)/$(notdir $@)" pytest tests/docker-stacks-foundation/test_outdated.py
check-outdated-all: $(foreach I, $(ALL_IMAGES), check-outdated/$(I)) ## check all the stacks for outdated packages
-cont-clean-all: cont-stop-all cont-rm-all ## clean all containers (stop + rm)
cont-stop-all: ## stop all containers
@echo "Stopping all containers ..."
-docker stop --time 0 $(shell docker ps --all --quiet) 2> /dev/null
cont-rm-all: ## remove all containers
@echo "Removing all containers ..."
-docker rm --force $(shell docker ps --all --quiet) 2> /dev/null
+cont-clean-all: cont-stop-all cont-rm-all ## clean all containers (stop + rm)
docs: ## build HTML documentation
sphinx-build -W --keep-going --color docs/ docs/_build/
-
linkcheck-docs: ## check broken links
sphinx-build -W --keep-going --color -b linkcheck docs/ docs/_build/
+hook/%: VARIANT?=default
hook/%: ## run post-build hooks for an image
- python3 -m tagging.tag_image --short-image-name "$(notdir $@)" --owner "$(OWNER)" && \
- python3 -m tagging.write_manifest --short-image-name "$(notdir $@)" --hist-line-dir /tmp/jupyter/hist_lines/ --manifest-dir /tmp/jupyter/manifests/ --owner "$(OWNER)"
+ python3 -m tagging.write_tags_file --short-image-name "$(notdir $@)" --tags-dir /tmp/jupyter/tags/ --registry "$(REGISTRY)" --owner "$(OWNER)" --variant "$(VARIANT)" && \
+ python3 -m tagging.write_manifest --short-image-name "$(notdir $@)" --hist-lines-dir /tmp/jupyter/hist_lines/ --manifests-dir /tmp/jupyter/manifests/ --registry "$(REGISTRY)" --owner "$(OWNER)" --variant "$(VARIANT)" && \
+ python3 -m tagging.apply_tags --short-image-name "$(notdir $@)" --tags-dir /tmp/jupyter/tags/ --platform "$(shell uname -m)" --variant "$(VARIANT)" --registry "$(REGISTRY)" --owner "$(OWNER)"
hook-all: $(foreach I, $(ALL_IMAGES), hook/$(I)) ## run post-build hooks for all images
-img-clean: img-rm-dang img-rm ## clean dangling and jupyter images
img-list: ## list jupyter images
@echo "Listing $(OWNER) images ..."
docker images "$(OWNER)/*"
-img-rm: ## remove jupyter images
- @echo "Removing $(OWNER) images ..."
- -docker rmi --force $(shell docker images --quiet "$(OWNER)/*") 2> /dev/null
+ docker images "*/$(OWNER)/*"
img-rm-dang: ## remove dangling images (tagged None)
@echo "Removing dangling images ..."
-docker rmi --force $(shell docker images -f "dangling=true" --quiet) 2> /dev/null
-
-
-
-pre-commit-all: ## run pre-commit hook on all files
- @pre-commit run --all-files --hook-stage manual
-pre-commit-install: ## set up the git hook scripts
- @pre-commit --version
- @pre-commit install
+img-rm-jupyter: ## remove jupyter images
+ @echo "Removing $(OWNER) images ..."
+ -docker rmi --force $(shell docker images --quiet "$(OWNER)/*") 2> /dev/null
+ -docker rmi --force $(shell docker images --quiet "*/$(OWNER)/*") 2> /dev/null
+img-rm: img-rm-dang img-rm-jupyter ## remove dangling and jupyter images
pull/%: ## pull a jupyter image
- docker pull $(OWNER)/$(notdir $@)
+ docker pull "$(REGISTRY)/$(OWNER)/$(notdir $@)"
pull-all: $(foreach I, $(ALL_IMAGES), pull/$(I)) ## pull all images
-
-
push/%: ## push all tags for a jupyter image
- docker push --all-tags $(OWNER)/$(notdir $@)
+ docker push --all-tags "$(REGISTRY)/$(OWNER)/$(notdir $@)"
push-all: $(foreach I, $(ALL_IMAGES), push/$(I)) ## push all tagged images
run-shell/%: ## run a bash in interactive mode in a stack
- docker run -it --rm $(OWNER)/$(notdir $@) $(SHELL)
-
-run-sudo-shell/%: ## run a bash in interactive mode as root in a stack
- docker run -it --rm --user root $(OWNER)/$(notdir $@) $(SHELL)
+ docker run -it --rm "$(REGISTRY)/$(OWNER)/$(notdir $@)" $(SHELL)
+run-sudo-shell/%: ## run bash in interactive mode as root in a stack
+ docker run -it --rm --user root "$(REGISTRY)/$(OWNER)/$(notdir $@)" $(SHELL)
test/%: ## run tests against a stack
- python3 -m tests.run_tests --short-image-name "$(notdir $@)" --owner "$(OWNER)"
+ python3 -m tests.run_tests --short-image-name "$(notdir $@)" --registry "$(REGISTRY)" --owner "$(OWNER)"
test-all: $(foreach I, $(ALL_IMAGES), test/$(I)) ## test all stacks
diff --git a/README.md b/README.md
index 19bacd8128..6eb4981089 100644
--- a/README.md
+++ b/README.md
@@ -3,50 +3,53 @@
[![GitHub actions badge](https://github.com/helxplatform/jupyter-docker-stacks/actions/workflows/docker.yml/badge.svg)](https://github.com/helxplatform/jupyter-docker-stacks/actions/workflows/docker.yml "Docker images build status")
-Jupyter Docker Stacks are a set of ready-to-run [Docker images](https://hub.docker.com/u/jupyter) containing Jupyter applications and interactive computing tools.
+Jupyter Docker Stacks are a set of ready-to-run [Docker images](https://quay.io/organization/jupyter) containing Jupyter applications and interactive computing tools.
You can use a stack image to do any of the following (and more):
- Start a personal Jupyter Server with the JupyterLab frontend (default)
- Run JupyterLab for a team using JupyterHub
-- Start a personal Jupyter Notebook server in a local Docker container
+- Start a personal Jupyter Server with the Jupyter Notebook frontend in a local Docker container
- Write your own project Dockerfile
## Quick Start
-You can try a [relatively recent build of the jupyter/base-notebook image on mybinder.org](https://mybinder.org/v2/gh/jupyter/docker-stacks/main?urlpath=lab/tree/README.ipynb)
-by simply clicking the preceding link.
-Otherwise, the examples below may help you get started if you [have Docker installed](https://docs.docker.com/get-docker/),
-know [which Docker image](https://jupyter-docker-stacks.readthedocs.io/en/latest/using/selecting.html) you want to use
-and want to launch a single Jupyter Server in a container.
+You can [try a relatively recent build of the quay.io/jupyter/base-notebook image on mybinder.org](https://mybinder.org/v2/gh/jupyter/docker-stacks/main?urlpath=lab/tree/README.ipynb).
+Otherwise, the examples below may help you get started if you [have Docker installed](https://docs.docker.com/get-started/get-docker/),
+know [which Docker image](https://jupyter-docker-stacks.readthedocs.io/en/latest/using/selecting.html) you want to use, and want to launch a single Jupyter Application in a container.
The [User Guide on ReadTheDocs](https://jupyter-docker-stacks.readthedocs.io/en/latest/) describes additional uses and features in detail.
-**Example 1:**
+```{note}
+Since `2023-10-20` our images are only pushed to `Quay.io` registry.
+Older images are available on Docker Hub, but they will no longer be updated.
+```
+
+### Example 1
-This command pulls the `jupyter/scipy-notebook` image tagged `2023-06-01` from Docker Hub if it is not already present on the local host.
-It then starts a container running a Jupyter Server and exposes the container's internal port `8888` to port `10000` of the host machine:
+This command pulls the `jupyter/scipy-notebook` image tagged `2024-10-07` from Quay.io if it is not already present on the local host.
+It then starts a container running a Jupyter Server with the JupyterLab frontend and exposes the container's internal port `8888` to port `10000` of the host machine:
```bash
-docker run -p 10000:8888 jupyter/scipy-notebook:2023-06-01
+docker run -p 10000:8888 quay.io/jupyter/scipy-notebook:2024-10-07
```
-You can modify the port on which the container's port is exposed by [changing the value of the `-p` option](https://docs.docker.com/engine/reference/run/#expose-incoming-ports) to `-p 8888:8888`.
+You can modify the port on which the container's port is exposed by [changing the value of the `-p` option](https://docs.docker.com/engine/containers/run/#exposed-ports) to `-p 8888:8888`.
Visiting `http://:10000/?token=` in a browser loads JupyterLab,
where:
-- `hostname` is the name of the computer running Docker
-- `token` is the secret token printed in the console.
+- The `hostname` is the name of the computer running Docker
+- The `token` is the secret token printed in the console.
-The container remains intact for restart after the Jupyter Server exits.
+The container remains intact for restart after the Server exits.
-**Example 2:**
+### Example 2
-This command pulls the `jupyter/datascience-notebook` image tagged `2023-06-01` from Docker Hub if it is not already present on the local host.
-It then starts an _ephemeral_ container running a Jupyter Server and exposes the server on host port 10000.
+This command pulls the `jupyter/datascience-notebook` image tagged `2024-10-07` from Quay.io if it is not already present on the local host.
+It then starts an _ephemeral_ container running a Jupyter Server with the JupyterLab frontend and exposes the server on host port 10000.
```bash
-docker run -it --rm -p 10000:8888 -v "${PWD}":/home/jovyan/work jupyter/datascience-notebook:2023-06-01
+docker run -it --rm -p 10000:8888 -v "${PWD}":/home/jovyan/work quay.io/jupyter/datascience-notebook:2024-10-07
```
The use of the `-v` flag in the command mounts the current working directory on the host (`${PWD}` in the example command) as `/home/jovyan/work` in the container.
@@ -54,72 +57,53 @@ The server logs appear in the terminal.
Visiting `http://:10000/?token=` in a browser loads JupyterLab.
-Due to the usage of [the flag `--rm`](https://docs.docker.com/engine/reference/run/#clean-up---rm) Docker automatically cleans up the container and removes the file
-system when the container exits, but any changes made to the `~/work` directory and its files in the container will remain intact on the host.
-[The `-it` flag](https://docs.docker.com/engine/reference/commandline/run/#name) allocates pseudo-TTY.
-
-## Contributing
-
-Please see the [Contributor Guide on ReadTheDocs](https://jupyter-docker-stacks.readthedocs.io/en/latest/)
-for information about how to contribute recipes, features, tests, and community maintained stacks.
-
-## Maintainer Help Wanted
+Due to the usage of [the `--rm` flag](https://docs.docker.com/reference/cli/docker/container/run/#rm)
+Docker automatically cleans up the container and removes the file system when the container exits,
+but any changes made to the `~/work` directory and its files in the container will remain intact on the host.
+[The `-i` flag](https://docs.docker.com/reference/cli/docker/container/run/#interactive) keeps the container's `STDIN` open, and lets you send input to the container through standard input.
+[The `-t` flag](https://docs.docker.com/reference/cli/docker/container/run/#tty) attaches a pseudo-TTY to the container.
-We value all positive contributions to the Docker stacks project,
-from [bug reports](https://jupyter-docker-stacks.readthedocs.io/en/latest/contributing/issues.html)
-to [pull requests](https://jupyter-docker-stacks.readthedocs.io/en/latest/contributing/features.html#submitting-a-pull-request)
-to help with answering questions.
-We'd also like to invite members of the community to help with two maintainer activities:
+```{note}
+By default, [jupyter's root_dir](https://jupyter-server.readthedocs.io/en/latest/other/full-config.html) is `/home/jovyan`.
+So, new notebooks will be saved there, unless you change the directory in the file browser.
-- **Issue triaging**: Reading and providing a first response to issues, labeling issues appropriately,
- redirecting cross-project questions to Jupyter Discourse
-- **Pull request reviews**: Reading proposed documentation and code changes, working with the submitter
- to improve the contribution, deciding if the contribution should take another form (e.g., a recipe
- instead of a permanent change to the images)
-
-Anyone in the community can jump in and help with these activities anytime.
-We will happily grant additional permissions (e.g., the ability to merge PRs) to anyone who shows an ongoing interest in working on the project.
+To change the default directory, you must specify `ServerApp.root_dir` by adding this line to the previous command: `start-notebook.py --ServerApp.root_dir=/home/jovyan/work`.
+```
-## Jupyter Notebook Deprecation Notice
+## Choosing Jupyter frontend
-Following [Jupyter Notebook notice](https://github.com/jupyter/notebook#notice), JupyterLab is now the default for all the Jupyter Docker stack images.
+JupyterLab is the default for all the Jupyter Docker Stacks images.
It is still possible to switch back to Jupyter Notebook (or to launch a different startup command).
You can achieve this by passing the environment variable `DOCKER_STACKS_JUPYTER_CMD=notebook` (or any other valid `jupyter` subcommand) at container startup;
more information is available in the [documentation](https://jupyter-docker-stacks.readthedocs.io/en/latest/using/common.html#alternative-commands).
-According to the Jupyter Notebook project status and its compatibility with JupyterLab,
-these Docker images may remove the classic Jupyter Notebook interface altogether in favor of another _classic-like_ UI built atop JupyterLab.
-
-This change is tracked in the issue [#1217](https://github.com/jupyter/docker-stacks/issues/1217); please check its content for more information.
-
-## Alternatives
-
-- [jupyter/repo2docker](https://github.com/jupyterhub/repo2docker) - Turn git repositories into
- Jupyter-enabled Docker Images
-- [openshift/source-to-image](https://github.com/openshift/source-to-image) - A tool for
- building artifacts from source and injecting them into docker images
-- [jupyter-on-openshift/jupyter-notebooks](https://github.com/jupyter-on-openshift/jupyter-notebooks) -
- OpenShift compatible S2I builder for basic notebook images
-
## Resources
- [Documentation on ReadTheDocs](https://jupyter-docker-stacks.readthedocs.io/en/latest/)
- [Issue Tracker on GitHub](https://github.com/jupyter/docker-stacks/issues)
- [Jupyter Discourse Forum](https://discourse.jupyter.org/)
- [Jupyter Website](https://jupyter.org)
-- [Images on DockerHub](https://hub.docker.com/u/jupyter)
+- [Images on Quay.io](https://quay.io/organization/jupyter)
+
+## Acknowledgments
+
+- Starting from `2022-07-05`, `aarch64` self-hosted runners were sponsored by [`@mathbunnyru`](https://github.com/mathbunnyru/).
+ Please, consider [sponsoring his work](https://github.com/sponsors/mathbunnyru) on GitHub
+- Starting from `2023-10-31`, `aarch64` self-hosted runners are sponsored by an amazing [`2i2c non-profit organization`](https://2i2c.org)
## CPU Architectures
- We publish containers for both `x86_64` and `aarch64` platforms
-- Single-platform images have either `aarch64-` or `x86_64-` tag prefixes, for example, `jupyter/base-notebook:aarch64-python-3.10.5`
+- Single-platform images have either `aarch64-` or `x86_64-` tag prefixes, for example, `quay.io/jupyter/base-notebook:aarch64-python-3.11.6`
- Starting from `2022-09-21`, we create multi-platform images (except `tensorflow-notebook`)
-- Starting from `2023-06-01`, we create multi-platform `tensorflow-notebook` image as well
+- Starting from `2023-06-01`, we create a multi-platform `tensorflow-notebook` image as well
+- Starting from `2024-02-24`, we create CUDA enabled variants of `pytorch-notebook` image for `x86_64` platform
+- Starting from `2024-03-26`, we create CUDA enabled variant of `tensorflow-notebook` image for `x86_64` platform
## Using old images
This project only builds one set of images at a time.
-If you want to use older `Ubuntu` and/or `python` version, you can use following images:
+If you want to use the older `Ubuntu` and/or `Python` version, you can use the following images:
| Build Date | Ubuntu | Python | Tag |
| ------------ | ------ | ------ | -------------- |
@@ -131,4 +115,22 @@ If you want to use older `Ubuntu` and/or `python` version, you can use following
| 2022-10-09 | 22.04 | 3.8 | `7285848c0a11` |
| 2022-10-09 | 22.04 | 3.9 | `ed2908bbb62e` |
| 2023-05-30 | 22.04 | 3.10 | `4d70cf8da953` |
-| weekly build | 22.04 | 3.11 | `latest` |
+| 2024-08-26 | 22.04 | 3.11 | `00987883e58d` |
+| weekly build | 24.04 | 3.11 | `latest` |
+
+## Contributing
+
+Please see the [Contributor Guide on ReadTheDocs](https://jupyter-docker-stacks.readthedocs.io/en/latest/)
+for information about how to contribute recipes, features, tests, and community-maintained stacks.
+
+## Alternatives
+
+- [rocker/binder](https://rocker-project.org/images/versioned/binder.html) -
+ From the R focused [rocker-project](https://rocker-project.org),
+ lets you run both RStudio and Jupyter either standalone or in a JupyterHub
+- [jupyter/repo2docker](https://github.com/jupyterhub/repo2docker) -
+ Turn git repositories into Jupyter-enabled Docker Images
+- [openshift/source-to-image](https://github.com/openshift/source-to-image) -
+ A tool for building artifacts from source code and injecting them into docker images
+- [jupyter-on-openshift/jupyter-notebooks](https://github.com/jupyter-on-openshift/jupyter-notebooks) -
+ OpenShift compatible S2I builder for basic notebook images
diff --git a/aarch64-runner/setup.sh b/aarch64-runner/setup.sh
index 96fb247abb..fb6e05b176 100755
--- a/aarch64-runner/setup.sh
+++ b/aarch64-runner/setup.sh
@@ -3,7 +3,7 @@ set -ex
GITHUB_RUNNER_USER="runner-user"
-if [ "$EUID" -ne 0 ]; then
+if [ "${EUID}" -ne 0 ]; then
echo "Please run as root"
exit 1
fi
@@ -14,14 +14,26 @@ apt-get upgrade --yes
echo "Setting up runner-user, who will run GitHub Actions runner"
adduser --disabled-password --gecos "" ${GITHUB_RUNNER_USER}
mkdir /home/${GITHUB_RUNNER_USER}/.ssh/
+set +e
cp "/home/${SUDO_USER}/.ssh/authorized_keys" "/home/${GITHUB_RUNNER_USER}/.ssh/authorized_keys"
-chown ${GITHUB_RUNNER_USER}:${GITHUB_RUNNER_USER} /home/${GITHUB_RUNNER_USER}/.ssh/authorized_keys
+set -e
+chown --recursive ${GITHUB_RUNNER_USER}:${GITHUB_RUNNER_USER} /home/${GITHUB_RUNNER_USER}/.ssh
echo "Setting up python3"
apt-get install --yes --no-install-recommends python3
curl -sS https://bootstrap.pypa.io/get-pip.py | python3
echo "Setting up docker"
-apt-get install --yes --no-install-recommends docker.io
+apt-get remove --yes docker.io docker-doc docker-compose podman-docker containerd runc
+apt-get update --yes
+apt-get install --yes ca-certificates curl gnupg
+install -m 0755 -d /etc/apt/keyrings
+curl -fsSL https://download.docker.com/linux/ubuntu/gpg | gpg --dearmor -o /etc/apt/keyrings/docker.gpg
+chmod a+r /etc/apt/keyrings/docker.gpg
+echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | \
+ tee /etc/apt/sources.list.d/docker.list > /dev/null
+apt-get update --yes
+apt-get install --yes docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
+
usermod -aG docker ${GITHUB_RUNNER_USER}
chmod 666 /var/run/docker.sock
diff --git a/base-notebook/docker_healthcheck.py b/base-notebook/docker_healthcheck.py
deleted file mode 100755
index 7c35a6b115..0000000000
--- a/base-notebook/docker_healthcheck.py
+++ /dev/null
@@ -1,21 +0,0 @@
-#!/usr/bin/env python3
-# Copyright (c) Jupyter Development Team.
-# Distributed under the terms of the Modified BSD License.
-import json
-import os
-from pathlib import Path
-
-import requests
-
-# A number of operations below deliberately don't check for possible errors
-# As this is a healthcheck, it should succeed or raise an exception on error
-
-runtime_dir = Path("/home/") / os.environ["NB_USER"] / ".local/share/jupyter/runtime/"
-json_file = next(runtime_dir.glob("*server-*.json"))
-
-url = json.loads(json_file.read_bytes())["url"]
-url = url + "api"
-
-r = requests.get(url, verify=False) # request without SSL verification
-r.raise_for_status()
-print(r.content)
diff --git a/base-notebook/start-notebook.sh b/base-notebook/start-notebook.sh
deleted file mode 100755
index 4f673d22a5..0000000000
--- a/base-notebook/start-notebook.sh
+++ /dev/null
@@ -1,22 +0,0 @@
-#!/bin/bash
-# Copyright (c) Jupyter Development Team.
-# Distributed under the terms of the Modified BSD License.
-
-set -e
-
-# The Jupyter command to launch
-# JupyterLab by default
-DOCKER_STACKS_JUPYTER_CMD="${DOCKER_STACKS_JUPYTER_CMD:=lab}"
-
-if [[ -n "${JUPYTERHUB_API_TOKEN}" ]]; then
- echo "WARNING: using start-singleuser.sh instead of start-notebook.sh to start a server associated with JupyterHub."
- exec /usr/local/bin/start-singleuser.sh "$@"
-fi
-
-wrapper=""
-if [[ "${RESTARTABLE}" == "yes" ]]; then
- wrapper="run-one-constantly"
-fi
-
-# shellcheck disable=SC1091,SC2086
-exec /usr/local/bin/start.sh ${wrapper} jupyter ${DOCKER_STACKS_JUPYTER_CMD} ${NOTEBOOK_ARGS} "$@"
diff --git a/base-notebook/start-singleuser.sh b/base-notebook/start-singleuser.sh
deleted file mode 100755
index a2166e2c6d..0000000000
--- a/base-notebook/start-singleuser.sh
+++ /dev/null
@@ -1,13 +0,0 @@
-#!/bin/bash
-# Copyright (c) Jupyter Development Team.
-# Distributed under the terms of the Modified BSD License.
-
-set -e
-
-# set default ip to 0.0.0.0
-if [[ "${NOTEBOOK_ARGS} $*" != *"--ip="* ]]; then
- NOTEBOOK_ARGS="--ip=0.0.0.0 ${NOTEBOOK_ARGS}"
-fi
-
-# shellcheck disable=SC1091,SC2086
-. /usr/local/bin/start.sh jupyterhub-singleuser ${NOTEBOOK_ARGS} "$@"
diff --git a/binder/Dockerfile b/binder/Dockerfile
index 4986e2f400..28ccdd9e62 100644
--- a/binder/Dockerfile
+++ b/binder/Dockerfile
@@ -1,10 +1,11 @@
# Copyright (c) Jupyter Development Team.
# Distributed under the terms of the Modified BSD License.
-# https://hub.docker.com/r/jupyter/base-notebook/tags
+# https://quay.io/repository/jupyter/base-notebook?tab=tags
+ARG REGISTRY=quay.io
ARG OWNER=jupyter
-ARG BASE_CONTAINER=$OWNER/base-notebook:2023-06-01
-FROM $BASE_CONTAINER
+ARG BASE_IMAGE=$REGISTRY/$OWNER/base-notebook:2024-10-07
+FROM $BASE_IMAGE
LABEL maintainer="Jupyter Project "
@@ -12,6 +13,8 @@ LABEL maintainer="Jupyter Project "
# Fix: https://github.com/koalaman/shellcheck/wiki/SC3014
SHELL ["/bin/bash", "-o", "pipefail", "-c"]
-ENV TAG="2023-06-01"
+ENV TAG="2024-10-07"
COPY --chown=${NB_UID}:${NB_GID} binder/README.ipynb "${HOME}"/README.ipynb
+
+RUN jupyter labextension disable "@jupyterlab/apputils-extension:announcements"
diff --git a/binder/README.ipynb b/binder/README.ipynb
index 4116a188ff..66630c9e80 100644
--- a/binder/README.ipynb
+++ b/binder/README.ipynb
@@ -28,7 +28,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "The notebook server is running as the following user."
+ "The Server is running as the following user."
]
},
{
@@ -128,7 +128,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
- "version": "3.9.10"
+ "version": "3.11.4"
}
},
"nbformat": 4,
diff --git a/docs/_static/contributing/stacks/docker-org-create-token.png b/docs/_static/contributing/stacks/docker-org-create-token.png
new file mode 100644
index 0000000000..21105bc297
Binary files /dev/null and b/docs/_static/contributing/stacks/docker-org-create-token.png differ
diff --git a/docs/_static/contributing/stacks/docker-org-security.png b/docs/_static/contributing/stacks/docker-org-security.png
new file mode 100644
index 0000000000..831b3d54f2
Binary files /dev/null and b/docs/_static/contributing/stacks/docker-org-security.png differ
diff --git a/docs/_static/contributing/stacks/docker-repo-name.png b/docs/_static/contributing/stacks/docker-repo-name.png
new file mode 100644
index 0000000000..baf24d5dbb
Binary files /dev/null and b/docs/_static/contributing/stacks/docker-repo-name.png differ
diff --git a/docs/_static/contributing/stacks/docker-user-dropdown.png b/docs/_static/contributing/stacks/docker-user-dropdown.png
new file mode 100644
index 0000000000..1cc107e913
Binary files /dev/null and b/docs/_static/contributing/stacks/docker-user-dropdown.png differ
diff --git a/docs/_static/contributing/stacks/github-actions-tab.png b/docs/_static/contributing/stacks/github-actions-tab.png
new file mode 100644
index 0000000000..1c233629bd
Binary files /dev/null and b/docs/_static/contributing/stacks/github-actions-tab.png differ
diff --git a/docs/_static/contributing/stacks/github-actions-workflow.png b/docs/_static/contributing/stacks/github-actions-workflow.png
new file mode 100644
index 0000000000..2dc4a9cf89
Binary files /dev/null and b/docs/_static/contributing/stacks/github-actions-workflow.png differ
diff --git a/docs/_static/contributing/stacks/github-create-secrets.png b/docs/_static/contributing/stacks/github-create-secrets.png
new file mode 100644
index 0000000000..1336d0dacb
Binary files /dev/null and b/docs/_static/contributing/stacks/github-create-secrets.png differ
diff --git a/docs/_static/contributing/stacks/github-secret-token.png b/docs/_static/contributing/stacks/github-secret-token.png
new file mode 100644
index 0000000000..5901c26473
Binary files /dev/null and b/docs/_static/contributing/stacks/github-secret-token.png differ
diff --git a/docs/_static/docker-github-settings.png b/docs/_static/docker-github-settings.png
deleted file mode 100644
index 8a07fe0c1b..0000000000
Binary files a/docs/_static/docker-github-settings.png and /dev/null differ
diff --git a/docs/_static/docker-org-create-token.png b/docs/_static/docker-org-create-token.png
deleted file mode 100644
index ae0163310b..0000000000
Binary files a/docs/_static/docker-org-create-token.png and /dev/null differ
diff --git a/docs/_static/docker-org-security.png b/docs/_static/docker-org-security.png
deleted file mode 100644
index 2dfa2a7851..0000000000
Binary files a/docs/_static/docker-org-security.png and /dev/null differ
diff --git a/docs/_static/docker-org-select.png b/docs/_static/docker-org-select.png
deleted file mode 100644
index 3d2a2b8ca7..0000000000
Binary files a/docs/_static/docker-org-select.png and /dev/null differ
diff --git a/docs/_static/docker-repo-name.png b/docs/_static/docker-repo-name.png
deleted file mode 100644
index b4088f9d54..0000000000
Binary files a/docs/_static/docker-repo-name.png and /dev/null differ
diff --git a/docs/_static/github-actions-tab.png b/docs/_static/github-actions-tab.png
deleted file mode 100644
index 932e3c0468..0000000000
Binary files a/docs/_static/github-actions-tab.png and /dev/null differ
diff --git a/docs/_static/github-actions-workflow.png b/docs/_static/github-actions-workflow.png
deleted file mode 100644
index 1af7c512ea..0000000000
Binary files a/docs/_static/github-actions-workflow.png and /dev/null differ
diff --git a/docs/_static/github-create-secrets.png b/docs/_static/github-create-secrets.png
deleted file mode 100644
index 3d1e35a4e1..0000000000
Binary files a/docs/_static/github-create-secrets.png and /dev/null differ
diff --git a/docs/_static/github-secret-token.png b/docs/_static/github-secret-token.png
deleted file mode 100644
index 68cb72c40e..0000000000
Binary files a/docs/_static/github-secret-token.png and /dev/null differ
diff --git a/docs/_static/github-secrets-completed.png b/docs/_static/github-secrets-completed.png
deleted file mode 100644
index c26cfd4280..0000000000
Binary files a/docs/_static/github-secrets-completed.png and /dev/null differ
diff --git a/docs/_static/using/troubleshooting/vscode-jupyter-settings.png b/docs/_static/using/troubleshooting/vscode-jupyter-settings.png
index 177c385522..6c60389ae5 100644
Binary files a/docs/_static/using/troubleshooting/vscode-jupyter-settings.png and b/docs/_static/using/troubleshooting/vscode-jupyter-settings.png differ
diff --git a/docs/conf.py b/docs/conf.py
index dad3b3cbeb..7957c246c2 100644
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -7,7 +7,7 @@
# https://www.sphinx-doc.org/en/master/usage/configuration.html#project-information
project = "docker-stacks"
-copyright = "2023, Project Jupyter"
+copyright = "2024, Project Jupyter"
author = "Project Jupyter"
version = "latest"
@@ -29,7 +29,7 @@
html_theme = "alabaster"
html_static_path = ["_static"]
-# File above was generated using sphinx 6.2.1 with this command:
+# The file above was generated using sphinx 7.2.6 with this command:
# sphinx-quickstart --project "docker-stacks" --author "Project Jupyter" -v "latest" -r "latest" -l en --no-sep --no-makefile --no-batchfile
# These are custom options for this project
@@ -37,16 +37,17 @@
html_title = "Docker Stacks documentation"
html_logo = "_static/jupyter-logo.svg"
html_theme_options = {
+ "logo": {
+ "text": html_title,
+ },
+ "navigation_with_keys": False,
"path_to_docs": "docs",
- "repository_url": "https://github.com/jupyter/docker-stacks",
"repository_branch": "main",
+ "repository_url": "https://github.com/jupyter/docker-stacks",
+ "use_download_button": True,
"use_edit_page_button": True,
"use_issues_button": True,
"use_repository_button": True,
- "use_download_button": True,
- "logo": {
- "text": html_title,
- },
}
html_last_updated_fmt = "%Y-%m-%d"
@@ -57,7 +58,7 @@
}
pygments_style = "sphinx"
-# MyST configuration reference: https://myst-parser.readthedocs.io/en/latest/sphinx/reference.html
+# MyST configuration reference: https://myst-parser.readthedocs.io/en/latest/configuration.html
myst_heading_anchors = 3
linkcheck_ignore = [
@@ -65,6 +66,7 @@
r"https://github\.com/jupyter/docker-stacks/settings/actions/runners/new\?arch=arm64\&os=linux", # only works for users with permissions to change runners
r"http://127\.0\.0\.1:.*", # various examples
r"https://mybinder\.org/v2/gh/.*", # lots of 500 errors
+ r"https://packages\.ubuntu\.com/search\?keywords=openjdk", # frequent read timeouts
]
linkcheck_allowed_redirects = {
diff --git a/docs/contributing/features.md b/docs/contributing/features.md
index 08c1331f27..7da372de13 100644
--- a/docs/contributing/features.md
+++ b/docs/contributing/features.md
@@ -1,36 +1,43 @@
# New Features
-Thank you for contributing to the Jupyter Docker Stacks! We review pull requests for new features
-(e.g., new packages, new scripts, new flags) to balance the value of the images to the Jupyter
-community with the cost of maintaining the images over time.
+Thank you for contributing to the Jupyter Docker Stacks!
+We review pull requests for new features (e.g., new packages, new scripts, new flags)
+to balance the value of the images to the Jupyter community with the cost of maintaining the images over time.
## Suggesting a New Feature
Please follow the process below to suggest a new feature for inclusion in one of the core stacks:
-1. Open a [GitHub feature request issue](https://github.com/jupyter/docker-stacks/issues/new?assignees=&labels=type%3AEnhancement&template=feature_request.md&title=)
+1. Open a [GitHub feature request issue](https://github.com/jupyter/docker-stacks/issues/new?assignees=&labels=type%3AEnhancement&projects=&template=feature_request.yml)
describing the feature you'd like to contribute.
-2. Discuss with the maintainers whether the addition makes sense in
- [one of the core stacks](../using/selecting.md#core-stacks), as a
- [recipe in the documentation](recipes.md), as a [community stack](stacks.md), or as something
- else entirely.
+2. Discuss with the maintainers whether the addition makes sense
+ in [one of the core stacks](../using/selecting.md#core-stacks),
+ as a [way to build a custom set of images](../using/custom-images.md),
+ as a [recipe in the documentation](recipes.md),
+ as a [community stack](stacks.md),
+ or as something else entirely.
## Selection Criteria
Roughly speaking, we evaluate new features based on the following criteria:
-- **Usefulness to Jupyter users**: Is the feature generally applicable across domains? Does it work
- with Jupyter Notebook, JupyterLab, JupyterHub, etc.?
-- **Fit with the image purpose**: Does the feature match the theme of the stack in which it will be
- added? Would it fit better in a new community stack?
-- **Complexity of build/runtime configuration**: How many lines of code does the feature require
- in one of the Dockerfiles or startup scripts? Does it require new scripts entirely? Do users need
- to adjust how they use the images?
-- **Impact on image metrics**: How many bytes does the feature and its dependencies add to the
- image(s)? How many minutes do they add to the build time?
-- **Ability to support the addition**: Can existing maintainers answer user questions and address
- future build issues? Are the contributors interested in helping with long-term maintenance? Can we
- write tests to ensure the feature continues to work over time?
+- **Usefulness to Jupyter users**:
+ Is the feature generally applicable across domains?
+ Does it work with JupyterLab, Jupyter Notebook, JupyterHub, etc.?
+- **Fit with the image purpose**:
+ Does the feature match the theme of the stack to which it will be added?
+ Would it fit better in a new community stack?
+- **The complexity of build/runtime configuration**:
+ How many lines of code does the feature require in one of the Dockerfiles or startup scripts?
+ Does it require new scripts entirely?
+ Do users need to adjust how they use the images?
+- **Impact on image metrics**:
+ How many bytes does the feature and its dependencies add to the image(s)?
+ How many minutes do they add to the build time?
+- **Ability to support the addition**:
+ Can existing maintainers answer user questions and address future build issues?
+ Are the contributors interested in helping with long-term maintenance?
+ Can we write tests to ensure the feature continues to work over the years?
## Submitting a Pull Request
@@ -43,7 +50,7 @@ If there's agreement that the feature belongs in one or more of the core stacks:
If you use `make`, call:
```bash
- make build/somestack-notebook
+ make build/
```
3. [Submit a pull request](https://github.com/PointCloudLibrary/pcl/wiki/A-step-by-step-guide-on-preparing-and-submitting-a-pull-request)(PR) with your changes.
diff --git a/docs/contributing/issues.md b/docs/contributing/issues.md
index 0f133c09f3..680d584b7d 100644
--- a/docs/contributing/issues.md
+++ b/docs/contributing/issues.md
@@ -9,7 +9,7 @@ Please review the following guidelines when reporting your problem.
- If you think your problem is unique to the Jupyter Docker Stacks images,
please search the [jupyter/docker-stacks issue tracker](https://github.com/jupyter/docker-stacks/issues)
to see if someone else has already reported the same problem.
- If not, please open a [GitHub bug report issue](https://github.com/jupyter/docker-stacks/issues/new?assignees=&labels=type%3ABug&template=bug_report.md&title=)
+ If not, please open a [GitHub bug report issue](https://github.com/jupyter/docker-stacks/issues/new?assignees=&labels=type%3ABug&projects=&template=bug_report.yml)
and provide all the information requested in the issue template.
Additionally, check the [Troubleshooting Common Problems](../using/troubleshooting.md) page in the documentation before submitting an issue.
- If the issue you're seeing is with one of the open-source libraries included in the Docker images and is reproducible outside the images,
diff --git a/docs/contributing/lint.md b/docs/contributing/lint.md
index 4fb83bf7e3..350f07dd27 100644
--- a/docs/contributing/lint.md
+++ b/docs/contributing/lint.md
@@ -2,7 +2,7 @@
To enforce some rules, **linters** are used in this project.
Linters can be run either during the **development phase** (by the developer) or the **integration phase** (by GitHub Actions).
-To integrate and enforce this process in the project lifecycle, we are using **git hooks** through [pre-commit][pre-commit].
+To integrate and enforce this process in the project lifecycle, we are using **git hooks** through [pre-commit](https://pre-commit.com/).
## Using pre-commit hooks
@@ -13,7 +13,7 @@ To achieve this, use the generic task to install all Python development dependen
```sh
# Install all development dependencies for the project
-pip install requirements-dev.txt
+pip install --upgrade -r requirements-dev.txt
# It can also be installed directly
pip install pre-commit
```
@@ -21,7 +21,7 @@ pip install pre-commit
Then the git hooks scripts configured for the project in `.pre-commit-config.yaml` need to be installed in the local git repository.
```sh
-make pre-commit-install
+pre-commit install
```
### Run
@@ -34,22 +34,29 @@ Hadolint pre-commit uses Docker to run, so `docker` should be running while runn
```
```sh
-make pre-commit-all
+pre-commit run --all-files --hook-stage manual
+```
+
+```{note}
+We're running `pre-commit` with `--hook-stage manual`, because `pre-commit` is run on modified files only, which doesn't work well with `mypy --follow-imports error`.
+More information can be found in [`.pre-commit-config.yaml` file](https://github.com/jupyter/docker-stacks/blob/main/.pre-commit-config.yaml)
```
## Image Lint
-To comply with [Docker best practices][dbp], we are using the [Hadolint][hadolint] tool to analyse each `Dockerfile`.
+To comply with [Docker best practices](https://docs.docker.com/build/building/best-practices/),
+we are using the [Hadolint](https://github.com/hadolint/hadolint) tool to analyze each `Dockerfile`.
### Ignoring Rules
-Sometimes it is necessary to ignore [some rules][rules].
+Sometimes it is necessary to ignore [some rules](https://github.com/hadolint/hadolint#rules).
The following rules are ignored by default for all images in the `.hadolint.yaml` file.
- [`DL3006`][dl3006]: We use a specific policy to manage image tags.
- - `base-notebook` `FROM` clause is fixed but based on an argument (`ARG`).
+ - The `docker-stacks-foundation` `FROM` clause is fixed but based on an argument (`ARG`).
- Building downstream images from (`FROM`) the latest is done on purpose.
- [`DL3008`][dl3008]: System packages are always updated (`apt-get`) to the latest version.
+- [`DL3013`][dl3013]: We always install the latest packages using `pip`
The preferred way to do it for other rules is to flag ignored ones in the `Dockerfile`.
@@ -64,9 +71,6 @@ FROM ubuntu
RUN cd /tmp && echo "hello!"
```
-[hadolint]: https://github.com/hadolint/hadolint
-[dbp]: https://docs.docker.com/develop/develop-images/dockerfile_best-practices
-[rules]: https://github.com/hadolint/hadolint#rules
[dl3006]: https://github.com/hadolint/hadolint/wiki/DL3006
[dl3008]: https://github.com/hadolint/hadolint/wiki/DL3008
-[pre-commit]: https://pre-commit.com/
+[dl3013]: https://github.com/hadolint/hadolint/wiki/DL3013
diff --git a/docs/contributing/packages.md b/docs/contributing/packages.md
index 45537bd354..d6b7706047 100644
--- a/docs/contributing/packages.md
+++ b/docs/contributing/packages.md
@@ -6,13 +6,13 @@ All this means that packages might have old versions.
Images are rebuilt weekly, so usually, packages receive updates quite frequently.
```{note}
-We pin major.minor version of python, so this will stay the same even after invoking the `mamba update` command.
+We pin major.minor version of Python, so this will stay the same even after invoking the `mamba update` command.
```
## Outdated packages
-To help to identify packages that can be updated, you can use the following helper tool.
-It will list all the updateable packages installed in the `Dockerfile` --
+To help identify packages that can be updated, you can use the following helper tool.
+It will list all the outdated packages installed in the `Dockerfile` --
dependencies are filtered to focus only on requested packages.
```bash
diff --git a/docs/contributing/recipes.md b/docs/contributing/recipes.md
index 313d755cc8..45afb60c50 100644
--- a/docs/contributing/recipes.md
+++ b/docs/contributing/recipes.md
@@ -1,10 +1,12 @@
# New Recipes
-We welcome contributions of [recipes](../using/recipes.md), short examples of using, configuring, or extending the Docker Stacks for inclusion in the documentation site.
+We welcome contributions of [recipes](../using/recipes.md), and short examples of using, configuring, or extending the Docker Stacks for inclusion in the documentation site.
Follow the process below to add a new recipe:
1. Open the `docs/using/recipes.md` source file.
-2. Add a second-level Markdown heading naming your recipe at the bottom of the file (e.g., `## Add the RISE extension`)
-3. Write the body of your recipe under the heading, including whatever command line, Dockerfile, links, etc. you need.
-4. [Submit a pull request](https://github.com/PointCloudLibrary/pcl/wiki/A-step-by-step-guide-on-preparing-and-submitting-a-pull-request) (PR) with your changes.
+2. Add a second-level Markdown heading naming your recipe at the bottom of the file (e.g., `## Slideshows with JupyterLab and RISE`)
+3. Write the body of your recipe under the heading, including whatever command line, links, etc. you need.
+4. If you have a Dockerfile, please put it in a `recipe_code` subdirectory.
+ This file will be built automatically by [contributed-recipes workflow](https://github.com/jupyter/docker-stacks/blob/main/.github/workflows/contributed-recipes.yml).
+5. [Submit a pull request](https://github.com/PointCloudLibrary/pcl/wiki/A-step-by-step-guide-on-preparing-and-submitting-a-pull-request) (PR) with your changes.
Maintainers will respond and work with you to address any formatting or content issues.
diff --git a/docs/contributing/stacks.md b/docs/contributing/stacks.md
index d93a6950f1..37b474e6f8 100644
--- a/docs/contributing/stacks.md
+++ b/docs/contributing/stacks.md
@@ -8,15 +8,15 @@ Following these steps will:
1. Set up a project on GitHub containing a Dockerfile based on any image we provide.
2. Configure GitHub Actions to build and test your image when users submit pull requests to your repository.
-3. Configure Docker Hub to build and host your images for others to use.
+3. Configure Quay.io to host your images for others to use.
4. Update the [list of community stacks](../using/selecting.md#community-stacks) in this documentation to include your image.
This approach mirrors how we build and share the core stack images.
-Feel free to follow it or pave your own path using alternative services and build tools.
+Feel free to follow it or pave your path using alternative services and build tools.
## Creating a Project
-First, install [cookiecutter](https://github.com/cookiecutter/cookiecutter) using _pip_ or _conda_:
+First, install [cookiecutter](https://github.com/cookiecutter/cookiecutter) using _pip_ or _mamba_:
```bash
pip install cookiecutter # or mamba install cookiecutter
@@ -35,17 +35,17 @@ This will serve as both the git repository name and the part of the Docker image
stack_name [my-jupyter-stack]:
```
-Enter the user or organization name under which this stack will reside on Docker Hub.
-You must have access to manage this Docker Hub organization to push images here and set up automated builds.
+Enter the user or organization name under which this stack will reside on Quay.io.
+You must have access to manage this Quay.io organization to push images here.
```text
stack_org [my-project]:
```
-Select an image from the jupyter/docker-stacks project that will serve as the base for your new image.
+Select an image from the `jupyter/docker-stacks` project that will serve as the base for your new image.
```text
-stack_base_image [jupyter/base-notebook]:
+stack_base_image [quay.io/jupyter/base-notebook]:
```
Enter a longer description of the stack for your README.
@@ -54,6 +54,7 @@ Enter a longer description of the stack for your README.
stack_description [my-jupyter-stack is a community-maintained Jupyter Docker Stack image]:
```
+Create a GitHub repository to store your project.
Initialize your project as a Git repository and push it to GitHub.
```bash
@@ -66,90 +67,88 @@ git remote add origin
git push -u origin main
```
-## Configuring GitHub actions
+## Exploring GitHub Actions
-The cookiecutter template comes with a `.github/workflows/docker.yml` file, which allows you to use GitHub actions to build your Docker image whenever you or someone else submits a pull request.
+1. By default, the newly `.github/workflows/docker.yaml` will trigger the CI pipeline whenever you push to your `main` branch
+ and when any Pull Requests are made to your repository.
+ For more details on this configuration, visit the [GitHub actions documentation on triggers](https://docs.github.com/en/actions/writing-workflows/choosing-when-your-workflow-runs/events-that-trigger-workflows).
-1. By default, the `.github/workflows/docker.yaml` file has the following triggers configuration:
+2. Go to your repository and click on the **Actions** tab.
+ From there, you can click on the workflows on the left-hand side of the screen.
- ```yaml
- on:
- pull_request:
- paths-ignore:
- - "*.md"
- push:
- branches:
- - main
- paths-ignore:
- - "*.md"
- ```
+ ![GitHub page for jupyter/docker-stacks with the Actions tab active and a rectangle around the "Build Docker Images" workflow in the UI](../_static/contributing/stacks/github-actions-tab.png)
- This will trigger the CI pipeline whenever you push to your `main` branch and when any Pull Requests are made to your repository.
- For more details on this configuration, visit the [GitHub actions documentation on triggers](https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows).
+ ```{note}
+ The first run is expected to fail because we haven't yet added Docker credentials to push the image
+ ```
-2. Commit your changes and push them to GitHub.
-3. Head back to your repository and click on the **Actions** tab.
- ![GitHub page for jupyter/docker-stacks with the Actions tab active and a rectangle around the "Build Docker Images" workflow in the UI](../_static/github-actions-tab.png)
- From there, you can click on the workflows on the left-hand side of the screen.
-4. In the next screen, you will see information about the workflow run and duration.
+3. In the next screen, you will see information about the workflow run and duration.
If you click the button with the workflow name again, you will see the logs for the workflow steps.
- ![GitHub Actions page showing the "Build Docker Images" workflow](../_static/github-actions-workflow.png)
+
+ ![GitHub Actions page showing the "Build Docker Images" workflow](../_static/contributing/stacks/github-actions-workflow.png)
## Configuring Docker Hub
+```{note}
+Jupyter Docker Stacks are hosted on Quay.io, but in this example, we show you how to host your image on Docker Hub.
+```
+
Now, configure Docker Hub to build your stack image and push it to the Docker Hub repository whenever
you merge a GitHub pull request to the main branch of your project.
1. Visit [https://hub.docker.com/](https://hub.docker.com/) and log in.
-2. Select the account or organization matching the one you entered when prompted with `stack_org` by the cookiecutter.
- ![DockerHub page zoomed into the user's settings and accounts menu.](../_static/docker-org-select.png)
-3. Scroll to the bottom of the page and click **Create repository**.
-4. Enter the name of the image matching the one you entered when prompted with `stack_name` by the cookiecutter.
- ![DockerHub - Create Repository page with the name field set to "My specialized jupyter stack"](../_static/docker-repo-name.png)
-5. Enter a description for your image.
-6. Click **GitHub** under the **Build Settings** and follow the prompts to connect your account if it is not already connected.
-7. Select the GitHub organization and repository containing your image definition from the dropdowns.
- ![Dockerhub - Create Repository page focusing on the "Select Repository" dropdown menu](../_static/docker-github-settings.png)
-8. Click the **Create and Build** button.
-9. Click on your avatar in the top-right corner and select Account settings.
- ![DockerHub page zoomed into the user's settings and accounts menu](../_static/docker-org-select.png)
-10. Click on **Security** and then click on the **New Access Token** button.
- ![DockerHub - Account page with the "Security" tab active and a rectangle highlighting the "New Access Token" button in the UI](../_static/docker-org-security.png)
-11. Enter a meaningful name for your token and click on **Create**
- ![DockerHub - New Access Token page with the name field set to "my-jupyter-docker-token"](../_static/docker-org-create-token.png)
-12. Copy the personal access token displayed on the next screen.
-
- ```{note}
- you will not be able to see it again after you close the pop-up window**.
- ```
-
-13. Head back to your GitHub repository and click on the **Settings tab**.
- ![GitHub page with the "Setting" tab active and a rectangle highlighting the "New repository secret" button in the UI](../_static/github-create-secrets.png)
-14. Click on the **Secrets** section and then on the **New repository secret** button in the top right corner (see image above).
-15. Create a **DOCKERHUB_TOKEN** secret and paste the Personal Access Token from DockerHub in the **value** field.
- ![GitHub - Actions/New secret page with the Name field set to "DOCKERHUB_TOKEN"](../_static/github-secret-token.png)
-16. Repeat the above step but creating a **DOCKERHUB_USERNAME** and replacing the _value_ field with your DockerHub username.
- Once you have completed these steps, your repository secrets section should look something like this:
- ![GitHub - Repository secrets page showing the existing "DOCKERHUB_TOKEN" and "DOCKERHUB_USERNAME" secrets](../_static/github-secrets-completed.png)
+2. Create a new repository - make sure to use the correct namespace (account or organization).
+ Enter the name of the image matching the one you entered when prompted with `stack_name` by the cookiecutter.
+
+ ![Docker Hub - 'Create repository' page with the name field set to "My specialized jupyter stack"](../_static/contributing/stacks/docker-repo-name.png)
+
+3. Enter a description for your image.
+4. Click on your avatar in the top-right corner and select Account Settings.
+
+ ![The Docker Hub page zoomed into the user's settings and accounts menu](../_static/contributing/stacks/docker-user-dropdown.png)
+
+5. Click on **Security** and then click on the **New Access Token** button.
+
+ ![Docker Hub - Account page with the "Security" tab active and a rectangle highlighting the "New Access Token" button in the UI](../_static/contributing/stacks/docker-org-security.png)
+
+6. Enter a meaningful name for your token and click on **Generate**
+
+ ![Docker Hub - New Access Token page with the name field set to "test-stack token"](../_static/contributing/stacks/docker-org-create-token.png)
+
+7. Copy the personal access token displayed on the next screen.
+
+ ```{note}
+ **You will not be able to see it again after you close the pop-up window**.
+ ```
+
+8. Head back to your GitHub repository and click on the **Settings tab**.
+9. Click on the **Secrets and variables->Actions** section and then on the **New repository secret** button in the top right corner.
+
+ ![GitHub page with the "Setting" tab active and a rectangle highlighting the "New repository secret" button in the UI](../_static/contributing/stacks/github-create-secrets.png)
+
+10. Create a **DOCKERHUB_TOKEN** secret and paste the Personal Access Token from Docker Hub in the **value** field.
+
+ ![GitHub - Actions/New secret page with the Name field set to "DOCKERHUB_TOKEN"](../_static/contributing/stacks/github-secret-token.png)
+
+11. Now you're ready to go and you can restart a failed workflow.
## Defining Your Image
-Make edits to the Dockerfile in your project to add third-party libraries and configure Jupyter
-applications.
-Refer to the Dockerfiles for the core stacks (e.g., [jupyter/datascience-notebook](https://github.com/jupyter/docker-stacks/blob/main/datascience-notebook/Dockerfile))
+Make edits to the Dockerfile in your project to add third-party libraries and configure Jupyter applications.
+Refer to the Dockerfiles for the core stacks (e.g., [jupyter/datascience-notebook](https://github.com/jupyter/docker-stacks/blob/main/images/datascience-notebook/Dockerfile))
to get a feel for what's possible and the best practices.
[Submit pull requests](https://github.com/PointCloudLibrary/pcl/wiki/A-step-by-step-guide-on-preparing-and-submitting-a-pull-request)
to your project repository on GitHub.
-Ensure your image builds correctly on GitHub actions before merging to the main branch.
-Refer to Docker Hub to build the main branch that you can `docker pull`.
+Ensure your image builds correctly on GitHub Actions before merging to the main branch.
+After merging to the main branch, your image will be built and pushed to the Docker Hub automatically.
## Sharing Your Image
Finally, if you'd like to add a link to your project to this documentation site, please do the following:
-1. Clone the [jupyter/docker-stacks](https://github.com/jupyter/docker-stacks) GitHub repository.
-2. Open the `docs/using/selecting.md` source file and locate the **Community Stacks** section.
-3. Add a table entry with a link to your project, a binder link and a short description of what your Docker image contains.
+1. Fork the [jupyter/docker-stacks](https://github.com/jupyter/docker-stacks) GitHub repository.
+2. Open the `docs/using/selecting.md` source file and locate the **Community Stacks** section in your fork.
+3. Add a table entry with a link to your project, a binder link, and a short description of what your Docker image contains.
4. [Submit a pull request](https://github.com/PointCloudLibrary/pcl/wiki/A-step-by-step-guide-on-preparing-and-submitting-a-pull-request)(PR) with your changes.
Maintainers will respond and work with you to address any formatting or content issues.
diff --git a/docs/contributing/tests.md b/docs/contributing/tests.md
index b27a3046b1..3d7888bf57 100644
--- a/docs/contributing/tests.md
+++ b/docs/contributing/tests.md
@@ -1,6 +1,6 @@
# Image Tests
-We greatly appreciate pull requests that extend the automated tests that vet the basic functionality of the Docker images.
+We greatly appreciate Pull Requests that extend the automated tests that vet the basic functionality of the Docker images.
## How the Tests Work
@@ -14,7 +14,7 @@ More info on `pytest` can be found [here](https://docs.pytest.org/en/latest/cont
The actual image-specific test files are located in folders like `tests//` (e.g., `tests/docker-stacks-foundation/`, `tests/minimal-notebook/`, etc.).
```{note}
-If your test is located in `tests//`, it will be run against `jupyter/` image and against all the [images inherited from this image](https://jupyter-docker-stacks.readthedocs.io/en/latest/using/selecting.html#image-relationships.
+If your test is located in `tests//`, it will be run against the `jupyter/` image and against all the [images inherited from this image](https://jupyter-docker-stacks.readthedocs.io/en/latest/using/selecting.html#image-relationships.
```
Many tests make use of global [pytest fixtures](https://docs.pytest.org/en/latest/reference/fixtures.html)
@@ -22,16 +22,16 @@ defined in the [conftest.py](https://github.com/jupyter/docker-stacks/blob/main/
## Unit tests
-You can add a unit test if you want to run a python script in one of our images.
-You should create a `tests//units/` directory, if it doesn't already exist and put your file there.
+You can add a unit test if you want to run a Python script in one of our images.
+You should create a `tests//units/` directory, if it doesn't already exist, and put your file there.
Files in this folder will be executed in the container when tests are run.
-You can see an [example for the TensorFlow package here](https://github.com/jupyter/docker-stacks/blob/HEAD/tests/tensorflow-notebook/units/unit_tensorflow.py).
+You can see an [TensorFlow package example here](https://github.com/jupyter/docker-stacks/blob/HEAD/tests/tensorflow-notebook/units/unit_tensorflow.py).
## Contributing New Tests
Please follow the process below to add new tests:
-1. Add your test code to one of the modules in `tests//` directory or create a new module.
+1. Add your test code to one of the modules in the `tests//` directory or create a new module.
2. Build one or more images you intend to test and run the tests locally.
If you use `make`, call:
diff --git a/docs/images/inherit.svg b/docs/images/inherit.svg
index 0f5467a403..e98db38019 100644
--- a/docs/images/inherit.svg
+++ b/docs/images/inherit.svg
@@ -1,6 +1,6 @@
-