Prefetching dependencies with Cachi2 #285

ccronca · 2025-01-27T15:00:53Z

This is an initial PR to get feedback on prefetching dependencies using Cachi2. Since Rapidast uses Konflux as the build system, this relies on Cachi2 support in Konflux

The konflux buildah-oci-ta.yaml v0.2 task, used to build the Rapidast image, mounts the prefetched dependencies into /cachi2. So, the Containerfile has been updated to retrieve dependencies from /cachi2 when prefetching is enabled.

To keep the Containerfile working for non-Konflux builds, I’ve added a build-time variable called PREFETCH. If PREFETCH=true, the build will try to use /cachi2 for dependencies. If not, it’ll just pull dependencies the same way it did before.

If this approach makes sense, I’ll add prefetching support for other dependencies

EDIT:
Dependencies covered by this PR:

Generic binaries: ZAP, Trivy, Kubecl, and Firefox
NPM dependencies

Dependencies not covered by this PR:

Pip dependencies: Prefetching these dependencies involves multiple steps, so I’d like to handle them in a separate PR
Zap addons: I'm not sure how to implement this yet, so it will be covered in a different PR if necessary
Rpm dependencies: Prefetching these dependencies involves multiple steps, so I’d like to handle them in a separate PR

sfowl · 2025-01-27T23:48:22Z

This is disappointing, especially since the docs make it seem like the feature should be available already:

task prefetch-dependencies has the status "Failed":

2025-01-27 15:12:26,470 ERROR UnsupportedFeature: Package manager(s) not yet supported: generic
Error: UnsupportedFeature: Package manager(s) not yet supported: generic
  But the good news is that we're already working on it!

The general approach of using a PREFETCH arg seems to fine to me though, FWIW.

ccronca · 2025-01-28T09:19:37Z

This is disappointing, especially since the docs make it seem like the feature should be available already:
task prefetch-dependencies has the status "Failed":
2025-01-27 15:12:26,470 ERROR UnsupportedFeature: Package manager(s) not yet supported: generic
Error: UnsupportedFeature: Package manager(s) not yet supported: generic
  But the good news is that we're already working on it!
The general approach of using a PREFETCH arg seems to fine to me though, FWIW.

Looks like this is fixed now! The main issue was that the task-prefetch-dependencies-oci-ta in the original bundle (task-prefetch-dependencies-oci-ta:0.1@sha256:3c11f5de6a0281bf93857f0c85bbbdfeda4cc118337da273fef0c138bda5eebb) was running cachi2 version 0.13, which doesn’t support the generic package manager. I updated it to the latest version of the image, and it’s working as expected now.

@sfowl since you're on board with this approach, I’ll go ahead and expand it to include prefetching support for other dependencies

…y and ff

ccronca · 2025-01-28T15:41:56Z

containerize/Containerfile


 ## redocly (https://github.com/Redocly/redocly-cli)
-RUN mkdir -p /tmp/redocly/node_modules && npm install --prefix /tmp/redocly @redocly/[email protected]
+COPY package.json package-lock.json /tmp/redocly/


Cachi2 relies on package.json and package-lock.json files to manage the prefetching of npm dependencies. So, instead of explicitly installing the redocly/cli package, it has been added to the package.json for automatic handling.

ccronca · 2025-01-28T15:47:09Z

containerize/Containerfile

-RUN mkdir -p /tmp/redocly/node_modules && npm install --prefix /tmp/redocly @redocly/[email protected]
+COPY package.json package-lock.json /tmp/redocly/
+RUN mkdir -p /tmp/redocly/node_modules && if [ "$PREFETCH" == "true" ]; then \
+    npm install --offline --prefix /tmp/redocly; \


The offline argument has been added to block any external access when installing npm dependencies, making sure everything comes from Cachi2. Essentially, Konflux will update the remote dependencies in the package-lock.json to point to the local file system using Cachi2’s inject-files feature

containerize/Containerfile

ccronca · 2025-01-28T16:03:18Z

containerize/Containerfile

+
+# These versions should be consistent with the listed in the artifacts.lock.yaml file
+ARG ZAP_VERSION=2.15.0
+ARG FF_VERSION=128.6.0esr


Firefox is now locked to a specific version. Previously, it was downloading the latest one, but in order to validate the checksum with Cachi2, we need to specify a fixed version. We could continue fetching the latest version when binaries aren’t pre-fetched, but I wanted to maintain consistency across both approaches.

Sounds good to me

.tekton/rapidast-pull-request.yaml

containerize/Containerfile

ccronca · 2025-01-28T16:18:41Z

package-lock.json

This file, along with the package.json, was generated by simply running npm install @redocly/[email protected]. This file includes a modification compared to the one generated directly, I've added the version key, as Cachi2 requires it. Without this key, it will fail to download dependencies:

2025-01-28 16:17:34,068 ERROR KeyError: 'version'

ccronca · 2025-01-30T11:15:32Z

This PR is ready for review. Other dependencies will be handled in separate PRs.

containerize/Containerfile

typo

package.json

jeremychoi

lgtm

ccronca added 2 commits January 27, 2025 15:43

Prefetch zap dependency

603e02c

Fix linting error

3ca7371

ccronca added 5 commits January 28, 2025 07:26

Test Trigger Build

c945297

Test prefetch-input using array as argument

1a6e03e

Update task-prefetch-dependencies-oci-ta

b4e139e

Update task-prefetch-dependencies-oci-ta to latest available

0d48796

Update task-prefetch-dependencies-oci-ta to latest available

958d80c

ccronca added 2 commits January 28, 2025 16:16

Add configuration for prefetching dependencies for npm, kubeclt, triv…

7755b1d

…y and ff

Fix linting error

57b1009

ccronca commented Jan 28, 2025

View reviewed changes

ccronca marked this pull request as ready for review January 28, 2025 16:27

ccronca requested a review from a team as a code owner January 28, 2025 16:27

jeremychoi reviewed Feb 3, 2025

View reviewed changes

containerize/Containerfile Outdated Show resolved Hide resolved

Update Containerfile

5c519c3

typo

jeremychoi reviewed Feb 3, 2025

View reviewed changes

package.json Outdated Show resolved Hide resolved

ccronca added 3 commits February 3, 2025 08:54

Bump trivy version to 0.59.0

14a8f23

Merge branch 'development' into prefetch-generic

177fa55

Fix the redoclys version to 1.9.1

12d7e78

ccronca self-assigned this Feb 3, 2025

jeremychoi approved these changes Feb 4, 2025

View reviewed changes

ccronca merged commit a4984c5 into development Feb 4, 2025
5 checks passed

ccronca mentioned this pull request Feb 5, 2025

Prefetching pip dependencies with Cachi2 #295

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Prefetching dependencies with Cachi2 #285

Prefetching dependencies with Cachi2 #285

ccronca commented Jan 27, 2025 •

edited

Loading

sfowl commented Jan 27, 2025

ccronca commented Jan 28, 2025 •

edited

Loading

ccronca Jan 28, 2025

ccronca Jan 28, 2025 •

edited

Loading

ccronca Jan 28, 2025

jeremychoi Feb 3, 2025

ccronca Jan 28, 2025

ccronca commented Jan 30, 2025

jeremychoi left a comment

Prefetching dependencies with Cachi2 #285

Prefetching dependencies with Cachi2 #285

Conversation

ccronca commented Jan 27, 2025 • edited Loading

sfowl commented Jan 27, 2025

ccronca commented Jan 28, 2025 • edited Loading

ccronca Jan 28, 2025

Choose a reason for hiding this comment

ccronca Jan 28, 2025 • edited Loading

Choose a reason for hiding this comment

ccronca Jan 28, 2025

Choose a reason for hiding this comment

jeremychoi Feb 3, 2025

Choose a reason for hiding this comment

ccronca Jan 28, 2025

Choose a reason for hiding this comment

ccronca commented Jan 30, 2025

jeremychoi left a comment

Choose a reason for hiding this comment

ccronca commented Jan 27, 2025 •

edited

Loading

ccronca commented Jan 28, 2025 •

edited

Loading

ccronca Jan 28, 2025 •

edited

Loading