Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Speed up Docker image builds using multi-stage parallel pipelines #987

Merged
merged 7 commits into from
Nov 25, 2024

Conversation

debanjum
Copy link
Member

@debanjum debanjum commented Nov 24, 2024

Objective

Improve build speed and size of khoj docker images

Changes

Improve docker image build speeds

Result: ~40% drop to 21mins from 35+mins for full build. ~5mins for rebuilds (as all layers can loaded from cache)

  • Decouple web app and server build steps
  • Build the web app and server in parallel
  • Cache docker layers for reuse across dockerize github workflow runs
    • Split Docker build layers for improved cacheability (e.g separate yarn install and yarn build steps)

Reduce size of khoj docker images

Result: ~34% drop to 4.78Gb from 7.25Gb for default build

  • Use an up-to-date .dockerignore to exclude unnecessary directories
  • Do not installing cuda python packages for cpu builds

Improve web app builds

  • Use consistent mechanism to get fonts for web app
  • Make tailwind extensions production instead of dev dependencies
  • Make next.js create production builds for the web app (via NODE_ENV=production env var)

@debanjum debanjum force-pushed the speed-up-khoj-docker-builds branch 7 times, most recently from 420b0b1 to 5a41ded Compare November 24, 2024 18:50
Decouple web app, server builds in parallel to speed up Docker
builds
@debanjum debanjum force-pushed the speed-up-khoj-docker-builds branch from 5a41ded to 0f0f5f5 Compare November 24, 2024 21:12
@debanjum debanjum force-pushed the speed-up-khoj-docker-builds branch 5 times, most recently from 1f47aa0 to 16463f3 Compare November 25, 2024 04:34
@debanjum debanjum force-pushed the speed-up-khoj-docker-builds branch from 16463f3 to 97de3b2 Compare November 25, 2024 04:41
@debanjum debanjum force-pushed the speed-up-khoj-docker-builds branch from 97de3b2 to 4a5646c Compare November 25, 2024 05:11
@debanjum debanjum added the maintain Maintain code, documentation or project label Nov 25, 2024
Copy link
Member

@sabaimran sabaimran left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WOOHOO!!! So excited to have faster build times.

@debanjum debanjum merged commit 5723a37 into master Nov 25, 2024
8 checks passed
@sabaimran sabaimran deleted the speed-up-khoj-docker-builds branch November 25, 2024 05:50
debanjum added a commit that referenced this pull request Nov 25, 2024
- Building arm64 image on an ubuntu arm64 runner reduces `yarn build'
  step time by 75% from 12mins to 3mins.
  - This is because no QEMU emulation for arm64 on x86 is required now
- Parallelizing x64 and arm64 platform builds halves build time on top
  - Revert to use standard ubuntu-latest runner as large x64 runner
    doesn't give much more speed improvements

This results an effective additional 50%-66% reduction in build time
on top of #987.

So a full dockerize workflow run now takes *10 mins* vs previous 35+mins.
This is a total of *72% improvement* in max dockerize run time.

Get additional speed improvements when docker layer cache hit.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
maintain Maintain code, documentation or project
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants