[Question] Builds inside local SSDs on GtiLab + GKE

[Laski]

As the title says, we're using Kaniko on GitLab CI/CD with a Kubernetes runner in GKE (using the official GitLab Runner Helm Chart). This works great and allows us to parallelize builds, use a cache repo and disable privileged mode on the runner pods.

However, we build many react applications and they take a long time to build even with full caching and redo snapshot mode. We've successfully identified disk throughput as the bottleneck on those jobs, so we changed the nodes' bootdisk to SSDs. This was a great improvement, but is still reaching the network throughput limits (persistent disks in GCP are networked).

GCP provides local SSDs in order to avoid the network limits entirely, which seems ideal for Kaniko as we don't need any kind of durability after the image is pushed. But local SSDs get mounted inside a specific folder in the node (/mnt/disks/ or something like that) and I wasn't able to find documentation on how to take advantage of them with Kaniko. My understanding is that I need to accomplish three, possibly unrelated, things:

• mount the SSD as a volume inside the job pod

• tell the runner to checkout the repository (and run the job) inside that dir

• tell Kaniko to make all the work inside that dir

Any hints on that? Has someone achieved something similar? Thanks in advance.

Disclaimer: I know that this is not a Kaniko specific issue, but this seems like a good place to find someone able to help. Feel free to close it I you disagree. I've made the same question in the GitLab Runner repository btw.

[tejal29]

Thanks @Laski for the issue.
This seems like a reasonable ask. I will find time to go through the documentation links and ways to integrate it with kaniko.

Thanks
Tejal

[Laski]

Thanks @tejal29. FYI, while investigating this I found a bug in the gitlab runner that makes this ¿impossible? for the time being https://gitlab.com/gitlab-org/gitlab-runner/-/issues/27011

[mitar]

I would suggest you use a ramdisk instead. Even faster than SSD. And already supported on GitLab CI. :-)

      builds_dir = "/ramdisk/builds"
      cache_dir = "/ramdisk/cache"

        [[runners.kubernetes.volumes.empty_dir]]
          # Mount ramdisk into build containers (8GB for n2d-highcpu-16).
          name = "ramdisk"
          mount_path = "/ramdisk"
          medium = "Memory"

[mitar]

I do not think though that this approach (SSD or ramdisk) will really help. Because I think Kaniko is not building/touching the GitLab build directory a lot (just for initial files, not for layers it is building). See: #1310 Or maybe I am mistaken? Where do build files go? Into /kaniko or into context dir?

[mitar]

Sadly, it turns out, GitLab CI does not yet properly do the ramdisk on Kubernetes executor: https://gitlab.com/gitlab-org/gitlab-runner/-/issues/27545