Makefile: prefetch crossgcc tarballs (ftpmirror.gnu.org failing downloads on coreboot builds) #947
Conversation
|
@Tonux599 This brings an interesting concept that could be simplified the other way around. I don't find the maintainability path sexy at all, and caching the files for non extracted coreboot build script is not interesting at all (which would be the alternative, scoping file versions under /heads/build/coreboot-*/util/crossgcc/buildgcc for yet non extracted file). Hashes could be made on coreboot module and coreboot_cross cache made on I could push a PR of that, where the mirror of gnu should not fail... normally. |
…4.8.1, 4.11 and 4.13
…g out much later if a tarball can not be fetched.
Tonux599
force-pushed
the
make-prefetch-packages
branch
from
d4478b8 to
df8f97b
Compare
Actually why stop with the tarballs, we may be able to cache the built crossgcc for each coreboot version? Will mess around more tomorrow. |
@Tonux599 The actual cache system is based on modules and patches directory checksums. |
|
As of right now: Digest creation of what we currently measure and cache consequently: Cache creation based on those digest: As you can see here, the coreboot packages are cached under Cache restore based on previous digests: |
|
@Tonux599 : What I was suggesting would be the introduction of a new cache between musl-cross and actual boards build cache, which could be limited to coreboot cache upon a successful run, which just like musl-cross cache only changing when that module signature changes, could be reused until coreboot module changes. It would look like something like the following: Add a new digest for coreboot module (and associated patches) Add coreboot module cache in between global cache and musl-cross for successful build (cache is build only if digest signature is different). The cache is made on save, and only if the digest is different from found, and hashed files in the digest created tmp file before. Restore the cache in order of bigger to smaller. Logic here is
Any better idea welcome. |
Per linuxboot#947 (comment) proposition
|
@Tonux599 Maybe we should take a different approach and have cache per boards? coreboot version? Silly idea, but https://app.circleci.com/pipelines/github/tlaurion/heads/677/workflows/c684d1ad-c557-4fb1-8aaa-ed364c270f34/jobs/736 is showing that the gnu mirror issue is not going away, while current successful build (so a specific coreboot version) are not cached (here, 4.8.1 coreboot under ./build could be cached and reused). I do not have a good angle as of right now to propose, while being conscious this holiday season is not over and gnu people have not answered. Another silly idea (hack) would be to resolve the mirror and sed buildgcc script to replace mirror with reolved name so that tarballs are downloaded.... But this is ugly also. |
The cache would have to include the tarballs, as even if crossgcc is already build Coreboot's make still verifies the tarball hash and extracts before checking to see if its already build. Maybe we are thinking too big for the moment, if I prepare a PR that simply has CircleCI cache only the crossgcc tarballs would you accept? At least with that once a build has been successful the gnu mirror doesn't need to be contacted again for another six months. |
|
Another approach could be to hack around the following output and cache only the files, as in OP (without hardcoding them) |
tlaurion
changed the title
The idea here is a cache to restore from (musl-cross from coreboot version bound crosscomipler, from which coreboot is built) 1- Reuse existing cache for all modules and patches created digest's hash. (If a single module or patch changes, we have cache miss) 2- Reuse existing coreboot and musl-cross-make created digest's hash (If a patch was added to current coreboot, or new coreboot version was added in coreboot module definition, we have a cache miss. Otherwise, we reuse the whole musl-cross-make and coreboot cache for current used versions.) 3- Reuse existing musl-cross-make created digest's hash matching cache (If musl-cross-make module didn't change, we don't rebuild it) Per linuxboot#947 (comment) proposition
The idea here is a cache to restore from (musl-cross from coreboot version bound crosscomipler, from which coreboot is built) 1- Reuse existing cache for all modules and patches created digest's hash past build matching cache. (If a single module or patch changes, we have cache miss.) 2- Reuse existing coreboot and musl-cross-make created digest's hash past build's matching cache (If a patch was added to current coreboot, or new coreboot version was added in coreboot module definition, we have a cache miss.) 3- Reuse existing musl-cross-make created digest's hash past build matching cache (If musl-cross-make module didn't change, we don't rebuild it.) Per linuxboot#947 (comment) proposition
|
@Tonux599 not sure how to implement/extend this hack, but it seems it could work (without adding maintenance like implied in this PR at each coreboot release.) Does this ignite some more proper hacks in your mind? And then continuing with boards compilation as in normal builds? Where to implement this, so that coreboot tarballs are extracted, prior of first board build? That could be in CI config, prior of first build of each version, requiring maintainer to move boards around (and comment CI per coreboot version)? EDIT: the precedent download a whole crap of unneeded stuff, including llvm gdb... |
The idea here is a cache to restore from (musl-cross from coreboot version bound crosscomipler, from which coreboot is built) 1- Reuse existing cache for all modules and patches created digest's hash past build matching cache. (If a single module or patch changes, we have cache miss.) 2- Reuse existing coreboot and musl-cross-make created digest's hash past build's matching cache (If a patch was added to current coreboot, or new coreboot version was added in coreboot module definition, we have a cache miss.) 3- Reuse existing musl-cross-make created digest's hash past build matching cache (If musl-cross-make module didn't change, we don't rebuild it.) Per linuxboot#947 (comment) proposition
Ever look at your CircleCI build for about 2 hours and then the whole thing fails because it failed to fetch some tarballs for a coreboot crossgcc version that hasn't been build yet? Hopefully this will help that. Fetches the tarballs required for coreboot's crossgcc for version 4.8.1, 4.11 and 4.13 (these appear to be the ones currently in use) early on.
Possible extra feature that I haven't implemented yet is checking the checksums early too but as it stands this should help a bit.