CI: Build cache issue

[kp992]

Hi @mmcky,
I was checking the build cache issue and looks like it is working as expected.
For example: https://github.com/QuantEcon/lecture-python-intro/actions/runs/11118983947/job/30893458986?pr=551

The logs show that for the build jobs: Build HTML and Build download notebooks things are working fine and it is using the cached notebooks. Build PDF from Latex does not seem to be using the cache and may the issue behind this is that in cache.yml, we are just building and uploading html build and so it couldn't find PDF cache.

Please let me know if you were pointing some other lecture series issue.

[mmcky]

@kp992 thanks for your thoughts on this.

The latex build is the first active build in the action, so it looks like it is executing all the notebooks and rebuilding the cache, so download notebooks and the html runs use that cache. The step download_cache should fetch the _build folder to enable the latex run to use the cache made by cache.yml but that doesn't seem to work.

the sphinx message indicates that the environment has changed?

[kp992]

yeah, you are right. Comparing two builds: https://github.com/QuantEcon/lecture-python-intro/actions/runs/11118983947/job/30893458986?pr=551 (2 weeks back) and https://github.com/QuantEcon/lecture-python-intro/actions/runs/10396786335/job/28791423058 (Aug 15), the difference is in the log loading pickled environment... checking bibtex cache... out of date. I don't see any changes in workflow files and so the regression is probably due to some upstream changes. I am trying to debug more into this.

[mmcky]

thanks @kp992 it is a really frustrating bug. No idea why the pickling would change.

[kp992]

I have even tried to match the same environment as 2 months but the issue still persists. Not sure what is the root cause because it's not even reproducible in previous environments. Any reason why aren't we using latest versions for each libraries and have pinned? I will try to remove all the pins locally and try to see if that fixes.

[mmcky]

Any reason why aren't we using latest versions for each libraries and have pinned? I will try to remove all the pins locally and try to see if that fixes.

You mean in environment.yml? I keep them up to date regularly but the pins have provided stability for us. I do a check execution each time we upgrade them.

[mmcky]

It's like sphinx is pickling something in the environment that is changing between cache runs and ci runs causing sphinx to think everything has changed?

[kp992]

I was debugging into it by trying various ways. I created a forked version where I added more logs to check the pickling info like path and all but that was never logged meaning it wasn't called anytime. So maybe not sure what is really wrong. sphinx isn't looking like a breaking library. There are many other extensions and may be something silently got changed in the existing version. Not sure. I tried many approaches.

[mmcky]

thanks @kp992 it is a tricky one to solve.

I don't fully understand why sphinx decides that the environment has changed sometimes. perhaps we should start to:

1. test the build without using Jupyter-book but myst-nb directly
2. then test myst-parser

if the problem persists it is something impacting the sphinx change logic.

[mmcky]

@kp992 I think I have figured out the issue

QuantEcon/lecture-python-programming.myst#329

[mmcky]

resolved by https://github.com/actions/upload-artifact/blob/main/docs/MIGRATION.md#hidden-files