-
Notifications
You must be signed in to change notification settings - Fork 2.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Spurious rebuilds due to filesystem rounding file modification timestamps #12060
Comments
The proposed solution sounds hacky initially but actually makes some sense. The precision of mtime timestamp varies in filesystem implementations I guess? I think we need research about what the precision is of each major file system handling timestamp. I know macbook once use 1-second accuracy HFS+ but not APFS is good at nanosecond level. I also wonder if there is any other factor affecting the round-up, perhaps different libc implementations? What if people in a less-accurate filesystem with powerful computing resources, leading to a lot of rebuild skipped. |
I'm also seeing this in docker for windows using wsl2 This has actually been something users using
I'm curious if the project would be interested in making the suggested workaround, or if maybe we could enable a way to tell cargo to be less exact in mtime checks (like a cargo flag, available through config) |
@weihanglo I'm unable to perform the research of filesystems and libc implementations that you suggested. I've been using a fork of Cargo with this change, and it worked fine for me. If someone else wants to develop a less hacky fix or investigate all platforms, please do. |
@kornelski, could you share the patch so we can weigh on how we can integrate that, or provide a config for low precision filesystem as a escape hatch. Using fingerprint instead of mtime is also under the consideration #6529. Since we have experimental SQLite integration, #6529 might be more possible to move forward. |
I think I have this issue too. |
Hi, @yuyang-ok. could you share what filesystem/OS/environment you are on? That would help evaluate possible solutions. |
I'm seeing this as well with z3:
I'm building in a Linux VM running on macOS host. The filesystem with the project is a virtfs passed to QEMU as:
Inside of the VM:
I'm not sure where the rounded timestamp comes from. Both VM's fs and remote fs produce ns precision timestamps:
Running Running it in a VM almost always produces a rebuild, but not 100% of the time, as it converges eventually:
|
Pragmatically this would be the best thing to do for the environment. I hate to think how much CO2 has been produced due to this 'feature' over the years of our inaction. |
I think I've found the cause of the problem: cargo/crates/cargo-util/src/paths.rs Lines 695 to 697 in bf79c8b
Cargo tries to write timestamps back to the file system. Cargo assumes that the precision of timestamps is the same in all I/O operations, but this assumption is empirically wrong: Docker's This means that files that were written naturally to the file system using regular I/O functions have high-precision timestamps (with microsecond precision), but ironically the invocation timestamps that Cargo tries to correct via Not so pretty workaround: master...kornelski:cargo:hack-mtimes |
That is really an impressive investigation. Thank you so much!! If I understand correctly, the detection only run once, and with rounding-up it makes a lower false-positive rate than #13955. But still, the "one-in-a-billion-chances-happen-at-scale" problem is there when the actual mtime has zero in nanosecond precision? |
This particular implementation, the workaround is biased towards doing nothing – if your timestamps happen to fall between 0-1 microseconds, it will do nothing and work in the flawed way that Cargo currently does. If it observes rounding, only then it will apply a workaround. In retrospect, instead of trying to detect the problem as it happens, it would be simpler to run a dedicated timestamp rounding test beforehand. |
Problem
The issue #2874 was not completely fixed by #2880. I'm still seeing the exact same problem in latest Cargo:
Steps
I'm using Docker and mounting
target/
dir as a Docker volume (in order to cache it). I think the docker volume rounds the timestamps to 1-second precision. However, this doesn't always cause spurious rebuilds, only maybe 25% of the time, and it's alwaysbuild_script_build
orbuild_script_main
, not other files.Possible Solution(s)
I assume Cargo wouldn't want to introduce imprecision in comparison, such as always comparing the integer number of seconds or allowing 1-second difference between timestamps. Therefore, I propose detecting this problem explicitly: there's only one in a billion chance that the nanosecond part will be exactly 0. So if either of the timestamps being compared has exactly 0 nanosecond part, compare only the integer number of seconds. Otherwise compare in full precision. I can make a PR with that.
Notes
No response
Version
The text was updated successfully, but these errors were encountered: