Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

On non-Unix platforms, use deep copying rather than symlinking #1136

Merged
merged 19 commits into from
Jan 10, 2025

Conversation

jnbooth
Copy link
Contributor

@jnbooth jnbooth commented Nov 27, 2024

Addresses #1120. cxx-qt-build's current symlinking works on Unix, fails on Windows unless Developer Mode is enabled, and panics on anything else. This PR modifies the non-Unix behavior to use deep file copying rather than symlinking.

I have tested this on Windows 11 and MacOS Sequoia 15.1.1.

Note: I'm not sure if this PR means these lines should be removed:

cxx-qt/README.md

Lines 87 to 93 in f5afefe

This repository contains symbolic links, which requires some setup on Windows 10 before cloning the repository.
First, [enable Windows Developer Mode](https://learn.microsoft.com/en-us/gaming/game-bar/guide/developer-mode)
to avoid needing administrator privileges to create symlinks. Then, enable symlinks in Git:
```shell
git config --global core.symlinks true
```

Copy link

codecov bot commented Nov 28, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 100.00%. Comparing base (f5627f8) to head (6dc6e17).
Report is 1 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff            @@
##              main     #1136   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           71        71           
  Lines        11967     11967           
=========================================
  Hits         11967     11967           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@LeonMatthesKDAB
Copy link
Collaborator

@jnbooth thank you for working on this.

Unfortunately the symlinks_conflict function won't work when a directory is copied, which is what CI is complaining about.
Basically, multiple dependencies may re-export the headers from e.g. cxx-qt-lib.
When we combine these dependencies in the main application, there should only be a single set of headers for cxx-qt-lib.
With the previous symlinks, this was rather simple to check, as in the end after resolving the symlinks they paths all point to the same headers that were initially created by cxx-qt-lib.

At the moment, I see two ways to fix this:

  1. When re-exporting dependencies, propagate the path to the original dependencies header files. Then when detecting the dependencies in cxx-qt-build, we need to store that original path and resolve conflicts that way.
  2. Adapt symlinks_conflict to recursively diff the directory and check the actual files inside on non-unix systems.

While option 1 is faster, option 2 is simpler, so option 2 is probably the better way forward at the moment.

There's also the option to remove the check entirely, but I'm a bit hesitant to do that, as accessing the wrong headers from a duplicate dependency sounds like a really bad time and super hard to detect without this check...

@LeonMatthesKDAB
Copy link
Collaborator

Ah, and side-note: The note from the README can be removed with this patch :)
This was the only reason we had it IIRC.

@jnbooth
Copy link
Contributor Author

jnbooth commented Nov 28, 2024

@LeonMatthesKDAB Thank you for taking a look at this so quickly! I'm taking most of today off for Thanksgiving, but I'll get to work on implementing those changes tomorrow.

@jnbooth
Copy link
Contributor Author

jnbooth commented Dec 1, 2024

@LeonMatthesKDAB Ready for re-review. Here's how I handled the diffing:

let mut source = File::open(source)?;
let mut dest = File::open(dest)?;
if source.metadata()?.len() != dest.metadata()?.len() {
return Ok(true);
}
let mut source_buf = [0; 1024];
let mut dest_buf = [0; 1024];
loop {
let source_n = source.read(&mut source_buf)?;
let dest_n = dest.read(&mut dest_buf)?;
if source_n == 0 && dest_n == 0 {
return Ok(false);
}
if &source_buf[..source_n] != &dest_buf[..dest_n] {
return Ok(true);
}
}

I think that should be fairly efficient. Using 2*1024 bytes of buffer seems safe enough. I noticed that in std, the default buffer size for BufReader is normally 8*1024 but set to only 512 for "bare metal" platforms, but I think those probably aren't relevant to cxx-qt.

Copy link
Collaborator

@LeonMatthesKDAB LeonMatthesKDAB left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jnbooth Thank you for keeping on top of this.

I'm quite happy with how the PR is turning out.
There's some small things left, especially the files_conflict function feels a bit more C-like than I would want.

Once those are resolved I'm happy to approve this :)

crates/cxx-qt-build/src/dir.rs Outdated Show resolved Hide resolved
crates/cxx-qt-build/src/dir.rs Outdated Show resolved Hide resolved
crates/cxx-qt-build/src/lib.rs Outdated Show resolved Hide resolved
crates/cxx-qt-build/src/lib.rs Show resolved Hide resolved
crates/cxx-qt-build/src/lib.rs Show resolved Hide resolved
@LeonMatthesKDAB
Copy link
Collaborator

@jnbooth sorry for the delay, took me a bit longer than I hoped to re-review this patch.

Unfortunately I also have to correct this statement:

Ah, and side-note: The note from the README can be removed with this patch :) This was the only reason we had it IIRC.

Indeed I did not recall correctly, because while depending on CXX-Qt now no longer requires symbolic links, the repository still contains some.
The qml_minimal example and the cargo_without_cmake share some code, which we do via symlinks to avoid duplicating files.
So can you please restore the note from the README 🙏
I'm sorry about the extra work...

@jnbooth jnbooth requested a review from ahayzen-kdab December 27, 2024 21:01
@jnbooth
Copy link
Contributor Author

jnbooth commented Jan 10, 2025

@LeonMatthesKDAB Is there anything else I should do for this PR?

Copy link
Collaborator

@LeonMatthesKDAB LeonMatthesKDAB left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jnbooth Again, sorry for the delay. The holidays had me firmly in their grasp 🎅 ☺️

Thank you for the contribution.

@LeonMatthesKDAB LeonMatthesKDAB enabled auto-merge (squash) January 10, 2025 08:58
@LeonMatthesKDAB LeonMatthesKDAB merged commit d0806b6 into KDAB:main Jan 10, 2025
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants