Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add parallelism and copy-on-write links to Robocopy task #462

Merged
merged 2 commits into from
Jul 17, 2023

Conversation

erikmav
Copy link
Collaborator

@erikmav erikmav commented Jul 11, 2023

  • Use the CopyOnWrite library and an action block to greatly speed up artifact copies. Use parallelism settings similar to those in the Microsoft.Build.CopyOnWrite SDK and allow turning off CoW linking with a task setting.
  • Fix an overcopying bug where the same source:destination pair would be copied multiple times.
  • Don't copy files onto themselves (seen in occasional cases).
  • Linearize same-destination copies from different sources.
  • Update major version of Artifacts.
  • Unify on 'DisableCopyOnWrite' property for Copy and Artifacts
  • Migrate CoW SDK Copy task to use CoW package for clone file compat checks.
  • Fix possibility of under-copying different-cased files/paths on Linux by varying path comparers by OS. NOTE: Did not update the CoW Copy task, and MSBuild itself seems to use IgnoreCase in most cases. Need feedback here.

@erikmav erikmav force-pushed the dev/erikmav/parallelCowRobocopy branch 2 times, most recently from 6b38165 to ea8319b Compare July 11, 2023 18:54
@erikmav
Copy link
Collaborator Author

erikmav commented Jul 11, 2023

@jeffkl I don't see how to increment the Microsoft.Build.CopyOnWrite SDK version - ideally the newer 0.3.3 CopyOnWrite should be used in a new version of that package.

@jeffkl
Copy link
Contributor

jeffkl commented Jul 11, 2023

@jeffkl I don't see how to increment the Microsoft.Build.CopyOnWrite SDK version - ideally the newer 0.3.3 CopyOnWrite should be used in a new version of that package.

Would just setting this to 1.1 be enough? https://github.com/microsoft/MSBuildSdks/blob/main/src/CopyOnWrite/version.json

The build number automatically increments for each commit but I manually adjust the minor version for feature releases.

@erikmav
Copy link
Collaborator Author

erikmav commented Jul 11, 2023

I think 1.0.NewBuildNumber would be fine as long as it gets published after this commit. Not sure how that works for this repo.

@erikmav
Copy link
Collaborator Author

erikmav commented Jul 11, 2023

One other note - tests ran successfully on a Dev Drive with CoW capability.

@erikmav erikmav force-pushed the dev/erikmav/parallelCowRobocopy branch from ea8319b to 24ce064 Compare July 11, 2023 20:08
@erikmav erikmav force-pushed the dev/erikmav/parallelCowRobocopy branch 2 times, most recently from 5083cc0 to 8e3a516 Compare July 11, 2023 22:48
@erikmav erikmav force-pushed the dev/erikmav/parallelCowRobocopy branch 7 times, most recently from 3621aa6 to dba67e4 Compare July 17, 2023 21:28
- Use the CopyOnWrite library and an action block to greatly speed up artifact copies. Use parallelism settings similar to those in the Microsoft.Build.CopyOnWrite SDK and allow turning off CoW linking with a task setting.
- Fix an overcopying bug where the same source:destination pair would be copied multiple times.
- Don't copy files onto themselves (seen in occasional cases).
- Linearize same-destination copies from different sources.
- Update major version of Artifacts.
- Unify on 'DisableCopyOnWrite' property for Copy and Artifacts
- Migrate CoW SDK Copy task to use CoW package for clone file compat checks.
- Fix possibility of under-copying different-cased files/paths on Linux by varying path comparers by OS. NOTE: Did not update the CoW Copy task, and MSBuild itself seems to use IgnoreCase in most cases. Need feedback here.
@erikmav erikmav force-pushed the dev/erikmav/parallelCowRobocopy branch from dba67e4 to 186fd91 Compare July 17, 2023 22:16
@AndyGerlicher AndyGerlicher merged commit 81e435b into microsoft:main Jul 17, 2023
@erikmav erikmav deleted the dev/erikmav/parallelCowRobocopy branch July 17, 2023 23:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants