[vcpkg] Use std::filesystem when Visual Studio is greater than 2015 #12774

JackBoosY · 2020-08-06T05:42:05Z

When the Visual Studio version is 2015, use std::experimental::filesystem
When the Visual Studio version is greater than 2015, use std::filesystem

etheory · 2020-08-06T12:22:33Z

In the process of testing this now, thank you.
It's not building for me.
I get tens of pages of warnings and errors, sorry.
vcpkgpaths.cpp(245,39): error C4996: 'std::filesystem::u8path'

warning STL4021: The std::filesystem::u8path() overloads are deprecated in C++20. T
he constructors of std::filesystem::path provide equivalent functionality via construction from u8string, u8string_view, or iterators with value_type char8_t.You can define _SILENCE_CXX20_U8PATH_DEPREC
ATION_WARNING or _SILENCE_ALL_CXX20_DEPRECATION_WARNINGS to acknowledge that you have received this warning. (compiling source file ..\src\vcpkg\vcpkgpaths.cpp) [Z:\Luke\Programming\Graphics\vcpkg2\too
lsrc\vcpkglib\vcpkglib.vcxproj]

and many others.

I'm using the latest:
Visual Studio 2019 Developer Command Prompt v16.6.5

strega-nil · 2020-08-06T13:37:52Z

Yeah, we need to turn off that warning, because that deprecation is not a thing we can deal with rn since we need to work back to VS2015

BillyONeal · 2020-08-06T17:47:46Z

I think this is just another reason to get off std::filesystem.

etheory · 2020-08-07T01:37:25Z

OK, just did a git pull, and it did re-build this time without errors.
Trying now to see if it works when trying to do a package install.
Will report back shortly....

etheory · 2020-08-07T01:41:57Z

So far so good.
The install is currently not giving me any rename or copy errors.
Will keep you posted if I can get all the way to the end of a full install.
Might take a while, but I will let you know.

etheory · 2020-08-07T01:47:01Z

Nope, the dreaded issue is back!
(Note I've manually edited the below file-paths just to remove references to my local drive, but safely assume on my end that is a full path)

PS **\Programming\Graphics\vcpkg2> .\vcpkg.exe install openimageio --triplet x64-windows
Computing installation plan...
A suitable version of cmake was not found (required v3.17.2). Downloading portable cmake v3.17.2...
Downloading cmake...
  https://github.com/Kitware/CMake/releases/download/v3.17.2/cmake-3.17.2-win32-x86.zip -> **\Programming\Graphics\vcpkg2\downloads\cmake-3.17.2-win32-x86.zip
Extracting cmake...
A suitable version of 7zip was not found (required v18.1.0). Downloading portable 7zip v18.1.0...
Downloading 7zip...
  https://www.nuget.org/api/v2/package/7-Zip.CommandLine/18.1.0 -> **\Programming\Graphics\vcpkg2\downloads\7-zip.commandline.18.1.0.nupkg
Extracting 7zip...
A suitable version of nuget was not found (required v5.5.1). Downloading portable nuget v5.5.1...
Downloading nuget...
  https://dist.nuget.org/win-x86-commandline/v5.5.1/nuget.exe -> **\Programming\Graphics\vcpkg2\downloads\22ea847d-nuget.exe
Failed to do post-extract rename-in-place.
fs.rename(**\Programming\Graphics\vcpkg2\downloads\tools\cmake-3.17.2-windows.partial.9060, **\Programming\Graphics\vcpkg2\downloads\tools\cmake-3.17.2-windows, Access is denied.)

FWIW I'm running this through a Windows Powershell using the very very latest Microsoft Visual Studio Comunity 2019 - 16.7.0.
Also didn't work via "x64 Native Tools Command Prompt for VS 2019".

BillyONeal · 2020-08-07T02:14:17Z

Can you try running it under procmon and see which process is holding the file locked?

JackBoosY · 2020-08-07T02:24:11Z

And can you try to use powershell with admin permission?

etheory · 2020-08-07T04:23:12Z

Admin permissions is not something I want to have to revert to sorry @JackBoosY and I can't anyway, since for some reason Windows hides my network drive in Admin mode (I never figured out why).

Could the issue be that I'm working from a network drive?
A friend of mine has suggested that file locking on network drives doesn't work on Windows properly.
Could this be the reason?

I'll try again from C: and let you know.

I wasn't aware that Windows performed so poorly when accessing network drives.

etheory · 2020-08-07T04:38:13Z

@JackBoosY

OK. Somewhat incredibly anticlimactically, this patch isn't required.

The issue is that vcpkg doesn't work with network drives.

I tried again with a fresh vcpkg install on C: and everything worked perfectly.

That's possibly the reason other users are having issues also....

It would be great if it DID work on network drives, but, it doesn't.

JackBoosY · 2020-08-07T05:32:12Z

From network driver... that's we didn't expected. I think that also affect WSL.
@BillyONeal What do you think about?

BillyONeal · 2020-08-07T07:32:55Z

Is it both VS2015 and being on a network drive needed to trigger the problem? We intend to remove all std::experimental paths when we start shipping binaries, always building vcpkg.exe with VS2019.

etheory · 2020-08-07T08:25:32Z

@BillyONeal no, only the location of vcpkg.
I have all VS2019 (I don't use VC2015) installations on C: and was trying to work from a network drive with my code on it.
I installed vcpkg itself, and it's packages to the network location.
That's what fails.

I'm not sure what would happen if both VS2019 AND vcpkg were on the network location, since that's not my use-case.

Moving all my stuff to C: is working perfectly now.

Neumann-A · 2020-08-07T08:38:44Z

I have all VS2019 (I don't use VC2015) installations on C: and was trying to work from a network drive with my code on it.
I installed vcpkg itself, and it's packages to the network location.

Have you tried subst'ing the network drive away?

e.g. subst N: <networkpath> but I highly discourage that usage since the linker/compile speed will suffer dramatically due to a lot of small files and network speed etc.

etheory · 2020-08-07T11:11:03Z

Have you tried subst'ing the network drive away?

e.g. subst N: <networkpath> but I highly discourage that usage since the linker/compile speed will suffer dramatically due to a lot of small files and network speed etc.

I already had used subst.
The drive letter was Z: and it mapped to a synology DiskStation, which is normally accessed as \\DiskStation.

And it didn't work with vcpkg.

toolsrc/src/vcpkg/base/files.cpp

BillyONeal · 2020-08-07T19:44:14Z

toolsrc/src/vcpkg/base/files.cpp

+#if defined(_WIN32)
+    return vcpkg::Strings::to_utf8(p.native());
+#else
+    return p.string();


I think this should be p.native() since we're making an assumption that it'll be UTF-8.

BillyONeal · 2020-08-07T19:45:00Z

toolsrc/src/vcpkg/base/files.cpp

+#if defined(_WIN32)
+    return vcpkg::Strings::to_utf8(p.generic_wstring());
+#else
+    return p.string();


That can't be right, it at least needs to be generic_string

toolsrc/include/vcpkg/base/files.h

…msbuild too.

# Conflicts: # toolsrc/vcpkglib/vcpkglib.vcxproj # toolsrc/vcpkgtest/vcpkgtest.vcxproj

etheory · 2020-08-09T04:29:28Z

If you need someone to test this once the PR has stabilized, please let me know!

strega-nil · 2020-08-09T19:05:21Z

@etheory it should be stable now; the one concern I want to fix is getting CMake to support compiling with VS 2015 (so we can continue to test VS2015 even without the VS build system), which will be fixed in a different PR.

BillyONeal · 2020-08-10T21:00:10Z

I see, so it's not just being a network share, it's also a "samba" server. It's possible that they don't implement our attempts to canonicalize the vcpkg root path. :/

…filesystem

etheory · 2020-08-15T07:03:49Z

I'm testing this now....
Unfortunately it doesn't work.
It produces exactly the same issue as before from a network location:

PS z:\Luke\Programming\Graphics\vcpkg> .\vcpkg.exe install openimageio --triplet x64-windows
Computing installation plan...
A suitable version of cmake was not found (required v3.17.2). Downloading portable cmake v3.17.2...
Downloading cmake...
  https://github.com/Kitware/CMake/releases/download/v3.17.2/cmake-3.17.2-win32-x86.zip -> Z:\Luke\Programming\Graphics\vcpkg\downloads\cmake-3.17.2-win32-x86.zip
Extracting cmake...
A suitable version of 7zip was not found (required v18.1.0). Downloading portable 7zip v18.1.0...
Downloading 7zip...
  https://www.nuget.org/api/v2/package/7-Zip.CommandLine/18.1.0 -> Z:\Luke\Programming\Graphics\vcpkg\downloads\7-zip.commandline.18.1.0.nupkg
Extracting 7zip...
A suitable version of nuget was not found (required v5.5.1). Downloading portable nuget v5.5.1...
Downloading nuget...
  https://dist.nuget.org/win-x86-commandline/v5.5.1/nuget.exe -> Z:\Luke\Programming\Graphics\vcpkg\downloads\22ea847d-nuget.exe
Failed to do post-extract rename-in-place.
fs.rename(Z:\Luke\Programming\Graphics\vcpkg\downloads\tools\cmake-3.17.2-windows.partial.11688, Z:\Luke\Programming\Graphics\vcpkg\downloads\tools\cmake-3.17.2-windows, Access is denied.)

…filesystem

etheory · 2020-08-18T01:17:12Z

Should I test this again? As far as I'm aware this PR has been approved but still doesn't fix the issue I'm encountering?

strega-nil · 2020-08-18T03:41:09Z

@etheory unfortunately, there's nothing much we can do if it doesn't fix your issue; fs::rename is a straight MoveFileEx -- if MoveFileEx doesn't work, nothing works.

BillyONeal · 2020-08-18T16:23:07Z

Thanks for your fix Jack!

etheory · 2020-08-20T14:03:38Z

@etheory unfortunately, there's nothing much we can do if it doesn't fix your issue; fs::rename is a straight MoveFileEx -- if MoveFileEx doesn't work, nothing works.

If that's the case, it might be pertinent to add to the vcpkg documentation that it doesn't work when run on network drives.

BillyONeal · 2020-08-20T19:52:04Z

It does work on network drives, just not those hosted on whatever software this Synology uses.

…icrosoft#12774) Co-authored-by: Nicole Mazzuca <[email protected]> Co-authored-by: Billy Robert O'Neal III <[email protected]>

[vcpkg] Use std::filesystem when Visual Studio is greater than 2015

6e83038

JackBoosY added category:vcpkg-bug The issue is with the vcpkg system (including helper scripts in `scripts/cmake/`) info:internal This PR or Issue was filed by the vcpkg team. labels Aug 6, 2020

JackBoosY requested review from BillyONeal and strega-nil August 6, 2020 05:42

JackBoosY assigned PhoebeHui Aug 6, 2020

JackBoosY mentioned this pull request Aug 6, 2020

Failed to rename files using fs.rename #9132

Closed

strega-nil added 3 commits August 6, 2020 11:31

fix check for experimental filesystem

a315433

fix building on C++20

de794ca

fix UB in u8path

2377359

JackBoosY mentioned this pull request Aug 7, 2020

Library "zlibd" was not found! #12788

Closed

clang-format

ad837ad

PhoebeHui mentioned this pull request Aug 7, 2020

vcpkg_fixup_pkgconfig failed to replace the string when root path contains Upper charactor on windows #12709

Closed

fix compile on unix

e60c96d

strega-nil suggested changes Aug 7, 2020

View reviewed changes

toolsrc/src/vcpkg/base/files.cpp Outdated Show resolved Hide resolved

toolsrc/src/vcpkg/base/files.cpp Outdated Show resolved Hide resolved

toolsrc/src/vcpkg/base/files.cpp Outdated Show resolved Hide resolved

toolsrc/src/vcpkg/base/files.cpp Outdated Show resolved Hide resolved

BillyONeal self-assigned this Aug 7, 2020

BillyONeal reviewed Aug 7, 2020

View reviewed changes

strega-nil self-requested a review August 7, 2020 19:51

BillyONeal reviewed Aug 7, 2020

View reviewed changes

toolsrc/include/vcpkg/base/files.h Outdated Show resolved Hide resolved

BillyONeal added 4 commits August 7, 2020 15:47

Move detection of VCPKG_USE_STD_FILESYSTEM into the build system for …

97df992

…msbuild too.

Use the right string.

e58b7c6

Version numbers are hard.

778115d

Merge remote-tracking branch 'origin/master' into vs2015

3ad78d9

# Conflicts: # toolsrc/vcpkglib/vcpkglib.vcxproj # toolsrc/vcpkgtest/vcpkgtest.vcxproj

BillyONeal approved these changes Aug 7, 2020

View reviewed changes

remove .sln file which should never be used

d571db2

Merge remote-tracking branch 'upstream/master' into dev/jack/use_std_…

f24f453

…filesystem

strega-nil added 2 commits August 17, 2020 10:38

Merge remote-tracking branch 'upstream/master' into dev/jack/use_std_…

cb6674f

…filesystem

fix C++20

5517851

ras0219 approved these changes Aug 17, 2020

View reviewed changes

strega-nil approved these changes Aug 18, 2020

View reviewed changes

BillyONeal merged commit 0bb8780 into microsoft:master Aug 18, 2020

JackBoosY deleted the dev/jack/use_std_filesystem branch August 19, 2020 02:07

ZhilkinSerg mentioned this pull request Nov 23, 2020

Manifest file not found with msbuild or Visual Studio #14735

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[vcpkg] Use std::filesystem when Visual Studio is greater than 2015 #12774

[vcpkg] Use std::filesystem when Visual Studio is greater than 2015 #12774

JackBoosY commented Aug 6, 2020 •

edited

Loading

etheory commented Aug 6, 2020 •

edited

Loading

strega-nil commented Aug 6, 2020

BillyONeal commented Aug 6, 2020

etheory commented Aug 7, 2020

etheory commented Aug 7, 2020

etheory commented Aug 7, 2020 •

edited

Loading

BillyONeal commented Aug 7, 2020

JackBoosY commented Aug 7, 2020

etheory commented Aug 7, 2020 •

edited

Loading

etheory commented Aug 7, 2020 •

edited

Loading

JackBoosY commented Aug 7, 2020 •

edited

Loading

BillyONeal commented Aug 7, 2020

etheory commented Aug 7, 2020

Neumann-A commented Aug 7, 2020

etheory commented Aug 7, 2020 •

edited

Loading

BillyONeal Aug 7, 2020

BillyONeal Aug 7, 2020

etheory commented Aug 9, 2020

strega-nil commented Aug 9, 2020

BillyONeal commented Aug 10, 2020

etheory commented Aug 15, 2020 •

edited

Loading

etheory commented Aug 18, 2020

strega-nil commented Aug 18, 2020

BillyONeal commented Aug 18, 2020

etheory commented Aug 20, 2020

BillyONeal commented Aug 20, 2020

[vcpkg] Use std::filesystem when Visual Studio is greater than 2015 #12774

[vcpkg] Use std::filesystem when Visual Studio is greater than 2015 #12774

Conversation

JackBoosY commented Aug 6, 2020 • edited Loading

etheory commented Aug 6, 2020 • edited Loading

strega-nil commented Aug 6, 2020

BillyONeal commented Aug 6, 2020

etheory commented Aug 7, 2020

etheory commented Aug 7, 2020

etheory commented Aug 7, 2020 • edited Loading

BillyONeal commented Aug 7, 2020

JackBoosY commented Aug 7, 2020

etheory commented Aug 7, 2020 • edited Loading

etheory commented Aug 7, 2020 • edited Loading

JackBoosY commented Aug 7, 2020 • edited Loading

BillyONeal commented Aug 7, 2020

etheory commented Aug 7, 2020

Neumann-A commented Aug 7, 2020

etheory commented Aug 7, 2020 • edited Loading

BillyONeal Aug 7, 2020

Choose a reason for hiding this comment

BillyONeal Aug 7, 2020

Choose a reason for hiding this comment

etheory commented Aug 9, 2020

strega-nil commented Aug 9, 2020

BillyONeal commented Aug 10, 2020

etheory commented Aug 15, 2020 • edited Loading

etheory commented Aug 18, 2020

strega-nil commented Aug 18, 2020

BillyONeal commented Aug 18, 2020

etheory commented Aug 20, 2020

BillyONeal commented Aug 20, 2020

JackBoosY commented Aug 6, 2020 •

edited

Loading

etheory commented Aug 6, 2020 •

edited

Loading

etheory commented Aug 7, 2020 •

edited

Loading

etheory commented Aug 7, 2020 •

edited

Loading

etheory commented Aug 7, 2020 •

edited

Loading

JackBoosY commented Aug 7, 2020 •

edited

Loading

etheory commented Aug 7, 2020 •

edited

Loading

etheory commented Aug 15, 2020 •

edited

Loading