Build system: _ITERATOR_DEBUG_LEVEL not respected in cuda test on MSVC

[emmenlau]

In order to build c++ against https://github.com/xtensor-stack/xtensor, it is required to set the _ITERATOR_DEBUG_LEVEL=0 on MSVC. We use the following environment variable for cmake to set this:

CPPFLAGS="/D_ITERATOR_DEBUG_LEVEL=0"

This setting is slightly cumbersome because it has to match in each and every source file of the project. Generally that works well with many projects, including fmt. However the current fmt CUDA test does not respect the setting, which in turn leads to a build failure because not all files use the same setting:

...
[55/55] Linking CXX executable bin\fmt-in-cuda-test.exe
FAILED: bin/fmt-in-cuda-test.exe 
cmd.exe /C "cd . && D:\Tools\bin\cmake.exe -E vs_link_exe --intdir=test\cuda-test\CMakeFiles\fmt-in-cuda-test.dir --rc=C:\PROGRA~2\WI3CF2~1\10\bin\100177~1.0\x64\rc.exe --mt=C:\PROGRA~2\WI3CF2~1\10\bin\100177~1.0\x64\mt.exe --manifests  -- D:\bda-ci\vs2017\VC\Tools\MSVC\14.16.27023\bin\Hostx64\x64\link.exe  test\cuda-test\CMakeFiles\fmt-in-cuda-test.dir\cuda-cpp14.cu.obj test\cuda-test\CMakeFiles\fmt-in-cuda-test.dir\cpp14.cc.obj test\cuda-test\CMakeFiles\fmt-in-cuda-test.dir\cmake_device_link.obj  /out:bin\fmt-in-cuda-test.exe /implib:test\cuda-test\fmt-in-cuda-test.lib /pdb:bin\fmt-in-cuda-test.pdb /version:0.0  /machine:x64  /MACHINE:X64 /DEBUG /debug /INCREMENTAL /subsystem:console -LIBPATH:D:\Tools\lib\x64 fmt.lib cudadevrt.lib cudart_static.lib kernel32.lib user32.lib gdi32.lib winspool.lib shell32.lib ole32.lib oleaut32.lib uuid.lib comdlg32.lib advapi32.lib && cd ."
LINK Pass 1: command "D:\bda-ci\vs2017\VC\Tools\MSVC\14.16.27023\bin\Hostx64\x64\link.exe test\cuda-test\CMakeFiles\fmt-in-cuda-test.dir\cuda-cpp14.cu.obj test\cuda-test\CMakeFiles\fmt-in-cuda-test.dir\cpp14.cc.obj test\cuda-test\CMakeFiles\fmt-in-cuda-test.dir\cmake_device_link.obj /out:bin\fmt-in-cuda-test.exe /implib:test\cuda-test\fmt-in-cuda-test.lib /pdb:bin\fmt-in-cuda-test.pdb /version:0.0 /machine:x64 /MACHINE:X64 /DEBUG /debug /INCREMENTAL /subsystem:console -LIBPATH:D:\Tools\lib\x64 fmt.lib cudadevrt.lib cudart_static.lib kernel32.lib user32.lib gdi32.lib winspool.lib shell32.lib ole32.lib oleaut32.lib uuid.lib comdlg32.lib advapi32.lib /MANIFEST /MANIFESTFILE:test\cuda-test\CMakeFiles\fmt-in-cuda-test.dir/intermediate.manifest test\cuda-test\CMakeFiles\fmt-in-cuda-test.dir/manifest.res" failed (exit code 1319) with the following output:
Microsoft (R) Incremental Linker Version 14.16.27034.0
Copyright (C) Microsoft Corporation.  All rights reserved.

cpp14.cc.obj : error LNK2038: mismatch detected for '_ITERATOR_DEBUG_LEVEL': value '0' doesn't match value '2' in cuda-cpp14.cu.obj
fmt.lib(format.cc.obj) : error LNK2038: mismatch detected for '_ITERATOR_DEBUG_LEVEL': value '0' doesn't match value '2' in cuda-cpp14.cu.obj
   Creating library test\cuda-test\fmt-in-cuda-test.lib and object test\cuda-test\fmt-in-cuda-test.exp
bin\fmt-in-cuda-test.exe : fatal error LNK1319: 2 mismatches detected

[vitaut]

Could you submit a PR to fix/workaround this?

[emmenlau]

Sadly not, I have no understanding of this cuda test, and I'm not experienced with cuda or why cmake may not pass compiler flags to it. I was able to make a small patch that makes cuda tests optional with a cmake option. Is that interesting?

[vitaut]

cc @luncliff & @risa2000 who know this test best

[risa2000]

@emmenlau
Which version of CMake do you use? Could you post the command line (with environment variables and CMake variables if any is set) you use to run cmake on fmt.

[luncliff]

it's a bit wierd but with my experience, linking issue can come from build with Debug or RelWithDebInfo with nvcc...

I will take a look and report here ASAP.

[luncliff]

@emmenlau I'd like to know how _ITERATOR_DEBUG_LEVEL is affecting the project. And which build configs you are using.
I read xtensor-stack/xtensor#1659 but I think your project has changed since the issue.

Please let us know both your environment and configuration.

Opinion about the source of this issue

I usually met _ITERATOR_DEBUG_LEVEL issues when build configuration is mismatched.
In this case, the configuration is manually specified. However, CUDA source file(.cu) didn't follow it (but C++ source .cc did).
Considering the build steps with NVCC, the source of the symptom should be one of the followings.

1. CUDA_NVCC_FLAGS and CUDA_NVCC_FLAGS_<CONFIG> is not configured properly.
   1. CMake dropped the configuration?
   2. Compiler option related to _ITERATOR_DEBUG_LEVEL is not in CUDA_NVCC_FLAGS
2. Mismatch of the build configuration
   1. CMAKE_BUILD_TYPE and actual build has difference. Like CMAKE_BUILD_TYPE=DEBUG + MinSizeRel

With the description of @emmenlau, I assumed the project structure is peering 'fmt' and 'xtensor'.

$ tree ./project/
./project
├── CMakeLists.txt
├── fmt
│   ├── CMakeLists.txt
│   └── ...
├── xtensor
│   ├── CMakeLists.txt
│   └── ... 
└...

If this is right, 2-1 case shouldn't matter when the command is like ...

cmake --build . --config minsizerel

Possible Fix?

I'm planning to change the CUDA_NVCC_FLAGS generation and set_property(SOURCE ...). Just like what @risa2000 did with set_property(SOURCE ...).

Or some generator expression can be used.

# test/cuda-test/CMakeLists.txt

list(APPEND CUDA_NVCC_FLAGS     "--verbose")
list(APPEND CUDA_NVCC_FLAGS_DEBUG           "-g" "-O0")
list(APPEND CUDA_NVCC_FLAGS_RELEASE         "-O3")
list(APPEND CUDA_NVCC_FLAGS_MINSIZEREL      "-O1")
list(APPEND CUDA_NVCC_FLAGS_RELWITHDEBINFO  "-g" "-O2")

if(...)
    set_property(SOURCE cuda-cpp14.cu
    APPEND PROPERTY
         COMPILE_OPTIONS "${CUDA_NVCC_FLAGS}" "${CUDA_NVCC_FLAGS_MINSIZEREL}"
    )
endif()

Currently checking known flags of NVCC. I think the work won't require many changes.

[vitaut]

Made cuda-test an opt-in in fd52de0. A PR to improve the test would be welcome too.