Running into Could NOT find CUDA (missing: CUDA_INCLUDE_DIRS) (found version "12.6") Error with CUDA 12.4 and 12.6 (part 3)
#59
Labels
bug
Something isn't working
Comments
|
Thanks for the continued persistence! 🙏
Once that is done, we have a |
|
Yes, I have the setup with |
|
I think we have enough info in debugging with the info from OP. @q10 The problem, IIUC (@robertmaynard can correct me), is that Caffe is still using the old FindCUDA routine, which has been long deprecated: As for a short-term solution, the CUDA headers are located in |
|
If it is PyTorch using the legacy |
Just one last question, what CMake variable should I use to pass in the directory as workaround? I've tried |
Both know how to look for headers and libraries based on those paths |
|
I added the cmake variables, but now I run into this error: Upon further investigation, it seems that How should I remedy this? |
|
We're seeing a similar situation in a JOSS review for https://github.com/Cambridge-ICCS/FTorch: Cambridge-ICCS/FTorch#214 Here's a reproducer [project]
authors = ["Matthew Feickert <[email protected]>"]
channels = ["conda-forge"]
description = "Add a short description here"
name = "example"
platforms = ["linux-64"]
version = "0.1.0"
[tasks]
# Selecting 6055a7dd35e394daff469673feac063f1bfaae01 as reference
clone = """
rm -rf FTorch && \
git clone https://github.com/Cambridge-ICCS/FTorch.git && \
cd FTorch && \
git reset --hard 6055a7dd35e394daff469673feac063f1bfaae01 && \
sed -i 's/cmake_minimum_required(VERSION 3.15 FATAL_ERROR)/cmake_minimum_required(VERSION 3.15...3.31 FATAL_ERROR)/g' src/CMakeLists.txt && \
sed -i 's/set(Python_FIND_VIRTUALENV FIRST)/set(Python_FIND_STRATEGY LOCATION)/g' src/CMakeLists.txt
"""
# Note that libtorch is at $CONDA_PREFIX/lib/libtorch.so
build = """
rm -rf build && \
cmake \
-DCMAKE_INSTALL_PREFIX=$CONDA_PREFIX \
-DCMAKE_PREFIX_PATH=$CONDA_PREFIX \
-DCMAKE_BUILD_TYPE=Release \
-DENABLE_CUDA=TRUE \
-DCUDA_TOOLKIT_ROOT_DIR=$CONDA_PREFIX/targets/x86_64-linux/include/cuda \
-DCUDA_nvrtc_LIBRARY=$CONDA_PREFIX/targets/x86_64-linux/lib/libnvrtc.so.12.6.85 \
-S FTorch/src/ \
-B build && \
cmake -LH build && \
cmake --build build --clean-first --parallel=10 && \
cmake --install build
"""
start = { depends-on = ["clone", "build"] }
[system-requirements]
cuda = "12.0"
[dependencies]
fortran-compiler = ">=1.8.0,<2"
cxx-compiler = ">=1.8.0,<2"
cuda-compiler = ">=12.6.3,<13"
cmake = ">=3.31.2,<4"
pytorch = ">=2.5.1,<3"
zlib = ">=1.3.1,<2"
git = ">=2.47.1,<3"
sed = ">=4.8,<5"where pixi info:pixi list:However, Full build log:Similar to @q10 I'm not clear on if this is an issue with PyTorch (e.g. pytorch/pytorch#76082) or if there is a missing dependency that has escaped my attention. The project's |
|
Is the goal here simply to build PyTorch? If so, would recommend starting from the conda-forge PyTorch recipe Note that it needs quite a few patches to fix different build issues. For example this patch fixes an NVTX detection issue probably similar to the one encountered here |
@jakirkham No, PyTorch is just providing (My interest here is that I'm the editor for a JOSS paper and I'm trying to help both the reviewers and the authors reach an easily installable environment for testing, hence going to conda-forge.) |
|
Users are recommended to move to the NVTX 3 header-only component introduced in CUDA 10.0, which can be picked up by using the target Am wondering if there is some reason the newer NVTX is not used in this case In terms of the search path issue, one thing to try would be to pass If that doesn't work, we may need verbose info from CMake to debug further |
$ grep -r 'CUDA::nvToolsExt'
...
.pixi/envs/default/share/cmake/Caffe2/FindCUDAToolkit.cmake:- ``CUDA::nvToolsExt``
.pixi/envs/default/share/cmake/Caffe2/public/cuda.cmake: set_property(TARGET torch::nvtoolsext PROPERTY INTERFACE_LINK_LIBRARIES CUDA::nvToolsExt)
.pixi/envs/default/share/cmake-3.31/Modules/FindCUDAToolkit.cmake:- ``CUDA::nvToolsExt``
.pixi/envs/default/share/cmake-3.31/Modules/FindCUDAToolkit.cmake: if(TARGET CUDA::nvToolsExt AND CMAKE_MINIMUM_REQUIRED_VERSION VERSION_GREATER_EQUAL 3.25)
.pixi/envs/default/share/cmake-3.31/Modules/FindCUDAToolkit.cmake: set_property(TARGET CUDA::nvToolsExt PROPERTY DEPRECATION "nvToolsExt has been superseded by nvtx3 since CUDA 10.0 and CMake 3.25. Use CUDA::nvtx3 and include <nvtx3/nvToolsExt.h> instead.")so these are apparently coming from Caffe2 in PyTorch $ pixi list pytorch
Package Version Build Size Kind Source
pytorch 2.5.1 cuda126_mkl_py312_hdbe889e_308 35.6 MiB conda pytorchbut as to the why I don't know. Though from ...
# nvToolsExt
if(USE_SYSTEM_NVTX)
find_path(nvtx3_dir NAMES nvtx3)
else()
find_path(nvtx3_dir NAMES nvtx3 PATHS "${PROJECT_SOURCE_DIR}/third_party/NVTX/c/include" NO_DEFAULT_PATH)
endif()
find_package_handle_standard_args(nvtx3 DEFAULT_MSG nvtx3_dir)
if(nvtx3_FOUND)
add_library(torch::nvtx3 INTERFACE IMPORTED)
target_include_directories(torch::nvtx3 INTERFACE "${nvtx3_dir}")
target_compile_definitions(torch::nvtx3 INTERFACE TORCH_CUDA_USE_NVTX3)
else()
message(WARNING "Cannot find NVTX3, find old NVTX instead")
add_library(torch::nvtoolsext INTERFACE IMPORTED)
set_property(TARGET torch::nvtoolsext PROPERTY INTERFACE_LINK_LIBRARIES CUDA::nvToolsExt)
endif()
...(on Pytorch's GitHub I can only find this in I now see that I can set Looking for anything # find . -type f -iname "*nvtx3*"
$CONDA_PREFIX/targets/x86_64-linux/include/cub/detail/nvtx3.hppBuild log with warnings: |
|
Thanks Matthew! 🙏 Right think In terms of the warnings, think we are missing the Mentioning this as lines like Also there are a few warnings like these. Not sure whether FTorch actually uses these. If not, this may not matter Fixing these would depend on providing the corresponding CUDA packages and possibly setting some environment variables from here or here |
|
Thanks very much for your help, @jakirkham. I was missing [project]
authors = ["Matthew Feickert <[email protected]>"]
channels = ["conda-forge"]
description = "Add a short description here"
name = "example"
platforms = ["linux-64"]
version = "0.1.0"
[tasks]
# Selecting 6055a7dd35e394daff469673feac063f1bfaae01 as reference
clone = """
rm -rf FTorch && \
git clone https://github.com/Cambridge-ICCS/FTorch.git && \
cd FTorch && \
git reset --hard 6055a7dd35e394daff469673feac063f1bfaae01 && \
sed -i 's/cmake_minimum_required(VERSION 3.15 FATAL_ERROR)/cmake_minimum_required(VERSION 3.15...3.31 FATAL_ERROR)/g' src/CMakeLists.txt
"""
# Note that libtorch is at $CONDA_PREFIX/lib/libtorch.so
build = """
rm -rf build && \
cmake \
$CMAKE_ARGS \
-DCMAKE_INSTALL_PREFIX=$CONDA_PREFIX \
-DCMAKE_PREFIX_PATH=$CONDA_PREFIX \
-DCMAKE_BUILD_TYPE=Release \
-DPython_EXECUTABLE=$CONDA_PREFIX/bin/python \
-DENABLE_CUDA=TRUE \
-Dnvtx3_dir=$CONDA_PREFIX/targets/x86_64-linux/include/nvtx3 \
-DCMAKE_VERBOSE_MAKEFILE=TRUE \
-S FTorch/src/ \
-B build && \
cmake -LH build && \
cmake --build build --clean-first --parallel=10 && \
cmake --install build
"""
start = { depends-on = ["clone", "build"] }
[system-requirements]
cuda = "12.0"
[dependencies]
fortran-compiler = ">=1.5.2,<2"
cxx-compiler = ">=1.5.2,<2"
cuda-compiler = ">=12.6.3,<13"
cuda-libraries-dev = ">=12.6.3,<13"
cuda-nvtx-dev = ">=12.6.77,<13"
cmake = ">=3.31.2,<4"
pytorch = ">=2.5.1,<3"
zlib = ">=1.3.1,<2"
# build tools
git = ">=2.47.1,<3"
sed = ">=4.8,<5"
# examples
torchvision = ">=0.20.1,<0.21"
mpi4py = ">=4.0.1,<5"
openmpi-mpifort = ">=5.0.6,<6"now provides a working (still with warnings, but I think tolerable warnings) build. $ pixi list | grep 'python\|pytorch\|cuda'
cpython 3.12.8 py312hd8ed1ab_1 43.7 KiB conda cpython
cuda-cccl_linux-64 12.6.77 ha770c72_0 1 MiB conda cuda-cccl_linux-64
cuda-compiler 12.6.3 hbad6d8a_0 19.7 KiB conda cuda-compiler
cuda-crt-dev_linux-64 12.6.85 ha770c72_0 86.7 KiB conda cuda-crt-dev_linux-64
cuda-crt-tools 12.6.85 ha770c72_0 26.5 KiB conda cuda-crt-tools
cuda-cudart 12.6.77 h5888daf_0 21.9 KiB conda cuda-cudart
cuda-cudart-dev 12.6.77 h5888daf_0 22.3 KiB conda cuda-cudart-dev
cuda-cudart-dev_linux-64 12.6.77 h3f2d84a_0 356.8 KiB conda cuda-cudart-dev_linux-64
cuda-cudart-static 12.6.77 h5888daf_0 21.9 KiB conda cuda-cudart-static
cuda-cudart-static_linux-64 12.6.77 h3f2d84a_0 744.5 KiB conda cuda-cudart-static_linux-64
cuda-cudart_linux-64 12.6.77 h3f2d84a_0 184.2 KiB conda cuda-cudart_linux-64
cuda-cuobjdump 12.6.77 hbd13f7d_1 240.8 KiB conda cuda-cuobjdump
cuda-cupti 12.6.80 hbd13f7d_0 1.9 MiB conda cuda-cupti
cuda-cuxxfilt 12.6.77 hbd13f7d_1 210.9 KiB conda cuda-cuxxfilt
cuda-driver-dev 12.6.77 h5888daf_0 21.7 KiB conda cuda-driver-dev
cuda-driver-dev_linux-64 12.6.77 h3f2d84a_0 34.9 KiB conda cuda-driver-dev_linux-64
cuda-libraries-dev 12.6.3 ha770c72_0 19.6 KiB conda cuda-libraries-dev
cuda-nvcc 12.6.85 hcdd1206_0 23.1 KiB conda cuda-nvcc
cuda-nvcc-dev_linux-64 12.6.85 he91c749_0 10.8 MiB conda cuda-nvcc-dev_linux-64
cuda-nvcc-impl 12.6.85 h85509e4_0 24.9 KiB conda cuda-nvcc-impl
cuda-nvcc-tools 12.6.85 he02047a_0 23 MiB conda cuda-nvcc-tools
cuda-nvcc_linux-64 12.6.85 h04802cd_0 24.7 KiB conda cuda-nvcc_linux-64
cuda-nvdisasm 12.6.77 hbd13f7d_1 47.6 MiB conda cuda-nvdisasm
cuda-nvprune 12.6.77 hbd13f7d_1 66.4 KiB conda cuda-nvprune
cuda-nvrtc 12.6.85 hbd13f7d_0 17.3 MiB conda cuda-nvrtc
cuda-nvrtc-dev 12.6.85 h5888daf_0 31.5 KiB conda cuda-nvrtc-dev
cuda-nvtx 12.6.77 hbd13f7d_0 30.6 KiB conda cuda-nvtx
cuda-nvtx-dev 12.6.77 ha770c72_0 49.1 KiB conda cuda-nvtx-dev
cuda-nvvm-dev_linux-64 12.6.85 ha770c72_0 24.6 KiB conda cuda-nvvm-dev_linux-64
cuda-nvvm-impl 12.6.85 he02047a_0 7.7 MiB conda cuda-nvvm-impl
cuda-nvvm-tools 12.6.85 he02047a_0 10.4 MiB conda cuda-nvvm-tools
cuda-opencl 12.6.77 hbd13f7d_0 29.4 KiB conda cuda-opencl
cuda-opencl-dev 12.6.77 h5888daf_0 93 KiB conda cuda-opencl-dev
cuda-profiler-api 12.6.77 h7938cbb_0 22.3 KiB conda cuda-profiler-api
cuda-version 12.6 h7480c83_3 20.4 KiB conda cuda-version
libtorch 2.5.1 cuda126_mkl_h2a4acef_308 491.1 MiB conda libtorch
python 3.12.8 h9e4cc4f_1_cpython 30.1 MiB conda python
python_abi 3.12 5_cp312 6.1 KiB conda python_abi
pytorch 2.5.1 cuda126_mkl_py312_hdbe889e_308 35.6 MiB conda pytorch
torchvision 0.20.1 cuda126_py312_h17ccbaa_4 2.7 MiB conda torchvision
triton 3.1.0 cuda126py312h776fbae_4 89 MiB conda tritonFull build log:$ pixi clean && rm -rf pixi.lock && pixi run start
removed /tmp/example/.pixi/envs ✨ Pixi task (clone): rm -rf FTorch && git clone https://github.com/Cambridge-ICCS/FTorch.git && cd FTorch && git reset --hard 6055a7dd35e394daff469673feac063f1bfaae01 && sed -i 's/cmake_minimum_required(VERSION 3.15 FATAL_ERROR)/cmake_minimum_required(VERSION 3.15...3.31 FATAL_ERROR)/g' src/CMakeLists.txt
Cloning into 'FTorch'...
remote: Enumerating objects: 5458, done.
remote: Counting objects: 100% (1658/1658), done.
remote: Compressing objects: 100% (584/584), done.
remote: Total 5458 (delta 1347), reused 1116 (delta 1074), pack-reused 3800 (from 2)
Receiving objects: 100% (5458/5458), 5.94 MiB | 7.24 MiB/s, done.
Resolving deltas: 100% (4215/4215), done.
HEAD is now at 6055a7d Unit tests (using pFunit) (#76)
✨ Pixi task (build): rm -rf build && cmake $CMAKE_ARGS -DCMAKE_INSTALL_PREFIX=$CONDA_PREFIX -DCMAKE_PREFIX_PATH=$CONDA_PREFIX -DCMAKE_BUILD_TYPE=Release -DPython_EXECUTABLE=$CONDA_PREFIX/bin/python -DENABLE_CUDA=TRUE -Dnvtx3_dir=$CONDA_PREFIX/targets/x86_64-linux/include/nvtx3 -DCMAKE_VERBOSE_MAKEFILE=TRUE -S FTorch/src/ -B build && cmake -LH build && cmake --build build --clean-first --parallel=10 && cmake --install build
-- The C compiler identification is GNU 11.4.0
-- The CXX compiler identification is GNU 11.4.0
-- The Fortran compiler identification is GNU 11.4.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /tmp/example/.pixi/envs/default/bin/x86_64-conda-linux-gnu-cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /tmp/example/.pixi/envs/default/bin/x86_64-conda-linux-gnu-c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Detecting Fortran compiler ABI info
-- Detecting Fortran compiler ABI info - done
-- Check for working Fortran compiler: /tmp/example/.pixi/envs/default/bin/x86_64-conda-linux-gnu-gfortran - skipped
-- Detecting Fortran/C Interface
-- Detecting Fortran/C Interface - Found GLOBAL and MODULE mangling
-- Verifying Fortran/CXX Compiler Compatibility
-- Verifying Fortran/CXX Compiler Compatibility - Success
-- Looking for a CUDA compiler
-- Looking for a CUDA compiler - /tmp/example/.pixi/envs/default/bin/nvcc
-- The CUDA compiler identification is NVIDIA 12.6.85 with host compiler GNU 11.4.0
-- Detecting CUDA compiler ABI info
-- Detecting CUDA compiler ABI info - done
-- Check for working CUDA compiler: /tmp/example/.pixi/envs/default/bin/nvcc - skipped
-- Detecting CUDA compile features
-- Detecting CUDA compile features - done
-- Found ZLIB: /tmp/example/.pixi/envs/default/lib/libz.so (found version "1.3.1")
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- Caffe2: Found protobuf with new-style protobuf targets.
-- Caffe2: Protobuf version 28.3.0
-- Found CUDA: /tmp/example/.pixi/envs/default (found version "12.6")
-- Found CUDAToolkit: /tmp/example/.pixi/envs/default/targets/x86_64-linux/include (found version "12.6.85")
-- Caffe2: CUDA detected: 12.6
-- Caffe2: CUDA nvcc is: /tmp/example/.pixi/envs/default/bin/nvcc
-- Caffe2: CUDA toolkit directory: /tmp/example/.pixi/envs/default
-- Caffe2: Header version is: 12.6
-- Found Python: /tmp/example/.pixi/envs/default/bin/python (found version "3.12.8") found components: Interpreter
CMake Warning at /tmp/example/.pixi/envs/default/share/cmake/Caffe2/public/cuda.cmake:140 (message):
Failed to compute shorthash for libnvrtc.so
Call Stack (most recent call first):
/tmp/example/.pixi/envs/default/share/cmake/Caffe2/Caffe2Config.cmake:86 (include)
/tmp/example/.pixi/envs/default/share/cmake/Torch/TorchConfig.cmake:68 (find_package)
CMakeLists.txt:54 (find_package)
CMake Warning (dev) at /tmp/example/.pixi/envs/default/share/cmake-3.31/Modules/FindPackageHandleStandardArgs.cmake:441 (message):
The package name passed to `find_package_handle_standard_args` (nvtx3) does
not match the name of the calling package (Caffe2). This can lead to
problems in calling code that expects `find_package` result variables
(e.g., `_FOUND`) to follow a certain pattern.
Call Stack (most recent call first):
/tmp/example/.pixi/envs/default/share/cmake/Caffe2/public/cuda.cmake:178 (find_package_handle_standard_args)
/tmp/example/.pixi/envs/default/share/cmake/Caffe2/Caffe2Config.cmake:86 (include)
/tmp/example/.pixi/envs/default/share/cmake/Torch/TorchConfig.cmake:68 (find_package)
CMakeLists.txt:54 (find_package)
This warning is for project developers. Use -Wno-dev to suppress it.
-- Found nvtx3: /tmp/example/.pixi/envs/default/targets/x86_64-linux/include/nvtx3
-- USE_CUDNN is set to 0. Compiling without cuDNN support
-- USE_CUSPARSELT is set to 0. Compiling without cuSPARSELt support
-- USE_CUDSS is set to 0. Compiling without cuDSS support
-- USE_CUFILE is set to 0. Compiling without cuFile support
-- Autodetected CUDA architecture(s): 8.9
-- Added CUDA NVCC flags for: -gencode;arch=compute_89,code=sm_89
-- Found Torch: /tmp/example/.pixi/envs/default/lib/libtorch.so
-- Configuring done (6.5s)
-- Generating done (0.0s)
-- Build files have been written to: /tmp/example/build
-- Caffe2: Found protobuf with new-style protobuf targets.
-- Caffe2: Protobuf version 28.3.0
-- Caffe2: CUDA detected: 12.6
-- Caffe2: CUDA nvcc is: /tmp/example/.pixi/envs/default/bin/nvcc
-- Caffe2: CUDA toolkit directory: /tmp/example/.pixi/envs/default
-- Caffe2: Header version is: 12.6
CMake Warning at /tmp/example/.pixi/envs/default/share/cmake/Caffe2/public/cuda.cmake:140 (message):
Failed to compute shorthash for libnvrtc.so
Call Stack (most recent call first):
/tmp/example/.pixi/envs/default/share/cmake/Caffe2/Caffe2Config.cmake:86 (include)
/tmp/example/.pixi/envs/default/share/cmake/Torch/TorchConfig.cmake:68 (find_package)
CMakeLists.txt:54 (find_package)
CMake Warning (dev) at /tmp/example/.pixi/envs/default/share/cmake-3.31/Modules/FindPackageHandleStandardArgs.cmake:441 (message):
The package name passed to `find_package_handle_standard_args` (nvtx3) does
not match the name of the calling package (Caffe2). This can lead to
problems in calling code that expects `find_package` result variables
(e.g., `_FOUND`) to follow a certain pattern.
Call Stack (most recent call first):
/tmp/example/.pixi/envs/default/share/cmake/Caffe2/public/cuda.cmake:178 (find_package_handle_standard_args)
/tmp/example/.pixi/envs/default/share/cmake/Caffe2/Caffe2Config.cmake:86 (include)
/tmp/example/.pixi/envs/default/share/cmake/Torch/TorchConfig.cmake:68 (find_package)
CMakeLists.txt:54 (find_package)
This warning is for project developers. Use -Wno-dev to suppress it.
-- USE_CUDNN is set to 0. Compiling without cuDNN support
-- USE_CUSPARSELT is set to 0. Compiling without cuSPARSELt support
-- USE_CUDSS is set to 0. Compiling without cuDSS support
-- USE_CUFILE is set to 0. Compiling without cuFile support
-- Autodetected CUDA architecture(s): 8.9
-- Added CUDA NVCC flags for: -gencode;arch=compute_89,code=sm_89
-- Configuring done (1.1s)
-- Generating done (0.0s)
-- Build files have been written to: /tmp/example/build
-- Cache values
// Path to a library.
C10_CUDA_LIBRARY:FILEPATH=/tmp/example/.pixi/envs/default/lib/libc10_cuda.so
// Choose the type of build, options are: None Debug Release RelWithDebInfo MinSizeRel ...
CMAKE_BUILD_TYPE:STRING=Release
// CUDA architectures
CMAKE_CUDA_ARCHITECTURES:STRING=52
// Install path prefix, prepended onto install directories.
CMAKE_INSTALL_PREFIX:PATH=/tmp/example/.pixi/envs/default
// Host side compiler used by NVCC
CUDA_HOST_COMPILER:FILEPATH=/tmp/example/.pixi/envs/default/bin/x86_64-conda-linux-gnu-cc
// Path to a program.
CUDA_NVCC_EXECUTABLE:FILEPATH=/tmp/example/.pixi/envs/default/bin/nvcc
//
CUDA_NVRTC_LIB:FILEPATH=/tmp/example/.pixi/envs/default/lib/libnvrtc.so
// Path to a file.
CUDA_SDK_ROOT_DIR:PATH=CUDA_SDK_ROOT_DIR-NOTFOUND
// Toolkit location.
CUDA_TOOLKIT_ROOT_DIR:PATH=/tmp/example/.pixi/envs/default
// The directory containing a CMake configuration file for Caffe2.
Caffe2_DIR:PATH=/tmp/example/.pixi/envs/default/share/cmake/Caffe2
// The directory containing a CMake configuration file for MKLDNN.
MKLDNN_DIR:PATH=MKLDNN_DIR-NOTFOUND
// The directory containing a CMake configuration file for MKL.
MKL_DIR:PATH=MKL_DIR-NOTFOUND
// The directory containing a CMake configuration file for Protobuf.
Protobuf_DIR:PATH=/tmp/example/.pixi/envs/default/lib/cmake/protobuf
// Path to a library.
TORCH_LIBRARY:FILEPATH=/tmp/example/.pixi/envs/default/lib/libtorch.so
// The directory containing a CMake configuration file for Torch.
Torch_DIR:PATH=/tmp/example/.pixi/envs/default/share/cmake/Torch
// The directory containing a CMake configuration file for absl.
absl_DIR:PATH=/tmp/example/.pixi/envs/default/lib/cmake/absl
// Path to a library.
c10_LIBRARY:FILEPATH=/tmp/example/.pixi/envs/default/lib/libc10.so
// Path to a file.
nvtx3_dir:PATH=/tmp/example/.pixi/envs/default/targets/x86_64-linux/include/nvtx3
// The directory containing a CMake configuration file for utf8_range.
utf8_range_DIR:PATH=/tmp/example/.pixi/envs/default/lib/cmake/utf8_range
Change Dir: '/tmp/example/build'
Run Clean Command: /tmp/example/.pixi/envs/default/bin/cmake -E env VERBOSE=1 /usr/bin/gmake -f Makefile -j10 clean
/usr/bin/gmake -f CMakeFiles/Makefile2 clean
gmake[1]: Entering directory '/tmp/example/build'
/usr/bin/gmake -f CMakeFiles/ftorch.dir/build.make CMakeFiles/ftorch.dir/clean
gmake[2]: Entering directory '/tmp/example/build'
/tmp/example/.pixi/envs/default/bin/cmake -P CMakeFiles/ftorch.dir/cmake_clean.cmake
gmake[2]: Leaving directory '/tmp/example/build'
gmake[1]: Leaving directory '/tmp/example/build'
Run Build Command(s): /tmp/example/.pixi/envs/default/bin/cmake -E env VERBOSE=1 /usr/bin/gmake -f Makefile -j10
/tmp/example/.pixi/envs/default/bin/cmake -S/tmp/example/FTorch/src -B/tmp/example/build --check-build-system CMakeFiles/Makefile.cmake 0
/tmp/example/.pixi/envs/default/bin/cmake -E cmake_progress_start /tmp/example/build/CMakeFiles /tmp/example/build//CMakeFiles/progress.marks
/usr/bin/gmake -f CMakeFiles/Makefile2 all
gmake[1]: Entering directory '/tmp/example/build'
/usr/bin/gmake -f CMakeFiles/ftorch.dir/build.make CMakeFiles/ftorch.dir/depend
gmake[2]: Entering directory '/tmp/example/build'
cd /tmp/example/build && /tmp/example/.pixi/envs/default/bin/cmake -E cmake_depends "Unix Makefiles" /tmp/example/FTorch/src /tmp/example/FTorch/src /tmp/example/build /tmp/example/build /tmp/example/build/CMakeFiles/ftorch.dir/DependInfo.cmake "--color="
Dependee "/tmp/example/build/CMakeFiles/ftorch.dir/DependInfo.cmake" is newer than depender "/tmp/example/build/CMakeFiles/ftorch.dir/depend.internal".
Dependee "/tmp/example/build/CMakeFiles/CMakeDirectoryInformation.cmake" is newer than depender "/tmp/example/build/CMakeFiles/ftorch.dir/depend.internal".
Scanning dependencies of target ftorch
gmake[2]: Leaving directory '/tmp/example/build'
/usr/bin/gmake -f CMakeFiles/ftorch.dir/build.make CMakeFiles/ftorch.dir/build
gmake[2]: Entering directory '/tmp/example/build'
[ 25%] Building Fortran object CMakeFiles/ftorch.dir/ftorch.F90.o
/tmp/example/.pixi/envs/default/bin/x86_64-conda-linux-gnu-gfortran -DPROTOBUF_USE_DLLS -DUNIX -DUSE_C10D_GLOO -DUSE_C10D_NCCL -DUSE_DISTRIBUTED -DUSE_RPC -DUSE_TENSORPIPE -Dftorch_EXPORTS -I/tmp/example/build/modules -I/tmp/example/.pixi/envs/default/include -I/tmp/example/.pixi/envs/default/include/torch/csrc/api/include -I/tmp/example/.pixi/envs/default/targets/x86_64-linux/include -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /tmp/example/.pixi/envs/default/include -O3 -Jmodules -fPIC -D_GLIBCXX_USE_CXX11_ABI=1 -c /tmp/example/FTorch/src/ftorch.F90 -o CMakeFiles/ftorch.dir/ftorch.F90.o
[ 50%] Building Fortran object CMakeFiles/ftorch.dir/ftorch_test_utils.f90.o
[ 75%] Building CXX object CMakeFiles/ftorch.dir/ctorch.cpp.o
/tmp/example/.pixi/envs/default/bin/x86_64-conda-linux-gnu-gfortran -DPROTOBUF_USE_DLLS -DUNIX -DUSE_C10D_GLOO -DUSE_C10D_NCCL -DUSE_DISTRIBUTED -DUSE_RPC -DUSE_TENSORPIPE -Dftorch_EXPORTS -I/tmp/example/build/modules -I/tmp/example/.pixi/envs/default/include -I/tmp/example/.pixi/envs/default/include/torch/csrc/api/include -I/tmp/example/.pixi/envs/default/targets/x86_64-linux/include -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /tmp/example/.pixi/envs/default/include -O3 -Jmodules -fPIC -D_GLIBCXX_USE_CXX11_ABI=1 -c /tmp/example/FTorch/src/ftorch_test_utils.f90 -o CMakeFiles/ftorch.dir/ftorch_test_utils.f90.o
/tmp/example/.pixi/envs/default/bin/x86_64-conda-linux-gnu-c++ -DPROTOBUF_USE_DLLS -DUNIX -DUSE_C10D_GLOO -DUSE_C10D_NCCL -DUSE_DISTRIBUTED -DUSE_RPC -DUSE_TENSORPIPE -Dftorch_EXPORTS -I/tmp/example/build/modules -isystem /tmp/example/.pixi/envs/default/include/torch/csrc/api/include -fvisibility-inlines-hidden -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /tmp/example/.pixi/envs/default/include -I/tmp/example/.pixi/envs/default/targets/x86_64-linux/include -O3 -DNDEBUG -std=c++17 -fPIC -D_GLIBCXX_USE_CXX11_ABI=1 -MD -MT CMakeFiles/ftorch.dir/ctorch.cpp.o -MF CMakeFiles/ftorch.dir/ctorch.cpp.o.d -o CMakeFiles/ftorch.dir/ctorch.cpp.o -c /tmp/example/FTorch/src/ctorch.cpp
/tmp/example/.pixi/envs/default/bin/cmake -E cmake_copy_f90_mod modules/ftorch_test_utils.mod CMakeFiles/ftorch.dir/ftorch_test_utils.mod.stamp GNU
/tmp/example/.pixi/envs/default/bin/cmake -E touch CMakeFiles/ftorch.dir/ftorch_test_utils.f90.o.provides.build
/tmp/example/.pixi/envs/default/bin/cmake -E cmake_copy_f90_mod modules/ftorch.mod CMakeFiles/ftorch.dir/ftorch.mod.stamp GNU
/tmp/example/.pixi/envs/default/bin/cmake -E touch CMakeFiles/ftorch.dir/ftorch.F90.o.provides.build
[100%] Linking CXX shared library libftorch.so
/tmp/example/.pixi/envs/default/bin/cmake -E cmake_link_script CMakeFiles/ftorch.dir/link.txt --verbose=1
/tmp/example/.pixi/envs/default/bin/x86_64-conda-linux-gnu-c++ -fPIC -fvisibility-inlines-hidden -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /tmp/example/.pixi/envs/default/include -I/tmp/example/.pixi/envs/default/targets/x86_64-linux/include -O3 -DNDEBUG -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,--allow-shlib-undefined -Wl,-rpath,/tmp/example/.pixi/envs/default/lib -Wl,-rpath-link,/tmp/example/.pixi/envs/default/lib -L/tmp/example/.pixi/envs/default/lib -L/tmp/example/.pixi/envs/default/targets/x86_64-linux/lib -L/tmp/example/.pixi/envs/default/targets/x86_64-linux/lib/stubs -shared -Wl,-soname,libftorch.so -o libftorch.so CMakeFiles/ftorch.dir/ctorch.cpp.o CMakeFiles/ftorch.dir/ftorch.F90.o CMakeFiles/ftorch.dir/ftorch_test_utils.f90.o -L/lib/intel64 -L/lib/intel64_win -L/lib/win-x64 -Wl,-rpath,/lib/intel64:/lib/intel64_win:/lib/win-x64::::::::::::::: /tmp/example/.pixi/envs/default/lib/libtorch.so /tmp/example/.pixi/envs/default/lib/libc10.so /tmp/example/.pixi/envs/default/lib/libnvrtc.so /tmp/example/.pixi/envs/default/lib/libc10_cuda.so -Wl,--no-as-needed,"/tmp/example/.pixi/envs/default/lib/libtorch_cpu.so" -Wl,--as-needed /tmp/example/.pixi/envs/default/lib/libprotobuf.so.28.3.0 /tmp/example/.pixi/envs/default/lib/libabsl_log_internal_check_op.so.2407.0.0 /tmp/example/.pixi/envs/default/lib/libabsl_leak_check.so.2407.0.0 /tmp/example/.pixi/envs/default/lib/libabsl_die_if_null.so.2407.0.0 /tmp/example/.pixi/envs/default/lib/libabsl_log_internal_conditions.so.2407.0.0 /tmp/example/.pixi/envs/default/lib/libabsl_log_internal_message.so.2407.0.0 /tmp/example/.pixi/envs/default/lib/libabsl_log_internal_nullguard.so.2407.0.0 /tmp/example/.pixi/envs/default/lib/libabsl_examine_stack.so.2407.0.0 /tmp/example/.pixi/envs/default/lib/libabsl_log_internal_format.so.2407.0.0 /tmp/example/.pixi/envs/default/lib/libabsl_log_internal_proto.so.2407.0.0 /tmp/example/.pixi/envs/default/lib/libabsl_log_internal_log_sink_set.so.2407.0.0 /tmp/example/.pixi/envs/default/lib/libabsl_log_sink.so.2407.0.0 /tmp/example/.pixi/envs/default/lib/libabsl_log_entry.so.2407.0.0 /tmp/example/.pixi/envs/default/lib/libabsl_flags_internal.so.2407.0.0 /tmp/example/.pixi/envs/default/lib/libabsl_flags_marshalling.so.2407.0.0 /tmp/example/.pixi/envs/default/lib/libabsl_flags_reflection.so.2407.0.0 /tmp/example/.pixi/envs/default/lib/libabsl_flags_config.so.2407.0.0 /tmp/example/.pixi/envs/default/lib/libabsl_flags_program_name.so.2407.0.0 /tmp/example/.pixi/envs/default/lib/libabsl_flags_private_handle_accessor.so.2407.0.0 /tmp/example/.pixi/envs/default/lib/libabsl_flags_commandlineflag.so.2407.0.0 /tmp/example/.pixi/envs/default/lib/libabsl_flags_commandlineflag_internal.so.2407.0.0 /tmp/example/.pixi/envs/default/lib/libabsl_log_initialize.so.2407.0.0 /tmp/example/.pixi/envs/default/lib/libabsl_log_internal_globals.so.2407.0.0 /tmp/example/.pixi/envs/default/lib/libabsl_log_globals.so.2407.0.0 /tmp/example/.pixi/envs/default/lib/libabsl_vlog_config_internal.so.2407.0.0 /tmp/example/.pixi/envs/default/lib/libabsl_log_internal_fnmatch.so.2407.0.0 /tmp/example/.pixi/envs/default/lib/libabsl_raw_hash_set.so.2407.0.0 /tmp/example/.pixi/envs/default/lib/libabsl_hash.so.2407.0.0 /tmp/example/.pixi/envs/default/lib/libabsl_city.so.2407.0.0 /tmp/example/.pixi/envs/default/lib/libabsl_low_level_hash.so.2407.0.0 /tmp/example/.pixi/envs/default/lib/libabsl_hashtablez_sampler.so.2407.0.0 /tmp/example/.pixi/envs/default/lib/libabsl_random_distributions.so.2407.0.0 /tmp/example/.pixi/envs/default/lib/libabsl_random_seed_sequences.so.2407.0.0 /tmp/example/.pixi/envs/default/lib/libabsl_random_internal_pool_urbg.so.2407.0.0 /tmp/example/.pixi/envs/default/lib/libabsl_random_internal_randen.so.2407.0.0 /tmp/example/.pixi/envs/default/lib/libabsl_random_internal_randen_hwaes.so.2407.0.0 /tmp/example/.pixi/envs/default/lib/libabsl_random_internal_randen_hwaes_impl.so.2407.0.0 /tmp/example/.pixi/envs/default/lib/libabsl_random_internal_randen_slow.so.2407.0.0 /tmp/example/.pixi/envs/default/lib/libabsl_random_internal_platform.so.2407.0.0 /tmp/example/.pixi/envs/default/lib/libabsl_random_internal_seed_material.so.2407.0.0 /tmp/example/.pixi/envs/default/lib/libabsl_random_seed_gen_exception.so.2407.0.0 /tmp/example/.pixi/envs/default/lib/libabsl_statusor.so.2407.0.0 /tmp/example/.pixi/envs/default/lib/libabsl_status.so.2407.0.0 /tmp/example/.pixi/envs/default/lib/libabsl_cord.so.2407.0.0 /tmp/example/.pixi/envs/default/lib/libabsl_cordz_info.so.2407.0.0 /tmp/example/.pixi/envs/default/lib/libabsl_cord_internal.so.2407.0.0 /tmp/example/.pixi/envs/default/lib/libabsl_cordz_functions.so.2407.0.0 /tmp/example/.pixi/envs/default/lib/libabsl_exponential_biased.so.2407.0.0 /tmp/example/.pixi/envs/default/lib/libabsl_cordz_handle.so.2407.0.0 /tmp/example/.pixi/envs/default/lib/libabsl_crc_cord_state.so.2407.0.0 /tmp/example/.pixi/envs/default/lib/libabsl_crc32c.so.2407.0.0 /tmp/example/.pixi/envs/default/lib/libabsl_crc_internal.so.2407.0.0 /tmp/example/.pixi/envs/default/lib/libabsl_crc_cpu_detect.so.2407.0.0 /tmp/example/.pixi/envs/default/lib/libabsl_bad_optional_access.so.2407.0.0 /tmp/example/.pixi/envs/default/lib/libabsl_strerror.so.2407.0.0 /tmp/example/.pixi/envs/default/lib/libabsl_str_format_internal.so.2407.0.0 /tmp/example/.pixi/envs/default/lib/libabsl_synchronization.so.2407.0.0 /tmp/example/.pixi/envs/default/lib/libabsl_stacktrace.so.2407.0.0 /tmp/example/.pixi/envs/default/lib/libabsl_symbolize.so.2407.0.0 /tmp/example/.pixi/envs/default/lib/libabsl_debugging_internal.so.2407.0.0 /tmp/example/.pixi/envs/default/lib/libabsl_demangle_internal.so.2407.0.0 /tmp/example/.pixi/envs/default/lib/libabsl_demangle_rust.so.2407.0.0 /tmp/example/.pixi/envs/default/lib/libabsl_decode_rust_punycode.so.2407.0.0 /tmp/example/.pixi/envs/default/lib/libabsl_utf8_for_code_point.so.2407.0.0 /tmp/example/.pixi/envs/default/lib/libabsl_graphcycles_internal.so.2407.0.0 /tmp/example/.pixi/envs/default/lib/libabsl_kernel_timeout_internal.so.2407.0.0 /tmp/example/.pixi/envs/default/lib/libabsl_malloc_internal.so.2407.0.0 /tmp/example/.pixi/envs/default/lib/libabsl_time.so.2407.0.0 /tmp/example/.pixi/envs/default/lib/libabsl_civil_time.so.2407.0.0 /tmp/example/.pixi/envs/default/lib/libabsl_time_zone.so.2407.0.0 /tmp/example/.pixi/envs/default/lib/libabsl_bad_variant_access.so.2407.0.0 /tmp/example/.pixi/envs/default/lib/libabsl_strings.so.2407.0.0 /tmp/example/.pixi/envs/default/lib/libabsl_int128.so.2407.0.0 /tmp/example/.pixi/envs/default/lib/libabsl_strings_internal.so.2407.0.0 /tmp/example/.pixi/envs/default/lib/libabsl_string_view.so.2407.0.0 /tmp/example/.pixi/envs/default/lib/libabsl_base.so.2407.0.0 /tmp/example/.pixi/envs/default/lib/libabsl_spinlock_wait.so.2407.0.0 -lpthread /tmp/example/.pixi/envs/default/lib/libabsl_throw_delegate.so.2407.0.0 /tmp/example/.pixi/envs/default/lib/libabsl_raw_logging_internal.so.2407.0.0 /tmp/example/.pixi/envs/default/lib/libabsl_log_severity.so.2407.0.0 -Wl,--no-as-needed,"/tmp/example/.pixi/envs/default/lib/libtorch_cuda.so" -Wl,--as-needed /tmp/example/.pixi/envs/default/lib/libc10_cuda.so /tmp/example/.pixi/envs/default/lib/libc10.so /tmp/example/.pixi/envs/default/lib/libcudart.so -Wl,--no-as-needed,"/tmp/example/.pixi/envs/default/lib/libtorch.so" -Wl,--as-needed -lgfortran -lquadmath
gmake[2]: Leaving directory '/tmp/example/build'
[100%] Built target ftorch
gmake[1]: Leaving directory '/tmp/example/build'
/tmp/example/.pixi/envs/default/bin/cmake -E cmake_progress_start /tmp/example/build/CMakeFiles 0
-- Install configuration: "Release"
-- Installing: /tmp/example/.pixi/envs/default/lib/libftorch.so
-- Set non-toolchain portion of runtime path of "/tmp/example/.pixi/envs/default/lib/libftorch.so" to "$ORIGIN/../lib:/lib/intel64:/lib/intel64_win:/lib/win-x64"
-- Installing: /tmp/example/.pixi/envs/default/include/ctorch.h
-- Installing: /tmp/example/.pixi/envs/default/lib/cmake/FTorch/FTorchConfig.cmake
-- Installing: /tmp/example/.pixi/envs/default/lib/cmake/FTorch/FTorchConfig-release.cmake
-- Installing: /tmp/example/.pixi/envs/default/include/ftorch/ftorch.mod
-- Installing: /tmp/example/.pixi/envs/default/include/ftorch/ftorch_test_utils.modSo I think I have everything I need now. 👍 @q10, I'm not sure if your questions are resolved now, but I would try and try again while setting |
|
Great, thanks Matthew! 🙏 The FWIW the equivalent of installing the entire CUDA Toolkit (except the driver) would be |
|
Hm. Curiously, the environment I arrived at in #59 (comment) seems brittle. I had run this under ReproducerIf I start from #59 (comment) and modify it in the following ways:
This results in the following pixi.toml:[project]
authors = ["Matthew Feickert <[email protected]>"]
channels = ["conda-forge"]
description = "Add a short description here"
name = "example"
platforms = ["linux-64"]
version = "0.1.0"
[tasks]
# Selecting 03c5475b6d05bf2c90cc1cd572187ef4b5946b3e as reference
clone = """
rm -rf FTorch && \
git clone https://github.com/Cambridge-ICCS/FTorch.git && \
cd FTorch && \
git reset --hard 03c5475b6d05bf2c90cc1cd572187ef4b5946b3e
"""
# Note that libtorch is at $CONDA_PREFIX/lib/libtorch.so
build = """
rm -rf build && \
cmake \
$CMAKE_ARGS \
-DCMAKE_INSTALL_PREFIX=$CONDA_PREFIX \
-DCMAKE_PREFIX_PATH=$CONDA_PREFIX \
-DCMAKE_BUILD_TYPE=Release \
-DPython_EXECUTABLE=$CONDA_PREFIX/bin/python \
-DENABLE_CUDA=TRUE \
-Dnvtx3_dir=$CONDA_PREFIX/targets/x86_64-linux/include/nvtx3 \
-DCMAKE_VERBOSE_MAKEFILE=TRUE \
-S FTorch/src/ \
-B build && \
cmake -LH build && \
cmake --build build --clean-first --parallel=10 && \
cmake --install build
"""
start = { depends-on = ["clone", "build"] }
[system-requirements]
cuda = "12.0"
[dependencies]
fortran-compiler = ">=1.5.2,<2"
cxx-compiler = ">=1.5.2,<2"
cuda-compiler = ">=12.6.3,<13"
cuda-libraries-dev = ">=12.6.3,<13"
cuda-nvtx-dev = ">=12.6.77,<13"
cuda-version = "12.6.*"
python = ">=3.12.8,<3.13"
cmake = ">=3.31.2,<4"
pytorch = ">=2.5.1,<3"
zlib = ">=1.3.1,<2"
# build tools
git = ">=2.47.1,<3"which in terms of a conda would be name: default
channels:
- conda-forge
- nodefaults
dependencies:
- fortran-compiler >=1.5.2,<2
- cxx-compiler >=1.5.2,<2
- cuda-compiler >=12.6.3,<13
- cuda-libraries-dev >=12.6.3,<13
- cuda-nvtx-dev >=12.6.77,<13
- cuda-version 12.6.*
- python >=3.12.8,<3.13
- cmake >=3.31.2,<4
- pytorch >=2.5.1,<3
- zlib >=1.3.1,<2
- git >=2.47.1,<3Full build log:$ pixi clean && rm -rf pixi.lock && pixi run start
removed /tmp/example/.pixi/envs ✨ Pixi task (clone): rm -rf FTorch && git clone https://github.com/Cambridge-ICCS/FTorch.git && cd FTorch && git reset --hard 03c5475b6d05bf2c90cc1cd572187ef4b5946b3e
Cloning into 'FTorch'...
remote: Enumerating objects: 5966, done.
remote: Counting objects: 100% (1653/1653), done.
remote: Compressing objects: 100% (701/701), done.
remote: Total 5966 (delta 1304), reused 1019 (delta 948), pack-reused 4313 (from 2)
Receiving objects: 100% (5966/5966), 6.40 MiB | 4.05 MiB/s, done.
Resolving deltas: 100% (4683/4683), done.
HEAD is now at 03c5475 Avoid double negatives in the run tests shell script to make things clearer.
✨ Pixi task (build): rm -rf build && cmake $CMAKE_ARGS -DCMAKE_INSTALL_PREFIX=$CONDA_PREFIX -DCMAKE_PREFIX_PATH=$CONDA_PREFIX -DCMAKE_BUILD_TYPE=Release -DPython_EXECUTABLE=$CONDA_PREFIX/bin/python -DENABLE_CUDA=TRUE -Dnvtx3_dir=$CONDA_PREFIX/targets/x86_64-linux/include/nvtx3 -DCMAKE_VERBOSE_MAKEFILE=TRUE -S FTorch/src/ -B build && cmake -LH build && cmake --build build --clean-first --parallel=10 && cmake --install build
-- The C compiler identification is GNU 13.3.0
-- The CXX compiler identification is GNU 13.3.0
-- The Fortran compiler identification is GNU 13.3.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /tmp/example/.pixi/envs/default/bin/x86_64-conda-linux-gnu-cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /tmp/example/.pixi/envs/default/bin/x86_64-conda-linux-gnu-c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Detecting Fortran compiler ABI info
-- Detecting Fortran compiler ABI info - done
-- Check for working Fortran compiler: /tmp/example/.pixi/envs/default/bin/x86_64-conda-linux-gnu-gfortran - skipped
-- Detecting Fortran/C Interface
-- Detecting Fortran/C Interface - Found GLOBAL and MODULE mangling
-- Verifying Fortran/CXX Compiler Compatibility
-- Verifying Fortran/CXX Compiler Compatibility - Success
-- Looking for a CUDA compiler
-- Looking for a CUDA compiler - /tmp/example/.pixi/envs/default/bin/nvcc
-- The CUDA compiler identification is NVIDIA 12.6.85 with host compiler GNU 13.3.0
-- Detecting CUDA compiler ABI info
-- Detecting CUDA compiler ABI info - done
-- Check for working CUDA compiler: /tmp/example/.pixi/envs/default/bin/nvcc - skipped
-- Detecting CUDA compile features
-- Detecting CUDA compile features - done
-- Found ZLIB: /tmp/example/.pixi/envs/default/lib/libz.so (found version "1.3.1")
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- Caffe2: Found protobuf with new-style protobuf targets.
-- Caffe2: Protobuf version 28.3.0
-- Could NOT find CUDA (missing: CUDA_INCLUDE_DIRS) (found version "12.6")
CMake Warning at /tmp/example/.pixi/envs/default/share/cmake/Caffe2/public/cuda.cmake:31 (message):
Caffe2: CUDA cannot be found. Depending on whether you are building Caffe2
or a Caffe2 dependent library, the next warning / error will give you more
info.
Call Stack (most recent call first):
/tmp/example/.pixi/envs/default/share/cmake/Caffe2/Caffe2Config.cmake:86 (include)
/tmp/example/.pixi/envs/default/share/cmake/Torch/TorchConfig.cmake:68 (find_package)
CMakeLists.txt:54 (find_package)
CMake Error at /tmp/example/.pixi/envs/default/share/cmake/Caffe2/Caffe2Config.cmake:90 (message):
Your installed Caffe2 version uses CUDA but I cannot find the CUDA
libraries. Please set the proper CUDA prefixes and / or install CUDA.
Call Stack (most recent call first):
/tmp/example/.pixi/envs/default/share/cmake/Torch/TorchConfig.cmake:68 (find_package)
CMakeLists.txt:54 (find_package)
-- Configuring incomplete, errors occurred!NotesIn terms of (what I thought was) the relevant subsection of the environment pixi list for python pytorch and cuda:$ pixi list | grep 'python\|pytorch\|cuda'
cpython 3.12.8 py312hd8ed1ab_1 43.7 KiB conda cpython
cuda-cccl_linux-64 12.6.77 ha770c72_0 1 MiB conda cuda-cccl_linux-64
cuda-compiler 12.6.3 hbad6d8a_0 19.7 KiB conda cuda-compiler
cuda-crt-dev_linux-64 12.6.85 ha770c72_0 86.7 KiB conda cuda-crt-dev_linux-64
cuda-crt-tools 12.6.85 ha770c72_0 26.5 KiB conda cuda-crt-tools
cuda-cudart 12.6.77 h5888daf_0 21.9 KiB conda cuda-cudart
cuda-cudart-dev 12.6.77 h5888daf_0 22.3 KiB conda cuda-cudart-dev
cuda-cudart-dev_linux-64 12.6.77 h3f2d84a_0 356.8 KiB conda cuda-cudart-dev_linux-64
cuda-cudart-static 12.6.77 h5888daf_0 21.9 KiB conda cuda-cudart-static
cuda-cudart-static_linux-64 12.6.77 h3f2d84a_0 744.5 KiB conda cuda-cudart-static_linux-64
cuda-cudart_linux-64 12.6.77 h3f2d84a_0 184.2 KiB conda cuda-cudart_linux-64
cuda-cuobjdump 12.6.77 hbd13f7d_1 240.8 KiB conda cuda-cuobjdump
cuda-cupti 12.6.80 hbd13f7d_0 1.9 MiB conda cuda-cupti
cuda-cuxxfilt 12.6.77 hbd13f7d_1 210.9 KiB conda cuda-cuxxfilt
cuda-driver-dev 12.6.77 h5888daf_0 21.7 KiB conda cuda-driver-dev
cuda-driver-dev_linux-64 12.6.77 h3f2d84a_0 34.9 KiB conda cuda-driver-dev_linux-64
cuda-libraries-dev 12.6.3 ha770c72_0 19.6 KiB conda cuda-libraries-dev
cuda-nvcc 12.6.85 hcdd1206_0 23.1 KiB conda cuda-nvcc
cuda-nvcc-dev_linux-64 12.6.85 he91c749_0 10.8 MiB conda cuda-nvcc-dev_linux-64
cuda-nvcc-impl 12.6.85 h85509e4_0 24.9 KiB conda cuda-nvcc-impl
cuda-nvcc-tools 12.6.85 he02047a_0 23 MiB conda cuda-nvcc-tools
cuda-nvcc_linux-64 12.6.85 h04802cd_0 24.7 KiB conda cuda-nvcc_linux-64
cuda-nvdisasm 12.6.77 hbd13f7d_1 47.6 MiB conda cuda-nvdisasm
cuda-nvprune 12.6.77 hbd13f7d_1 66.4 KiB conda cuda-nvprune
cuda-nvrtc 12.6.85 hbd13f7d_0 17.3 MiB conda cuda-nvrtc
cuda-nvrtc-dev 12.6.85 h5888daf_0 31.5 KiB conda cuda-nvrtc-dev
cuda-nvtx 12.6.77 hbd13f7d_0 30.6 KiB conda cuda-nvtx
cuda-nvtx-dev 12.6.77 ha770c72_0 49.1 KiB conda cuda-nvtx-dev
cuda-nvvm-dev_linux-64 12.6.85 ha770c72_0 24.6 KiB conda cuda-nvvm-dev_linux-64
cuda-nvvm-impl 12.6.85 he02047a_0 7.7 MiB conda cuda-nvvm-impl
cuda-nvvm-tools 12.6.85 he02047a_0 10.4 MiB conda cuda-nvvm-tools
cuda-opencl 12.6.77 hbd13f7d_0 29.4 KiB conda cuda-opencl
cuda-opencl-dev 12.6.77 h5888daf_0 93 KiB conda cuda-opencl-dev
cuda-profiler-api 12.6.77 h7938cbb_0 22.3 KiB conda cuda-profiler-api
cuda-version 12.6 h7480c83_3 20.4 KiB conda cuda-version
libtorch 2.5.1 cuda126_mkl_haa0cf67_310 491.5 MiB conda libtorch
python 3.12.8 h9e4cc4f_1_cpython 30.1 MiB conda python
python_abi 3.12 5_cp312 6.1 KiB conda python_abi
pytorch 2.5.1 cuda126_mkl_py312_hdbe889e_310 26.1 MiB conda pytorch
triton 3.1.0 cuda126py312h776fbae_5 89 MiB conda tritonthe only relevant differences (beyond that I'm not including $ diff -u pixi_list_2025-01-08.txt pixi_list_2025-01-28.txt
--- pixi_list_2025-01-08.txt 2025-01-28 23:41:01.092714388 -0700
+++ pixi_list_2025-01-28.txt 2025-01-28 23:39:48.732605805 -0700
@@ -33,9 +33,8 @@
cuda-opencl-dev 12.6.77 h5888daf_0 93 KiB conda cuda-opencl-dev
cuda-profiler-api 12.6.77 h7938cbb_0 22.3 KiB conda cuda-profiler-api
cuda-version 12.6 h7480c83_3 20.4 KiB conda cuda-version
-libtorch 2.5.1 cuda126_mkl_h2a4acef_308 491.1 MiB conda libtorch
+libtorch 2.5.1 cuda126_mkl_haa0cf67_310 491.5 MiB conda libtorch
python 3.12.8 h9e4cc4f_1_cpython 30.1 MiB conda python
python_abi 3.12 5_cp312 6.1 KiB conda python_abi
-pytorch 2.5.1 cuda126_mkl_py312_hdbe889e_308 35.6 MiB conda pytorch
-torchvision 0.20.1 cuda126_py312_h17ccbaa_4 2.7 MiB conda torchvision
-triton 3.1.0 cuda126py312h776fbae_4 89 MiB conda triton
+pytorch 2.5.1 cuda126_mkl_py312_hdbe889e_310 26.1 MiB conda pytorch
+triton 3.1.0 cuda126py312h776fbae_5 89 MiB conda tritonEven if I constrain the build numbers to be the same $ pixi add "pytorch [build_number=308]" "triton [build_number=4]"
✔ Added pytorch [build_number=308]
✔ Added triton [build_number=4]$ diff -u pixi_list_2025-01-08.txt pixi_list_2025-01-28_pinned.txt
--- pixi_list_2025-01-08.txt 2025-01-28 23:41:01.092714388 -0700
+++ pixi_list_2025-01-28_pinned.txt 2025-01-28 23:56:10.779856268 -0700
@@ -37,5 +37,4 @@
python 3.12.8 h9e4cc4f_1_cpython 30.1 MiB conda python
python_abi 3.12 5_cp312 6.1 KiB conda python_abi
pytorch 2.5.1 cuda126_mkl_py312_hdbe889e_308 35.6 MiB conda pytorch
-torchvision 0.20.1 cuda126_py312_h17ccbaa_4 2.7 MiB conda torchvision
triton 3.1.0 cuda126py312h776fbae_4 89 MiB conda tritonthe same build error happens. Taking the error message at face value, and noting comment #59 (comment), then $ grep -r "find_package(CUDAToolkit)"
$ grep -r "find_package(CUDA)"
.pixi/envs/default/share/cmake/Caffe2/Modules_CUDA_fix/upstream/FindCUDA.cmake:# use this module or call ``find_package(CUDA)``. This module
.pixi/envs/default/share/cmake/Caffe2/public/cuda.cmake:find_package(CUDA)
.pixi/envs/default/share/cmake-3.31/Modules/FindCUDA.cmake:It is no longer necessary to use this module or call ``find_package(CUDA)``
.pixi/envs/default/share/cmake-3.31/Help/policy/CMP0146.rst:The ``OLD`` behavior of this policy is for ``find_package(CUDA)`` toindicates that Caffe2 will check the legacy given $ find .pixi/envs/default/ -type f -name nvcc
.pixi/envs/default/bin/nvcchas no effect. Same goes for It seems that I've gotten myself back to where I started a few weeks ago, and I'm not really sure how to debug this particular problem. @jakirkham, do you happen to have any suggestions for basic debugging steps that I've missed so far? |
Ah, seems that I had the wrong value of 👍 So, for the purposes of Cambridge-ICCS/FTorch#214: Python and CUDA version constrained environmentpixi manifest pixi.toml:[project]
authors = ["Matthew Feickert <[email protected]>"]
channels = ["conda-forge"]
description = "Add a short description here"
name = "example"
platforms = ["linux-64"]
version = "0.1.0"
[tasks]
# Selecting 03c5475b6d05bf2c90cc1cd572187ef4b5946b3e as reference
clone = """
rm -rf FTorch && \
git clone https://github.com/Cambridge-ICCS/FTorch.git && \
cd FTorch && \
git reset --hard 03c5475b6d05bf2c90cc1cd572187ef4b5946b3e
"""
# Note that libtorch is at $CONDA_PREFIX/lib/libtorch.so
build = """
rm -rf build && \
cmake \
$CMAKE_ARGS \
-DCMAKE_INSTALL_PREFIX=$CONDA_PREFIX \
-DCMAKE_PREFIX_PATH=$CONDA_PREFIX \
-DCMAKE_BUILD_TYPE=Release \
-DPython_EXECUTABLE=$CONDA_PREFIX/bin/python \
-DENABLE_CUDA=TRUE \
-DCUDA_TOOLKIT_ROOT_DIR=$CONDA_PREFIX/targets/x86_64-linux \
-Dnvtx3_dir=$CONDA_PREFIX/targets/x86_64-linux/include/nvtx3 \
-DCMAKE_VERBOSE_MAKEFILE=TRUE \
-S FTorch/src/ \
-B build && \
cmake -LH build && \
cmake --build build --clean-first --parallel=10 && \
cmake --install build
"""
start = { depends-on = ["clone", "build"] }
[system-requirements]
cuda = "12.0"
[dependencies]
fortran-compiler = ">=1.5.2,<2"
cxx-compiler = ">=1.5.2,<2"
cuda-compiler = ">=12.6.3,<13"
cuda-libraries-dev = ">=12.6.3,<13"
cuda-nvtx-dev = ">=12.6.77,<13"
cuda-version = "12.6.*"
python = ">=3.12.8,<3.13"
cmake = ">=3.31.2,<4"
pytorch = ">=2.5.1,<3"
zlib = ">=1.3.1,<2"
# build tools
git = ">=2.47.1,<3"
# examples
torchvision = ">=0.20.1,<0.21"
mpi4py = ">=4.0.1,<5"
openmpi-mpifort = ">=5.0.6,<6"where gives name: default
channels:
- conda-forge
- nodefaults
dependencies:
- fortran-compiler >=1.5.2,<2
- cxx-compiler >=1.5.2,<2
- cuda-compiler >=12.6.3,<13
- cuda-libraries-dev >=12.6.3,<13
- cuda-nvtx-dev >=12.6.77,<13
- cuda-version 12.6.*
- python >=3.12.8,<3.13
- cmake >=3.31.2,<4
- pytorch >=2.5.1,<3
- zlib >=1.3.1,<2
- git >=2.47.1,<3
- torchvision >=0.20.1,<0.21
- mpi4py >=4.0.1,<5
- openmpi-mpifort >=5.0.6,<6unconstrained environmentpixi manifest pixi.toml:[project]
authors = ["Matthew Feickert <[email protected]>"]
channels = ["conda-forge"]
description = "Add a short description here"
name = "example"
platforms = ["linux-64"]
version = "0.1.0"
[tasks]
# Selecting 03c5475b6d05bf2c90cc1cd572187ef4b5946b3e as reference
clone = """
rm -rf FTorch && \
git clone https://github.com/Cambridge-ICCS/FTorch.git && \
cd FTorch && \
git reset --hard 03c5475b6d05bf2c90cc1cd572187ef4b5946b3e
"""
# Note that libtorch is at $CONDA_PREFIX/lib/libtorch.so
build = """
rm -rf build && \
cmake \
$CMAKE_ARGS \
-DCMAKE_INSTALL_PREFIX=$CONDA_PREFIX \
-DCMAKE_PREFIX_PATH=$CONDA_PREFIX \
-DCMAKE_BUILD_TYPE=Release \
-DPython_EXECUTABLE=$CONDA_PREFIX/bin/python \
-DENABLE_CUDA=TRUE \
-DCUDA_TOOLKIT_ROOT_DIR=$CONDA_PREFIX/targets/x86_64-linux \
-Dnvtx3_dir=$CONDA_PREFIX/targets/x86_64-linux/include/nvtx3 \
-DCMAKE_VERBOSE_MAKEFILE=TRUE \
-S FTorch/src/ \
-B build && \
cmake -LH build && \
cmake --build build --clean-first --parallel=10 && \
cmake --install build
"""
start = { depends-on = ["clone", "build"] }
[system-requirements]
cuda = "12.0"
[dependencies]
fortran-compiler = ">=1.5.2,<2"
cxx-compiler = ">=1.5.2,<2"
cuda-compiler = ">=12.6.3,<13"
cuda-libraries-dev = ">=12.6.3,<13"
cuda-nvtx-dev = ">=12.6.77,<13"
cmake = ">=3.31.2,<4"
pytorch = ">=2.5.1,<3"
zlib = ">=1.3.1,<2"
# build tools
git = ">=2.47.1,<3"
# examples
torchvision = ">=0.20.1,<0.21"
mpi4py = ">=4.0.1,<5"
openmpi-mpifort = ">=5.0.6,<6"where gives name: default
channels:
- conda-forge
- nodefaults
dependencies:
- fortran-compiler >=1.5.2,<2
- cxx-compiler >=1.5.2,<2
- cuda-compiler >=12.6.3,<13
- cuda-libraries-dev >=12.6.3,<13
- cuda-nvtx-dev >=12.6.77,<13
- cmake >=3.31.2,<4
- pytorch >=2.5.1,<3
- zlib >=1.3.1,<2
- git >=2.47.1,<3
- torchvision >=0.20.1,<0.21
- mpi4py >=4.0.1,<5
- openmpi-mpifort >=5.0.6,<6 |
4 tasks
1 task
That's a weakness of github's search not checking I'm noticing the failures with pytorch's CMake metadata now, perhaps newer CMake got stricter about something (because this didn't use to fail before)? I'm open to a patch in pytorch that replaces |
Thanks for noting this, @h-vetinari. I hadn't noticed that these got skipped. That's useful to know.
Unclear(?). The errors that I noticed where in the last weeks I needed to add
Thanks for also making useful notes RE: Yes, I suspect that it won't be trivially easy. I've subscribed to those issues, so if there's an opportunity to help I will try, but given my current timeline for the week that probably won't be in the next 4 days. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Solution to issue cannot be found in the documentation.
Issue
This is a followup to conda-forge/cuda-nvcc-feedstock#56 and #58. Unlike the other two issues, all packages have been installed using the conda-forge channel.
We are trying to build FBGEMM against the CUDA 12.6 (pytorch/FBGEMM#3503), and are running into the following error:
Contrary to conda-forge/cuda-nvcc-feedstock#56, we are now installing CUDA using the conda-forge channel, i.e.
We did notice that for CUDA 12.6, when we look for
cuda.h, we see:likewise with CUDA 12.4, we have:
Note that if we install CUDA 12.4 using the nvidia channel for example, i.e.:
then we get:
In other words, when installing using conda-forge, the CUDA headers do not appear to be installed onto
$CONDA_PREFIX/include, which seems to be the reason that cmake is unable to find the path.How do we resolve this?
Installed packages
Environment info
(build_env) bash-5.2# conda info active environment : build_env active env location : /root/miniconda/envs/build_env shell level : 2 user config file : /root/.condarc populated config files : /root/miniconda/.condarc conda version : 24.11.1 conda-build version : not installed python version : 3.12.8.final.0 solver : libmamba (default) virtual packages : __archspec=1=skylake __conda=24.11.1=0 __glibc=2.34=0 __linux=5.15.49=0 __unix=0=0 base environment : /root/miniconda (writable) conda av data dir : /root/miniconda/etc/conda conda av metadata url : None channel URLs : https://repo.anaconda.com/pkgs/main/linux-64 https://repo.anaconda.com/pkgs/main/noarch https://repo.anaconda.com/pkgs/r/linux-64 https://repo.anaconda.com/pkgs/r/noarch package cache : /root/miniconda/pkgs /root/.conda/pkgs envs directories : /root/miniconda/envs /root/.conda/envs platform : linux-64 user-agent : conda/24.11.1 requests/2.32.3 CPython/3.12.8 Linux/5.15.49-linuxkit amzn/2023.6.20241121 glibc/2.34 solver/libmamba conda-libmamba-solver/24.9.0 libmambapy/1.5.11 aau/0.5.0 c/. s/. e/. UID:GID : 0:0 netrc file : None offline mode : FalseThe text was updated successfully, but these errors were encountered: