Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CI] optimze bazel compile guidance #175

Merged
merged 1 commit into from
Oct 10, 2023
Merged

[CI] optimze bazel compile guidance #175

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .bazeliskrc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
USE_BAZEL_VERSION=5.3.1
USE_BAZEL_VERSION=5.0.0
4 changes: 2 additions & 2 deletions .bazelrc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,9 +15,9 @@ build --host_cxxopt=-std=c++17
build:cuda --crosstool_top=@local_config_cuda//crosstool:toolchain

# CUDA options
build:cuda --action_env GCC_HOST_COMPILER_PATH="/opt/rh/devtoolset-9/root/usr/bin/gcc"
build:cuda --action_env GCC_HOST_COMPILER_PATH="/usr/bin/gcc"
build:cuda --action_env CUDA_TOOLKIT_PATH="/usr/local/cuda"
build:cuda --action_env CUDA_VERSION="11"
build:cuda --action_env CUDNN_VERSION="8"
build:cuda --action_env CUDNN_INSTALL_PATH="/usr/local/cuda"
build:cuda --action_env CUDNN_INSTALL_PATH="/usr/"
build:cuda --action_env CUDA_COMPUTE_CAPABILITIES="7.5"
27 changes: 21 additions & 6 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -158,6 +158,13 @@ and also open for public contributions, bug fixes, and documentation. [[Contribu

Basically, HierarchicalKV is a headers only library, the commands below only create binaries for benchmark and unit testing.

Your environment must meet the following requirements:

- CUDA version >= 11.2
- NVIDIA GPU with compute capability 8.0, 8.6, 8.7 or 9.0
- GCC supports `C++17' standard or later.
- Bazel version >= 3.7.2 (Bazel compile only)

### with cmake
```shell
git clone --recursive https://github.com/NVIDIA-Merlin/HierarchicalKV.git
Expand All @@ -176,20 +183,28 @@ For Unit Test:
```

### with bazel

- DON'T use the option of `--recursive` for `git clone`.
- Please modify the environment variables in the `.bazelrc` file in advance if using the customized docker images.
- The docker images maintained on `nvcr.io/nvidia/tensorflow` are highly recommended.

Pull the docker image:
```shell
git clone --recursive https://github.com/NVIDIA-Merlin/HierarchicalKV.git
cd HierarchicalKV && bazel build --config=cuda //...
docker pull nvcr.io/nvidia/tensorflow:22.09-tf2-py3
docker run --gpus all -it --rm nvcr.io/nvidia/tensorflow:22.09-tf2-py3
```

Compile in docker container:
```shell
git clone https://github.com/NVIDIA-Merlin/HierarchicalKV.git
cd HierarchicalKV && bash bazel_build.sh
```

For Benchmark:
```shell
./benchmark_util
```

Your environment must meet the following requirements:

- CUDA version >= 11.2
- NVIDIA GPU with compute capability 8.0, 8.6, 8.7 or 9.0

## Benchmark & Performance(W.I.P)

Expand Down
40 changes: 25 additions & 15 deletions build_deps/gpus/find_cuda_config.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -306,9 +306,11 @@ def _find_cublas_config(base_paths, required_version, cuda_version):
if _at_least_version(cuda_version, "10.1"):

def get_header_version(path):
version = (_get_header_version(path, name)
for name in ("CUBLAS_VER_MAJOR", "CUBLAS_VER_MINOR",
"CUBLAS_VER_PATCH"))
version = (v for v in (_get_header_version(path, name)
for name in ("CUBLAS_VER_MAJOR",
"CUBLAS_VER_MINOR",
"CUBLAS_VER_PATCH",
"CUBLAS_VER_BUILD")) if v != "")
return ".".join(version)

header_path, header_version = _find_header(base_paths, "cublas_api.h",
Expand Down Expand Up @@ -338,9 +340,11 @@ def _find_cusolver_config(base_paths, required_version, cuda_version):
if _at_least_version(cuda_version, "11.0"):

def get_header_version(path):
version = (_get_header_version(path, name)
for name in ("CUSOLVER_VER_MAJOR", "CUSOLVER_VER_MINOR",
"CUSOLVER_VER_PATCH"))
version = (v for v in (_get_header_version(path, name)
for name in ("CUSOLVER_VER_MAJOR",
"CUSOLVER_VER_MINOR",
"CUSOLVER_VER_PATCH",
"CUSOLVER_VER_BUILD")) if v != "")
return ".".join(version)

header_path, header_version = _find_header(base_paths,
Expand Down Expand Up @@ -369,9 +373,11 @@ def _find_curand_config(base_paths, required_version, cuda_version):
if _at_least_version(cuda_version, "11.0"):

def get_header_version(path):
version = (_get_header_version(path, name)
for name in ("CURAND_VER_MAJOR", "CURAND_VER_MINOR",
"CURAND_VER_PATCH"))
version = (v for v in (_get_header_version(path, name)
for name in ("CURAND_VER_MAJOR",
"CURAND_VER_MINOR",
"CURAND_VER_PATCH",
"CURAND_VER_BUILD")) if v != "")
return ".".join(version)

header_path, header_version = _find_header(base_paths, "curand.h",
Expand All @@ -398,9 +404,11 @@ def _find_cufft_config(base_paths, required_version, cuda_version):
if _at_least_version(cuda_version, "11.0"):

def get_header_version(path):
version = (_get_header_version(path, name)
for name in ("CUFFT_VER_MAJOR", "CUFFT_VER_MINOR",
"CUFFT_VER_PATCH"))
version = (v for v in (_get_header_version(path, name)
for name in ("CUFFT_VER_MAJOR",
"CUFFT_VER_MINOR",
"CUFFT_VER_PATCH",
"CUFFT_VER_BUILD")) if v != "")
return ".".join(version)

header_path, header_version = _find_header(base_paths, "cufft.h",
Expand Down Expand Up @@ -451,9 +459,11 @@ def _find_cusparse_config(base_paths, required_version, cuda_version):
if _at_least_version(cuda_version, "11.0"):

def get_header_version(path):
version = (_get_header_version(path, name)
for name in ("CUSPARSE_VER_MAJOR", "CUSPARSE_VER_MINOR",
"CUSPARSE_VER_PATCH"))
version = (v for v in (_get_header_version(path, name)
for name in ("CUSPARSE_VER_MAJOR",
"CUSPARSE_VER_MINOR",
"CUSPARSE_VER_PATCH",
"CUSPARSE_VER_BUILD")) if v != "")
return ".".join(version)

header_path, header_version = _find_header(base_paths, "cusparse.h",
Expand Down