triton build failing to access https://tritonlang.blob.core.windows.net/llvm-builds/ with HTTP Error 409

[lisaong]

This was working earlier, so recently happened.

PublicAccessNotPermitted Public access is not permitted on this storage account. RequestId:372d399d-901e-0036-4086-ef20cc000000 Time:2024-08-16T02:46:44.4328589Z

[] #186 27.76 Building wheels for collected packages: triton
[] #186 27.76   Building wheel for triton (pyproject.toml): started
[] #186 28.67   error: subprocess-exited-with-error
[] #186 28.67   
[] #186 28.67   × Building wheel for triton (pyproject.toml) did not run successfully.
[] #186 28.67   │ exit code: 1
[] #186 28.67   ╰─> [7 lines of output]
[] #186 28.67       running bdist_wheel
[] #186 28.67       running build
[] #186 28.67       running build_py
[] #186 28.67       running build_ext
[] #186 28.67       downloading and extracting https://github.com/pybind/pybind11/archive/refs/tags/v2.11.1.tar.gz ...
[] #186 28.67       downloading and extracting https://tritonlang.blob.core.windows.net/llvm-builds/llvm-49af6502-ubuntu-x64.tar.gz ...
[] #186 28.67       error: HTTP Error 409: Public access is not permitted on this storage account.
[] #186 28.67       [end of output]

[prarit]

I'm seeing this too. The link to the tarball is now restricted.

[prarit]

In case anyone is looking into this: I've tried accessing https://tritonlang.blob.core.windows.net via an internal redhat.com network, a private network and via an online anonymizer. It looks like the site is locked down for some reason and access to the tarball is restricted as indicated in the initial report by @lisaong

[prarit]

Is there a way to provide a 1-time download for the tarball? Perhaps through a direct commit to the triton repo itself?

[atalman]

We are also seeing this when trying to build triton in pytorch: https://github.com/pytorch/pytorch/actions/runs/10421322991/job/28863284428

cc @ptillet @Jokeren

[jayfurmanek]

That URL was changed back in June:
06e6799

Perhaps torch is on an older commit?

[ytaous]

The url is till blocked - any updates?

[TheTrustedComputer]

I also faced this HTTP 409 error when building PyTorch 2.4 + ROCm. I think this problem of Triton not building points to AOTriton (AMD's fork) using the old upstream commit containing the now broken URL.

[fei-xx]

I'm also seeing the similar issue. Any updates?

[cantor-set]

This is still an issue, AOTriton still can not be build

[prarit]

For anyone attempting to build aotriton from ROCm. aotriton references a DIFFERENT repository than triton-lang, and I have filed an issue here: ROCm#631

[hyhuang00]

That URL was changed back in June: 06e6799

Perhaps torch is on an older commit?

Thank you! Using the new URL solved this problem.

[prarit]

@jayfurmanek torch is indeed on an older commit. Is there any plan to provide the older llvm-49af6502-ubuntu-x64.tar.gz tarball at the new oatriton download link? If not, is there a way I and others could get a copy?

[vaenyr]

For anyone who might be in a similar situation:

I've run into the same issue recently when trying to build PT for ROCm from source. I've tried a couple of ideas, including upgrading LLVM in AMD's triton fork, etc., all in vain. In the end what worked well for me was to follow the Building with custom LLVM part of README.

1. Clone and checkout LLVM to commit 49af6502c6dcb4a7f7520178bd14df396f78240c.
2. Build with AMDGPU support
3. set the env variables as explained in the readme (I used export ...)
4. build pytorch as usual

No more HTTP 409/missing blobs/version mismatch problems - everything went smoothly.

[prarit]

Thanks. I had actually stumbled across this today and was trying it out :) It seems like it works -- I'm just worried that the AMD llvm may have additional flags we're not compiling with. Hopefully someone from the AMD ROCm team can confirm the above instructions are valid.

[TheTrustedComputer]

I also had mixed results on building Triton (upstream commit cb3d79a185e40c9d8a579bea07747a8a8d157d52 and AMD's triton commit 9b73a543a5545960bcaf2830900b0560eec443c5) with this particular LLVM commit (49af6502c6dcb4a7f7520178bd14df396f78240c). I seem to cannot get past this error installing the Python package:

CMake Error at CMakeLists.txt:61 (find_package):
Found package configuration file:

    /root/triton-llvm-build/lib/cmake/mlir/MLIRConfig.cmake

but it set MLIR_FOUND to FALSE so package "MLIR" is considered to be NOT
FOUND.  Reason given by package:

The following imported targets are referenced, but are missing:
LLVMNVPTXCodeGen LLVMNVPTXDesc LLVMNVPTXInfo



-- Configuring incomplete, errors occurred

I built LLVM with CMake using the following flags:

-B /root/triton-llvm-build \
-DCMAKE_BUILD_TYPE=Release \
-DLLVM_DEFAULT_TARGET_TRIPLE="x86_64-pc-linux-gnu" \
-DLLVM_TARGETS_TO_BUILD="AMDGPU;NVPTX;Native" \
-DLLVM_ENABLE_PROJECTS="mlir" \
-DLLVM_ENABLE_ASSERTIONS=ON \
-DLLVM_CCACHE_BUILD=ON \
-G Ninja

The find command shows these targets are present, despite what CMake says.

find /root/triton-llvm-build -name "*LLVMNVPTXCodeGen*"

/root/triton-llvm-build/lib/Target/NVPTX/CMakeFiles/LLVMNVPTXCodeGen.dir
/root/triton-llvm-build/lib/libLLVMNVPTXCodeGen.a

find /root/triton-llvm-build -name "*LLVMNVPTXDesc*"

/root/triton-llvm-build/lib/Target/NVPTX/MCTargetDesc/CMakeFiles/LLVMNVPTXDesc.dir
/root/triton-llvm-build/lib/libLLVMNVPTXDesc.a

find /root/triton-llvm-build -name "*LLVMNVPTXInfo*"

/root/triton-llvm-build/lib/Target/NVPTX/TargetInfo/CMakeFiles/LLVMNVPTXInfo.dir
/root/triton-llvm-build/lib/libLLVMNVPTXInfo.a

Environment variables used:

LLVM_LIBRARY_DIR=/root/triton-llvm-build/lib
LLVM_INCLUDE_DIRS=/root/triton-llvm-build/include
LLVM_SYSPATH=/root/triton-llvm-build

[pramenku]

https://tritonlang.blob.core.windows.net/llvm-builds/llvm-ed4e505c-ubuntu-x64.tar.gz, this link is working again which was not working earlier.

$ wget https://tritonlang.blob.core.windows.net/llvm-builds/llvm-ed4e505c-ubuntu-x64.tar.gz
--2024-08-27 03:25:50-- https://tritonlang.blob.core.windows.net/llvm-builds/llvm-ed4e505c-ubuntu-x64.tar.gz
Resolving tritonlang.blob.core.windows.net (tritonlang.blob.core.windows.net)... 20.47.62.100, 20.47.62.112, 20.47.62.60
Connecting to tritonlang.blob.core.windows.net (tritonlang.blob.core.windows.net)|20.47.62.100|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 977588953 (932M) [application/x-tar]
Saving to: ‘llvm-ed4e505c-ubuntu-x64.tar.gz’

llvm-ed4e505c-ubuntu-x64.tar.gz 0%[ ] 31.63K 11.9KB/s

[TheTrustedComputer]

I can also confirm the links are working again.