-
-
Notifications
You must be signed in to change notification settings - Fork 5.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
CI: add TSAN to the sanitizers pipelines #42444
Merged
Merged
Changes from all commits
Commits
Show all changes
6 commits
Select commit
Hold shift + click to select a range
9913683
Setup CI for TSAN
tkf 6ed9b7f
Don't build sysimage with TSAN
tkf a09febb
in the Buildkite cache, don't cache the "registries" directory in the…
tkf b92db8b
Merge branch 'master' into tkf/tsan-ci
tkf f22d188
We only run the `tsan` job on Julia 1.8 and later
tkf ee11eb4
Use build target julia-src-debug
tkf File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,16 @@ | ||
| TOOLCHAIN=$(BUILDROOT)/../toolchain | ||
| BINDIR=$(TOOLCHAIN)/usr/bin | ||
| TOOLDIR=$(TOOLCHAIN)/usr/tools | ||
|
|
||
| # use our new toolchain | ||
| USECLANG=1 | ||
| override CC=$(BINDIR)/clang | ||
| override CXX=$(TOOLDIR)/clang++ | ||
|
|
||
| USE_BINARYBUILDER_LLVM=1 | ||
|
|
||
| override SANITIZE=1 | ||
| override SANITIZE_THREAD=1 | ||
|
|
||
| # default to a debug build for better line number reporting | ||
| override JULIA_BUILD_MODE=debug | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,61 @@ | ||
| #!/bin/bash | ||
| # This file is a part of Julia. License is MIT: https://julialang.org/license | ||
|
|
||
| # | ||
| # Usage: | ||
| # contrib/tsan/build.sh <path> [<make_targets>...] | ||
| # | ||
| # Build TSAN-enabled julia. Given a workspace directory <path>, build | ||
| # TSAN-enabled julia in <path>/tsan. Required toolss are install under | ||
| # <path>/toolchain. Note that the same <path> passed to `contrib/asan/build.sh` | ||
| # can be used to share the toolchain used for ASAN. This scripts also takes | ||
| # optional <make_targets> arguments which are passed to `make`. The default | ||
| # make target is `debug`. | ||
|
|
||
| set -ue | ||
|
|
||
| # `$WORKSPACE` is a directory in which we create `toolchain` and `tsan` | ||
| # sub-directories. | ||
| WORKSPACE="$1" | ||
| shift | ||
| if [ "$WORKSPACE" = "" ]; then | ||
| echo "Workspace directory must be specified as the first argument" >&2 | ||
| exit 2 | ||
| fi | ||
|
|
||
| mkdir -pv "$WORKSPACE" | ||
| WORKSPACE="$(cd "$WORKSPACE" && pwd)" | ||
| if [ "$WORKSPACE" = "" ]; then | ||
| echo "Failed to create the workspace directory." >&2 | ||
| exit 2 | ||
| fi | ||
|
|
||
| HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" | ||
| JULIA_HOME="$HERE/../../" | ||
|
|
||
| echo | ||
| echo "Installing toolchain..." | ||
|
|
||
| TOOLCHAIN="$WORKSPACE/toolchain" | ||
| if [ ! -d "$TOOLCHAIN" ]; then | ||
| make -C "$JULIA_HOME" configure O=$TOOLCHAIN | ||
| cp "$HERE/../asan/Make.user.tools" "$TOOLCHAIN/Make.user" | ||
| fi | ||
|
|
||
| make -C "$TOOLCHAIN/deps" install-clang install-llvm-tools | ||
|
|
||
| # TODO: https://github.com/JuliaPackaging/Yggdrasil/issues/3359 | ||
| rm "$TOOLCHAIN/usr/tools/clang++" | ||
| ln -s "$TOOLCHAIN/usr/bin/clang" "$TOOLCHAIN/usr/tools/clang++" | ||
|
|
||
| echo | ||
| echo "Building Julia..." | ||
|
|
||
| BUILD="$WORKSPACE/tsan" | ||
| if [ ! -d "$BUILD" ]; then | ||
| make -C "$JULIA_HOME" configure O="$BUILD" | ||
| cp "$HERE/Make.user.tsan" "$BUILD/Make.user" | ||
| fi | ||
|
|
||
| cd "$BUILD" # so that we can pass `-C src` to `make` | ||
| make "$@" |
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This won't work. You need to build LLVM with tsan instrumentation since it uses locks.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we still use a normal-build LLVM with
deadlock:llvmin theTSAN_OPTIONS=suppressions=...configuration? Or there's no way to work around this fundamentally?If we need LLVM with TSAN, I guess we'd need to start adding it in BB before trying this in CI. But it'd be a much bigger project than this quick PR.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah I don't want to build LLVM from source in the Base Julia CI. We'd need to have an "LLVM with TSAN" JLL available from Ygg.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It might be possible, but I'd probably recommend against it. I don't know if that can introduce false positives.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
OK, so maybe it makes sense to close this PR until a JLL is ready.
Or, maybe we can still just run the build process (
-C src), so that some rudimentary bugs can be caught?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think a
LLVM_tsan_jllmakes the most sense. Then it only needs to be built very occasionally in Ygg, instead of building LLVM from source on every PR that is opened to JuliaLang/julia.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't really want to support "yet" another version of LLVM in Yggdrasil and slow the update cycle down even more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hmmm. That's a fair point. Let's see if @staticfloat has any ideas.
At least we can keep the first part, which doesn't require building LLVM from source.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What's the source of slowness in yet another LLVM JLL vs ccache? Aren't they equivalent in terms of machine time? Is it that, with the JLL approach, there need to be humans involved in the process waiting for the JLL and updating the URL/version/checksum? If that's the case, it can also potentially be solved by more automation?
I kinda liked the JLL approach since it'd make it easier for more casual users to try sanitizers. OTOH, I guess the ccache approach would be more hackable for core devs since it'd give us a "healthy" CI'ed build script that is easy to flip some LLVM build options. Maybe the upside of the ccache approach wins, since there's no single set of build options that covers all the needs when you debugging things.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, the maintenance of the LLVM JLL is a lot of human time, some of it waiting, some of it making sure that all the version number bits line up. Sheparding things through to the registry.
I made attempts at trying to automate it, but currently it is pareto optimal in that I find is really annoying, but not annoying enough to descend to arcane wizardy. c.f JuliaPackaging/Yggdrasil#726 (comment)