Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Enzyme inexact error #1294

Closed
swilliamson7 opened this issue Feb 14, 2024 · 12 comments
Closed

Enzyme inexact error #1294

swilliamson7 opened this issue Feb 14, 2024 · 12 comments

Comments

@swilliamson7
Copy link
Collaborator

With or without the checkpointing lines in this script I'm getting the following error when trying to run autodiff:

julia> include("run_checkpointing.jl")
WARNING: replacing module ShallowWaters.
ERROR: LoadError: InexactError: check_top_bit(UInt32, -1)
Stacktrace:
  [1] throw_inexacterror(f::Symbol, ::Type{UInt32}, val::Int32)
    @ Core ./boot.jl:634
  [2] check_top_bit
    @ Core ./boot.jl:648 [inlined]
  [3] toUInt32
    @ Core ./boot.jl:747 [inlined]
  [4] UInt32
    @ Core ./boot.jl:788 [inlined]
  [5] Integer
    @ CEnum ~/.julia/packages/CEnum/B6ahP/src/CEnum.jl:19 [inlined]
  [6] cconvert
    @ CEnum ~/.julia/packages/CEnum/B6ahP/src/CEnum.jl:20 [inlined]
  [7] LLVMAddCallSiteAttribute(C::LLVM.CallInst, Idx::LLVM.API.__JL_Ctag_85, A::LLVM.StringAttribute)
    @ LLVM.API ~/.julia/packages/LLVM/ycL7C/lib/15/libLLVM.jl:2536
  [8] check_ir!(job::GPUCompiler.CompilerJob{Enzyme.Compiler.EnzymeTarget, Enzyme.Compiler.EnzymeCompilerParams}, errors::Vector{Tuple{…}}, imported::Set{String}, inst::LLVM.CallInst, calls::Vector{Any})
    @ Enzyme.Compiler ~/.julia/packages/Enzyme/KJgKj/src/compiler/validation.jl:515
  [9] check_ir!(job::GPUCompiler.CompilerJob{Enzyme.Compiler.EnzymeTarget, Enzyme.Compiler.EnzymeCompilerParams}, errors::Vector{Tuple{String, Vector{…}, Any}}, imported::Set{String}, f::LLVM.Function)
    @ Enzyme.Compiler ~/.julia/packages/Enzyme/KJgKj/src/compiler/validation.jl:208
 [10] check_ir!(job::GPUCompiler.CompilerJob{Enzyme.Compiler.EnzymeTarget, Enzyme.Compiler.EnzymeCompilerParams}, errors::Vector{Tuple{String, Vector{Base.StackTraces.StackFrame}, Any}}, mod::LLVM.Module)
    @ Enzyme.Compiler ~/.julia/packages/Enzyme/KJgKj/src/compiler/validation.jl:178
 [11] check_ir
    @ ~/.julia/packages/Enzyme/KJgKj/src/compiler/validation.jl:157 [inlined]
 [12] codegen(output::Symbol, job::GPUCompiler.CompilerJob{…}; libraries::Bool, deferred_codegen::Bool, optimize::Bool, toplevel::Bool, strip::Bool, validate::Bool, only_entry::Bool, parent_job::Nothing)
    @ Enzyme.Compiler ~/.julia/packages/Enzyme/KJgKj/src/compiler.jl:4384
 [13] codegen
    @ ~/.julia/packages/Enzyme/KJgKj/src/compiler.jl:4348 [inlined]
 [14] _thunk(job::GPUCompiler.CompilerJob{Enzyme.Compiler.EnzymeTarget, Enzyme.Compiler.EnzymeCompilerParams}, postopt::Bool) (repeats 2 times)
    @ Enzyme.Compiler ~/.julia/packages/Enzyme/KJgKj/src/compiler.jl:5362
 [15] cached_compilation
    @ ~/.julia/packages/Enzyme/KJgKj/src/compiler.jl:5396 [inlined]
 [16] (::Enzyme.Compiler.var"#509#510"{DataType, DataType, DataType, Enzyme.API.CDerivativeMode, Tuple{Bool, Bool}, Int64, Bool, Bool, UInt64, DataType})(ctx::LLVM.Context)
    @ Enzyme.Compiler ~/.julia/packages/Enzyme/KJgKj/src/compiler.jl:5462
 [17] JuliaContext(f::Enzyme.Compiler.var"#509#510"{DataType, DataType, DataType, Enzyme.API.CDerivativeMode, Tuple{Bool, Bool}, Int64, Bool, Bool, UInt64, DataType})
    @ GPUCompiler ~/.julia/packages/GPUCompiler/U36Ed/src/driver.jl:47
 [18] #s1056#508
    @ ~/.julia/packages/Enzyme/KJgKj/src/compiler.jl:5414 [inlined]
 [19] 
    @ Enzyme.Compiler ./none:0
 [20] (::Core.GeneratedFunctionStub)(::UInt64, ::LineNumberNode, ::Any, ::Vararg{Any})
    @ Core ./boot.jl:602
 [21] autodiff
    @ Enzyme ~/.julia/packages/Enzyme/KJgKj/src/Enzyme.jl:209 [inlined]
 [22] autodiff(mode::ReverseMode{true, FFIABI}, f::Const{typeof(checkpoint_function)}, args::Duplicated{Main.ShallowWaters.ModelSetup{Float32, Float32}})
    @ Enzyme ~/.julia/packages/Enzyme/KJgKj/src/Enzyme.jl:238
 [23] autodiff
    @ ~/.julia/packages/Enzyme/KJgKj/src/Enzyme.jl:224 [inlined]
 [24] run_checkpointing()
    @ Main ~/Documents/GitHub/ShallowWaters.jl/my_scripts/run_checkpointing.jl:357
 [25] macro expansion
    @ ./timing.jl:279 [inlined]
 [26] top-level scope
    @ ~/Documents/GitHub/ShallowWaters.jl/my_scripts/run_checkpointing.jl:269
 [27] include(fname::String)
    @ Base.MainInclude ./client.jl:489
 [28] top-level scope
    @ REPL[8]:1
in expression starting at /Users/swilliamson/Documents/GitHub/ShallowWaters.jl/my_scripts/run_checkpointing.jl:403
Some type information was truncated. Use `show(err)` to see complete types.

I had this happen on Enzyme v0.11.14, v0.11.15, and v0.11.15#main. The code ran on Enzyme v0.11.14 based on my Github history
@swilliamson7 swilliamson7 changed the title Enzyme error on previously working code Enzyme inexact error Feb 14, 2024
@wsmoses
Copy link
Member

wsmoses commented Feb 14, 2024

Yeah LLVM.jl had an accidentally breaking release (eg no Enzyme code change caused). This should be fixed in the latest release though?

@swilliamson7
Copy link
Collaborator Author

swilliamson7 commented Feb 14, 2024 •
edited
Loading

I can cause this error on Enzyme v0.11.4 and LLVM v6.5.0, I'm not sure which LLVM had the problem though. Edit to add that it also happens on Enzyme v0.11.15 and LLVM v6.5.0

@wsmoses
Copy link
Member

wsmoses commented Feb 14, 2024

You need enzyme 11.15 which was released yesterday with this fix

@swilliamson7
Copy link
Collaborator Author

I tried with both 0.11.5 and 0.11.5#main and it still seemed to happen

@wsmoses
Copy link
Member

wsmoses commented Feb 14, 2024

Can you paste the result of st

@swilliamson7
Copy link
Collaborator Author 

Status `~/Documents/GitHub/ShallowWaters.jl/my_scripts/Project.toml`
  [eb46d486] Checkpointing v0.8.3 `https://github.com/Argonne-National-Laboratory/Checkpointing.jl.git#main`
  [35d6a980] ColorSchemes v3.24.0
  [717857b8] DSP v0.7.9
  [7da242da] Enzyme v0.11.15 `https://github.com/EnzymeAD/Enzyme.jl.git#main`
  [7a1cc6ca] FFTW v1.8.0
  [a98d9a8b] Interpolations v0.15.1
  [033835bb] JLD2 v0.4.45
  [929cbde3] LLVM v6.5.0
  [b964fa9f] LaTeXStrings v1.3.1
  [30363a11] NetCDF v0.11.7
  [d96e819e] Parameters v0.12.3
  [91a5bcdd] Plots v1.40.1
  [90137ffa] StaticArrays v1.9.2
  [ade2ca70] Dates
  [de0858da] Printf
  [2f01184e] SparseArrays v1.10.0

@wsmoses
Copy link
Member

wsmoses commented Feb 14, 2024 via email

@swilliamson7
Copy link
Collaborator Author 

(my_scripts) pkg> st
Status `~/Documents/GitHub/ShallowWaters.jl/my_scripts/Project.toml`
  [eb46d486] Checkpointing v0.8.3 `https://github.com/Argonne-National-Laboratory/Checkpointing.jl.git#main`
  [35d6a980] ColorSchemes v3.24.0
  [717857b8] DSP v0.7.9
  [7da242da] Enzyme v0.11.15
  [7a1cc6ca] FFTW v1.8.0
  [a98d9a8b] Interpolations v0.15.1
  [033835bb] JLD2 v0.4.45
  [929cbde3] LLVM v6.5.0
  [b964fa9f] LaTeXStrings v1.3.1
  [30363a11] NetCDF v0.11.7
  [d96e819e] Parameters v0.12.3
  [91a5bcdd] Plots v1.40.1
  [90137ffa] StaticArrays v1.9.2
  [ade2ca70] Dates
  [de0858da] Printf
  [2f01184e] SparseArrays v1.10.0

julia> include("run_checkpointing.jl")
WARNING: replacing module ShallowWaters.
[ Info: Revolve: Number of checkpoints: 9
[ Info: Revolve: Number of steps: 88
[ Info: Prediction:
[ Info: Forward steps   : 198
[ Info: Overhead factor : 2.2247191011235956
ERROR: LoadError: InexactError: check_top_bit(UInt32, -1)
Stacktrace:
  [1] throw_inexacterror(f::Symbol, ::Type{UInt32}, val::Int32)
    @ Core ./boot.jl:634
  [2] check_top_bit
    @ Core ./boot.jl:648 [inlined]
  [3] toUInt32
    @ Core ./boot.jl:747 [inlined]
  [4] UInt32
    @ Core ./boot.jl:788 [inlined]
  [5] Integer
    @ CEnum ~/.julia/packages/CEnum/B6ahP/src/CEnum.jl:19 [inlined]
  [6] cconvert
    @ CEnum ~/.julia/packages/CEnum/B6ahP/src/CEnum.jl:20 [inlined]
  [7] LLVMAddCallSiteAttribute(C::LLVM.CallInst, Idx::LLVM.API.__JL_Ctag_85, A::LLVM.StringAttribute)
    @ LLVM.API ~/.julia/packages/LLVM/ycL7C/lib/15/libLLVM.jl:2536
  [8] check_ir!(job::GPUCompiler.CompilerJob{…}, errors::Vector{…}, imported::Set{…}, inst::LLVM.CallInst, calls::Vector{…})
    @ Enzyme.Compiler ~/.julia/packages/Enzyme/KJgKj/src/compiler/validation.jl:515
  [9] check_ir!(job::GPUCompiler.CompilerJob{…}, errors::Vector{…}, imported::Set{…}, f::LLVM.Function)
    @ Enzyme.Compiler ~/.julia/packages/Enzyme/KJgKj/src/compiler/validation.jl:208
 [10] check_ir!(job::GPUCompiler.CompilerJob{Enzyme.Compiler.EnzymeTarget, Enzyme.Compiler.EnzymeCompilerParams}, errors::Vector{Tuple{…}}, mod::LLVM.Module)
    @ Enzyme.Compiler ~/.julia/packages/Enzyme/KJgKj/src/compiler/validation.jl:178
 [11] check_ir
    @ ~/.julia/packages/Enzyme/KJgKj/src/compiler/validation.jl:157 [inlined]
 [12] codegen(output::Symbol, job::GPUCompiler.CompilerJob{…}; libraries::Bool, deferred_codegen::Bool, optimize::Bool, toplevel::Bool, strip::Bool, validate::Bool, only_entry::Bool, parent_job::Nothing)
    @ Enzyme.Compiler ~/.julia/packages/Enzyme/KJgKj/src/compiler.jl:4384
 [13] codegen
    @ ~/.julia/packages/Enzyme/KJgKj/src/compiler.jl:4348 [inlined]
 [14] _thunk(job::GPUCompiler.CompilerJob{Enzyme.Compiler.EnzymeTarget, Enzyme.Compiler.EnzymeCompilerParams}, postopt::Bool) (repeats 2 times)
    @ Enzyme.Compiler ~/.julia/packages/Enzyme/KJgKj/src/compiler.jl:5362
 [15] cached_compilation
    @ ~/.julia/packages/Enzyme/KJgKj/src/compiler.jl:5396 [inlined]
 [16] (::Enzyme.Compiler.var"#509#510"{DataType, DataType, DataType, Enzyme.API.CDerivativeMode, Tuple{…}, Int64, Bool, Bool, UInt64, DataType})(ctx::LLVM.Context)
    @ Enzyme.Compiler ~/.julia/packages/Enzyme/KJgKj/src/compiler.jl:5462
 [17] JuliaContext(f::Enzyme.Compiler.var"#509#510"{DataType, DataType, DataType, Enzyme.API.CDerivativeMode, Tuple{…}, Int64, Bool, Bool, UInt64, DataType})
    @ GPUCompiler ~/.julia/packages/GPUCompiler/U36Ed/src/driver.jl:47
 [18] #s1056#508
    @ ~/.julia/packages/Enzyme/KJgKj/src/compiler.jl:5414 [inlined]
 [19] 
    @ Enzyme.Compiler ./none:0
 [20] (::Core.GeneratedFunctionStub)(::UInt64, ::LineNumberNode, ::Any, ::Vararg{Any})
    @ Core ./boot.jl:602
 [21] autodiff
    @ Enzyme ~/.julia/packages/Enzyme/KJgKj/src/Enzyme.jl:209 [inlined]
 [22] autodiff(::ReverseMode{…}, ::Const{…}, ::Duplicated{…}, ::Revolve{…})
    @ Enzyme ~/.julia/packages/Enzyme/KJgKj/src/Enzyme.jl:238
 [23] autodiff
    @ ~/.julia/packages/Enzyme/KJgKj/src/Enzyme.jl:224 [inlined]
 [24] run_checkpointing()
    @ Main ~/Documents/GitHub/ShallowWaters.jl/my_scripts/run_checkpointing.jl:412
 [25] macro expansion
    @ ./timing.jl:279 [inlined]
 [26] top-level scope
    @ ~/Documents/GitHub/ShallowWaters.jl/my_scripts/run_checkpointing.jl:269
 [27] include(fname::String)
    @ Base.MainInclude ./client.jl:489
 [28] top-level scope
    @ REPL[39]:1
in expression starting at /Users/swilliamson/Documents/GitHub/ShallowWaters.jl/my_scripts/run_checkpointing.jl:458
Some type information was truncated. Use `show(err)` to see complete types.

@wsmoses
Copy link
Member

wsmoses commented Feb 14, 2024 via email

@swilliamson7
Copy link
Collaborator Author

Trust me when I say I'm also deeply confused, this worked yesterday 😢

@wsmoses
Copy link
Member

wsmoses commented Feb 14, 2024

Can you remove all manifest.toml files and retry

@swilliamson7
Copy link
Collaborator Author

That worked (fingers crossed), thank you!

@wsmoses wsmoses closed this as not planned Won't fix, can't repro, duplicate, stale Feb 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@wsmoses @swilliamson7