GAP crash after Julia TypeError

[ThomasBreuer]

I have observed reproducible crashes when I start GAP (current master branch, with Julia GC and using Julia 1.1.0), load JuliaInterface (current master branch), and then run computations involving a lot of Julia objects.
The error messages are more than 700 lines long, they start with

Internal error: encountered unexpected error in runtime:
TypeError(func=:Tuple, context="parameter", expected=Type, got=ForeignGAP.MPtr())
rec_backtrace at /buildworker/worker/package_linux64/build/src/stackwalk.c:94
record_backtrace at /buildworker/worker/package_linux64/build/src/task.c:217 [inlined]
jl_throw at /buildworker/worker/package_linux64/build/src/task.c:417
jl_type_error_rt at /buildworker/worker/package_linux64/build/src/rtutils.c:118
jl_f_apply_type at /buildworker/worker/package_linux64/build/src/builtins.c:841
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2219
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1571 [inlined]
jl_f__apply at /buildworker/worker/package_linux64/build/src/builtins.c:556
tuple_tfunc at ./compiler/tfuncs.jl:1107
matching_cache_argtypes at ./compiler/inferenceresult.jl:103
Type at ./compiler/inferenceresult.jl:17 [inlined]
Type at ./compiler/inferenceresult.jl:12 [inlined]
typeinf_ext at ./compiler/typeinfer.jl:574
typeinf_ext at ./compiler/typeinfer.jl:613
jfptr_typeinf_ext_1.clone_1 at /users/oscar/julia-1.1.0/lib/julia/sys.so (unknown line)
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2219
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1571 [inlined]
jl_type_infer at /buildworker/worker/package_linux64/build/src/gf.c:277
jl_compile_method_internal at /buildworker/worker/package_linux64/build/src/gf.c:1819 [inlined]
jl_fptr_trampoline at /buildworker/worker/package_linux64/build/src/gf.c:1863
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2219
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1571 [inlined]
jl_call3 at /buildworker/worker/package_linux64/build/src/jlapi.c:249
DoCallJuliaFunc at /users/oscar/GAP.jl/pkg/GAPJulia/JuliaInterface/src/calls.c:114 [inlined]
DoCallJuliaFunc3Arg at /users/oscar/GAP.jl/pkg/GAPJulia/JuliaInterface/src/calls.c:152
CALL_3ARGS at /users/oscar/gap-master/gap/src/calls.h:321 [inlined]
EvalOrExecCall at /users/oscar/gap-master/gap/src/funcs.c:170 [inlined]
ExecProccall3args at /users/oscar/gap-master/gap/src/funcs.c:228
EXEC_STAT at /users/oscar/gap-master/gap/src/stats.c:52 [inlined]
ExecForRangeHelper at /users/oscar/gap-master/gap/src/stats.c:542 [inlined]
ExecForRange at /users/oscar/gap-master/gap/src/stats.c:555
EXEC_STAT at /users/oscar/gap-master/gap/src/stats.c:52 [inlined]
ExecForRangeHelper at /users/oscar/gap-master/gap/src/stats.c:542 [inlined]
ExecForRange at /users/oscar/gap-master/gap/src/stats.c:555
EXEC_STAT at /users/oscar/gap-master/gap/src/stats.c:52 [inlined]

The GAP code below is the result of my attempts to reduce the problem to a small number of input code lines.
My first question is whether others are able to reproduce the crash.
My second question is how we can fix this crash.

LoadPackage( "JuliaInterface" );

JuliaIncludeFile( "funs.jl" );  # this file must be made available, see below

# block a suitable huge amount of memory
glob:= [];;
mat:= IdentityMat( 100 );;
for i in [ 1 .. 4800 ] do
  Add( glob, StructuralCopy( mat ) );
od;

test:= function( r, z )
    local mat, i, j;

    Julia.LoewyStructure.OverflowCount( r );
    Julia.LoewyStructure.BasesOfRadicalPowers( r );
    Julia.LoewyStructure.BasesOfSocleSeries( r );
    Julia.LoewyStructure.BasesOfPPowers( r );

    mat:= Julia.LoewyStructure.OverflowTable( r );;
    for i in [ 1 .. z+1 ] do
      for j in [ 1 .. z+1 ] do
        Julia.Base.getindex( mat, i, j );
      od;
    od;
end;

cand:= [
[ 1040, 447, 4, 47, 4 ],
[ 1040, 127, 12, 63, 12 ],
[ 1040, 223, 12, 63, 12 ],
[ 1040, 223, 12, 127, 12 ],
[ 1040, 123, 12, 67, 12 ],
[ 1040, 397, 12, 293, 12 ],
[ 1043, 4, 222, 2, 444 ],
[ 1043, 16, 111, 2, 444 ],
];;

for entry in cand do
Print( "#I  deal with ", entry, "\n" );
  z:= entry[1];
  A1:= Julia.LoewyStructure.LoewyLayersData( entry[2], entry[3], z );
  A2:= Julia.LoewyStructure.LoewyLayersData( entry[4], entry[5], z );
  Julia.LoewyStructure.OverflowTable( A1 );
  Julia.LoewyStructure.OverflowTable( A2 );
  test( A1, z );
  test( A2, z );
od;

And here is the necessary Julia file funs.jl.

module LoewyStructure

function CoefficientsQadicReversed( k::Int, z::Int, q::Int, n::Int )
    # represent the integer <i> as <q>-adic number
    v::Array{Int,1} = zeros( Int, n )

    if k == z
      for i in 1:n
        v[i] = q-1
      end
    else
      r = k
      for i in 1:n
        d, r = divrem( r * q, z )
        v[i] = d
      end
    end

    return v
end

function islessorequal( mon1::Array{Int,1}, mon2::Array{Int,1}, n::Int )
  local i::Int

  for i in 1:n
    if mon2[i] < mon1[i]
      return false
    end
  end

  return true
end

function LoewyLayersData( q::Int, n::Int, z::T ) where {T<:Integer}
    if ( z != 1 && powermod( q, n, z ) != 1 )
      error( "<z> must divide <q>^<n> - 1" )
    end

    zeromon = zeros( Int, n )
    monomials = [ zeromon ]
    layers = Int[ 0 ]
    degrees = Int[ 0 ]
    predecessors = Int[ 0 ]

    m = n * (q-1)
    for i in 1:z
      mon = CoefficientsQadicReversed( i, z, q, n )
      lambda = 0
      pred = 1
      mm = sum( mon )
      for j in 2:i
        if lambda < layers[j] && degrees[j] < mm &&
                                 islessorequal( monomials[j], mon, n )
          lambda = layers[j]
          pred = j
        end
      end
      push!( monomials, mon )
      push!( layers, lambda + 1 )
      push!( degrees, mm )
      push!( predecessors, pred )

      if lambda == 0
        if mm < m
          m = mm
        end
      end
    end

    i = length( monomials )
    pred = Int[]
    while i > 0
      push!( pred, i )
      i = predecessors[i]
    end

    return Dict( :monomials => monomials,
                 :layers => layers,
                 :chain => pred,
                 :m => m,
                 :ll => layers[ z+1 ] + 1,
                 :inputs => [ q, n, z ] )
end

function OverflowTable( data::Dict )
    if ! haskey( data, :overflowTable )

      monoms = data[ :monomials ]
      d = length( monoms )
      mat = falses( d, d )
      q = data[ :inputs ][1]
      n = data[ :inputs ][2]
      for i in 1:d
        for j in 1:(i-1)
          res = all( x -> x < q, monoms[i] + monoms[j] )
          mat[ i, j ] = res
          mat[ j, i ] = res
        end
        mat[ i, i ] = all( x -> x < q, monoms[i] + monoms[i] )
      end

      data[ :overflowTable ] = mat
    end

    return data[ :overflowTable ]
end

function OverflowCount( data::Dict )
    if ! haskey( data, :overflowCount )
      mat = OverflowTable( data )
      list = [ count( mat[i,:] ) for i in 1:size( mat, 1 ) ]
      data[ :overflowCount ] = list
    end

    return data[ :overflowCount ]
end

function BasesOfRadicalPowers( data::Dict )
    lay = data[ :layers ]
    ll = data[ :ll ]
    jbylayer = map( i -> findall( x -> x == i, lay ), 1:(ll-1) )
    J = [ jbylayer[ ll-1 ] ]
    for i in 2:(ll-1)
      push!( J, sort( union( J[ i-1 ], jbylayer[ ll-i ] ) ) )
    end

    return reverse( J )
end

function BasesOfSocleSeries( data::Dict )
    lay = reverse( data[ :layers ] )
    ll = data[ :ll ]
    sbylayer = map( i -> findall( x -> x == i, lay ), 0:(ll-2) )
    S = [ sbylayer[1] ]
    for i in 2:(ll-1)
      push!( S, sort( union( S[ i-1 ], sbylayer[i] ) ) )
    end

    return S
end

function BasesOfPPowers( data::Dict )
    result = []
    p = 2
    while true
      res = map( x -> Int[], 1:data[ :ll ] )
      q = data[ :inputs ][1]
      for k in 2:length( data[ :monomials ] )
        kk = k - 1
        mx = maximum( data[ :monomials ][k] )
        pos = 1
        lhs = p * mx
        kk = p * kk
        while lhs < q
          push!( res[ pos ], kk + 1 )
          lhs = p*lhs
          kk = p * kk
          pos = pos + 1
        end
      end
      if all( i -> isempty( res[i] ), 1:data[ :ll ] )
        break
      end

      push!( result, res )
      if p == 2
        p = 3
      else
        break
      end
    end

    return result
end

end

[ThomasBreuer]

@fingolfin @rbehrends : Do you have an idea what is going on here?

[rbehrends]

Not yet. I've seen the bug report, but haven't had an opportunity to look at possible causes yet.

[fingolfin]

Sorry, I didn't see this issue before :-(

The error looks as if we are missing to catch a julia exception somewhere. I'll investigate ASAP (but right now I am about to give a talk)

[fingolfin]

@ThomasBreuer I cannot reproduce this with Julia 1.1.1 nor in 1.2.0. There, I get this instead:

#I  deal with [ 1040, 447, 4, 47, 4 ]
Error, MethodError: no method matching OverflowCount(::Array{Core.Compiler.NewNode,1})
Closest candidates are:
  OverflowCount(!Matched::Dict) at /Users/mhorn/Projekte/OSCAR/GAP.jl/funs.jl:109 in
  Julia.LoewyStructure.OverflowCount( r ); at *stdin*:15 called from
test( A2, z ); at *stdin*:45 called from
<function "unknown">( <arguments> )
 called from read-eval loop at *stdin*:46
type 'quit;' to quit to outer loop

You said that you are using 1.1.0 -- any reason for this? 1.1.1 was made months ago and fixed some critical bugs. And pretty soon, we'll have to require Julia 1.3 or later for OSCAR anyway (see also the recent email by Bill)

[ThomasBreuer]

@fingolfin The reasons for using Julia 1.1.0 in this issue are that this is the Julia version that is "officially" recommended at https://oscar.computeralgebra.de/install/, and that this is the Julia version where I can reproduce the problem reliably.

In fact, I get a crash with the same input also with Julia 1.3.0-DEV.433, but only in one of about ten attempts; in the other cases, GAP prints eight "deal with" lines and then the GAP prompt appears.
In particular, I do not run into the MethodError problem which you report.
(The error messages in the crash situation with Julia 1.3 are different from the abovementioned ones which I get with Julia 1.1.0.)

[fingolfin]

The URL you refer to is not actively maintained (esp since Sebastian left). Please don't use 1.1.0, use at least 1.1.1; better would be 1.2 (and certainly don't use DEV versions of Julia; 1.3.0 final was released a few weeks ago).

Can you also reproduce the issue with any of 1.1.1, 1.2.0, 1.3.0?

[ThomasBreuer]

@fingolfin The example from this issue works fine with the Julia versions 1.1.1, 1.2.0, 1.3.0.
Moreover, I have run the long computations from which I had extracted this example with Julia 1.3.0, and I did not get a crash.
(By the way, these computations were used to obtain part of the data shown at http://www.math.rwth-aachen.de/~Thomas.Breuer/singeralg/data/.)

Thus it looks as if the Julia fixes you mentioned are the reason for the changed behaviour. From my point of view, this issue can be closed.

[fingolfin]

Thanks.

I also update oscar-system/oscar-website#25 to emphasis that the install instructions really need to update (that issue has been open for a long time now, though).