Assertion failure in type system

[simonster]

On JuliaIO/JLD2.jl@dfca1f0 (current JLD2 master) with 0.6 debug build:

  | | |_| | | | (_| |  |  Version 0.6.1-pre.0 (2017-06-19 13:06 UTC)
 _/ |\__'_|_|_|\__'_|  |  Commit dcf39a1 (8 days old release-0.6)
|__/                   |  x86_64-apple-darwin16.5.0

julia> using JLD2
Assertion failed: (!jl_subtype((jl_value_t*)sig, (jl_value_t*)type)), function check_ambiguous_visitor, file /usr/local/julia-release-0.6/src/gf.c, line 1129.

signal (6): Abort trap: 6
while loading /Users/simon/.julia/v0.6/JLD2/src/data.jl, in expression starting on line 914
__pthread_kill at /usr/lib/system/libsystem_kernel.dylib (unknown line)
Allocations: 3993141 (Pool: 3991608; Big: 1533); GC: 7
fish: 'julia-debug' terminated by signal SIGABRT (Abort)

The assertion failure occurs when loading this line. Appears to operate properly with a non-debug build.

[simonster]

Minimal reproducer:

f(::UnionAll) = nothing
f(::DataType) = nothing
f{T<:DataType}(::Type{T}) = nothing

[JeffBezanson]

Seems to be fixed on master? @simonster could you confirm whether JLD2 works?

[timholy]

On a debug build of current master, here's a case that triggers the same assertion:

module CVS

abstract type Colorant{T,N} end
abstract type Color{T, N} <: Colorant{T,N} end
abstract type AbstractRGB{T} <: Color{T,3} end

AbstractGray{T} = Color{T,1}

MathTypes{T,C} = Union{AbstractRGB{T},AbstractGray{T}}

plus(a::AbstractGray, b::AbstractGray) = 1
plus(c::AbstractGray) = c
plus(a::MathTypes, b::MathTypes) = plus(promote(a, b)...)

end

It's a stripped-down version of ColorVectorSpace.

[timholy]

It seems to be a problem of type specificity between the unary variant and the binary MathTypes variant:

julia: /home/tim/src/julia-1.0/src/gf.c:1152: check_ambiguous_visitor: Assertion `!jl_subtype((jl_value_t*)sig, (jl_value_t*)type)' failed.

Thread 1 "julia" received signal SIGABRT, Aborted.
0x00007ffff6b1e428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
54      ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) f 4
#4  0x00007ffff7519df2 in check_ambiguous_visitor (oldentry=0x7fffea3562c0, closure0=0x7fffffffb890) at /home/tim/src/julia-1.0/src/gf.c:1152
1152            assert(!jl_subtype((jl_value_t*)sig, (jl_value_t*)type));
(gdb) call jl_type_morespecific(type, sig)
$1 = 0
(gdb) call jl_type_morespecific(sig, type)
$2 = 1
(gdb) call jl_(closure->newentry)
TypeMapEntry(sig=Tuple{typeof(CVS.plus), Union{CVS.AbstractRGB{T}, CVS.Color{T, 1}} where T, Union{CVS.AbstractRGB{T}, CVS.Color{T, 1}} where T}, simplesig=nothing, guardsigs=svec(), min_world=24860, max_world=-1, func=CVS.plus(...), isleafsig=false, issimplesig=false, va=false, next=↩︎
  TypeMapEntry(sig=Tuple{typeof(CVS.plus), CVS.Color{T, 1} where T}, simplesig=nothing, guardsigs=svec(), min_world=24859, max_world=-1, func=CVS.plus(...), isleafsig=false, issimplesig=false, va=false, next=↩︎
  TypeMapEntry(sig=Tuple{typeof(CVS.plus), CVS.Color{T, 1} where T, CVS.Color{T, 1} where T}, simplesig=nothing, guardsigs=svec(), min_world=24858, max_world=-1, func=CVS.plus(...), isleafsig=false, issimplesig=false, va=false, next=↩︎
  nothing)))
(gdb) call jl_(closure->newentry->next->sig)
Tuple{typeof(CVS.plus), CVS.Color{T, 1} where T}
(gdb) call jl_(type)
Tuple{typeof(CVS.plus), Union{CVS.AbstractRGB{T}, CVS.Color{T, 1}} where T, Union{CVS.AbstractRGB{T}, CVS.Color{T, 1}} where T}
(gdb) call jl_type_morespecific(type, closure->newentry->next->sig)
$3 = 1

So it seems that the method table is being sorted like this (in order of "most specific to least specific"):

plus(a::MathTypes, b::MathTypes) = plus(promote(a, b)...)
plus(c::AbstractGray) = c
plus(a::AbstractGray, b::AbstractGray) = 1

As evidence that this is correct, all I have to do is change the order of their definition to

plus(a::AbstractGray, b::AbstractGray) = 1
plus(a::MathTypes, b::MathTypes) = plus(promote(a, b)...)
plus(c::AbstractGray) = c

and then I don't trigger the assertion anymore, and the method table is

julia> methods(CVS.plus)
# 3 methods for generic function "plus":
[1] plus(c::Main.CVS.Color{T,1} where T) in Main.CVS at /tmp/CVS.jl:13
[2] plus(a::Main.CVS.Color{T,1} where T, b::Main.CVS.Color{T,1} where T) in Main.CVS at /tmp/CVS.jl:11
[3] plus(a::Union{Main.CVS.AbstractRGB{T}, Main.CVS.Color{T,1}} where T, b::Union{Main.CVS.AbstractRGB{T}, Main.CVS.Color{T,1}} where T) in Main.CVS at /tmp/CVS.jl:12

[JeffBezanson]

I believe this was fixed a while ago; @simonster please reopen if you're still able to reproduce this. The ColorVectorSpace issue is separate, #29136.