Simplify lowering of typed comprehension

[c42f]

For implementing special syntax it's useful for typed comprehension to
lower to collect(T, gen). This changes typed comprehensions to use the
same lowering pattern as normal comprehensions.

@andyferris

[JeffBezanson]

See this comment.

[vtjnash]

yes. and to expand on that a little, we use this to make it possible to use comprehensions in reflection and inference. we wouldn't be able to use comprehensions there if it wasn't for this guarantee.

[andyferris]

OK, I guess that makes sense; I just noticed I can even return a typed comprehension from a @generated function (for obvious reasons). There seems to be a bunch of Any[... for ...] comprehensions in the compiler.

On the other hand, it seems these days that collect is getting some overloads and therefore untyped comprehensions aren't so tied to Array anymore. For example, is it bad that [f(x) for x in a::StaticArray] isa StaticArray? This is juxtaposed with !(Float64[f(x) for x in a::StaticArray) isa StaticArray). I'm wondering if we can get a bit more consistency? (As in - maybe the solution we want is the opposite of this PR, and enforce the untyped version to always create an Array?)

I note that I don't see an aweful lot of typed comprehensions in user code (maybe I'm wrong and that's just me?) so do you know if outside of Julia internals does this lowering "optimization" has much of an impact on latency in the wild? If not, could the desired behavior be viewed as more of a bootstrap vs not bootstrap issue?

[c42f]

Yes I definitely read this comment! At face value it says "this is a performance hack" and I wondered whether it was still necessary. Especially because comprehensions seem more common than typed comprehensions for user code.

Thanks @vtjnash for expanding. Here's what I think you're saying:

• Normal comprehensions are prohibited in base/compiler due to the call to return_type here:

  julia/base/array.jl

  Line 594 in 442d159

  Core.Compiler.return_type(first, Tuple{typeof($I)})

• Simple typed comprehensions are allowed because they lower to a simple form which doesn't re-enter inference.
• The comment greatly reduce the # of functions and load on the compiler refers to the performance of comprehensions in the compiler itself rather than the performance of user code (??)

But I don't understand why lowering to collect(T, gen) is not ok - this doesn't call return_type; it goes via the _collect methods here:

julia/base/array.jl

Line 519 in 442d159

_collect(::Type{T}, itr, isz::HasLength) where {T} = copyto!(Vector{T}(undef, Int(length(itr)::Integer)), itr)

If this is about compiler performance, what should I be measuring? Is the time to compile Base relevant?

[c42f]

Ok, AFAICT a good reason not to do this yet is that it would break the use of typed comprehensions within the AST produced by @generated functions. So we should wait until that limitation is lifted if possible (cf. #21094 (comment)).

Other than that, the typed version of collect lowered to here doesn't re-enter inference so I can't see a fundamental problem with doing this. Some concern about performance has been raised but it's not yet clear what kind of benchmark should be run for that.

With the first point in mind, I think I'll just close this for now.