Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Simplify lowering of typed comprehension #32709

Closed
wants to merge 1 commit into from
Closed

Simplify lowering of typed comprehension #32709

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion contrib/generate_precompile.jl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -159,7 +159,7 @@ function generate_precompile_statements()
include_time = @elapsed for statement in sort(collect(statements))
# println(statement)
# Work around #28808
occursin("\"YYYY-mm-dd\\THH:MM:SS\"", statement) && continue
occursin("\"YYYY-mm-dd\\THH:MM:SS", statement) && continue
statement == "precompile(Tuple{typeof(Base.show), Base.IOContext{Base.TTY}, Type{Vararg{Any, N} where N}})" && continue
try
Base.include_string(PrecompileStagingArea, statement)
Expand Down
12 changes: 1 addition & 11 deletions src/julia-syntax.scm
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2283,17 +2283,7 @@

'typed_comprehension
(lambda (e)
(expand-forms
(or (and (eq? (caaddr e) 'generator)
(let ((ranges (cddr (caddr e))))
(if (any (lambda (x) (eq? x ':)) ranges)
(error "comprehension syntax with `:` ranges has been removed"))
(and (every (lambda (x) (and (pair? x) (eq? (car x) '=)))
ranges)
;; TODO: this is a hack to lower simple comprehensions to loops very
;; early, to greatly reduce the # of functions and load on the compiler
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See this comment.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes. and to expand on that a little, we use this to make it possible to use comprehensions in reflection and inference. we wouldn't be able to use comprehensions there if it wasn't for this guarantee.

Copy link
Member

@andyferris andyferris Jul 29, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, I guess that makes sense; I just noticed I can even return a typed comprehension from a @generated function (for obvious reasons). There seems to be a bunch of Any[... for ...] comprehensions in the compiler.

On the other hand, it seems these days that collect is getting some overloads and therefore untyped comprehensions aren't so tied to Array anymore. For example, is it bad that [f(x) for x in a::StaticArray] isa StaticArray? This is juxtaposed with !(Float64[f(x) for x in a::StaticArray) isa StaticArray). I'm wondering if we can get a bit more consistency? (As in - maybe the solution we want is the opposite of this PR, and enforce the untyped version to always create an Array?)

I note that I don't see an aweful lot of typed comprehensions in user code (maybe I'm wrong and that's just me?) so do you know if outside of Julia internals does this lowering "optimization" has much of an impact on latency in the wild? If not, could the desired behavior be viewed as more of a bootstrap vs not bootstrap issue?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes I definitely read this comment! At face value it says "this is a performance hack" and I wondered whether it was still necessary. Especially because comprehensions seem more common than typed comprehensions for user code.

Thanks @vtjnash for expanding. Here's what I think you're saying:

  • Normal comprehensions are prohibited in base/compiler due to the call to return_type here:

    julia/base/array.jl

    Line 594 in 442d159

    Core.Compiler.return_type(first, Tuple{typeof($I)})
  • Simple typed comprehensions are allowed because they lower to a simple form which doesn't re-enter inference.
  • The comment greatly reduce the # of functions and load on the compiler refers to the performance of comprehensions in the compiler itself rather than the performance of user code (??)

But I don't understand why lowering to collect(T, gen) is not ok - this doesn't call return_type; it goes via the _collect methods here:

julia/base/array.jl

Line 519 in 442d159

_collect(::Type{T}, itr, isz::HasLength) where {T} = copyto!(Vector{T}(undef, Int(length(itr)::Integer)), itr)

If this is about compiler performance, what should I be measuring? Is the time to compile Base relevant?

(lower-comprehension (cadr e) (cadr (caddr e)) ranges))))
`(call (top collect) ,(cadr e) ,(caddr e)))))))
(expand-forms `(call (top collect) ,(cadr e) ,(caddr e))))))

(define (has-return? e)
(expr-contains-p return? e (lambda (x) (not (function-def? x)))))
Expand Down