Code generation callback for non-standard AST types.

[DrTodd13]

As @hongborong and @ninegua have mentioned in recent posts, we are doing heavy analysis and modification of typed AST and in many cases we then force that modified AST back into the method structure. At the moment, we have modified Julia internally to add a flag to that structure so that if you try to compile that method it goes through the J2C path rather than through LLVM. This was mainly to generate code for Xeon Phi where a LLVM compiler was lacking. With newer Xeon Phis where the instruction set is supported by LLVM we would like the option of going through the LLVM path again.

As part of our AST modifications, we introduce a few new custom AST node types which encode information that would otherwise be lost if they were lowered to existing AST types. This introduces a problem when it comes time to convert the AST to LLVM as these custom AST types would be unrecognized by codegen.cpp.

We would like to request a new feature whereby we could register a callback to deal with these custom AST types. If codegen doesn't recognize an AST type, it would then try each of the registered callbacks until one of them responsible for that custom AST type returned some generated LLVM. I imagine there might also need to be some state kept by these callbacks and so it would also be nice to also get a callback at the start and end of code generation for a function to give them a chance to initialize and clean-up state.

With these callbacks in place, we believe it would be possible to produce third-party AST optimizations packages without further disturbance to Julia internals.

[JeffBezanson]

llvmcall (#5046) might be a good approach to this. It lets you include inline LLVM code.

[JeffBezanson]

Something else that might work is using :meta nodes, and putting the needed features directly in the existing code generator. We very much want to have a single coherent platform, and not various competing extensions. For example in @ArchRobison 's work on SIMD optimizations, we simply added a :simdloop AST node to the system. If there are other valuable features like that we need, it might be best just to add them. Perhaps they should be grouped under :meta nodes, but that's a detail.

[DrTodd13]

Thanks for the llvmcall idea. I think we are going to pursue that to start with.

I understand you don't want a bunch of competing extensions but an enormous core Julia that tries to be everything to everyone is probably not a good idea either. Some of our AST nodes are pretty generic but others are proprietary and tied to our runtime. Some of the ones that are generic may represent a change in the underlying Julia philosophy. So, it is something to consider but maybe the path there is to do it as a separate package and if you want to integrate later then we can discuss it.

[ViralBShah]

That seems like the right balance. It would be nice if the work were open source so that others interested can follow along / contribute / etc.

[tkelman]

Making code generation easily extensible is a cool idea, whether via callbacks or some other mechanism. Would be good to have a visible proof-of-concept execution of the idea to point at first, as motivation for what kind of internal changes could be done in mainline Julia to make it easier. Answer the "what would this enable" kind of question with concrete examples.

[Keno]

Closing this as non-specific. Our compiler-tricks toolbag has improved since this issue was filed. At this point we'll need more specific issues for extensions to the existing setup.