Failing to build with llvm3.3

[Narrat]

Using the system llvm package is failing since it got updated to 3.3

src/codegen.cpp: julia won't find some of the headers as they got moved into a subfolder (llvm/DerivedTypes.h, LLVMContext.h, Module.h, Intrinsics.h, Attributes.h, IRBuilder.h moved into llvm/IR/)

But fixing this doesn't solve it
Here is the log for the failure http://pastie.org/8050095

Edit: Forgot the system info

Distro: Arch Linux

[StefanKarpinski]

Can you give some more context? What OS are you running? Is this OS X? I was unaware that LLVM 3.3 had even been released.

[Narrat]

Eh, sorry for leaving that information out.

My distro is Arch Linux and here's the distro specific packagelist for llvm: https://www.archlinux.org/packages/?q=llvm
The source files can be found there http://llvm.org/releases/3.3/
I'm also stumbled about the fact, that there is no news on the main-page, but nevertheless my distribution has updated to it

[JeffBezanson]

We will move to LLVM 3.3 when it is officially released. That doesn't seem to be the case yet, but it will happen soon. Thanks for the heads up.

[JeffBezanson]

Ok, it looks like LLVM 3.3 is now official. Since we hope to release 0.2 in a couple weeks, I'd propose waiting until after that to move to 3.3. cc @StefanKarpinski @ViralBShah

[Keno]

I agree. As part as moving to LLVM 3.3, we should also move to the new MCJIT engine, since the old one is now deprecated and will be removed in 3.4

[StefanKarpinski]

That might be a significant undertaking, but it's a worthwhile one since MCJIT has much better support for doing things with jitted code like debugging it and saving it to binaries. So it's probably a crucial step forward.

[ihnorton]

Where did you see the deprecation notice? The only thing I found is that exception handling will be deprecated in 3.4.

MCJIT does not support incremental Function compilation which might be painful - at very best - to adapt to:

"Basically, with MCJIT once code has been generated for a module nothing else can be added to the Module."
http://llvm.1065342.n5.nabble.com/MCJIT-and-Kaleidoscope-Tutorial-td58225.html

There are plans to support multiple modules in one instance, presumably with runtime linking, but this is not yet implemented. Relevant discussions:
http://llvm.1065342.n5.nabble.com/Old-JIT-Status-i-e-can-we-delete-it-td51730.html
http://llvm.1065342.n5.nabble.com/MCJIT-and-Lazy-Compilation-td54448.html

(I hope I'm wrong! MCJIT will bring some great things if it is feasible to switch. The ObjectCache facility should fix #260 almost for free)

[JeffBezanson]

If you can't interleave compilation and execution, one wonders what they mean by "JIT".

[Keno]

Indeed. I was under the impression that MCJIT was a replacement for the old JIT just using the new MC backend, but that impression seems to be misguided. I also may have misinterpreted the deprecation of EH for the deprecation of the JIT.

[ViralBShah]

I like the idea of using llvm 3.3 for the 0.2 release. We should just give enough time for release candidates so that it can be widely tested, and also so that packages have enough time to update to 0.2.

[StefanKarpinski]

Let's not cram any more major changes into 0.2. The whole MCJIT discussion is worrying, but this at least is encouraging:

http://llvm.1065342.n5.nabble.com/MCJIT-and-Lazy-Compilation-td54448.html#a55039

[JeffBezanson]

That is encouraging. I think I had to do something like that on .NET; you just keep making new modules as necessary. We might have to dump 100 .so's, but that's not the end of the world.

[StefanKarpinski]

Ugh. That would be really annoying – merging .so files seems to be a "hard problem". However, I suspect that with LLVM there will be the option to generate bitcode instead, which should be mergable.

[vtjnash]

For what it's worth, we've never really needed the true-JIT feature of the compiler (the current JIT delays compiling a function until it is called for the first time -- but we don't generate the function until we need it anyways). Also, I tried turning on MCJIT a year or so ago and it worked on linux easily and failed completely on Mac.

[StefanKarpinski]

That's good to know. Presumably it's more stable across platforms at this point. Especially since LLVM is largely backed by Apple, we can be pretty confident that things will eventually work well on OS X.

[Keno]

The problem is that none of the developers of LLVM really need the JIT, the sole raison d'être of the MCJIT project is to be used with lli in which case none of this matters.

[StefanKarpinski]

Hopefully, even if that's the case, they recognize that JIT functionality
is essential to the momentum of the LLVM project and that sabotaging it
would alienate a significant number of users, like Pure and Julia, and
generate a huge amount of bad PR and ill will. There are also projects like
Cling, which I suspect need real JIT functionality even if they are based
on C/C++.

On Tuesday, June 18, 2013, Keno Fischer wrote:

The problem is that none of the developers of LLVM really need the JIT,
the sole raison d'être of the MCJIT project is to be used with lli in
which case none of this matters.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/3418#issuecomment-19622265
.

[staticfloat]

Just wanted to jump in and say Homebrew has moved to llvm 3.3, and now homebrew users are noticing that their installations are failing with the compilation failure noted above. Is there any simple fix we can push to allow for compiling against llvm 3.3 even without the large changes noted here?

[Keno]

Yeah, I'll work on it. (I have it pretty much done on my kf/ccall3 branch anyway, so I'll extract the relevant changes).

[JeffBezanson]

Thanks @loladiro . Yes, we will move to llvm 3.3 soon, but nobody should ever have an expectation that they can build julia against whatever version of llvm happens to be lying around. The idea of finding libraries only by name and not version is totally broken in the case of llvm, and probably for many other libraries too. Right now it's not a big problem, but the day may come when we decide not to upgrade llvm, or that doing so takes a very long time. Homebrew should not be casually dropping in new versions and expecting everything to just work.

[staticfloat]

Thanks @loladiro! :D

[staticfloat]

@JeffBezanson I'm not certain the Homebrew maintainers realize the incompatibility between llvm versions. I'll see if we can't get formula such as llvm3.2 and llvm3.3 etc, so that our formulae don't have versions changing underneath them.

We can, of course, maintain our own llvm formulae, and it's almost trivial to install a previous version of llvm even without any extra work, but Homebrew by default always runs on the latest versions of everything.