fix #12853: kill detached compilation process on an exception

[stevengj]

This fixes #12853 by the simple expedient of wrapping the compilation in a try block and making sure that things are cleaned up on exit (julia subprocess is killed and file is closed).

[stevengj]

Seems to have some Travis problems... restarting Travis to see if that helps.

[stevengj]

Weird, I'm getting There are newer queued builds for this pull request, failing early. from Travis. Maybe I'll try rebasing and force-pushing to start a new check.

[stevengj]

I'm getting Worker 4 terminated. ERROR (unhandled task failure): EOFError: read end of file From worker 4: * examples on Linux x86_64, which is weird....

[yuyichao]

That's the typical output for OOM error.

[stevengj]

But why would I be having an OOM here? Grrr.

[yuyichao]

Because we add too many tests recently?

[stevengj]

@yuyichao, but then how do I fix it?

[kshyatt]

@stevengj Take away my commit access?

[stevengj]

@kshyatt, I think we'd have to do that retroactively...

[kshyatt]

More seriously, it might be worth running the test suite through the allocation profiler and see where we're using a lot of memory.

[yuyichao]

More seriously, it might be worth running the test suite through the allocation profiler and see where we're using a lot of memory.

Hint, focusing on llvm as a start.

In my original issue about Travis OOM @amitmurthy brought up the possibility to more aggressively restart worker processes. I'm not really familiar with the parallel part but it might worth doing.

[stevengj]

In the short term, how are people getting PRs through? Should I just keep force-pushing and hope that it randomly works?

[kshyatt]

I sign in to Travis and restart the build.

[stevengj]

Restarting the build from Travis yields There are newer queued builds for this pull request, failing early.

[stevengj]

Trying rebase and push -f.

[yuyichao]

@tkelman Is the fact that travis doesn't assign a new build number if you restart a build conflict with some logic in the travis fail fast script?

[stevengj]

Well, the good news is that the OOM killer is predictable. The bad news is that it is predictably (3 times in a row) killing (only) the x86_64-Linux build of this branch.

[stevengj]

Just pushed a commit to this branch that reduces the number of worker processes in runtests.jl, let me see if that works around the problem.

[stevengj]

Oh, that won't help, the .travis.yml file already exports JULIA_CPU_CORES=4

[stevengj]

Okay, reducing the number of workers fixed it, so it seems to definitely be the old OOM error.

Can we merge this, and increase the number of Travis cores later when a better solution to the OOM problem is found?

[stevengj]

Going to merge this later today if there are no objections.

[sbromberger]

Thank you all for fixing this issue and working through the Travis failures!

[ScottPJones]

What do I need to do to get my 5 code coverage tests PRs restarted now that this (hopefully) has been fixed? ( #12842 #12880 #12882 #12884 #12898 )

[sjkelly]

@ScottPJones I believe a rebase (probably just pull --rebase) and force push (push -f) should add the PR to the Travis queue.

[ScottPJones]

OK, thanks!

[stevengj]

It's not really fixed, just worked around (temporarily, until our tests get so memory-hungry that even two processes is not enough) at the cost of increased Travis latency. Please take further discussions to #11553.