Pull LuaJIT v2.1 and import complete LuaJIT git history

[lukego]

Here is an unusual PR :-) 2,125 commits!

There are two merge commits here that each do something related:

• bd41b1d imports the latest changes from the LuaJIT v2.1 branch. These are squashed into a single commit (736cc36) as we have done in the past.
• b535cf8 imports the LuaJIT git history without making any changes to the code. This means that every commit from the LuaJIT v2.1 branch is represented in the Snabb repo under the lib/luajit/ subtree.

So why import the history? This is so that we can work on LuaJIT within the Snabb source tree: make our own changes, pull changes from upstream, pull changes from new feature branches, etc. This does not work so well if we only have the squashed version of the history: we can't pull from multiple branches without them conflicting and rolling back each others' changes. More discussion over on #1109 (comment).

Have to look at the potential costs/benefits of increasing the size of our commit history, including individual LuaJIT changes in procedures like git bisect, potentially increasing the size of the repo on a fresh checkout, etc.

Overall I would like to be able to hack on LuaJIT directly in the Snabb tree and have it covered by all of our usual workflows and CI tests and so on. I see our own workflows and infrastructure being much more powerful than what LuaJIT upstream offers at the moment. So I think it makes sense to develop our subtree as a first-class citizen and pull changes from all other branches of interest.

Feedback welcome!

Edit with more Git info

The upshot of having the complete history is that we should be able to pull changes from any branch of LuaJIT like this:

git subtree pull --prefix lib/luajit https://github.com/reponame/luajit branchname

So for example we could pull in and CI-benchmark-torture experimental features like vmprofile (LuaJIT/LuaJIT#290), intrinsics (LuaJIT/LuaJIT#116), new garbage collectors that have been written by @corsix and @fsfod, and so on. If they are good we can merge them directly into Snabb. If other people are interested in our LuaJIT changes we/they can use git subtree push to push them out to a branch of a LuaJIT repo (i.e. separate from the Snabb code and repo.)

[lukego]

Here is a Hydra run showing that the performance impact of the new LuaJIT changes from upstream is neutral: https://hydra.snabb.co/build/1077739/download/2/report.html.

Have to get Hydra automatically testing PRs!

[kbara]

I like the concept. Would you like it merged to kbara-next?

[lukego]

@kbara Thanks but I have merged it directly onto the luajit branch and I will take it to next from there. If we decide we want more hands and eyes on maintaining out LuaJIT fork then we can manage that by sharing the luajit branch somehow (e.g. multiple committers and LGTM convention or spinning it off to a separate Github organization or ...).

Just thinking it will be easier if over time the branches become more topic-specific (e.g. LuaJIT) and reviewers work in smaller groups with common subject matter interest (e.g. luajit, aarch64, etc.) This way we won't spread the *-next branches too thin (still acting as a valuable catch-all of course.)

Feeling the way along as always... shout if I mess up!

[kbara]

Sounds good!