luarocks: migrate into lua formulae

[DomT4]

This PR ‘pythonises’ the Lua setup. There have been requests for us to do this for a while, and it’s a good idea - It’s just taken me a while to finish ironing out.

The individual luarocks formula dies entirely, in favour of luarocks being vendored into the two Lua formulae in Homebrew/homebrew with versioned trees. This provides roughly the same experience we deliver with pip and co from Python.

Luajit is not yet supported, because it’s being a pain, but Luajit support will land in the semi-near future, as will support for Lua53 in Versions. All 4 installations can sit side-by-side with no non-binary conflict. Binary conflict is handled roughly the same way Python handles it sadly, which is regrettable but presently unavoidable.

[dunn]

Would turning Library/Formula/luarocks.rb into an alias for Lua cause problems?

[DomT4]

Would turning Library/Formula/luarocks.rb into an alias for Lua cause problems?

Theoretically, no. We've done that before when we migrated npm back into the node formula - Slightly different circumstances, but same general idea.

The slightly confusing element to end-users perhaps here is that we'd be actively moving away from a Luarocks that only supports one Lua to potentially 4 Luarocks supporting their own Lua version - So despite supporting multiple Luarocks brew install luarocks would still get you luarocks linked to lua (5.2).

[MikeMcQuaid]

I like this idea generally, seems like a step forward. A couple of questions:

• Can we get away with only installing luarocks for Lua 5.2?
• If not, can we combine the code more somehow?
• Will packages be preserved across upgrades and will they be separated nicely between Lua versions?

Thanks!

[DomT4]

Can we get away with only installing luarocks for Lua 5.2?

We could, but like the Python situation I think it would blunt the usefulness of merging luarocks in. Luarocks is an important enough part of the Lua 'experience', with such a wide range of modules that don't necessarily play nicely with differing Lua versions, that IMO it's worth supporting the multi-setup.

If not, can we combine the code more somehow?

It's difficult because the build options used and the way Luarocks builds versioned directories requires such different configure options. I'm open to suggestions, but not sure there's an easier way to handle the requirement to version, sadly.

Will packages be preserved across upgrades and will they be separated nicely between Lua versions?

It's pretty similar to Python. The libraries will never conflict - They'll be installed in HOMEBREW_PREFIX/lib/luarocks/rocks/lua-#{version}/, and they aren't symlinked or execed anywhere else.

The binaries suffer from the Python problem, where one version overrides the other - This only applies to the main HOMEBREW_PREFIX/bin though - Luarocks will actually install the rocks directly into HOMEBREW_PREFIX/lib/luarocks/rocks/lua-#{version}/ and then creates an exec script to the main bin. It's that exec script that gets overwritten when you install the same binary on a different Lua version.

There's some caveats in the formula about how to handle this. Luarocks attempts to automatically migrate conflict by changing the exec script names to prevent conflict - In practice though, this method can be a little flawed.

The Rocks themselves, and the directories they are installed in will work across updates to either Lua or the internalised Luarocks. We could probably fix the exec script problem by moving the luarocks install to postinstall, but then we lose the ability to bottle, which is a pain.

The issue as such is that brew doesn't create that opt directory until the very last step, when it links everything, so I can't tell luarocks internally to use the opt prefix during compile, I have to tell it to use the prefix, which of course is the full versioned Cellar path. I'm going to sniff around the conf file and see if we can fix that, but ultimately it's something I'll end up probably asking upstream for.

[DomT4]

Apologies for the long answer. Hopefully it's more useful than ramble.

[MikeMcQuaid]

Ok, that makes sense to me. Last bit that confuses me: how will I know which version of luarocks to use? Just depending on the project I'm working on?

[DomT4]

how will I know which version of luarocks to use?

As in, how the heck do I know which modules I should use which Luarocks for? 😸. That's kind of the frustrating thing about Luarocks - The central websites involved don't do a PyPi-esque This works with 2.7 or 3.1 message, you usually have to go into individual module websites to hunt the answer down. The two most consistently popular Luarocks available both specify in detail their compatibility with various Lua, but there's no central place for that information yet as far as I've found.

It's frustrating, and something that would ideally change long-term.

A lot of it is still trial-and-error, aside from checking the websites themselves. If I do luarocks-5.2 install example does it work, or does it break? kind of thing, sadly.

[DomT4]

Looks like I may be able inreplace the Lua path inside the Luarocks-5.2 and Luarocks-5.1 and so on binaries in postinstall, which might fix the exec script breaking across updates problem. The main Luarocks 'binary' is actually a really short script that is essentially a wrapper around internal commands. I'll play with it a bit more this evening.

[DomT4]

Fixed the exec persistence issue. Took a couple of extra blocks of code, but nothing too hideous/hacky. I'm relatively happy for this to ⛵ if you are @MikeMcQuaid.

[dunn]

"[handled] after the main install"

[DomT4]

Crumbs. Well spotted. I read that line about 10 times and still missed the typo 😿

[DomT4]

@MikeMcQuaid Anything else need doing here? No rush to merge; Just checking through my 'old' open PRs, seeing what's still relevant, what can be closed, etc.

[MikeMcQuaid]

I believe you can just use pkg_version here which will do what you want.

[DomT4]

Neat. That'd certainly be preferable. Will check it out.

[MikeMcQuaid]

One comment and then I'm good to merge this.

[MikeMcQuaid]

Oh, last thing would be checking the upgrade path for this works well?

[DomT4]

Oh, last thing would be checking the upgrade path for this works well?

As in does it work with brew upgrade? It's been a reasonably long day, feel free to explain it like I'm a 2 year old if I'm being stupid 😸

[MikeMcQuaid]

As in does it work with brew upgrade? It's been a reasonably long day, feel free to explain it like I'm a 2 year old if I'm being stupid

Aye, what happens if you're using the old formula and upgrade. Will all your packages be preserved etc. and what happens if the luarocks formula isn't uninstalled?

[DomT4]

Blurgh. We're going to need to enforce the removal of luarocks otherwise brew link breaks. We could add a one-version/revision check for Luarocks and then require people to remove it prior to upgrading.

Still think it's a worthwhile thing to do, even if not being able to kill Luarocks entirely ourselves is a pain. What did Homebrew do when the npm formula was abolished and merged into Node?

Will all your packages be preserved etc

Yeah they won't get wiped, but they'll be ignored. You'll need to migrate them to the new system once. Essentially the current luarocks tree is in $HOME, and we're deliberately specifying a tree elsewhere here to facilitate multi-trees.

[MikeMcQuaid]

Blurgh. We're going to need to enforce the removal of luarocks otherwise brew link breaks. We could add a one-version/revision check for Luarocks and then require people to remove it prior to upgrading.

You can't really enforce removal, unfortunately. The best way to do that is probably have the upgrade install nothing (or make it keg-only) so the linking works.

Yeah they won't get wiped, but they'll be ignored. You'll need to migrate them to the new system once. Essentially the current luarocks tree is in $HOME, and we're deliberately specifying a tree elsewhere here to facilitate multi-trees.

Cool, that's fine but we may want to add a caveat about it.

[DomT4]

The best way to do that is probably have the upgrade install nothing (or make it keg-only) so the linking works.

There's an existing PR to update Luarocks's version. Maybe we could keg_only Luarocks with that PR and then merge this after? Should resolve the problem relatively non-invasively.

[DomT4]

@MikeMcQuaid Xu merged the keg_only formula, so if you're happy at this point, once the rebase here passes the bot we should be good to roll.

[DomT4]

Cool, that's fine but we may want to add a caveat about it.

Added 👍

[thomasjachmann]

From my point of view, making luarocks keg-only outside of this PR wasn't a good idea. It took me a while to figure out why luarocks wasn't available anymore after doing a brew upgrade. I had to force link luarocks to make it work again since this PR isn't merged yet.

[DomT4]

This was intended to be merged very shortly after #38620. I'm not sure whether the trigger was pulled too soon on #38620, or whether there's an issue holding this up.

[MikeMcQuaid]

Merged now, thanks @DomT4!

[DomT4]

Thanks Mike!