Maintenance of clang-format wheel?

[nightlark]

Hi @mgevaert, I noticed there hasn't been much GitHub activity in a few years or updates to the clang-format PyPI wheel in a few years. Are you around to accept PRs and make releases on PyPI? If not, would you be willing to add me to the PyPI project?

I'm interested in getting the package updated to include wheels for newer versions of clang-format, along with binary wheels for macOS and Windows -- it would be much more convenient to use with pre-commit hooks than a Docker image (not to mention the new pre-commit.ci service doesn't support docker); part of this would involve getting working sdist that can build clang-format on Windows/Linux/macOS, and setting up GitHub actions build and publish binary wheels for those platforms to PyPI.

Regards,
Ryan

(I sent an email to the one listed on PyPI for the wheel, though I wasn't sure if you still own the name due to the expired SSL cert)

[mgevaert]

Hi @mgevaert, I noticed there hasn't been much GitHub activity in a few years or updates to the clang-format PyPI wheel in a few years. Are you around to accept PRs and make releases on PyPI? If not, would you be willing to add me to the PyPI project?

I'm very much willing to accept PRs, especially for platforms that I don't have much experience with.

I'm interested in getting the package updated to include wheels for newer versions of clang-format, along with binary wheels for macOS and Windows

That sounds great, looking forward to any PRs.

it would be much more convenient to use with pre-commit hooks than a Docker image (not to mention the new pre-commit.ci service doesn't support docker)

Interesting, at the time, GH Actions were pretty new, so that's what I went with. I've been using https://cibuildwheel.readthedocs.io/en/stable/ for other projects, and that's probably what I'd use now. pre-commit.ci is something I wanted to look into, so I'm game (asottile does nice work.)

part of this would involve getting working sdist that can build clang-format on Windows/Linux/macOS, and setting up GitHub actions build and publish binary wheels for those platforms to PyPI.

The llvm+clang source tree is pretty enormous, so I'm not sure this will work, but it's worth a shot. The itch I scratched with this was a small wheel w/ few dependencies for linux. Part of the reason for not doing macOS/Windows was that there were binaries available, so it didn't seem as pressing. I did put in a bunch of effort into macOS ones, but IIRC, I couldn't build it staticially across all the versions I wanted to support, and I found the final package size to be large. At the time, I pondered if it would make sense to take apart and wrap the macOS & windows binaries, but IIRC they didn't always release them - perhaps that's changed now. Finally, I'm wary of supply-chain attacks, which is why I try to make the building process as transparent as possible, FYI. Thanks for your interest!

[dokempf]

Hey @mgevaert @nightlark
I am also interested in having clang-format pip-installable for all platforms (I use it through pre-commit). I had some time to dig into this and I implemented a version based on scikit-build and cibuildwheel: https://github.com/dokempf/clang-format-wheel
It draws its inspiration from how e.g. ninja and cmake do it. I have deployed the current version to TestPyPI: https://test.pypi.org/project/clang-format/

The approach has some advantages:

• cibuildwheel builds a variety of platform-specific wheels, see the list on PyPI: https://test.pypi.org/project/clang-format/#files Some of these are created using the runners provided in Github Actions, some are emulated using QEMU.
• There is an sdist distribution available (you can try it by running pip with --no-binary :all:). It should be mentioned though that the "source" is only the CMakeLists.txt file that accesses the LLVM source code from GitHub (Ninja does a similar thing). I tried a version that uploads the LLVM sources to PyPI, but it hit PyPI's upload file size limit.
• scikit-build allows this type of packaging with a relatively small amount of low level code which gives me a good feeling about long term maintenance.

I am open to any feedback regarding my approach. If you like it, we can work out a roadmap of how to proceed with it.

[nightlark]

Hey @dokempf that looks awesome! That's pretty much the exact same approach I had in mind. The 20-30 minutes for a native build seems pretty reasonable.. I think I was expecting them to take closer to the 3 hours like the cross-compiled arm/ppc/s390 builds.

Do you want me to make PRs to your repo for now?

[mgevaert]

Very nice! Distribution size is svelte; and the build time is way faster than when I tried on a local old mac the first time I tried this.
Let me find time to look at this, and hopefully we can find some way to make this available.
Cheers.

[mgevaert]

@dokempf: I had a chance to look at your wheel; can you comment on what the version scheme is?

The test pypi has 0.0.7, but if I read it right, it's clang-format 12.x?

I think having the version match the clang version would be helpful. In addition, having the wheels for different versions of clang-format would be good; how hard would it be to release the previous versions?

Thanks!

[nightlark]

I think a version scheme similar to other packaging projects (PyPI CMake/ninja) would work where the version is <clang-format version>.postN, where postN is used to indicate that the change is just to the package and not the included copy of clang-format. Pretty sure the 0.0.7 is just a placeholder for testing.

I have a PR open for versioneer to make it handle cases where a tagged version already contains a post release segment, but I think with that patch it should be useable for handling clang-format-wheel versioning.

If the clang build system hasn't changed, it should be easy to release previous versions by making a branch that pulls the source tree from the old version and tagging a release -- those could either be kept around or cleaned up by merging to main using --ours to ignore the version number change used to create the old release.

[nightlark]

I tried out building a manylinux1 compatible wheel, but the 12.0.1 version of clang-format requires a host gcc version of at least 5.1; it might be possible to build a manylinux1 compatible wheel for an older clang-format. Though now there should hardly be anyone running such an old OS.

[mgevaert]

.postN, where postN is used to indicate that the change

Cool, that's how I was doing the original ones, and I think it makes the most sense.

[...] manylinux1 compatible wheel for an older clang-format. Though now there should hardly be anyone running such an old OS.

I don't think we need to target manylinux1, I agree that it's too old. manylinux2010 would be nice, but even that is a bit long in tooth, so if it's too much work, manylinux2014 would be acceptable, IMO.

[dokempf]

The 0.0.x release scheme was purely for my experimentation to not spoil any actual releases. I would favor the same scheme that @nightlark proposed. Building for older versions of clang-format will be very easy, as we are working with release tags of the llvm-project git repository.

[henryiii]

FYI, besides cibuildwheel (which was mentioned Edit: already in use in the fork, nice), I'd also recommend using the normal CMake package, as we've been supporting manylinux1 for quite a while now - and if you go to manylinux2010, then even more reason to not bother with building CMake. You can also look at the cmake or ninja packages for a wheel rename script that correctly does the tag change in the wheel - I'm planning to make a PR to wheel in the near future to make that easier (initial design was well received), cibuildwheel might gain enhanced support for that in the future too.

Please use <clang-version>.<N> rather than <clang-version>.post<N>, as pip will not update to a newer post release if one is already downloaded. Post releases are only intended for metadata changes, never code changes. A four digit release is fine, typing-extensions (has done?) does that.

[henryiii]

PS: the main reason to do manylinux1 really has nothing to do with the OS, it's due to the pip version. Pip 9 can't install manylinux2010, and pip 9 is the default pip on Ubuntu 18.04, RHEL/CentOS 7 & 8, and some other places. If you upgrade your package manager's pip (which is unsupported), or better yet, use a venv then update the pip, then you can get manylinux2014 just about everywhere. This is generally not a library dependency, so I'd target 2010 unless there's a reason that 2014 is easier.

Or better yet, just use PyPA's pipx. It's even available by default in GHA.

Manylinux1 images will be discontinued Jan 1, 2022.

[dokempf]

@henryiii Thanks for the input! With the discontinuation of manylinux1 being so close, I would rather not target it at all. Having the wheel renaming implemented in wheel would definitely be the cleanest solution, thanks for working on that!

[dokempf]

@mgevaert I think it is time to discuss the mode and timeline of putting the new version into production. I did rewrite the project from scratch, which leaves us with the open question of future maintainership. Here are two very different takes on it:

• I leave the code that I wrote to you, @mgevaert and you do with it whatever you consider best: Fully adopt it, partially adopt it, do not adopt it at all. (Of course the license already allows just that, I am just repeating to say that I have no hard feelings about it).
• I take over (co-)maintainership of the code and the PyPI package. In that case I would like to move the repository into my work institution's namespace: https://github.com/ssciwr/

I am fine with both of above solutions and most likely also with some compromise in between. If you decide not to adopt it at all, I might at some point publish it under a different name on PyPI though.

[nightlark]

@henryiii where is the proposal for the feature getting added to wheel? With the way the platform tag is getting set right now we don't need any renaming script -- I think ideally wheel would have an option to override the platform tags using a config file (pyproject.toml section?) so that wheels automatically built in other package repositories will also get the right tags set.

As far as I can tell piwheels.org (and any similar package repositories that might be out there) don't provide a way to pass command line arguments to the build wheel command, so a config file option could be the only way to set the tags.

[dokempf]

I think @henryiii is referring to this: pypa/wheel#407

[henryiii]

It's not a perfect or final solution (a config setting would be best, ideally in pyproject.toml), it would allow this common need to be done after-the-build, which currently is not easy to do. Though I see you've set this using setup.py customization, that's fine too. There are other reasons to do this though, such as adding native compatibility tags to universal2 wheels, etc.

[mgevaert]

I take over (co-)maintainership of the code and the PyPI package. In that case I would like to move the repository into my work institution's namespace: https://github.com/ssciwr/

Thanks for the suggestion/offer!

I think this is a good solution - the Scientific Software Center looks like a good fit for long term maintenance of the project, and you probably have the resources to maintain it more than I do.

If, for whatever reason, it turns out that it needs to be 'un-transferred', I'd like the project to be transferred back to me, but other than that, let's move forward with that solution.

Thanks to everyone else for the input on this ticket, it's greatly appreciated.

[dokempf]

Thanks to @mgevaert for the trust and all the maintenance work you put into this in the past.

Here is my TODO list for the transition process:

Missing Implementation:

• Adopt final naming scheme
• Adjust/complete setup.py metadata

Infrastructure:

• Move new implementation from https://github.com/dokempf/clang-format-wheel into repository
• Migrate repository into ssciwr organization
• Correct PyPI URL to actual PyPI
• Add new PyPI deploy secret

Publish versions (skipping over outdated patch releases):

• 10.0.1
• 11.0.1
• 11.1.0
• 12.0.1

[nightlark]

I tried out building 10.0.1 on GitHub Actions, and the qemu jobs hit the 6 hour timeout limit, while jobs for other versions queued around the same time finished in the usual ~4 hours. It might have been bad luck with slow VMs, though it could be that the older versions just take longer to build (maybe some options can be adjusted to speed up the process).

[henryiii]

You can manually build & upload the QEMU wheels locally if you want to, FYI. I've not had success doing QEMU from macOS, but a few of the CMake wheels were uploaded from Linux when we had Travis credit problems (before moving to emulation on GHA).

[nightlark]

It looks like pip installing 12.0.1 on my systems is working -- I think this issue can probably be closed now.

[dokempf]

I have finalized the transition and built the missing packages for versions 10, 11 and 12. There is one exception: The wheels that require emulation are only available for version 12, as LLVM changed its dependency structure (the number of necessary compilation units dropped by 30%) such that version 12 can be built within the 6 hour time limit of GHA, while previous versions couldn't. I am currently not planning to work a lot on older builds (diminishing returns) - please switch to newer versions of clang-format instead.

Thanks to everybody who participated in this process (@mgevaert @nightlark @henryiii ). Please use the packages, recommend them to others and report any issues you might find.