Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Proposal: Remove Python & Go bindings to their own repositories #267

Closed
Gadgetoid opened this issue Jan 25, 2018 · 55 comments
Closed

Proposal: Remove Python & Go bindings to their own repositories #267

Gadgetoid opened this issue Jan 25, 2018 · 55 comments

Comments

@Gadgetoid
Copy link
Collaborator

I don't believe having the Python & Go code here is a good idea, for many reasons but most prominent of all is that they add unnecessary burden to the maintainer, @jgarff, who must deal with PRs and issues against the language ports, in addition to the core C library.

It also sets a precedent that other language ports should be PR'd into this repository, rather than maintained by domain experts in their own repositories. This doesn't result in a very good separation of concerns. Although I appreciate it catches the edge-case where an issue with the core library might be reported against a specific port.

Do we really need to be funnelling fixes/issues for Python, Go, Mono - https://github.com/jgarff/rpi_ws281x/pull/58/files - and Rust - #265 - bindings for this library through one person?

I have the solution for Python- While it will almost certainly come across as self-serving to some, I propose that we instead focus on Pimoroni's hard fork of the Python bindings which currently lives here: https://github.com/pimoroni/rpi_ws281x-python

Why?

  • It's part of my job to maintain these bindings, so I have a vested interest in keeping on top of PRs and issues
  • These bindings are packaged and distributed via pypi, they're what you're installing if you type sudo pip install rpi_ws281x. There is no distribution for the Python code in this repository.
  • Right now we have duplication of effort- eg: Fix the default DMA in python code rpi-ws281x/rpi-ws281x-python#1 versus: Fix the default DMA in python code #266
  • I am still planning to distribute these bindings, and rpi_ws281x itself, as deb packages and submit them for inclusion in the Raspbian repository (I already do this for many Python libraries).

Right now this repository has multiple outstanding PRs and issues concerning the Python bindings:
#258
#192
#163

#252
#247
#225

Frankly, this stuff is making this repository a confusing mess. So this is my impassioned petition to ask @jgarff to find a maintainer for the Go bindings in this library, and for us to work together to EOL the Go and Python bindings maintained here.

Feedback, particularly from the major contributors here is much appreciated;

And in general @penfold42 and @jgarff

@cpheps
Copy link
Contributor

cpheps commented Jan 25, 2018

I think this is a good idea, though I don't have the time to maintain the Go version of it. Developing against this was non-trivial for Go and it would be nice to see it broken out and given some proper love.

@supcik
Copy link
Contributor

supcik commented Jan 25, 2018

I agree. I already have a dedicated repo for go: https://github.com/supcik/ws2811. I can complete the repo with a README file and review/update the code if you want. Then we can just link our projects.

@jgarff
Copy link
Owner

jgarff commented Jan 25, 2018 via email

@eloots
Copy link

eloots commented Jan 25, 2018 via email

@supcik
Copy link
Contributor

supcik commented Jan 25, 2018

Can we agree on repository names? Are rpi_ws281x_{{language}} ok for you?

@penfold42
Copy link
Contributor

Good idea to split them out as separate repositories.

Another suggestion - create a new organisation and have repositories under that.
rpi-ws281x / rpi_ws281x
rpi-ws281x / go
rpi-ws281x / python

@penfold42
Copy link
Contributor

@jgarff - I’ve created it already to stop a cheeky person grabbing it.
If this is the approach you want to take, I’ll hand it over to you

@pietrodn
Copy link
Contributor

I was mentioned for Golang but I don’t know Go at all :P
I can give a hand on Python, though.

@penfold42
Copy link
Contributor

Would each binding repository reference the “C” repository as a submodule ?

This has worked well for Hyperion - we can point it at a specific commit / tag and adjust over time when new features or breaking API changes occur

@siggy
Copy link

siggy commented Jan 26, 2018

@jgarff If you're interested in keeping everything in one repo, you can use https://help.github.com/articles/about-codeowners/ to designate owners of specific parts of the repo.

@Gadgetoid
Copy link
Collaborator Author

I'm totally on-board with @penfold42 and would love to move (and possibly rename) rpi_ws281x-python into this org if you'll have me!

I also need to redirect the submodule to the official rpi_ws281x, since I've been working from a fork that lets me be more responsive to new Pi model additions and other changes I need to expedite. I have upstreamed pretty much every feature required for Unicorn HAT/pHAT so it should be an easy switch over.

An org is how I structured the WiringPi library ports, which is a very similar case to this one: a C library with various language ports. Each port submodules the WiringPi library and binds accordingly.

See: https://github.com/wiringpi

I guess I need to:

  • Transfer https://github.com/pimoroni/rpi_ws281x-python into the org (and rename if necessary)
  • Replace the submodule with a reference to the upstream rpi_ws281x
  • Fork this back out to Pimoroni, so I can make and propose changes in a PR with proper due process
  • Start to improve the library with release notes etc and get a proper release train rolling

@supcik
Copy link
Contributor

supcik commented Jan 31, 2018

I worked on the golang wrapper (https://github.com/supcik/rpi_ws281x_go) for the library. Any decision on the "official" repo names?

@JMurph2015
Copy link

I don't mind having federated model of maintaining bindings. Let me know where and when to put stuff. Also, we should collectively start working on public API documentation so that it's easier to keep on top of what features need to be exposed through bindings. That's something I've brushed with a couple times while making Rust bindings.

@JMurph2015
Copy link

Also, I've had good luck in the past with using Git submodules (like I did with the Rust bindings I've made). It makes it easy to track another repo, and in my case I was even able to set up my build system to ensure that all submodules were initialized and downloaded before compilation. So I'm all for at least my repo using a submodule to grab the C source.

@Gadgetoid
Copy link
Collaborator Author

Reflecting upon this, one thing that specifically needs to happen with rpi_ws281x is versioning.

One particular benefit of this would be that we can release versioned shared library packages for Raspbian and bindings can depend upon these. See: #217

This also gives better visibility on fixed/outstanding bugs, new features, etc.

It's clear from the outstanding Issues and PRs that this repository needs a little more regular TLC and perhaps another eager maintainer before we try to push these ideas forward, though.

@jgarff
Copy link
Owner

jgarff commented Mar 1, 2018 via email

@Gadgetoid
Copy link
Collaborator Author

@jgarff it's okay! It happens. I don't blame you. I've done the same thing to repositories. Sometimes other concerns take priority and that's fine! We're not robots!

I'm conflicted about whether I should be a maintainer, since:

  1. I have a similar problem: Time. I'm busy with a hundred or so repositories already.
  2. I don't thoroughly understand the C code which this library is founded upon.

But on the flip side, I have a vested interest in this library and its future since Unicorn HAT and pHAT - and the code we ship to drive them - are built upon it. Aww shucks!

I suppose having a hand on the wheel would let me put the above plans into action - as we seem to have a reasonable consensus - and draw a line under the current repository state, tag it as a release and publish it to Raspbian at the very least.

@JMurph2015
Copy link

Also @Gadgetoid, I was reading your comment about versioning. So, one thing that you can do if I remember correctly, is make git submodules that point to branches instead of commits so that you could make some "1.1" branch so that someone like me could track that. Then if you wanted to do a patch, you'd just start a branch for it, then make sure the relavent minor version branch gets updated.

It's a bit convoluted, but I'm not entirely sure how best to do it. There isn't exactly a C package manager.

@SlySven
Copy link

SlySven commented Mar 6, 2018

ℹ️ One gotcha if using git submodules on GitHub: they are not included in the archives make available from the "download the source" buttons/links should an interested party want to work from a tarball of a particular release. Then, without using git (which someone who downloads a .zip or .tar.gz copy is not likely to be using) it is impossible for the person who obtains the main project to identify which tarball-ed commit of the sub-module(s) are the ones that a particular main repository branch is linked to - if there is any sort of strong linkage to the use of particulars versions.

However, if the the main releases are given identifiable tag version markers it is possible to hand-craft additional URLs to the (suggested) submodules as other tarballs in the content on the "Releases" page of the "Code" tab here when releases are made for the main repository...

@JMurph2015
Copy link

@SlySven I think the solution to that would be as you said to make tarballs ourselves in the form of "releases". I don't see a much better (in the sense of something sustainable and standardized for all of the various bindings maintainers) solution, but I'm definitely willing to be convinced. The only other way I'd thought of doing it for my Rust bindings was to use the build script to fetch a tarball itself, but then you get into how does x repo grab the source vs y repo etc. tl;dr, it would end up looking pretty different across each binding repo.

@Gadgetoid
Copy link
Collaborator Author

I'm very much in support of leveraging the "Releases" page to distribute proper tarballs or appropriate source/binary distributions. For my Python libraries (not specifically rpi_ws281x-python) I use the files functionality of Releases to include .deb files and Python binary .whl files and a .tar.gz of the source is also generated during the process of building those.

Well maintained releases will be key here, and depending upon how a library links against/wraps the rpi_ws281x package it may not even be necessary to include a submodule. If we're shipping an rpi_ws281x deb package and dev package then it could be managed as a dependency via apt.

@JMurph2015
Copy link

@penfold42 Do you think you could add some of us to the rpi-ws281x org so we could start to test the waters on this strategy?

@penfold42
Copy link
Contributor

Done

@Gadgetoid
Copy link
Collaborator Author

I've transferred rpi_w281x-python into the new org. TODO:

Move over rpi_ws281x itself

I think only @jgarff can do this? Needs shunted over and forked back.

Nomenclature.

Should we opt for rpi_ws281x-languagename or something else?

Surgery.

We'll have to remove the Python code from this repository and update the documentation as necessary.

Python examples need updating to pep8 and migrated to an examples folder in https://github.com/rpi-ws281x/rpi_ws281x-python

I've made pains to drop the Adafruit brand-name "NeoPixel" from the Python library, although a dummy module is included for backwards compatibility.

Transfer/create other language repositories

Once we've figured out the naming scheme we need to transfer in or create empty repositories for:

  1. go - needs to be grafted from: https://github.com/jgarff/rpi_ws281x/tree/master/golang
  2. mono - mono wrapper #58
  3. rust - Rust Support #265

@supcik
Copy link
Contributor

supcik commented Mar 14, 2018

I have an updated repository for go here: https://github.com/supcik/rpi_ws281x_go. If you agree, (and as soon as we have a consistent naming convention) I will move it into the new organization.

@JMurph2015
Copy link

Can we do either all hyphens or all underscores? Currently we have all but one permutation of that ( underscore-hyphen, hyphen-hyphen, underscore-underscore), I think ideally we would have some consistent naming scheme.

@Gadgetoid
Copy link
Collaborator Author

Gadgetoid commented Mar 16, 2018

My personal preference is all hyphens. When naming my version of the Python wrapper I ended up with a mix of what I consider the upstream name: "rpi_ws281x" and then the correct way to append the language: "-python" which gives a hyphen/underscore mix. I don't really have a good argument for that particular pattern, other than it's what I did with "WiringPi-Python" "WiringPi-Ruby" etc.

I'm for renaming "rpi_ws281x" to "rpi-ws281x" when it's moved into the org (which is incidentally named "rpi-ws218x" too, if we can get some consensus? Which would make the structure:

  • rpi-ws281x (org)
    • rpi-ws281x
    • rpi-ws281x-go
    • rpi-ws281x-mono
    • rpi-ws281x-rust
    • rpi-ws281x-python

It looks like rpi-ws281x-rust already fits this pattern.

@penfold42 could you rename "rpi_ws281x-python" to "rpi-ws281x-python" to be consistent with Rust, please? I don't have access.

@supcik
Copy link
Contributor

supcik commented Mar 16, 2018

It is OK for me.

@penfold42
Copy link
Contributor

Haven’t seen an email.

Does it need access to the whole or or just a repository ?

@supcik
Copy link
Contributor

supcik commented Mar 17, 2018

You granted access to CircleCi and everything is fine for me, thank you.

@penfold42
Copy link
Contributor

Yep - found the email in an old account

@Gadgetoid
Copy link
Collaborator Author

To avoid any duplication of effort we should push a deprecation warning, directing people to the new golang bindings, into the README at: https://github.com/jgarff/rpi_ws281x/tree/master/golang

@Urmel11
Copy link

Urmel11 commented Mar 22, 2018

The mono / C# wrapper in PR #58 is not up-to-date and does not work with the latest "C" implementation.
Therefore I made a new wrapper (see https://github.com/Urmel11/rpi-ws281x-csharp).
Maybe we can transfer this project to the org and decline PR #58.

What do you think @Gadgetoid, @penfold42 and @jgarff?

@Gadgetoid
Copy link
Collaborator Author

Sheesh I just looked at the date on #58!

I'd certainly welcome your C# wrapper, although I'm not qualified to comment upon its quality or accuracy.

Yours is a good example of a wrapper that could easily depend upon a pre-packed version of this library.

@penfold42
Copy link
Contributor

penfold42 commented Mar 22, 2018

@Urmel11 good idea, I’ll add you to the org

@Urmel11
Copy link

Urmel11 commented Mar 23, 2018

@Gadgetoid
Yes, this wrapper totally benefits from a pre-packed library version.
Therefore I am really excited of issue #217.

@penfold42
Did you already added me to to org? I just wanted to transfer the ownership of my project but I get an error that I have no permissions to create a repository in the org.

@penfold42
Copy link
Contributor

Sorry - busy day at work.

Just done it now

@Urmel11
Copy link

Urmel11 commented Mar 23, 2018

Thank you. I transferred the ownership now.
Only a few more hours, then it is weekend ;)

@Gadgetoid Gadgetoid mentioned this issue Mar 27, 2018
@Gadgetoid
Copy link
Collaborator Author

Looks like we've got an empty Rust repository in the org.

@JMurph2015 do you have a game plan for getting the Rust code there?

I've closed #265 for now, but I don't mind re-opening it with a "Help Wanted" tag, since you had asked for help with hand-writing some nicer bindings.

@JMurph2015
Copy link

@Gadgetoid
I'm overhaulimg the Rust bindings to make them less like dirty C wrappers and more like a typical Rust crate. Call it a 0.2.x release. So I figured I'd move it then. Though I don't suppose it'd be all that difficult to move it now.

@Gadgetoid
Copy link
Collaborator Author

I believe we're at a point now where - pending the current PR - https://github.com/rpi-ws281x/rpi-ws281x-python is ready to take on the mantle as the way-to-do-rpi_ws281x-in-Python. I'm going to write up a deprecation warning to add to the readme here: https://github.com/jgarff/rpi_ws281x/tree/master/python

This will direct Python users to the pip package and the rpi_ws281x-python GitHub repository so we can start to migrate Python support requests, bug fixes, improvements and general progress away from this repository.

I still have some maintenance and documentation to do on the Python front, but that's of no consequence to its suitability as the new way forward.

@JMurph2015 I'm easy! It's your code and your time ;) Manage it how you see fit.

@JMurph2015
Copy link

@Gadgetoid The repo is moved over, README and LICENSE also added. Still haven't gotten around to finishing the API overhaul (college is hard) but will probably be able to hit that in the early summer.

@mbelling
Copy link

@penfold42 do you need to give me access to https://github.com/rpi-ws281x/rpi-ws281x-java so that I can move my repository from https://github.com/mbelling/rpi-ws281x-java as mentioned in #205?

I was thinking I would do a repository transfer, but I am not sure if that works with a repo already in place or not.

@penfold42
Copy link
Contributor

Added you as collaborator to the java repo

@mbelling
Copy link

Thanks. Since your repo already exists, should I just push my repo to it instead of transferring?

@penfold42
Copy link
Contributor

Yep

@DanielSSilva
Copy link

Hello folks. So with the PowerShell being able to run on RPi, I will start a new implementation for it, probably based on the @Urmel11 Wrapper. Is it possible to be added to the organization? That way it is easier to commit directly over there, instead of committing to a repository of mine and them move it there

@penfold42
Copy link
Contributor

rpi-ws281x-powershell ?

@DanielSSilva
Copy link

Sound like a good name, and it follows the "convention" :)

@penfold42
Copy link
Contributor

Done

@JMurph2015
Copy link

How are we doing on this effort? Was the plan to eventually migrate the core, or are we going to leave it here?

@Gadgetoid
Copy link
Collaborator Author

I've raised a PR to deprecate the go and python bindings in this repository in favour of the separately maintained ones- #317

As I note in my PR, it would probably be simpler just to delete them and leave a note in their respective READMEs directing users to the maintained ports.

Additionally @ladyada would like to regain control of the "neopixel" namespace for their own Pi/CircuitPython cross-compatibility project. I've removed "neopixel" from rpi-ws281x-python and shipped v4.0.0 to pypi, but the outdated bindings in this repository would also conflict with their package. Mothballing these old Python examples/bindings ASAP would probably save us some support issues with clashes in future since Adafruit are keen to forge ahead with a unified CircuitPython API that will include examples/tutorials covering the use of neopixels, but not with this library.

@Gadgetoid
Copy link
Collaborator Author

The Golang and Python bindings have been unceremoniously dropped in #479

I think that concludes this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests