Migration to aioredis 2.0

[Andrew-Chen-Wang]

Hi all,

After #891, aioredis will be making a major jump to aioredis==2.0.0. The PR made aioredis an enormous async port of redis-py. For the migration docs, please refer to: https://github.com/aio-libs/aioredis-py/blob/master/docs/migration.md. To install:

pip install aioredis==2.0.0

Old pip install instructions for beta left for posterity

pip install aioredis==2.0.0b1

To install aioredis at a specific commit:

$ pip install git+git://github.com/aio-libs/aioredis-py.git@ eb4c001a598c01d9a0e5f2a24065c486c823d1b0

To install at master:

pip install git+git://github.com/aio-libs/aioredis-py.git@master

We'd love to hear how well this new version of aioredis integrates with your current production code! @abrookins and I will be updating this specific GitHub issue with any updates that you should be aware of; any breakages can be reported as a separate GitHub issue or in this one, too.

Thanks everyone!

[bmoscon]

I've been using this for a week or two (just running off a commit hash from that time period). It would be nice to be able to push my project up to pypi, but I can't because pypi wont let you use a direct dependency. Have you considered releasing a 2.0.0a preview version (or something like that)?

[Andrew-Chen-Wang]

Yes we've considered it, but I don't have release permission only seandstewart, but @abrookins might? @abrookins I suggest we push out the alpha version. It won't show up as an upgrade to people but it at least makes the installation easier to do.

[bmoscon]

You can see the list of those with access here, there are only 3

https://pypi.org/project/aioredis/

[bmoscon]

Hey @asvetlov - you're on the pypi account, any chance you could give access to the two Andrews so we can continue release work while the Sean is AFK? He made them admins on the repo, I doubt he'd be opposed to having them have pypi access

[seandstewart]

@bmoscon -

thanks for stepping in with your vote of confidence. I’ve been afk for the last ten days because my father passed away and I’m helping my mother with everything. I commented on the Maintainership ticket letting folks know.

#822 (comment)

@Andrew-Chen-Wang the GitHub action for automatic deployment from a tag is live in master so should only require a tagged commit on master - note I haven’t tested the action so it may require tweaking to get it right. If you’re unable to publish an alpha package via the action, let me know and I’ll step online to to get one published to pypi.

[bmoscon]

@seandstewart wow sorry to hear that! Take care of yourself/your family!

[abrookins]

I'll be more available to help out in May, but in the meantime I agree @Andrew-Chen-Wang, let's push an alpha 2.0 release. Pip should prevent people from installing an alpha unless they ask for it IIRC. Sounds like we need to tag 2.0.0a1 but also change the version in master to use the alpha signifier. I can work on that.

[abrookins]

@Andrew-Chen-Wang Can you review? Should be short. :D #934

[bmoscon]

alas, it looks like its hung. Hopefully there is an easy way for it to retry. . . .

[abrookins]

Our first 2.0.0 alpha release is now available on PyPI: https://pypi.org/project/aioredis/2.0.0a1/

[bmoscon]

I'm already using it :D

[rafaelSorel]

Hi guys, is there any release timeline for aioredis 2.0 ? the alpha works great so far, but using an alpha release on my main software trunk line is not the best thing to consider as an alpha version may experience major changes. Is it the documentation that delays the release?

[Andrew-Chen-Wang]

@rafaelSorel I would say ETA middle of June at worst, optimistically maybe beginning of May when abrookins takes over or middle of April when seandsteward returns and there isn't anything major.

TL;DR the following portion: I feel I need a second/third opinion before making the final huge leap to 2.0.0 and would need the RTD to be updated first.

Justification (click me)

Currently, it's mostly me (with finals and lots of essays coming up), so I'm taking this time to get feedback on those who are migrating or currently using it (some bugs did come up already, but they weren't major). Additionally, yes, the docs need to be updated and Idk who has power over that. I'd say the major prevention is the RTD, but that could be due to being an alpha version. Holistically speaking, there needs to be more than one person helping make major decisions and tying things together for this huge migration. (I have PTSD from PyJWT 1 to PyJWT 2 migration that broke an entire package because PyJWT 1 hadn't been updated in a 1.5 years).

[agronholm]

Hi, I was just pointed to this issue when discussing candidates for the first AnyIO enabled redis client. Would you be willing to consider letting me base this project on it? I would pitch in with packaging and testing setup too, as I consider those my strong suits.

[abrookins]

@rafaelSorel We don't have a release timeline yet, but we'll make sure to update this issue when we do. This release needs more testing in live environments, and we also need access to the docs site.

[agronholm]

I've gone over the code and found some dangerous code: __del__ calls code async code even though it could be triggered in any thread.

Is there a gitter room / discord server / irc channel you guys collaborate on? This would potentially go forward quicker than just posting messages here now and then.

[agronholm]

Ah, that Gitter room must be "aio-libs" which I have already joined, right?

[webknjaz]

and we also need access to the docs site.

FWIW latest gets updates on pushes to master: https://readthedocs.org/projects/aioredis/builds/

[seandstewart]

and we also need access to the docs site.

FWIW latest gets updates on pushes to master: https://readthedocs.org/projects/aioredis/builds/

Yes, but if you look those docs are empty because we switched to mkdocs from Sphinx and it’s still configured for Sphinx.

[popravich]

and we also need access to the docs site.

@seandstewart , you're now a maintainer

[mattyweb]

Is there going to be a call for testers at some point soon? I have an open source project using aioredis that's currently pinned to Python 3.7/Redis 5 but looking to update. Was wondering if there are specific stacks you're looking for feedback on.

It might be only semantics but would prefer using something with a beta label to alpha.

Stoked you're so close to release!

[seandstewart]

and we also need access to the docs site.

@seandstewart , you're now a maintainer

@popravich thanks for that - the new documentation is now live: https://aioredis.readthedocs.io/en/latest/

[abrookins]

Ah, that Gitter room must be "aio-libs" which I have already joined, right?

Hi @agronholm - that Gitter room seems pretty quiet. I'm new on this project and not sure if there's a proper real-time chat somewhere, but I hang out in the Redis discord server: http://discord.gg/redis

I don't think the server has client-specific rooms, but we have language-specific rooms like Python, etc.

[abrookins]

I'm holding off on the full 2.0.0 release until next week because I didn't have time to vet some of the open issues I wanted to get to before the release. I'll target spending more time on those next week, with a goal that we release 2.0.0 by the end of the week.

We're getting close!

[Andrew-Chen-Wang]

Note: aioredis 2.0 release will have to push back to the end of this week. I'd like to support Redis 6.2.X, and redis-py has had many awesome changes recently that we'll be merging here. Thanks for everyone's patience.

[abrookins]

@Andrew-Chen-Wang Our team is adding PRs to redis-py for 6.2.x at a blistering pace that might be hard to catch up with. What do you think about a 2.0 release this week followed closely by 6.2.x compatible commands next week? I'd like to get 2.0 out this week.

For the release, I'd like to get #1068 merged, fix #1065, and update the docs to explain that you should await subscribe() with PubSub before trying to read messages. I want to support Tasks better with that interface, but doing so will take some time to figure out the details. (Also, maybe it's not even worth the effort, I'm not sure. 😂 )

[Andrew-Chen-Wang]

@abrookins sounds good to me. A release this week at least gets people to migrate. Gettin the changes for 6.2.X can be in a minor version. For the locking mechanism, people should just follow the docs for now. Even though the tasks seems like it should just work, it's not officially supported in the docs for now. I think now's a good time to release if no more bugs have arisen.

[aviramha]

Hey, someone has mentioned before that we should have a different package name - can we consider having two at the same time? It'd ease the migration for many (including my team). We have some common infra code shared across microservices that relies on aioredis, and migrating all services at once is a bit of a mess.

[Andrew-Chen-Wang]

@aviramha That's the point of semver. Major will be breaking. Pin the package to aioredis<2.

[graingert]

@Andrew-Chen-Wang some people will want to use aioredis 1 and 2 in the same project

[aviramha]

@graingert Exactly. We have a library that uses redis for certain operations, and the service itself uses redis. If we update the library, all the other services need to be updated. and we can't update only the "service" redis library..

[bmoscon]

any time ANY library has a breaking change this would be an issue, please don't rename it for this. I haven't seen any other major python package change their library name when this happens

[seandstewart]

@graingert @aviramha -

If you have a use-case where you really need to support both major versions, then you can vendor 1.X under a versioned name (e.g., aioredis-py-v1) and direct pip to install this as a separate dependency.

I have seen this done for major breaking changes where there was a need in a monolithic code-base to have a gradual rollout. I wouldn't recommend except in extreme cases, as there's an equal risk of just having two major versions of the same library and never successfully migrating, but it's a valid option.

[graingert]

@graingert @aviramha @bmoscon We're not going to be changing the package name. People who don't know the major change and, if the 2.0 version changed its name, then no one knows about the migration process. If the 1.3.1 version package is renamed, then people's pip installation will break especially in containerized services.

What I can do rn is create a new package called aioredis-1 with package name aioredis_1 so that you can have both. It won't be maintained, but at least it'll make your guys' teams' transition easier.

Sounds great thanks!

[Andrew-Chen-Wang]

@graingert @aviramha I like Sean's answer much better than mine. Please vendor it into your own repositories. Or fork aioredis, rename setup.py's package to whatever you want (hence renaming the package), then install from your forked git repository. Especially since there were multiple changes that we did before deciding on the migration to 2.0.0 (such as support for Python 3.8 back in November). You can fix bugs and adjust to your needs thereafter.

[abrookins]

It is time. #1075

Prepare the sacrifice...

[seandstewart]

[jneines]

Huge congrats to everyone involved. I just came to study the implementation in more detail earlier the week, as I had some issues with the 1.3.1 release, just to notice that the forthcoming 2.0 will wipe all my problems away. And there it is!

This is awesome work!

[abrookins]

We've now released version 2.0.0 to PyPI. 🥳

Thanks to @seandstewart for all your hard work on the port and @Andrew-Chen-Wang for many great PRs, issues, and replies to users. (Same goes for Sean, of course!)

Thanks also to @bmerry for many great PRs and especially insights into asyncio specifics.

And thanks to all the aioredis users who tested the alpha and beta releases to help us find bugs.

Next, we plan on adding support for new commands added to Redis since version 6.0. You can use these already by calling execute_command(), but helpers are better.

We will also explore ways to make some areas of the aioredis API more easily used with the asyncio create_task() and gather() APIs.

[bauti-defi]

Round of applause for the DEV team!

[bmerry]

If anyone is used to using aioredis 1.x with fakeredis, you'll be happy to know that I'm adding support for aioredis 2.0: PR.

[gnat]

Wanted to drop a success story here- I've already transitioned my production over to aioredis 2.0, and have upgraded to Redis 6.

This effort by the community, @seandstewart and @abrookins at Redis Labs saved this library.

It's wonderful to have been a witness to this effort and I'm very happy async python continues to have a solid, maintainable, go-to Redis library. This is essential infrastructure!

[jgmarcel]

I echo @gnat’s report of success in transitioning to v2.0 and his high praise to the team of developers. Thank you, guys!

[Andrew-Chen-Wang]

Further discussion should be moved to #1225. Thank you all for the support, help, PR, and testing through these few months!