New version of redis package is the cause of the Error

[egregors]

What happened?

After redeploy, we got redis==3.0.1 instead redis==2.10.6. So, when you try to work with cache for example, we got the exception:

DataError: Invalid input of type: 'CacheKey'. Convert to a byte, string or number first.
  File "django/core/handlers/exception.py", line 35, in inner
    response = get_response(request)
  File "django/core/handlers/base.py", line 128, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "django/core/handlers/base.py", line 126, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "python3.6/contextlib.py", line 52, in inner
    return func(*args, **kwds)
  File "django/utils/decorators.py", line 134, in _wrapped_view
    result = middleware.process_request(request)
  File "django/middleware/cache.py", line 133, in process_request
    cache_key = get_cache_key(request, self.key_prefix, 'GET', cache=self.cache)
  File "django/utils/cache.py", line 360, in get_cache_key
    headerlist = cache.get(cache_key)
  File "django_redis/cache.py", line 32, in _decorator
    return method(self, *args, **kwargs)
  File "django_redis/cache.py", line 81, in get
    client=client)
  File "django_redis/client/default.py", line 202, in get
    value = client.get(key)
  File "redis/client.py", line 1207, in get
    return self.execute_command('GET', name)
  File "redis/client.py", line 754, in execute_command
    connection.send_command(*args)
  File "redis/connection.py", line 619, in send_command
    self.send_packed_command(self.pack_command(*args))
  File "redis/connection.py", line 659, in pack_command
    for arg in imap(self.encoder.encode, args):
  File "redis/connection.py", line 124, in encode
    "byte, string or number first." % typename)

The problem exactly in the redis==3.0.1, as we may see there: https://stackoverflow.com/questions/53331405/django-compress-error-invalid-input-of-type-cachekey

It is because of this pkg sets in the requiriments/base.txt as redis>=2.10.5 # https://github.com/antirez/redis

What should've happened instead?

All should work well

Steps to reproduce

Just rebuild compose.

(Ubuntu 18.04, Docker 17.05.0-ce)

[egregors]

This is all about this issue: jazzband/django-redis#342

[browniebroke]

Hi, thanks for the report. I believe it was fixed upstream in django-redis. I've just merged #1870 which should solve this.

[egregors]

@browniebroke Unfortunately, I still get Error, even with django-redis==4.10.0 and redis==3

Do you still think pinning redis to 2.10.4, <3 is a bad idea? At least as a temporary solution.

celeryworker_1  | [2018-11-20 13:30:07,838: WARNING/MainProcess] celery@cd4e42f8b190 ready.
celeryworker_1  | [2018-11-20 13:30:07,927: ERROR/MainProcess] Unrecoverable error: AttributeError("'str' object has no attribute 'items'",)
celeryworker_1  | Traceback (most recent call last):
celeryworker_1  |   File "/usr/local/lib/python3.6/site-packages/celery/worker/__init__.py", line 206, in start
celeryworker_1  |     self.blueprint.start(self)
celeryworker_1  |   File "/usr/local/lib/python3.6/site-packages/celery/bootsteps.py", line 123, in start
celeryworker_1  |     step.start(parent)
celeryworker_1  |   File "/usr/local/lib/python3.6/site-packages/celery/bootsteps.py", line 374, in start
celeryworker_1  |     return self.obj.start()
celeryworker_1  |   File "/usr/local/lib/python3.6/site-packages/celery/worker/consumer.py", line 280, in start
celeryworker_1  |     blueprint.start(self)
celeryworker_1  |   File "/usr/local/lib/python3.6/site-packages/celery/bootsteps.py", line 123, in start
celeryworker_1  |     step.start(parent)
celeryworker_1  |   File "/usr/local/lib/python3.6/site-packages/celery/worker/consumer.py", line 884, in start
celeryworker_1  |     c.loop(*c.loop_args())
celeryworker_1  |   File "/usr/local/lib/python3.6/site-packages/celery/worker/loops.py", line 76, in asynloop
celeryworker_1  |     next(loop)
celeryworker_1  |   File "/usr/local/lib/python3.6/site-packages/kombu/async/hub.py", line 340, in create_loop
celeryworker_1  |     cb(*cbargs)
celeryworker_1  |   File "/usr/local/lib/python3.6/site-packages/kombu/transport/redis.py", line 1019, in on_readable
celeryworker_1  |     self._callbacks[queue](message)
celeryworker_1  |   File "/usr/local/lib/python3.6/site-packages/kombu/transport/virtual/__init__.py", line 534, in _callback
celeryworker_1  |     self.qos.append(message, message.delivery_tag)
celeryworker_1  |   File "/usr/local/lib/python3.6/site-packages/kombu/transport/redis.py", line 146, in append
celeryworker_1  |     pipe.zadd(self.unacked_index_key, delivery_tag, time()) \
celeryworker_1  |   File "/usr/local/lib/python3.6/site-packages/redis/client.py", line 2263, in zadd
celeryworker_1  |     for pair in iteritems(mapping):
celeryworker_1  |   File "/usr/local/lib/python3.6/site-packages/redis/_compat.py", line 123, in iteritems
celeryworker_1  |     return iter(x.items())
celeryworker_1  | AttributeError: 'str' object has no attribute 'items'

[browniebroke]

Well, the error is different. Technically, it is now coming from Celery/Kombu, not from django-celery. They did a similar fix: celery/kombu#947 but it's not yet released.

[angerrp]

@browniebroke Unfortunately, I still get Error, even with django-redis==4.10.0 and redis==3

Do you still think pinning redis to 2.10.4, <3 is a bad idea? At least as a temporary solution.

celeryworker_1  | [2018-11-20 13:30:07,838: WARNING/MainProcess] celery@cd4e42f8b190 ready.
celeryworker_1  | [2018-11-20 13:30:07,927: ERROR/MainProcess] Unrecoverable error: AttributeError("'str' object has no attribute 'items'",)
celeryworker_1  | Traceback (most recent call last):
celeryworker_1  |   File "/usr/local/lib/python3.6/site-packages/celery/worker/__init__.py", line 206, in start
celeryworker_1  |     self.blueprint.start(self)
celeryworker_1  |   File "/usr/local/lib/python3.6/site-packages/celery/bootsteps.py", line 123, in start
celeryworker_1  |     step.start(parent)
celeryworker_1  |   File "/usr/local/lib/python3.6/site-packages/celery/bootsteps.py", line 374, in start
celeryworker_1  |     return self.obj.start()
celeryworker_1  |   File "/usr/local/lib/python3.6/site-packages/celery/worker/consumer.py", line 280, in start
celeryworker_1  |     blueprint.start(self)
celeryworker_1  |   File "/usr/local/lib/python3.6/site-packages/celery/bootsteps.py", line 123, in start
celeryworker_1  |     step.start(parent)
celeryworker_1  |   File "/usr/local/lib/python3.6/site-packages/celery/worker/consumer.py", line 884, in start
celeryworker_1  |     c.loop(*c.loop_args())
celeryworker_1  |   File "/usr/local/lib/python3.6/site-packages/celery/worker/loops.py", line 76, in asynloop
celeryworker_1  |     next(loop)
celeryworker_1  |   File "/usr/local/lib/python3.6/site-packages/kombu/async/hub.py", line 340, in create_loop
celeryworker_1  |     cb(*cbargs)
celeryworker_1  |   File "/usr/local/lib/python3.6/site-packages/kombu/transport/redis.py", line 1019, in on_readable
celeryworker_1  |     self._callbacks[queue](message)
celeryworker_1  |   File "/usr/local/lib/python3.6/site-packages/kombu/transport/virtual/__init__.py", line 534, in _callback
celeryworker_1  |     self.qos.append(message, message.delivery_tag)
celeryworker_1  |   File "/usr/local/lib/python3.6/site-packages/kombu/transport/redis.py", line 146, in append
celeryworker_1  |     pipe.zadd(self.unacked_index_key, delivery_tag, time()) \
celeryworker_1  |   File "/usr/local/lib/python3.6/site-packages/redis/client.py", line 2263, in zadd
celeryworker_1  |     for pair in iteritems(mapping):
celeryworker_1  |   File "/usr/local/lib/python3.6/site-packages/redis/_compat.py", line 123, in iteritems
celeryworker_1  |     return iter(x.items())
celeryworker_1  | AttributeError: 'str' object has no attribute 'items'

@egregors did u find a proper fix for this? having the same issue