KeyError while running pylint with parallel job option using dill-0.3.5

[karthikhebbar]

07:35 dds $ find . -type f -name "*.py" | xargs pylint --output-format=colorized -j 0
Traceback (most recent call last):
File "/usr/local/bin/pylint", line 8, in
sys.exit(run_pylint())
File "/usr/local/lib/python3.9/site-packages/pylint/init.py", line 22, in run_pylint
PylintRun(argv or sys.argv[1:])
File "/usr/local/lib/python3.9/site-packages/pylint/lint/run.py", line 358, in init
linter.check(args)
File "/usr/local/lib/python3.9/site-packages/pylint/lint/pylinter.py", line 1087, in check
check_parallel(
File "/usr/local/lib/python3.9/site-packages/pylint/lint/parallel.py", line 148, in check_parallel
jobs, initializer=initializer, initargs=[dill.dumps(linter)]
File "/usr/local/lib/python3.9/site-packages/dill/_dill.py", line 364, in dumps
dump(obj, file, protocol, byref, fmode, recurse, **kwds)#, strictio)
File "/usr/local/lib/python3.9/site-packages/dill/_dill.py", line 336, in dump
Pickler(file, protocol, **_kwds).dump(obj)
File "/usr/local/lib/python3.9/site-packages/dill/_dill.py", line 620, in dump
StockPickler.dump(self, obj)
File "/usr/local/Cellar/[email protected]/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 487, in dump
self.save(obj)
File "/usr/local/Cellar/[email protected]/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 603, in save
self.save_reduce(obj=obj, *rv)
File "/usr/local/Cellar/[email protected]/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 717, in save_reduce
save(state)
File "/usr/local/Cellar/[email protected]/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 560, in save
f(self, obj) # Call unbound method with explicit self
File "/usr/local/lib/python3.9/site-packages/dill/_dill.py", line 1251, in save_module_dict
StockPickler.save_dict(pickler, obj)
File "/usr/local/Cellar/[email protected]/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 971, in save_dict
self._batch_setitems(obj.items())
File "/usr/local/Cellar/[email protected]/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 997, in _batch_setitems
save(v)
File "/usr/local/Cellar/[email protected]/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 603, in save
self.save_reduce(obj=obj, *rv)
File "/usr/local/Cellar/[email protected]/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 717, in save_reduce
save(state)
File "/usr/local/Cellar/[email protected]/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 560, in save
f(self, obj) # Call unbound method with explicit self
File "/usr/local/lib/python3.9/site-packages/dill/_dill.py", line 1251, in save_module_dict
StockPickler.save_dict(pickler, obj)
File "/usr/local/Cellar/[email protected]/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 971, in save_dict
self._batch_setitems(obj.items())
File "/usr/local/Cellar/[email protected]/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 997, in _batch_setitems
save(v)
File "/usr/local/Cellar/[email protected]/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 560, in save
f(self, obj) # Call unbound method with explicit self
File "/usr/local/lib/python3.9/site-packages/dill/_dill.py", line 1251, in save_module_dict
StockPickler.save_dict(pickler, obj)
File "/usr/local/Cellar/[email protected]/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 971, in save_dict
self._batch_setitems(obj.items())
File "/usr/local/Cellar/[email protected]/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 997, in _batch_setitems
save(v)
File "/usr/local/Cellar/[email protected]/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 603, in save
self.save_reduce(obj=obj, *rv)
File "/usr/local/Cellar/[email protected]/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 687, in save_reduce
save(cls)
File "/usr/local/Cellar/[email protected]/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 560, in save
f(self, obj) # Call unbound method with explicit self
File "/usr/local/lib/python3.9/site-packages/dill/_dill.py", line 1791, in save_type
defaults = [obj._field_defaults[field] for field in obj._fields]
File "/usr/local/lib/python3.9/site-packages/dill/_dill.py", line 1791, in
defaults = [obj._field_defaults[field] for field in obj._fields]
KeyError: 'color'

Issue is not seen with 0.3.4 dill package

[fdebrabander]

Also seeing this issue since dill 0.3.5 (pylint 2.13.9 and python 3.8.10), using "jobs=1" in pylintrc file can be used as workaround.

[mmckerns]

Thanks for reporting. That's hitting very new code, so I expect it is in need of a patch (@anivegesana).

[mmckerns]

Issued a patch release

[DanielNoord]

@mmckerns Would you consider yanking 0.3.5? Otherwise packages will need to specify that they are not compatible with it.

[mmckerns]

yes, will see about removing the assets

[DanielNoord]

Thanks! And thank you very much for the quick fix! 😁

[anivegesana]

Ah yes. I didn't consider the possibility that named tuples could have both nondefault and default values when I added support for _field_defaults. @mmckerns thanks for fixing the bug so quickly.

[anivegesana]

@mmckerns Friendly reminder to yank dill 0.3.5 proper.

https://snarky.ca/what-to-do-when-you-botch-a-release-on-pypi/amp/

[mmckerns]

Thanks, @anivegesana I haven't forgotten. Your super useful recent PRs also introduced a few bugs and other compatibility issues, as you know. I'm not certain that due to this bug we are in yank the release territory, but if not, it's close. I'm assessing based on all the issues introduced in 0.3.5.

[anivegesana]

yes, will see about removing the assets

Ah sorry. Got confused by this comment. 👍