Seeing KeyError while running pylint with parallel job option (on dill package 0.3.5)

[karthikhebbar]

Bug description

Seeing KeyError while running pylint with parallel job option (on dill package 0.3.5)
New dill package came out yesterday post which seeing this issue.

This issue is not seen when parallel execution is not used.

07:39 $ pip install --upgrade pylint==2.13.9
DEPRECATION: Configuring installation scheme with distutils config files is deprecated and will no longer work in the near future. If you are using a Homebrew or Linuxbrew Python, please see discussion at Homebrew/homebrew-core#76621
Requirement already satisfied: pylint==2.13.9 in /usr/local/lib/python3.9/site-packages (2.13.9)
Requirement already satisfied: platformdirs>=2.2.0 in /usr/local/lib/python3.9/site-packages (from pylint==2.13.9) (2.5.1)
Requirement already satisfied: typing-extensions>=3.10.0 in /usr/local/lib/python3.9/site-packages (from pylint==2.13.9) (4.1.1)
Requirement already satisfied: dill>=0.2 in /usr/local/lib/python3.9/site-packages (from pylint==2.13.9) (0.3.5)
Requirement already satisfied: astroid<=2.12.0-dev0,>=2.11.5 in /usr/local/lib/python3.9/site-packages (from pylint==2.13.9) (2.11.5)
Requirement already satisfied: tomli>=1.1.0 in /usr/local/lib/python3.9/site-packages (from pylint==2.13.9) (2.0.1)
Requirement already satisfied: isort<6,>=4.2.5 in /usr/local/lib/python3.9/site-packages (from pylint==2.13.9) (5.10.1)
Requirement already satisfied: mccabe<0.8,>=0.6 in /usr/local/lib/python3.9/site-packages (from pylint==2.13.9) (0.6.1)
Requirement already satisfied: lazy-object-proxy>=1.4.0 in /usr/local/lib/python3.9/site-packages (from astroid<=2.12.0-dev0,>=2.11.5->pylint==2.13.9) (1.7.1)
Requirement already satisfied: wrapt<2,>=1.11 in /usr/local/lib/python3.9/site-packages (from astroid<=2.12.0-dev0,>=2.11.5->pylint==2.13.9) (1.13.3)
Requirement already satisfied: setuptools>=20.0 in /usr/local/lib/python3.9/site-packages (from astroid<=2.12.0-dev0,>=2.11.5->pylint==2.13.9) (60.5.0)

Configuration

No response

Command used

07:40 dds $ find . -type f -name "*.py" | xargs pylint --output-format=colorized -j 0

Pylint output

Traceback (most recent call last):
  File "/usr/local/bin/pylint", line 8, in <module>
    sys.exit(run_pylint())
  File "/usr/local/lib/python3.9/site-packages/pylint/__init__.py", line 22, in run_pylint
    PylintRun(argv or sys.argv[1:])
  File "/usr/local/lib/python3.9/site-packages/pylint/lint/run.py", line 358, in __init__
    linter.check(args)
  File "/usr/local/lib/python3.9/site-packages/pylint/lint/pylinter.py", line 1087, in check
    check_parallel(
  File "/usr/local/lib/python3.9/site-packages/pylint/lint/parallel.py", line 148, in check_parallel
    jobs, initializer=initializer, initargs=[dill.dumps(linter)]
  File "/usr/local/lib/python3.9/site-packages/dill/_dill.py", line 364, in dumps
    dump(obj, file, protocol, byref, fmode, recurse, **kwds)#, strictio)
  File "/usr/local/lib/python3.9/site-packages/dill/_dill.py", line 336, in dump
    Pickler(file, protocol, **_kwds).dump(obj)
  File "/usr/local/lib/python3.9/site-packages/dill/_dill.py", line 620, in dump
    StockPickler.dump(self, obj)
  File "/usr/local/Cellar/[email protected]/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 487, in dump
    self.save(obj)
  File "/usr/local/Cellar/[email protected]/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 603, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/local/Cellar/[email protected]/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 717, in save_reduce
    save(state)
  File "/usr/local/Cellar/[email protected]/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/usr/local/lib/python3.9/site-packages/dill/_dill.py", line 1251, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/usr/local/Cellar/[email protected]/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 971, in save_dict
    self._batch_setitems(obj.items())
  File "/usr/local/Cellar/[email protected]/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 997, in _batch_setitems
    save(v)
  File "/usr/local/Cellar/[email protected]/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 603, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/local/Cellar/[email protected]/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 717, in save_reduce
    save(state)
  File "/usr/local/Cellar/[email protected]/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/usr/local/lib/python3.9/site-packages/dill/_dill.py", line 1251, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/usr/local/Cellar/[email protected]/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 971, in save_dict
    self._batch_setitems(obj.items())
  File "/usr/local/Cellar/[email protected]/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 997, in _batch_setitems
    save(v)
  File "/usr/local/Cellar/[email protected]/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/usr/local/lib/python3.9/site-packages/dill/_dill.py", line 1251, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/usr/local/Cellar/[email protected]/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 971, in save_dict
    self._batch_setitems(obj.items())
  File "/usr/local/Cellar/[email protected]/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 997, in _batch_setitems
    save(v)
  File "/usr/local/Cellar/[email protected]/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 603, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/local/Cellar/[email protected]/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 687, in save_reduce
    save(cls)
  File "/usr/local/Cellar/[email protected]/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/usr/local/lib/python3.9/site-packages/dill/_dill.py", line 1791, in save_type
    defaults = [obj._field_defaults[field] for field in obj._fields]
  File "/usr/local/lib/python3.9/site-packages/dill/_dill.py", line 1791, in <listcomp>
    defaults = [obj._field_defaults[field] for field in obj._fields]
KeyError: 'color'

Expected behavior

Pylint is supposed to run, or dependencies needs to be updated to not use 0.3.5 until there is solution.

Pylint version

07:40 dds $ pylint --version
pylint 2.13.9
astroid 2.11.5
Python 3.9.10 (main, Jan 15 2022, 11:48:00) 
[Clang 13.0.0 (clang-1300.0.29.3)]

OS / Environment

Mac

Additional dependencies

07:41 dds $ pip freeze
aenum==3.1.5
astroid==2.11.5
attrs==21.4.0
bcrypt==3.2.0
boto3==1.21.41
botocore==1.24.41
certifi==2021.10.8
cffi==1.15.0
charset-normalizer==2.0.12
configparser==5.2.0
coverage==6.1.1
cryptography==36.0.1
decorator==5.1.1
dill==0.3.5
dnspython==2.1.0
future==0.18.2
git-pylint-commit-hook==2.6.1
idna==3.3
iniconfig==1.1.1
isort==5.10.1
jmespath==1.0.0
lazy-object-proxy==1.7.1
lxml==4.7.1
mccabe==0.6.1
netaddr==0.8.0
packaging==21.3
paramiko==2.9.2
pexpect==4.8.0
platformdirs==2.5.1
pluggy==1.0.0
polling==0.3.2
ptyprocess==0.7.0
py==1.11.0
pycparser==2.21
pydash==5.1.0
pylint==2.13.9
PyNaCl==1.5.0
pyOpenSSL==21.0.0
pyparsing==3.0.7
pytest==6.2.5
pytest-cov==2.12.1
python-dateutil==2.8.2
python-hosts==1.0.2
python-nmap==0.7.1
pytz==2022.1
requests==2.27.1
responses==0.17.0
s3transfer==0.5.2
six==1.16.0
toml==0.10.2
tomli==2.0.1
typing_extensions==4.1.1
urllib3==1.26.8
validators==0.18.2
win-inet-pton==1.1.0
wrapt==1.13.3

[fdebrabander]

Also seeing this issue since dill 0.3.5 (pylint 2.13.9 and python 3.8.10), using "jobs=1" in pylintrc file can be used as workaround.

[DanielNoord]

Regression also reported here: uqfoundation/dill#478
Let's see if this is a regression or if we are doing something wrong.

[fdebrabander]

Dill have created a possible fix with uqfoundation/dill@5ee1206 and released dill version 0.3.5.1 which includes this fix.

Have not yet verified the fix myself.

[DanielNoord]

Closing this as a patch release has been released and the broken release is going to get yanked. Thanks for reporting!