Possible issue with parsing setup.py (pdm.models.in_process.parse_setup_py)

[sr-murthy]

• [Y ] I have searched the issue tracker and believe that this is not a duplicate.

Steps to reproduce

In a project running in a Python 3.9.6 virtual env., without a preexisting pyproject.toml, with a valid setup.py, and no previous runs of pdm, run the following from the project root:

$ pdm import -v setup.py

Actual behavior

Produces this output:

['<project src dir>', '<project root dir>', '<project venv>/lib/python3.9/site-packages/pdm/models/in_process', '<user home dir>/.pyenv/versions/3.9.6/lib/python39.zip', '<user home dir>/.pyenv/versions/3.9.6/lib/python3.9', '<user home dir>/.pyenv/versions/3.9.6/lib/python3.9/lib-dynload', '<project venv dir>/lib/python3.9/site-packages']
<valid setup.py JSON> 

json.decoder.JSONDecodeError: Expecting value: line 1 column 2 (char 1)

Expected behavior

I expect to see something like:

Changes are written to pyproject.toml.

Environment Information

PDM version:
  2.7.0
Python Interpreter:
  <project venv>/bin/python (3.9)
Project Root:
  <project dir>/
Local Packages:

{
  "implementation_name": "cpython",
  "implementation_version": "3.9.6",
  "os_name": "posix",
  "platform_machine": "x86_64",
  "platform_release": "22.5.0",
  "platform_system": "Darwin",
  "platform_version": "Darwin Kernel Version 22.5.0: Mon Apr 24 20:51:50 PDT 2023; root:xnu-8796.121.2~5/RELEASE_X86_64",
  "python_full_version": "3.9.6",
  "platform_python_implementation": "CPython",
  "python_version": "3.9",
  "sys_platform": "darwin"
}
 0 {'implementation_name': 'cpython', 'implementation_version': '3.9.6', 'os_name': 'posix', 'platform_machine': 'x86_64', 'platform_release': '22.5.0', 'platform_system': 'Darwin', 'platform_version': 'Darwin Kernel Version 22.5.0: Mon Apr 24 20:51:50 PDT 2023; root:xnu-8796.121.2~5/RELEASE_X86_64', 'python_full_version': '3.9.6', 'platform_python_implementation': 'CPython', 'python_version': '3.9', 'sys_platform': 'darwin'} 450
{
  "implementation_name": "cpython",
  "implementation_version": "3.9.6",
  "os_name": "posix",
  "platform_machine": "x86_64",
  "platform_release": "22.5.0",
  "platform_system": "Darwin",
  "platform_version": "Darwin Kernel Version 22.5.0: Mon Apr 24 20:51:50 PDT 2023; root:xnu-8796.121.2~5/RELEASE_X86_64",
  "python_full_version": "3.9.6",
  "platform_python_implementation": "CPython",
  "python_version": "3.9",
  "sys_platform": "darwin"
}

I've checked and re-checked setup.py, and it is completely valid in its original form, and as JSON (no JSON validation errors).

Using the debuggger I looked at how pdm.models.in_process.parse_setup_py is processing the output of parsing setup.py here:

    with _in_process_script("parse_setup.py") as script:
        cmd = [executable, "-Es", script, path]
        return json.loads(subprocess.check_output(cmd))

Here, the executed command, cmd, is:

['<project venv>/bin/python', '-Es', '<project venv>/lib/python3.9/site-packages/pdm/models/in_process/parse_setup.py', 'setup.py']

and output of subprocess.check_output(cmd) is the byte string:

b'[\'<project root dir>/src\', \'<project root dir>\', \'<project venv dir>/lib/python3.9/site-packages/pdm/models/in_process\', \'<user home dir>/.pyenv/versions/3.9.6/lib/python39.zip\', \'<user home dir>/.pyenv/versions/3.9.6/lib/python3.9\', \'<user home dir>/.pyenv/versions/3.9.6/lib/python3.9/lib-dynload\', \'<project venv dir>/lib/python3.9/site-packages\']\n<valid setup.py JSON>\n'

Here, json.loads cannot process this as valid jSON, because you have various directory paths being prepended to the valid setup.py JSON inside the byte string.
If I modify parse_setup_py to the following then the parsing works, and the pyproject.toml is generated:

def parse_setup_py(executable: str, path: str) -> dict[str, Any]:
    """Parse setup.py and return the kwargs"""
    with _in_process_script("parse_setup.py") as script:
        cmd = [executable, "-Es", script, path]
        s = subprocess.check_output(cmd).decode()[477:]
        return json.loads(s)

The 477 happens to be the index of the start of the valid setup.py JSON substring inside the byte string produced by subprocess.check_output.

Am I running the pdm import correctly, or is parse_setup_py running the parse command incorrectly?

[frostming]

That's because your setup.py prints extra info to the stdout.

You just can't use the importer here. You need to migrate the file yourself.

[sr-murthy]

Yes, the setup.py was migrated manually, using the intial version generated by the import, after getting rid of the extra output in cmd.