"pyproject.toml does not exist" when pdm add git+https from a subdirectory of a repository #1036

cye18 · 2022-04-12T21:05:10Z

I have searched the issue tracker and believe that this is not a duplicate.

Make sure you run commands with -v flag before pasting the output.

Steps to reproduce

In an empty directory
pdm init
pdm add -v git+https://username:[email protected]/path/to/my/project.git#egg=package_name#subdirectory=python_utils

Actual behavior

======== Start resolving requirements ========
  package_name-subdirectory-python-utils @ git+https://username:[email protected]/path/to/my/project.git
  python>=3.8,<3.10.1
  Adding requirement package_name-subdirectory-python-utils @ git+https://username:[email protected]/path/to/my/project.git
Preparing isolated env for PEP 517 build...
Traceback (most recent call last):
  File "/home/ds/.local/share/pdm/venv/lib/python3.8/site-packages/pdm/pep517/metadata.py", line 72, in _read_pyproject
    with self.filepath.open("rb") as f:
  File "/usr/lib/python3.8/pathlib.py", line 1222, in open
    return io.open(self, mode, buffering, encoding, errors, newline,
  File "/usr/lib/python3.8/pathlib.py", line 1078, in _opener
    return self._accessor.open(self, flags, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/pdm-build-inloop56/pyproject.toml'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ds/.local/bin/pdm", line 8, in <module>
    sys.exit(main())
  File "/home/ds/.local/share/pdm/venv/lib/python3.8/site-packages/pdm/core.py", line 233, in main
    return Core().main(args)
  File "/home/ds/.local/share/pdm/venv/lib/python3.8/site-packages/pdm/core.py", line 168, in main
    raise cast(Exception, err).with_traceback(traceback)
  File "/home/ds/.local/share/pdm/venv/lib/python3.8/site-packages/pdm/core.py", line 163, in main
    f(options.project, options)
  File "/home/ds/.local/share/pdm/venv/lib/python3.8/site-packages/pdm/cli/commands/add.py", line 53, in handle
    actions.do_add(
  File "/home/ds/.local/share/pdm/venv/lib/python3.8/site-packages/pdm/cli/actions.py", line 251, in do_add
    resolved = do_lock(project, strategy, tracked_names, reqs, dry_run=dry_run)
  File "/home/ds/.local/share/pdm/venv/lib/python3.8/site-packages/pdm/cli/actions.py", line 90, in do_lock
    mapping, dependencies = resolve(
  File "/home/ds/.local/share/pdm/venv/lib/python3.8/site-packages/pdm/resolver/core.py", line 31, in resolve
    result = resolver.resolve(requirements, max_rounds)
  File "/home/ds/.local/share/pdm/venv/lib/python3.8/site-packages/resolvelib/resolvers.py", line 481, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
  File "/home/ds/.local/share/pdm/venv/lib/python3.8/site-packages/resolvelib/resolvers.py", line 348, in resolve
    self._add_to_criteria(self.state.criteria, r, parent=None)
  File "/home/ds/.local/share/pdm/venv/lib/python3.8/site-packages/resolvelib/resolvers.py", line 168, in _add_to_criteria
    candidates=build_iter_view(matches),
  File "/home/ds/.local/share/pdm/venv/lib/python3.8/site-packages/resolvelib/structs.py", line 164, in build_iter_view
    matches = list(matches)
  File "/home/ds/.local/share/pdm/venv/lib/python3.8/site-packages/pdm/resolver/providers.py", line 237, in find_matches
    yield from super().find_matches(identifier, requirements, incompatibilities)
  File "/home/ds/.local/share/pdm/venv/lib/python3.8/site-packages/pdm/resolver/providers.py", line 145, in find_matches
    candidates = self._find_candidates(reqs[0])
  File "/home/ds/.local/share/pdm/venv/lib/python3.8/site-packages/pdm/resolver/providers.py", line 125, in _find_candidates
    can.prepare(self.repository.environment).prepare_metadata()
  File "/home/ds/.local/share/pdm/venv/lib/python3.8/site-packages/pdm/models/candidates.py", line 397, in prepare_metadata
    self._metadata_dir = builder(
  File "/home/ds/.local/share/pdm/venv/lib/python3.8/site-packages/pdm/builders/base.py", line 185, in __init__
    self.init_build_system(build_system)
  File "/home/ds/.local/share/pdm/venv/lib/python3.8/site-packages/pdm/builders/base.py", line 203, in init_build_system
    self.ensure_setup_py()
  File "/home/ds/.local/share/pdm/venv/lib/python3.8/site-packages/pdm/builders/base.py", line 318, in ensure_setup_py
    return builder.ensure_setup_py().as_posix()
  File "/home/ds/.local/share/pdm/venv/lib/python3.8/site-packages/pdm/pep517/base.py", line 433, in ensure_setup_py
    setup_py_path.write_text(self.format_setup_py(), encoding="utf-8")
  File "/home/ds/.local/share/pdm/venv/lib/python3.8/site-packages/pdm/pep517/base.py", line 278, in format_setup_py
    meta = self.meta
  File "/home/ds/.local/share/pdm/venv/lib/python3.8/site-packages/pdm/pep517/base.py", line 143, in meta
    self._meta = Metadata(self.location / "pyproject.toml")
  File "/home/ds/.local/share/pdm/venv/lib/python3.8/site-packages/pdm/pep517/metadata.py", line 68, in __init__
    self._read_pyproject()
  File "/home/ds/.local/share/pdm/venv/lib/python3.8/site-packages/pdm/pep517/metadata.py", line 75, in _read_pyproject
    raise ProjectError("pyproject.toml does not exist.")
pdm.pep517.exceptions.ProjectError: pyproject.toml does not exist.

If I install other packages using pdm add, it completes with no error. (I tried pandas)

Expected behavior

Install the package and add the corresponding dependency in pyproject.toml

Environment Information

# Paste the output of `pdm info && pdm info --env` below:
$ pdm info && pdm info --env
PDM version:        1.14.0
Python Interpreter: /usr/bin/python3.8 (3.8)
Project Root:       /path/to/my/project
Project Packages:   /path/to/my/project/__pypackages__/3.8
{
  "implementation_name": "cpython",
  "implementation_version": "3.8.10",
  "os_name": "posix",
  "platform_machine": "x86_64",
  "platform_release": "5.4.0-1040-gcp",
  "platform_system": "Linux",
  "platform_version": "#43-Ubuntu SMP Fri Mar 19 17:49:48 UTC 2021",
  "python_full_version": "3.8.10",
  "platform_python_implementation": "CPython",
  "python_version": "3.8",
  "sys_platform": "linux"
}

The text was updated successfully, but these errors were encountered:

frostming · 2022-04-13T01:36:20Z

A package must contain either pyproject.toml or setup.py to be installable. The same applies to a subdirectory.

cye18 · 2022-04-13T02:51:14Z

I double checked and confirmed that the subdirectory does have a setup.py. I was able to install it with Pipenv or pip, so it should be installable.

I found another potential issue while debugging this one.

Steps to reproduce

In an empty folder
Create requirements.txt and insert git+https://username:[email protected]/path/to/my/project.git#subdirectory=python_utils to it.
pdm init
It will ask if I want to import dependencies from requirements.txt. Select Yes.

Actual behavior

It added package_name@ git+https://username:[email protected]/path/to/my/project.git to the dependencies. The subdirectory is missing.

I ran pdm install and it gave me the same "pyproject.toml does not exist" error. It makes sense, because PDM might not be able to know about the subdirectory. So I ran pdm add -v git+https://username:[email protected]/path/to/my/project.git#egg=package_name#subdirectory=python_utils, and it still showed the same error, which is what I originally reported in this issue.

Expected behavior

It loads the subdirectory information to pyproject.toml from requirements.txt when I run pdm init.

cye18 · 2022-04-13T02:53:31Z

I found a solution to the original issue. If I manually add package_name@ git+https://username:[email protected]/path/to/my/project.git#subdirectory=python_utils to pyproject.toml dependencies, it works when I run pdm install.

However, it doesn't work with pdm add git+https://username:[email protected]/path/to/my/project.git#egg=package_name#subdirectory=python_utils

frostming · 2022-04-13T03:13:13Z

git+https://username:[email protected]/path/to/my/project.git#egg=package_name#subdirectory=python_utils

This is not valid. The second # should be a &. It should be:

git+https://username:[email protected]/path/to/my/project.git#egg=package_name&subdirectory=python_utils

which is equal to

package_name@ git+https://username:[email protected]/path/to/my/project.git#subdirectory=python_utils

It added package_name@ git+https://username:[email protected]/path/to/my/project.git to the dependencies. The subdirectory is missing.

This looks wrong and needs to be fixed.

cye18 · 2022-04-13T04:35:54Z

git+https://username:[email protected]/path/to/my/project.git#egg=package_name#subdirectory=python_utils

This is not valid. The second # should be a &. It should be:
git+https://username:[email protected]/path/to/my/project.git#egg=package_name&subdirectory=python_utils
which is equal to
package_name@ git+https://username:[email protected]/path/to/my/project.git#subdirectory=python_utils

Good catch! I run pdm add git+https://username:[email protected]/path/to/my/project.git#egg=package_name&subdirectory=python_utils and still get the same pdm.pep517.exceptions.ProjectError: pyproject.toml does not exist., while pip install works.

It added package_name@ git+https://username:[email protected]/path/to/my/project.git to the dependencies. The subdirectory is missing.

This looks wrong and needs to be fixed.

Thanks! 👍

frostming · 2022-04-13T04:54:12Z

It works fine for me:

pdm add "git+https://github.com/techalchemy/test-project.git#egg=pep508-package&subdirectory=parent_folder/pep508-package"

You can directly run this command. You should wrap the requirement in quotes.

cye18 · 2022-04-13T14:31:09Z

You should wrap the requirement in quotes.

The quotes are certainly needed.

It's wierd that installing pep508-package works, but it doesn't work for my package 😂. I'm sure there is a setup.py, but it keeps saying pyproject.toml does not exist. It shouldn't be PDM's problem though. I'll keep you posted if I could reproduce the issue in other scenarios. Thanks.

cye18 · 2022-04-13T15:33:33Z

A note to anyone who has the same issue. As mentioned in the comment, while pdm add "git+https" could give you the error, manually adding package_name@ git+https://username:[email protected]/path/to/my/project.git#subdirectory=python_utils to dependencies in pyproject.toml works for me.

werneric · 2022-10-07T13:19:39Z

I had the exact same problem, although @cye18's workaround with manual addition didn't do it for me.
I've tried with 2.1.0 and 2.1.5 versions and none of those worked.
Then, miraculously, installing 2.1.3 and retrying helped.

cye18 added the 🐛 bug Something isn't working label Apr 12, 2022

frostming mentioned this issue Apr 13, 2022

Encode the requirments.txt from ireq #1037

Merged

2 tasks

frostming closed this as completed in #1037 Apr 13, 2022

menzenski mentioned this issue Jul 11, 2022

Add setup.py file to glue-1.0 branch awslabs/aws-glue-libs#144

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

"pyproject.toml does not exist" when pdm add git+https from a subdirectory of a repository #1036

"pyproject.toml does not exist" when pdm add git+https from a subdirectory of a repository #1036

cye18 commented Apr 12, 2022

frostming commented Apr 13, 2022

cye18 commented Apr 13, 2022

cye18 commented Apr 13, 2022

frostming commented Apr 13, 2022 •

edited

Loading

cye18 commented Apr 13, 2022

frostming commented Apr 13, 2022 •

edited

Loading

cye18 commented Apr 13, 2022

cye18 commented Apr 13, 2022

werneric commented Oct 7, 2022

"pyproject.toml does not exist" when pdm add git+https from a subdirectory of a repository #1036

"pyproject.toml does not exist" when pdm add git+https from a subdirectory of a repository #1036

Comments

cye18 commented Apr 12, 2022

Steps to reproduce

Actual behavior

Expected behavior

Environment Information

frostming commented Apr 13, 2022

cye18 commented Apr 13, 2022

Steps to reproduce

Actual behavior

Expected behavior

cye18 commented Apr 13, 2022

frostming commented Apr 13, 2022 • edited Loading

cye18 commented Apr 13, 2022

frostming commented Apr 13, 2022 • edited Loading

cye18 commented Apr 13, 2022

cye18 commented Apr 13, 2022

werneric commented Oct 7, 2022

frostming commented Apr 13, 2022 •

edited

Loading

frostming commented Apr 13, 2022 •

edited

Loading