Add a more detailed --tree-json-full output option

[sschuberth]

What you were trying to do (and why)

Output the dependency tree in JSON format.

What happened (including command output)

Compare

$ pipgrip --tree-json pipgrip
{"pipgrip": {"anytree": {"six>=1.9.0": {}}, "click": {}, "enum34": {}, "packaging>=17": {"pyparsing>=2.0.2": {}, "six": {}}, "pip>=7.1.0": {}, "pkginfo>=1.4.2": {}, "setuptools>=38.3": {}, "typing": {}, "wheel": {}}}

to

$ pipgrip --tree-ascii pipgrip
pipgrip (0.5.0)
|-- anytree (2.8.0)
|   +-- six>=1.9.0 (1.15.0)
|-- click (7.1.2)
|-- enum34 (1.1.10)
|-- packaging>=17 (20.4)
|   |-- pyparsing>=2.0.2 (2.4.7)
|   +-- six (1.15.0)
|-- pip>=7.1.0 (20.1.1)
|-- pkginfo>=1.4.2 (1.5.0.1)
|-- setuptools>=38.3 (44.1.1)
|-- typing (3.7.4.3)
+-- wheel (0.34.2)

to see that e.g. the wheel version "0.34.2" is missing from JSON output while it's present in ASCII output.

What you expected to happen

Each JSON node probably should have two properties, version and dependencies, where version is the use version of the package itself, and dependencies a list of its dependencies (where each dependency is of the same type as the parent node).

[ddelange]

Hey @sschuberth, I thought about data structure as well, and decided to avoid structures where each node has multiple attributes, as it makes recursing unnecessarily painful for most use-cases. Also, combining the stated version ranges in the tree (a static statement) with exact version pins (dynamic, dependent on runtime) in the dependency tree for a package feels wrong.

However, for the exact version pins, I added --tree-json-exact in the same plain structure, and for your use case I imagine a --tree-json-full flag could do the trick if you want a nested multi-attribute JSON tree with all the data about the Nodes built by pipgrip.

Anytree supports dict export, with all attributes of the Node nested inside like you suggested. That would be something like:

from anytree.exporter import DictExporter
from collections import OrderedDict

exporter = DictExporter(dictcls=OrderedDict, attriter=sorted)

tree_dict_full = exporter.export(tree_root)["__root__"]["children"]

and then for the new flag something like https://github.com/ddelange/pipgrip/blob/0.5.0/src/pipgrip/cli.py#L380-L383:

elif tree_json_full:
    output = dumps(tree_dict_full, sort_keys=sort)

The parent attr will be omitted (anytree builtin), and metadata attr which I added and is currently unused can be removed altogether (to keep the resulting output more or less readable).

Default Node attribute is the package name (e.g. aiobotocore). Others:

• pip_string (e.g. aiobotocore[awscli]<1)
• extras_name (aiobotocore[awscli])
• version (resolved, e.g. 0.12.0)
• optionally cyclic

Check out https://github.com/ddelange/pipgrip/blob/0.5.0/src/pipgrip/cli.py#L119 and https://github.com/ddelange/pipgrip/blob/0.5.0/src/pipgrip/cli.py#L135 to compare to the current outputs of --tree and --tree-json & --tree-json-exact respectively.

A PR is very welcome!

[sschuberth]

Thanks for pointing out the new --tree-json-exact. I was wondering what it's there for, and with a bit of string parsing, that will do the trick for my use case.

However, in general I believe the new plethora of tree-related options to be confusing, and probably unnecessary. For example, I regard --tree-ascii just as a work-around for --tree not being aware of the character encoding used by the OS. You know the input encoding used in your Python files (probably UTF-8), and you can query the output encoding used by the OS, so you should always be able to print the correct tree / branch characters. Maybe this StackOverflow answer helps.

Similarly, I still believe --tree-json and --tree-json-exact should be combined into one by using multiple attributes per node. Maybe more attributes will follow in the future (like the declared license or so), and it's just much more convenient from a user perspective.

Just FYI, in the context of the OSS Review Toolkit project I requested something similar for Haskell's Stack, and the JSON format looks like this.

[ddelange]

Interesting take on it, probably merged too soon ;) Thanks for the wisdom, yes I can relate to that, also probably json wouldn't be used by anything that can't subsequently cherry pick the relevant info for them. I'm quite swamped currently, but I'll leave this open and once these changes are sorted, do either a 1.0.0 listing breaking changes to --tree-json, or avoid breaking changes, hide --tree-json and -exact options (but keep them), and introduce --json --json combo to give full output with all relevant attributes.

So without breaking changes, that would be:

• shown in help:

  --tree
  --tree-ascii
  --tree --json  (currently disallowed)
  

• hidden

  --tree-json
  --tree-json-exact
  

Proposed addition:

$ pipgrip --json --tree 'aiobotocore[awscli,boto3]~=0.1' 'urllib3<1.25.10'
[
    {
        "aiobotocore[awscli,boto3]~=0.1rc1": {
            "name": "aiobotocore",
            "extras_name": "aiobotocore[awscli,boto3]",
            "pip_string": "aiobotocore[awscli,boto3]~=0.1rc1",
            "version": "0.12.0",
            "extras": ["awscli", "boto3"],
            "dependencies": [...]
        }
    },
    {"urllib3<1.25.10": {...}}
]

that better?

[ddelange]

also, @sschuberth, what would you suggest in the Keras case in the README? cyclic attr will be there, so just empty array under the dependencies key just like for depencency-less packages?

[sschuberth]

that better?

Much better 😃

what would you suggest in the Keras case in the README? cyclic attr will be there, so just empty array under the dependencies key just like for depencency-less packages?

Good question. Yes, I think setting a cyclic attribute to true and therefore omitting the dependencies attributes (to distinguish from packages without dependencies) is a good approach.

[ddelange]

This is (prettyprinted) output for pipgrip --json --tree keras==2.2.2 (cyclic example from the README). PR up in a sec. Looking good @sschuberth? Anytree's dict structure turned out to be slightly different from what I suggested above.

[{'dependencies': [{'dependencies': [{'extras_name': 'numpy',
                                      'name': 'numpy',
                                      'pip_string': 'numpy>=1.7',
                                      'version': '1.16.6'},
                                     {'extras_name': 'six',
                                      'name': 'six',
                                      'pip_string': 'six',
                                      'version': '1.13.0'}],
                    'extras_name': 'h5py',
                    'name': 'h5py',
                    'pip_string': 'h5py',
                    'version': '2.10.0'},
                   {'dependencies': [{'dependencies': [{'extras_name': 'numpy',
                                                        'name': 'numpy',
                                                        'pip_string': 'numpy>=1.7',
                                                        'version': '1.16.6'},
                                                       {'extras_name': 'six',
                                                        'name': 'six',
                                                        'pip_string': 'six',
                                                        'version': '1.13.0'}],
                                      'extras_name': 'h5py',
                                      'name': 'h5py',
                                      'pip_string': 'h5py',
                                      'version': '2.10.0'},
                                     {'cyclic': True,
                                      'extras_name': 'keras',
                                      'name': 'keras',
                                      'pip_string': 'keras>=2.1.6',
                                      'version': '2.2.2'},
                                     {'extras_name': 'numpy',
                                      'name': 'numpy',
                                      'pip_string': 'numpy>=1.9.1',
                                      'version': '1.16.6'}],
                    'extras_name': 'keras-applications',
                    'name': 'keras-applications',
                    'pip_string': 'keras-applications==1.0.4',
                    'version': '1.0.4'},
                   {'dependencies': [{'cyclic': True,
                                      'extras_name': 'keras',
                                      'name': 'keras',
                                      'pip_string': 'keras>=2.1.6',
                                      'version': '2.2.2'},
                                     {'extras_name': 'numpy',
                                      'name': 'numpy',
                                      'pip_string': 'numpy>=1.9.1',
                                      'version': '1.16.6'},
                                     {'dependencies': [{'extras_name': 'numpy',
                                                        'name': 'numpy',
                                                        'pip_string': 'numpy>=1.8.2',
                                                        'version': '1.16.6'}],
                                      'extras_name': 'scipy',
                                      'name': 'scipy',
                                      'pip_string': 'scipy>=0.14',
                                      'version': '1.2.2'},
                                     {'extras_name': 'six',
                                      'name': 'six',
                                      'pip_string': 'six>=1.9.0',
                                      'version': '1.13.0'}],
                    'extras_name': 'keras-preprocessing',
                    'name': 'keras-preprocessing',
                    'pip_string': 'keras-preprocessing==1.0.2',
                    'version': '1.0.2'},
                   {'extras_name': 'numpy',
                    'name': 'numpy',
                    'pip_string': 'numpy>=1.9.1',
                    'version': '1.16.6'},
                   {'extras_name': 'pyyaml',
                    'name': 'pyyaml',
                    'pip_string': 'pyyaml',
                    'version': '5.3'},
                   {'dependencies': [{'extras_name': 'numpy',
                                      'name': 'numpy',
                                      'pip_string': 'numpy>=1.8.2',
                                      'version': '1.16.6'}],
                    'extras_name': 'scipy',
                    'name': 'scipy',
                    'pip_string': 'scipy>=0.14',
                    'version': '1.2.2'},
                   {'extras_name': 'six',
                    'name': 'six',
                    'pip_string': 'six>=1.9.0',
                    'version': '1.13.0'}],
  'extras_name': 'keras',
  'name': 'keras',
  'pip_string': 'keras==2.2.2',
  'version': '2.2.2'}]

[sschuberth]

That basically looks good to me! Many just one nit, personally I'd prefer this order of attributes: name, extras_name, version, pip_string, dependencies / cyclic.

[ddelange]

@sschuberth nit incorporated! the example above can still be obtained by additionally passing --sort.

[{'name': 'keras',
  'extras_name': 'keras',
  'version': '2.2.2',
  'pip_string': 'keras==2.2.2',
  'dependencies': [{'name': 'h5py',
                    'extras_name': 'h5py',
                    'version': '2.10.0',
                    'pip_string': 'h5py',
                    'dependencies': [{'name': 'numpy',
                                      'extras_name': 'numpy',
                                      'version': '1.19.1',
                                      'pip_string': 'numpy>=1.7'},
                                     {'name': 'six',
                                      'extras_name': 'six',
                                      'version': '1.15.0',
                                      'pip_string': 'six'}]},
                   {'name': 'keras-applications',
                    'extras_name': 'keras-applications',
                    'version': '1.0.4',
                    'pip_string': 'keras-applications==1.0.4',
                    'dependencies': [{'name': 'h5py',
                                      'extras_name': 'h5py',
                                      'version': '2.10.0',
                                      'pip_string': 'h5py',
                                      'dependencies': [{'name': 'numpy',
                                                        'extras_name': 'numpy',
                                                        'version': '1.19.1',
                                                        'pip_string': 'numpy>=1.7'},
                                                       {'name': 'six',
                                                        'extras_name': 'six',
                                                        'version': '1.15.0',
                                                        'pip_string': 'six'}]},
                                     {'name': 'keras',
                                      'extras_name': 'keras',
                                      'version': '2.2.2',
                                      'pip_string': 'keras>=2.1.6',
                                      'cyclic': True},
                                     {'name': 'numpy',
                                      'extras_name': 'numpy',
                                      'version': '1.19.1',
                                      'pip_string': 'numpy>=1.9.1'}]},
                   {'name': 'keras-preprocessing',
                    'extras_name': 'keras-preprocessing',
                    'version': '1.0.2',
                    'pip_string': 'keras-preprocessing==1.0.2',
                    'dependencies': [{'name': 'keras',
                                      'extras_name': 'keras',
                                      'version': '2.2.2',
                                      'pip_string': 'keras>=2.1.6',
                                      'cyclic': True},
                                     {'name': 'numpy',
                                      'extras_name': 'numpy',
                                      'version': '1.19.1',
                                      'pip_string': 'numpy>=1.9.1'},
                                     {'name': 'scipy',
                                      'extras_name': 'scipy',
                                      'version': '1.5.2',
                                      'pip_string': 'scipy>=0.14',
                                      'dependencies': [{'name': 'numpy',
                                                        'extras_name': 'numpy',
                                                        'version': '1.19.1',
                                                        'pip_string': 'numpy>=1.14.5'}]},
                                     {'name': 'six',
                                      'extras_name': 'six',
                                      'version': '1.15.0',
                                      'pip_string': 'six>=1.9.0'}]},
                   {'name': 'numpy',
                    'extras_name': 'numpy',
                    'version': '1.19.1',
                    'pip_string': 'numpy>=1.9.1'},
                   {'name': 'pyyaml',
                    'extras_name': 'pyyaml',
                    'version': '5.3.1',
                    'pip_string': 'pyyaml'},
                   {'name': 'scipy',
                    'extras_name': 'scipy',
                    'version': '1.5.2',
                    'pip_string': 'scipy>=0.14',
                    'dependencies': [{'name': 'numpy',
                                      'extras_name': 'numpy',
                                      'version': '1.19.1',
                                      'pip_string': 'numpy>=1.14.5'}]},
                   {'name': 'six',
                    'extras_name': 'six',
                    'version': '1.15.0',
                    'pip_string': 'six>=1.9.0'}]}]

[sschuberth]

the example above can still be obtained by additionally passing --sort.

Thanks, that's looking good, although I have to admit I see no value in sorting the JSON attributes.

[ddelange]

yes, generally JSON is order agnostic but the addition was small so I went for it 😄

thanks for all the feedback!