Package scanning update #482

JonoYang · 2022-08-02T01:51:54Z

This PR updates application package scanning in scancode.io to reflect the updates made to package scanning in scancode-toolkit. Most notably, the DiscoveredDependency model has been added to scancode.io (as the analogue to the Dependency model from scancode-toolkit), and the scan_for_application_packages scancode pipe function has been updated to run the assemble() methods from scancode-toolkit Package handlers. This allows us to create DiscoveredDependencies for DiscoveredPackages and to properly associate application packages to CodebaseResources.

The Project API has been updated with the following:

dependency_count
- The number of DiscoveredDependencies associated with the project.
discovered_dependency_summary
- A mapping that contains following fields:
  - total
    - The number of DiscoveredDependencies associated with the project.
  - is_runtime
    - The number of runtime dependencies.
  - is_optional
    - The number of optional dependencies.
  - is_resolved
    - The number of resolved dependencies.

The Project detail view has been updated to show the DiscoveredDependencies stats and a DiscoveredDependency view, but more work on the UI will be needed. In particular, the field labels (I'm not sure what the best labels are) and colors used for boolean values should be updated.

The DiscoveredDependency view field labels, column widths, and search box (doesn't filter) need to be updated as well.

This feature requires aboutcode-org/scancode-toolkit#3035, which will be available in scancode-toolkit 31.0.0rc4

* Update scan_for_application_packages to save detected Package data to the CodebaseResource it is from, then iterate through the CodebaseResources with Package data and use the proper Package handler to process the Package data * Create DiscoveredDependency model * Add package_data JSON field to CodebaseResource Signed-off-by: Jono Yang <[email protected]>

* Increase field sizes in DiscoveredDependency Signed-off-by: Jono Yang <[email protected]>

Signed-off-by: Jono Yang <[email protected]>

* Create DiscoveredDependencySerializer Signed-off-by: Jono Yang <[email protected]>

Signed-off-by: Jono Yang <[email protected]>

* We never run into the situation where we are updating a DiscoveredDependency when scanning packages * Clean up migrations Signed-off-by: Jono Yang <[email protected]>

* Update scancode-toolkit to 31.0.0rc3 #447 * Update expected test results * Update code formatting Signed-off-by: Jono Yang <[email protected]>

* This is to reflect the changes made to the Package model in scancode-toolkit * Update test expectations Signed-off-by: Jono Yang <[email protected]>

* purl and dependency_uid are the only required fields on DiscoveredDependency * Update migrations * Check for created DiscoveredDependency in tests Signed-off-by: Jono Yang <[email protected]>

* Create DiscoveredDependencies in load_codebase pipeline * Update tests to check for DiscoveredDependencies * Update test expectations Signed-off-by: Jono Yang <[email protected]>

* Update test expectations Signed-off-by: Jono Yang <[email protected]>

* This will fail until scancode-toolkit 31.0.0rc4 is released * Update test expectations Signed-off-by: Jono Yang <[email protected]>

Signed-off-by: Jono Yang <[email protected]>

JonoYang · 2022-08-02T20:26:05Z

Some more to do's:

ManyToMany field on DiscoveredPackage to DiscoveredDependencies
Implement dependency detail view
- Have hyperlink to the Package a dependency is from
~~Resources tab in package detail view does not work~~ I can click on the Resources tab in firefox, but not in chrome

* Remove related dependencies when resetting a project Signed-off-by: Jono Yang <[email protected]>

tdruez · 2022-08-03T11:56:28Z

ManyToMany field on DiscoveredPackage to DiscoveredDependencies

Let's try to make it part of the same migration to limit the number of migration files.

I can click on the Resources tab in firefox, but not in chrome

Can you try to refresh/reset your cache? The tab system is working fine on my side in chrome.

On a more general note, I think there's a bit too much within this one PR and it makes it harder to review and test.
It would have been better to split the resource name, the package assemble, and the dependencies (model and UI) in separate PRs. Not a big deal, but let's try to keep 1 PR per issue in the future. Code quality is great though ;)

tdruez

Code quality is great! I think we are also missing a few tests for the new functions and methods.

tdruez · 2022-08-03T12:07:29Z

scanpipe/models.py

+    def parent_path(self):
+        """
+        Return the parent path for this CodebaseResource or None.
+        """
+        return parent_directory(self.path, with_trail=False)
+
+    def has_parent(self):
+        """
+        Return True if this CodebaseResource has a parent CodebaseResource or
+        False otherwise.
+        """
+        parent_path = self.parent_path()
+        if not parent_path:
+            return False
+        if self.project.codebaseresources.filter(path=parent_path).exists():
+            return True
+        return False
+
+    def parent(self, codebase=None):
+        """
+        Return the parent CodebaseResource object for this CodebaseResource or
+        None.
+
+        `codebase` is not used in this context but required for compatibility
+        with the commoncode.resource.Codebase class API.
+        """
+        parent_path = self.parent_path()
+        return parent_path and self.project.codebaseresources.get(path=parent_path)
+
+    def has_siblings(self, codebase=None):
+        """
+        Return True is this CodebaseResource has siblings.
+
+        `codebase` is not used in this context but required for compatibility
+        with the commoncode.resource.Codebase class API.
+        """
+        return self.has_parent() and self.parent(codebase).has_children()
+
+    def siblings(self, codebase=None):
+        """
+        Return a sequence of sibling Resource objects for this Resource
+        or an empty sequence.
+
+        `codebase` is not used in this context but required for compatibility
+        with the commoncode.resource.Codebase class API.
+        """
+        if self.has_parent():
+            return self.parent(codebase).children(codebase)
+        return []
+


Are those new method related to this PR?

These methods are called in the .assembly() methods of different Package handlers in packagedcode. Some of these methods has logic that determine the Resources of a Package based on the files around a Package manifest. e.g. https://github.com/nexB/scancode-toolkit/blob/develop/src/packagedcode/pypi.py#L185

I've also updated the CodebaseResource walk test to also test for these methods

Ok let's make sure we have full test coverage for those.

scanpipe/pipes/scancode.py

tdruez · 2022-08-03T12:16:55Z

scanpipe/pipes/scancode.py

    _scan_and_save(
        resource_qs=resource_qs,
        scan_func=scan_for_package_data,
        save_func=save_scan_package_results,
    )

+    # Iterate through CodebaseResources with Package data and handle them using
+    # the proper Package handler from packagedcode
+    assemble_packages(project=project)


How are the performances of the new assemble? Do we have some stats about the time spent to assemble compared to the scan_and_save processing?

I don't have any metrics on the performance impact that assemble_packages has on application package scanning. What is the process you do to collect performance metrics on different scancode.io parts?

We run the pipeline using various inputs pre-changes then again post-changes and we can compare the execution time of the relevant steps.

Also, a profiling utility is available to go deeper into optimizations: https://github.com/nexB/scancode.io/blob/main/scanpipe/pipelines/__init__.py#L155

How are the performances of the new assemble? Do we have some stats about the time spent to assemble compared to the scan_and_save processing?

The new assemble is a must have functional change that we should get working and tested before we can optimize if we ever need to optimize? ... @tdruez did you experience an issue?

scanpipe/pipes/scancode.py

scanpipe/templates/scanpipe/package_list.html

scanpipe/templates/scanpipe/project_detail.html

Signed-off-by: Jono Yang <[email protected]>

* Show and link relations in the views Signed-off-by: Jono Yang <[email protected]>

* Remove dependency field from CodebaseResourceSerializer output * Update dependency donut chart labels * Revert Package URL hover to package_uid value in package_list view Signed-off-by: Jono Yang <[email protected]>

* Update walk test to also test for new parent and sibling related methods on CodebaseResource * Remove has_sibling method from CodebaseResource Signed-off-by: Jono Yang <[email protected]>

JonoYang · 2022-08-04T01:20:58Z

@tdruez Thanks! I've responded to your comments. Now, I am wondering about the best approach for #444, where we want to represent individual instances of Packages and Dependencies. Right now, in the case where we have detected the same package multiple times, we create unique packages for each instance even though it is essentially the same package.

JonoYang · 2022-08-04T18:49:40Z

I am closing this PR and splitting it into two:

One where we implement the package assembly step
One where we create the DiscoveredDependency model

JonoYang added 17 commits July 20, 2022 17:03

Add has_single_resource to Project #436 #447

f60f6c2

* Increase field sizes in DiscoveredDependency Signed-off-by: Jono Yang <[email protected]>

Add codebase argument to CodebaseResource.save #447

51ed7ce

Signed-off-by: Jono Yang <[email protected]>

Display DiscoveredDependencies in JSON output #437

9632137

* Create DiscoveredDependencySerializer Signed-off-by: Jono Yang <[email protected]>

Create initial Project dependency view #447

d647390

Signed-off-by: Jono Yang <[email protected]>

Update test expectations #447

8e618de

Signed-off-by: Jono Yang <[email protected]>

Show dependency stats in project detail #447

3f03d96

Signed-off-by: Jono Yang <[email protected]>

Remove update_from_data method #447

6a8be49

* We never run into the situation where we are updating a DiscoveredDependency when scanning packages * Clean up migrations Signed-off-by: Jono Yang <[email protected]>

Store filenames with extensions in name field #467

56b948b

* Update scancode-toolkit to 31.0.0rc3 #447 * Update expected test results * Update code formatting Signed-off-by: Jono Yang <[email protected]>

Remove dependencies from DiscoveredPackage #447

d8fe7c3

* This is to reflect the changes made to the Package model in scancode-toolkit * Update test expectations Signed-off-by: Jono Yang <[email protected]>

Update model field attributes #447

7629d55

* purl and dependency_uid are the only required fields on DiscoveredDependency * Update migrations * Check for created DiscoveredDependency in tests Signed-off-by: Jono Yang <[email protected]>

Re-add update_or_create_dependencies #447

472eb07

* Create DiscoveredDependencies in load_codebase pipeline * Update tests to check for DiscoveredDependencies * Update test expectations Signed-off-by: Jono Yang <[email protected]>

Add more fields to dependency API summary #447

f9a1d71

* Update test expectations Signed-off-by: Jono Yang <[email protected]>

Use scancode.io specific package adder #447

cd2f566

* This will fail until scancode-toolkit 31.0.0rc4 is released * Update test expectations Signed-off-by: Jono Yang <[email protected]>

Merge branch 'main' into package-scanning-update

69fb4e2

Update test expectations #447

053e311

Signed-off-by: Jono Yang <[email protected]>

Update CHANGELOG.rst #447

80a9745

Signed-off-by: Jono Yang <[email protected]>

JonoYang requested a review from tdruez August 2, 2022 01:51

JonoYang added 3 commits August 1, 2022 18:58

Don't save resource instance after add #447

75bd938

Signed-off-by: Jono Yang <[email protected]>

Update scancode-toolkit version #447

0faa61b

Signed-off-by: Jono Yang <[email protected]>

Update changelog formatting #447

310097a

Signed-off-by: Jono Yang <[email protected]>

Update Dependency field labels #447

f240333

* Remove related dependencies when resetting a project Signed-off-by: Jono Yang <[email protected]>

tdruez requested changes Aug 3, 2022

View reviewed changes

Add dependency detail view #447

2640c2f

Signed-off-by: Jono Yang <[email protected]>

JonoYang force-pushed the package-scanning-update branch from 5671373 to 2640c2f Compare August 3, 2022 17:42

JonoYang added 2 commits August 3, 2022 13:03

Relate dependencies to packages #447

ac82864

* Show and link relations in the views Signed-off-by: Jono Yang <[email protected]>

Add has_package_data to CodebaseResourceQuerySet #447

1981cb2

* Remove dependency field from CodebaseResourceSerializer output * Update dependency donut chart labels * Revert Package URL hover to package_uid value in package_list view Signed-off-by: Jono Yang <[email protected]>

JonoYang force-pushed the package-scanning-update branch from 83f545d to 4da9b62 Compare August 4, 2022 00:59

Create tests for new CodebaseResource methods #447

205dddf

* Update walk test to also test for new parent and sibling related methods on CodebaseResource * Remove has_sibling method from CodebaseResource Signed-off-by: Jono Yang <[email protected]>

JonoYang force-pushed the package-scanning-update branch from 4da9b62 to 205dddf Compare August 4, 2022 01:03

JonoYang closed this Aug 4, 2022

JonoYang mentioned this pull request Aug 4, 2022

Implement DiscoveredDependency model #486

Merged

tdruez deleted the package-scanning-update branch September 12, 2022 09:42

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Package scanning update #482

Package scanning update #482

JonoYang commented Aug 2, 2022

JonoYang commented Aug 2, 2022 •

edited

Loading

tdruez commented Aug 3, 2022 •

edited

Loading

tdruez left a comment

tdruez Aug 3, 2022

JonoYang Aug 3, 2022

JonoYang Aug 4, 2022

tdruez Aug 4, 2022

tdruez Aug 3, 2022

JonoYang Aug 4, 2022

tdruez Aug 4, 2022

pombredanne Aug 4, 2022

JonoYang commented Aug 4, 2022

JonoYang commented Aug 4, 2022

Package scanning update #482

Package scanning update #482

Conversation

JonoYang commented Aug 2, 2022

JonoYang commented Aug 2, 2022 • edited Loading

tdruez commented Aug 3, 2022 • edited Loading

tdruez left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JonoYang commented Aug 4, 2022

JonoYang commented Aug 4, 2022

JonoYang commented Aug 2, 2022 •

edited

Loading

tdruez commented Aug 3, 2022 •

edited

Loading