Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Upgrade ScanCode-toolkit to latest v32 #569 #715

Merged
merged 57 commits into from
May 18, 2023
Merged

Upgrade ScanCode-toolkit to latest v32 #569 #715

merged 57 commits into from
May 18, 2023

Conversation

tdruez
Copy link
Contributor

@tdruez tdruez commented May 5, 2023

No description provided.

tdruez added 30 commits May 4, 2023 19:23
@tdruez
Upgrade ScanCode-toolkit to v32.0.0rc4 #569
4ef592e 
Signed-off-by: Thomas Druez <[email protected]>
@tdruez
Migrate DiscoveredPackage model to toolkit v32 changes #569
1756c3a 
Signed-off-by: Thomas Druez <[email protected]>
@tdruez
Add new field in API serializer #569
b887491 
Signed-off-by: Thomas Druez <[email protected]>
@tdruez
Fix the scan_package pipeline #569
e278f85 
Signed-off-by: Thomas Druez <[email protected]>
@tdruez
Fix unit tests and update expected files #569
7ff0a5e 
Signed-off-by: Thomas Druez <[email protected]>
@tdruez
Fix unit tests and update expected files #569
9a02875 
Signed-off-by: Thomas Druez <[email protected]>
@tdruez
Adjust new JSONField default as list #569
87cfd93 
Signed-off-by: Thomas Druez <[email protected]>
@tdruez
Multiple adjustments for the v32 update #569
8797ea5 
Signed-off-by: Thomas Druez <[email protected]>
@tdruez
Refine get_declared_license_expression_spdx logic #569
c135721 
Signed-off-by: Thomas Druez <[email protected]>
@tdruez
Remove the manifest_path and contains_source_code fields #569
434b0e0 
Signed-off-by: Thomas Druez <[email protected]>
@tdruez
Data migration for declared_license_expression_spdx #569
60ec5a1 
Signed-off-by: Thomas Druez <[email protected]>
@tdruez
Fix rendering for license_detections and other_license_detections #569
c73a0bb 
Signed-off-by: Thomas Druez <[email protected]>
@tdruez
Add get_declared_license_expression method #569
0103927 
Signed-off-by: Thomas Druez <[email protected]>
@tdruez
Regen asgiref-3.3.0_toolkit_scan.json in RegenTestData #569
3a36ba4 
Signed-off-by: Thomas Druez <[email protected]>
@tdruez
Base set of changes for the CodebaseResource migration #569
a79cd7e 
Signed-off-by: Thomas Druez <[email protected]>
@tdruez
Consolidate migrations #569
de7f6a1 
Signed-off-by: Thomas Druez <[email protected]>
@tdruez
Update code to resource model changes and fix parts of tests #569
09ebed8 
Signed-off-by: Thomas Druez <[email protected]>
@tdruez
Update test expected results files #569
03cd314 
Signed-off-by: Thomas Druez <[email protected]>
@tdruez
Fix unit test #569
e538e70 
Signed-off-by: Thomas Druez <[email protected]>
@tdruez
Refine migration files in 3 steps #569
2a294e8 
Signed-off-by: Thomas Druez <[email protected]>
@tdruez
Add data mgiration for detected_license_expression #569
4a087bf 
Signed-off-by: Thomas Druez <[email protected]>
@tdruez
Merge branch 'main' into 569-toolkit-upgrade-v32
bca062e 
Signed-off-by: Thomas Druez <[email protected]>

# Conflicts:
#	scanpipe/pipes/output.py
@tdruez
Remove licenses_categories #569
f2b301f 
Signed-off-by: Thomas Druez <[email protected]>
@tdruez
Migrate attribution to new model fields #569
4d96857 
Signed-off-by: Thomas Druez <[email protected]>
@tdruez
Migrate charts for detected_license_expression #569
f494896 
Signed-off-by: Thomas Druez <[email protected]>
@tdruez
Migrate spdx to new model fields #569
bd17cb3 
Signed-off-by: Thomas Druez <[email protected]>
@tdruez
Add proper help text for new fields #569
6736f87 
Signed-off-by: Thomas Druez <[email protected]>
@tdruez
Fix chart and filter for detected_license_expression #569
3337314 
Signed-off-by: Thomas Druez <[email protected]>
@tdruez
Fix the policy and compliance with new field #569
e029568 
Signed-off-by: Thomas Druez <[email protected]>
@tdruez
Refine loaded_license_expression using the expression #569
66fb266 
Signed-off-by: Thomas Druez <[email protected]>
tdruez added 22 commits May 15, 2023 12:04
@tdruez
Display warning if one pipeline was run with old toolkit #569
9b7eb48 
Signed-off-by: Thomas Druez <[email protected]>
@tdruez
Merge main
2e1e417
@tdruez
Merge main
ab71bd6
@tdruez
Add package_assembly_codebase.json in the regen stack #569
17d416b 
Signed-off-by: Thomas Druez <[email protected]>
@tdruez
Do not run the data migration operation during tests #569
48eb798 
Signed-off-by: Thomas Druez <[email protected]>
@tdruez
Add changelog entry #569
6f5762a 
Signed-off-by: Thomas Druez <[email protected]>
@tdruez
Update "Custom Pipeline Example" in documentation #569
13b5c8f 
Signed-off-by: Thomas Druez <[email protected]>
@tdruez
Add unit test for spdx_package_to_discovered_package_data #569
a41a389 
Signed-off-by: Thomas Druez <[email protected]>
@tdruez
Fix _get_license_matches_grouped and improve testing #569
0edc4c2 
Signed-off-by: Thomas Druez <[email protected]>
@tdruez
Enhance unit tests #569
49a9eb6 
Signed-off-by: Thomas Druez <[email protected]>
@tdruez
Ensure the policies index is empty #569
c4e277f 
Signed-off-by: Thomas Druez <[email protected]>
@tdruez
Store AttributeError as ProjectError during data migration #569
932046f 
Signed-off-by: Thomas Druez <[email protected]>
@tdruez
Merge branch 'main' into 569-toolkit-upgrade-v32
2aa8c70 
Signed-off-by: Thomas Druez <[email protected]>
@tdruez
Use proper model creation for ProjectError in data migration #569
d01a9e1 
Signed-off-by: Thomas Druez <[email protected]>
@tdruez
Refactored the compliance alert implementation #569
5dc9499 
Signed-off-by: Thomas Druez <[email protected]>
@tdruez
Fix problematic path in expected files #569
62549d6 
Signed-off-by: Thomas Druez <[email protected]>
@tdruez
Add debug statement #569
acc96ca 
Signed-off-by: Thomas Druez <[email protected]>
@tdruez
Fix failing test expected file #569
ab8df20 
Signed-off-by: Thomas Druez <[email protected]>
@tdruez
Improve performances of the charts view (25x faster) #569
7b54a4d 
Signed-off-by: Thomas Druez <[email protected]>
@tdruez
Add missing DB indexes on Project model #569
3cf3d58 
Signed-off-by: Thomas Druez <[email protected]>
@tdruez
Bump version to not display warning on new runs #569
f94a76e 
Signed-off-by: Thomas Druez <[email protected]>
@tdruez
Replace Counter.total as not available before py3.10 #569
4293f91 
Signed-off-by: Thomas Druez <[email protected]>
This was referenced May 18, 2023
Open
Open
pombredanne
pombredanne approved these changes May 18, 2023
View reviewed changes
Copy link
Member

@pombredanne pombredanne left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All good, I added a few nitpickings on the doc and two follow up issues:

CHANGELOG.rst Outdated Show resolved Hide resolved
scanpipe/models.py Outdated Show resolved Hide resolved
scanpipe/tests/data/is-npm-1.0.0_scan_codebase.json
"other_license_expression": "",
"other_license_expression_spdx": "",
"other_license_detections": [],
"extracted_license_statement": "['MIT']",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a bug!! we should not have this ugly repr-like in the extracted license statement but proper YAML like this was done before with the SCTK 31.x in "declared_license" . @AyanSinhaMahapatra ping :)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll fix this in scancode-toolkit v32.0.0rc5, but this is not in the way of the upgrade, as the extracted license statement will continue to be a string, but it will be using yaml (and now parse-able) instead of the repr used now.

Btw, @pombredanne this was not yaml in v31 at all, see https://github.com/nexB/scancode-toolkit/blob/v31.2.6/tests/packagedcode/data/plugin/bower-package-expected.json#LL143C11-L145C13 for example. It used to be everything from dicts/list/strings etc.

AyanSinhaMahapatra
AyanSinhaMahapatra approved these changes May 18, 2023
View reviewed changes
Copy link
Member

@AyanSinhaMahapatra AyanSinhaMahapatra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tdruez thanks! I've tested with scanning and using multiple and different pipelines, migrating old data, and the views. Everything works as expected and nicely. I'll also open follow up issues on more upgrades for later.

@AyanSinhaMahapatra AyanSinhaMahapatra mentioned this pull request May 18, 2023
Merged
6 tasks
@tdruez tdruez merged commit 657c1af into main May 18, 2023
@tdruez tdruez deleted the 569-toolkit-upgrade-v32 branch May 18, 2023 09:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@AyanSinhaMahapatra AyanSinhaMahapatra AyanSinhaMahapatra approved these changes

@pombredanne pombredanne pombredanne approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@tdruez @pombredanne @AyanSinhaMahapatra