Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Combine license matches in new LicenseDetection #2961

Merged
merged 86 commits into from
Nov 11, 2022
Merged

Combine license matches in new LicenseDetection #2961

merged 86 commits into from
Nov 11, 2022

Conversation

AyanSinhaMahapatra
Copy link
Member

@AyanSinhaMahapatra AyanSinhaMahapatra commented May 17, 2022 •
edited
Loading

See #2878

Tasks

  • Reviewed contribution guidelines
  • PR is descriptively titled 📑 and links the original issue above 🔗
  • Tests pass -- look for a green checkbox ✔️ a few minutes after opening your PR
    Run tests locally to check for errors.
  • Commits are in uniquely-named feature branch and has no merge conflicts 📁

@AyanSinhaMahapatra
Import functions from scancode-analyzer
a6e2941 
Signed-off-by: Ayan Sinha Mahapatra <[email protected]>
@AyanSinhaMahapatra
Modify LicenseDetection attributes
ce84de8 
Remove primary license expression attributes, move matches to last.

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>
@AyanSinhaMahapatra
Create LicenseDetection serialization function
466e0e0 
Move the LicenseMatch serialization functions into
the LicenseDetection function.

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>
@AyanSinhaMahapatra
Enable LicenseDetection in API
8de3111 
Create LicenseDetection functions and enable them in the API.

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>
@AyanSinhaMahapatra
Remove the --unknown-licenses option
77ccbc9 
Removes the unknown licenses CLI option, as this should be
default behaviour.

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>
@AyanSinhaMahapatra
Update local license dereferencing code
3bd00bc 
Update local unknown license reference code to use
LicenseDetection functions.

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>
@AyanSinhaMahapatra
Update unknown license dereferencing for key-files
56facc3 
This updates the unknown license dereferencing code to also
look for the referenced filename in root of the codebase.

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>
@AyanSinhaMahapatra
Update intro rules with is_license_intro as True
391383c 
Signed-off-by: Ayan Sinha Mahapatra <[email protected]>
@AyanSinhaMahapatra
Add test files for LicenseDetection
5b0dc11 
Signed-off-by: Ayan Sinha Mahapatra <[email protected]>
@AyanSinhaMahapatra
Add LicenseDetection full scan tests
ce19d7a 
Adds tests for:

- unknown license intros
- unknown local license references

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>
@AyanSinhaMahapatra
Add function to create index from test rules folders
578bd8d 
Signed-off-by: Ayan Sinha Mahapatra <[email protected]>
pombredanne
pombredanne requested changes May 18, 2022
View reviewed changes
Copy link
Member

@pombredanne pombredanne left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! that's quite a piece of work! 🙇 I provided some comments, but I did not review everything just yet. Of note you should use LicenseMatch and objects everywhere and not carry around dictionaries of scan data.
Instead convert these at the boundaries as early as possible deserialized to objects.

src/licensedcode/cache.py Outdated Show resolved Hide resolved
src/licensedcode/cache.py Outdated Show resolved Hide resolved
src/licensedcode/data/rules/lead-in_unknown_2.yml Outdated Show resolved Hide resolved
src/scancode/api.py Outdated Show resolved Hide resolved
src/licensedcode/detection.py Outdated Show resolved Hide resolved
src/licensedcode/detection.py Outdated Show resolved Hide resolved
src/licensedcode/detection.py Outdated Show resolved Hide resolved
src/licensedcode/detection.py Outdated Show resolved Hide resolved
src/licensedcode/detection.py Outdated Show resolved Hide resolved
src/licensedcode/detection.py Show resolved Hide resolved
pombredanne
pombredanne requested changes May 18, 2022
View reviewed changes
Copy link
Member

@pombredanne pombredanne left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here is some feedback on the format. We need to streamline this to the essence and avoid repeating details to remove noise when possible.

tests/licensedcode/data/plugin_license/license-expression/scan.expected.json Outdated Show resolved Hide resolved
tests/licensedcode/data/plugin_license/license-expression/scan.expected.json Outdated Show resolved Hide resolved
tests/licensedcode/data/plugin_license/license-expression/scan.expected.json Outdated Show resolved Hide resolved
tests/licensedcode/data/plugin_license/license-expression/scan.expected.json Outdated Show resolved Hide resolved
tests/licensedcode/data/plugin_license/license-expression/scan.expected.json Outdated Show resolved Hide resolved
tests/licensedcode/data/plugin_license/license-expression/scan.expected.json Outdated Show resolved Hide resolved
@AyanSinhaMahapatra
Remove unwanted code, update docstrings
dc25cd4 
Updates implementation to address review comments.

- Remove code that wasn't being used
- Update docstrings
- Use constants and Enums
- Fix case where local reference can't be found
- Move back constants to API
- Modify test expectations
- Other misc changes

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>
@AyanSinhaMahapatra
Move LicenseDetection tests to new file
d70acde 
Also adds a test without the --license-references option.

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>
@AyanSinhaMahapatra
Modify LicenseMatch data in results #2416
b4ddc77 
LicenseMatch data now is based on a license-expression instead
of a license key.

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>
@AyanSinhaMahapatra
Add --licenses-reference option
6a441da 
Add new command line option to not inline license, licenseDB and license
detection level information.

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>
@AyanSinhaMahapatra
Add docs for LicenseDetection and detail referencing
bca7f4b 
Signed-off-by: Ayan Sinha Mahapatra <[email protected]>
@AyanSinhaMahapatra
Do not summurize license detections
d57f390 
Signed-off-by: Ayan Sinha Mahapatra <[email protected]>
@AyanSinhaMahapatra
Apply LicenseDetection everywhere
cc945a0 
Modify code to align to the LicenseDetection model everywhere.

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>
@AyanSinhaMahapatra
Regenerate test expectations for LicenseDetection
6a91773 
Signed-off-by: Ayan Sinha Mahapatra <[email protected]>
@AyanSinhaMahapatra AyanSinhaMahapatra changed the title [WIP] Add LicenseDetection Add LicenseDetection Jun 1, 2022
@AyanSinhaMahapatra
Fix SPDX and other output plugins
3e68f76 
Modify and update spdx and other output plugins to use LicenseDetection
output data format for licenses.

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>
@AyanSinhaMahapatra
Rename license fields for resource
4fc7d24 
* Rename `licenses` to `license_detections`
* Rename `license_expressions` to `detected_license_expression`
* Rename `spdx_license_expressions` to `detected_license_expression_spdx`

Also fix other uses of these attributes around scancode.

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>
@AyanSinhaMahapatra
Update test expectations for license field renaming
5e20984 
Signed-off-by: Ayan Sinha Mahapatra <[email protected]>
@AyanSinhaMahapatra
Improve unknown reference to package dereferencing #2965 #1379
8721bdf 
* in case of unknown references being present without top-level
  detected, dereference using license detections in legalese/readme
  files at codebase root.
* add example cases from samba/samba, sugarlabs/physics, debian fusiondirectory,
  and paddlenlp as tests.

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>
This was referenced Oct 18, 2022
Open
Closed
@pombredanne pombredanne mentioned this pull request Oct 28, 2022
Open
@AyanSinhaMahapatra
Rename detection_rules to detection_log
b1c999f 
Signed-off-by: Ayan Sinha Mahapatra <[email protected]>
@AyanSinhaMahapatra
Update license clues heuristics
f0213ec 
Signed-off-by: Ayan Sinha Mahapatra <[email protected]>
@AyanSinhaMahapatra
Add rule_url and update scancode URLs
5de2744 
* add a new attribute `rule_url` to LicenseMatch results.
* deleted License attributes 'scancode_data_url' and 'scancode_text_url' and
  replaced them with a single License attribute 'scancode_url'
* regen test expectations accordingly

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>
@AyanSinhaMahapatra
Restore the --unknown-licenses experimental CLI option
fd3f1d7 
Signed-off-by: Ayan Sinha Mahapatra <[email protected]>
@AyanSinhaMahapatra
Adjust unknown licenses heuristics
637c4dd 
Signed-off-by: Ayan Sinha Mahapatra <[email protected]>
@AyanSinhaMahapatra
Replace package in referenced_filename
01b1de4 
replaces the `package` in referenced_filename with a temporary
`INHERIT_LICENSE_FROM_PACKAGE` which will be removed later in
favour of a flag.

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>
@AyanSinhaMahapatra
Update docstrings and improve readability
5768c8c 
Signed-off-by: Ayan Sinha Mahapatra <[email protected]>
@AyanSinhaMahapatra
Copy link
Member Author

@pombredanne this is all ready!

AyanSinhaMahapatra and others added 6 commits November 10, 2022 19:43
@AyanSinhaMahapatra
Update changelog and docs
0e00d28 
Signed-off-by: Ayan Sinha Mahapatra <[email protected]>
@pombredanne
Improve CHANGELOG
5ebca43 
Signed-off-by: Philippe Ombredanne <[email protected]>
@AyanSinhaMahapatra
Update CHANGELOG to remove duplications
18aca01 
Signed-off-by: Ayan Sinha Mahapatra <[email protected]>
@pombredanne
Bump output format to v3
05d163a 
Signed-off-by: Philippe Ombredanne <[email protected]>
@pombredanne
Update CHANGELOG
d264aec 
Signed-off-by: Philippe Ombredanne <[email protected]>
@pombredanne
Bump version
8f07fdf 
Signed-off-by: Philippe Ombredanne <[email protected]>
pombredanne
pombredanne approved these changes Nov 11, 2022
View reviewed changes
CHANGELOG.rst Outdated Show resolved Hide resolved
CHANGELOG.rst Outdated Show resolved Hide resolved
@pombredanne pombredanne merged commit a410f6e into develop Nov 11, 2022
@pombredanne pombredanne deleted the add-license-detection branch November 11, 2022 17:07
@AyanSinhaMahapatra AyanSinhaMahapatra mentioned this pull request Jul 3, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@bwjohnson-ss bwjohnson-ss bwjohnson-ss left review comments

@pombredanne pombredanne pombredanne approved these changes

@JonoYang JonoYang Awaiting requested review from JonoYang

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@AyanSinhaMahapatra @JonoYang @pombredanne @bwjohnson-ss