Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Combine license matches in new LicenseDetection #2961

Merged
merged 86 commits into from
Nov 11, 2022
Merged
Show file tree
Hide file tree
Changes from 11 commits
Commits
Show all changes
86 commits
Select commit Hold shift + click to select a range
a6e2941
Import functions from `scancode-analyzer`
AyanSinhaMahapatra May 17, 2022
ce84de8
Modify LicenseDetection attributes
AyanSinhaMahapatra May 17, 2022
466e0e0
Create LicenseDetection serialization function
AyanSinhaMahapatra May 17, 2022
8de3111
Enable LicenseDetection in API
AyanSinhaMahapatra May 17, 2022
77ccbc9
Remove the --unknown-licenses option
AyanSinhaMahapatra May 17, 2022
3bd00bc
Update local license dereferencing code
AyanSinhaMahapatra May 17, 2022
56facc3
Update unknown license dereferencing for key-files
AyanSinhaMahapatra May 17, 2022
391383c
Update intro rules with is_license_intro as True
AyanSinhaMahapatra May 17, 2022
5b0dc11
Add test files for LicenseDetection
AyanSinhaMahapatra May 17, 2022
ce19d7a
Add LicenseDetection full scan tests
AyanSinhaMahapatra May 17, 2022
578bd8d
Add function to create index from test rules folders
AyanSinhaMahapatra May 17, 2022
dc25cd4
Remove unwanted code, update docstrings
AyanSinhaMahapatra May 24, 2022
d70acde
Move LicenseDetection tests to new file
AyanSinhaMahapatra May 24, 2022
b4ddc77
Modify LicenseMatch data in results #2416
AyanSinhaMahapatra May 24, 2022
6a441da
Add --licenses-reference option
AyanSinhaMahapatra May 25, 2022
bca7f4b
Add docs for LicenseDetection and detail referencing
AyanSinhaMahapatra May 25, 2022
d57f390
Do not summurize license detections
AyanSinhaMahapatra May 31, 2022
cc945a0
Apply LicenseDetection everywhere
AyanSinhaMahapatra May 31, 2022
6a91773
Regenerate test expectations for LicenseDetection
AyanSinhaMahapatra May 31, 2022
5826429
Merge branch 'develop' into add-license-detection
AyanSinhaMahapatra May 31, 2022
3e68f76
Fix SPDX and other output plugins
AyanSinhaMahapatra Jun 1, 2022
4fc7d24
Rename license fields for resource
AyanSinhaMahapatra Jun 8, 2022
5e20984
Update test expectations for license field renaming
AyanSinhaMahapatra Jun 8, 2022
2e3dd3e
Rename and Add package license attributes
AyanSinhaMahapatra Jul 12, 2022
855cdef
Add functions for package LicenseDetection
AyanSinhaMahapatra Jul 12, 2022
802c085
Modify package parsers to adopt new LicenseDetection
AyanSinhaMahapatra Jul 12, 2022
ab677c6
Align to package LicenseDetection
AyanSinhaMahapatra Jul 12, 2022
552f173
Modify system packages to use LicenseDetection
AyanSinhaMahapatra Jul 12, 2022
e94a19e
Add new license key `undetected-license`
AyanSinhaMahapatra Jul 12, 2022
0462ef4
Regenerate test expectations for package LicenseDetection
AyanSinhaMahapatra Jul 12, 2022
fd9cd57
Merge branch 'develop' into add-license-detection
AyanSinhaMahapatra Jul 12, 2022
1147381
Fix test failures
AyanSinhaMahapatra Jul 12, 2022
8929571
Fix pypi setup.py email bug
AyanSinhaMahapatra Jul 17, 2022
9df811e
Add manifest license references detection
AyanSinhaMahapatra Jul 18, 2022
c47ce89
Add license from file if empty manifest license
AyanSinhaMahapatra Jul 18, 2022
fce8e7c
Add feature to get package license from sibling file
AyanSinhaMahapatra Jul 18, 2022
eb88b56
Allow package LicenseDetection without --licenses
AyanSinhaMahapatra Jul 21, 2022
67872a4
Regen datadriven LicenseDetections
AyanSinhaMahapatra Jul 21, 2022
9a93552
Revert to `rule_identifier`
AyanSinhaMahapatra Jul 21, 2022
ce11ae4
Remove `undetected-license` in favour of `unknown`
AyanSinhaMahapatra Jul 21, 2022
7fc32ee
Regen test expectations and fix tests
AyanSinhaMahapatra Jul 22, 2022
5aae457
Reorder license expressions and detections in result
AyanSinhaMahapatra Jul 22, 2022
4c1c129
Add fucntions to test license detection with subset of rules
AyanSinhaMahapatra Jul 31, 2022
86ec441
Fix datadriven license test errors
AyanSinhaMahapatra Jul 31, 2022
ca823b7
Fix debian_copyright test failure
AyanSinhaMahapatra Jul 31, 2022
81793dd
Address review feedback
AyanSinhaMahapatra Aug 1, 2022
fb8f492
Add tests from eclipse foundation issues
AyanSinhaMahapatra Aug 3, 2022
08cb42d
Merge branch 'develop' into add-license-detection
AyanSinhaMahapatra Aug 4, 2022
dca0371
Regenerate test expectations after merging develop
AyanSinhaMahapatra Aug 4, 2022
174a097
Add `other_license*` attributes for packages #2065
AyanSinhaMahapatra Aug 8, 2022
c8ca7a3
Remove `compute_normalized_license` functions
AyanSinhaMahapatra Aug 9, 2022
539ebed
Add `default_license_relation` attribute to handlers
AyanSinhaMahapatra Aug 9, 2022
6f42f6f
Support NuGet license URLs #3037
AyanSinhaMahapatra Aug 11, 2022
4f29860
Fix test failures
AyanSinhaMahapatra Aug 11, 2022
0450138
Fix rpm tests
AyanSinhaMahapatra Aug 18, 2022
21648a5
Merge branch 'develop' into add-license-detection
AyanSinhaMahapatra Aug 18, 2022
57a3c2a
Fix test failures and expectations after merging develop
AyanSinhaMahapatra Aug 18, 2022
f1ee5f6
Do not return empty dict as exctracted license
AyanSinhaMahapatra Aug 19, 2022
bae1a30
Fix csv output after adding LicenseDetection`
AyanSinhaMahapatra Aug 19, 2022
b1e422c
Tag intro rule properly
AyanSinhaMahapatra Sep 5, 2022
1b9a8f7
Fix package license expression None bug
AyanSinhaMahapatra Sep 5, 2022
eadb7bd
Also classify license intro in false positives list
AyanSinhaMahapatra Sep 6, 2022
64bd3d8
Merge branch 'develop' into add-license-detection
AyanSinhaMahapatra Sep 6, 2022
8edfb92
Make LicenseMatch grouping affected by presence of license intro
AyanSinhaMahapatra Sep 6, 2022
0d8fba9
Modify license references to also effect clues
AyanSinhaMahapatra Sep 19, 2022
08afdfd
Restore returning whole_lines by default
AyanSinhaMahapatra Sep 19, 2022
6e2cad3
Add libxml files as unknown license reference test
AyanSinhaMahapatra Sep 20, 2022
a5c52fa
Tag license intro rules correctly
AyanSinhaMahapatra Sep 29, 2022
02ab56c
Update false positives and unknown intro heuristics
AyanSinhaMahapatra Oct 3, 2022
7d5c647
Add unknown license reference to package dereferencing #2965 #1379
AyanSinhaMahapatra Oct 13, 2022
8721bdf
Improve unknown reference to package dereferencing #2965 #1379
AyanSinhaMahapatra Oct 18, 2022
5b0efe4
Merge branch 'develop' into add-license-detection
AyanSinhaMahapatra Oct 18, 2022
afd4025
Merge branch 'develop' into add-license-detection
AyanSinhaMahapatra Nov 3, 2022
b1c999f
Rename `detection_rules` to `detection_log`
AyanSinhaMahapatra Nov 6, 2022
f0213ec
Update license clues heuristics
AyanSinhaMahapatra Nov 7, 2022
5de2744
Add `rule_url` and update scancode URLs
AyanSinhaMahapatra Nov 7, 2022
fd3f1d7
Restore the --unknown-licenses experimental CLI option
AyanSinhaMahapatra Nov 8, 2022
637c4dd
Adjust unknown licenses heuristics
AyanSinhaMahapatra Nov 8, 2022
01b1de4
Replace `package` in referenced_filename
AyanSinhaMahapatra Nov 8, 2022
5768c8c
Update docstrings and improve readability
AyanSinhaMahapatra Nov 9, 2022
0e00d28
Update changelog and docs
AyanSinhaMahapatra Nov 10, 2022
5ebca43
Improve CHANGELOG
pombredanne Nov 11, 2022
18aca01
Update CHANGELOG to remove duplications
AyanSinhaMahapatra Nov 11, 2022
05d163a
Bump output format to v3
pombredanne Nov 11, 2022
d264aec
Update CHANGELOG
pombredanne Nov 11, 2022
8f07fdf
Bump version
pombredanne Nov 11, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 42 additions & 0 deletions src/licensedcode/cache.py
Original file line number Diff line number Diff line change
Expand Up @@ -138,6 +138,40 @@ def load_or_build(
raise


class MockCacheFromTestRules(LicenseCache):
AyanSinhaMahapatra marked this conversation as resolved.
Show resolved Hide resolved

@staticmethod
def build_from_test_rules(test_rules_dir):

from licensedcode.models import licenses_data_dir as ldd
from licensedcode.models import load_licenses

licenses_data_dir = licenses_data_dir or ldd
rules_data_dir = test_rules_dir

licenses_db = load_licenses(licenses_data_dir=licenses_data_dir)

index = build_index(
licenses_db=licenses_db,
licenses_data_dir=licenses_data_dir,
rules_data_dir=rules_data_dir,
)

spdx_symbols = build_spdx_symbols(licenses_db=licenses_db)
unknown_spdx_symbol = build_unknown_spdx_symbol(licenses_db=licenses_db)
licensing = build_licensing(licenses_db=licenses_db)

license_cache = LicenseCache(
db=licenses_db,
index=index,
licensing=licensing,
spdx_symbols=spdx_symbols,
unknown_spdx_symbol=unknown_spdx_symbol,
)

return license_cache


def build_index(
licenses_db=None,
licenses_data_dir=None,
Expand Down Expand Up @@ -353,6 +387,14 @@ def get_index(force=False, index_all_languages=False):
return get_cache(force=force, index_all_languages=index_all_languages).index


def get_index_with_test_rules(test_rules_dir):
AyanSinhaMahapatra marked this conversation as resolved.
Show resolved Hide resolved
"""
Return and eventually build and cache a LicenseIndex from a directory with
a few test rules for a specific test.
"""
return MockCacheFromTestRules.build_from_test_rules(test_rules_dir).index


get_cached_index = get_index


Expand Down
2 changes: 1 addition & 1 deletion src/licensedcode/data/rules/lead-in_unknown_2.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
license_expression: unknown-license-reference
is_license_notice: yes
is_license_intro: yes
AyanSinhaMahapatra marked this conversation as resolved.
Show resolved Hide resolved
relevance: 100
2 changes: 1 addition & 1 deletion src/licensedcode/data/rules/lead-in_unknown_30.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
license_expression: unknown-license-reference
is_license_notice: yes
is_license_intro: yes
relevance: 100
2 changes: 1 addition & 1 deletion src/licensedcode/data/rules/lead-in_unknown_31.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
license_expression: unknown-license-reference
is_license_notice: yes
is_license_intro: yes
relevance: 100
Loading