Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add license dump option #3100

Merged
merged 52 commits into from
Nov 2, 2022
Merged

Add license dump option #3100

merged 52 commits into from
Nov 2, 2022

Conversation

AyanSinhaMahapatra
Copy link
Member

@AyanSinhaMahapatra AyanSinhaMahapatra commented Sep 12, 2022 •
edited
Loading

  1. Modifies the License and Rule files to have their data as YAML frontmatter.
  • .RULE files and .LICENSE files now also contain their data.
  • git history is preserved
  • loading/dumping methods are modified to support YAML frontmatter
  • License/Rule now has different methods to point to their file.
  1. Adds a new command line option: --get-license-data to:
  • Dump license data in JSON, YAML and HTML formats.
  • Also dumps the .LICENSE file with text and data as YAML frontmatter.
  • Generates an index and a static website to view the data.

This is reusing code originally located at:
https://github.com/nexB/scancode-licensedb

Reference: #2738
Reference: #3049

This also contains PR: #3063 btw.

Tasks

  • Reviewed contribution guidelines
  • PR is descriptively titled 📑 and links the original issue above 🔗
  • Tests pass -- look for a green checkbox ✔️ a few minutes after opening your PR
    Run tests locally to check for errors.
  • Commits are in uniquely-named feature branch and has no merge conflicts 📁

tdruez and others added 30 commits December 4, 2020 16:30
@tdruez
Initial commit
07b0269 
Signed-off-by: Thomas Druez <[email protected]>
@tdruez
Add the generated static website
27c255c 
Signed-off-by: Thomas Druez <[email protected]>
@pombredanne
Create CNAME
b39763d
@tdruez
Upgrade ScanCode-toolkit to latest 21.2.9 version
3bf65b9 
Signed-off-by: Thomas Druez <[email protected]>
@tdruez
Do not display "LicenseRef-" SPDX key as links
e97dc5d 
Signed-off-by: Thomas Druez <[email protected]>
@tdruez
Create CNAME
76c145d
@tdruez
Inject the search value in the browser href
07316b6 
Signed-off-by: Thomas Druez <[email protected]>
@tdruez
Merge branch 'main' of https://github.com/nexB/scancode-licensedb int…
3e56341 
…o main
@tdruez
Do not remove the CNAME file on Make clean
4e8389b 
Signed-off-by: Thomas Druez <[email protected]>
@tdruez
Upgrade ScanCode-toolkit to latest version
9164413
@tdruez
Add "publish" command in Makefile
cf386c8 
Signed-off-by: Thomas Druez <[email protected]>
@JonoYang
Expose SPDX license info in index.json #7
efdb626 
    * Expose spdx_license_key, other_spdx_license_keys, and is_exception

Signed-off-by: Jono Yang <[email protected]>
@tdruez
Merge pull request #8 from nexB/7-expose-spdx-information
c6a64d1 
Expose SPDX license info in index.json #7
@tdruez
Fix header layout for smaller display sizes
f98b976 
Signed-off-by: tdruez <[email protected]>
@tdruez
Merge branch
8c69070
@tdruez
Upgrade ScanCode-toolkit to latest version
d2f3920
@tdruez
Upgrade ScanCode-toolkit to latest version
16dd721
@tdruez
Enhance the details view UI and display URLs as link #5
13e9166 
Signed-off-by: tdruez <[email protected]>
@tdruez
Do not display deprecated licenses in main listing by default #2
2a4e4cb 
Signed-off-by: tdruez <[email protected]>
@JonoYang
Expose is_deprecated in index.json #9
40c5a8d 
Signed-off-by: Jono Yang <[email protected]>
@tdruez
Merge pull request #10 from nexB/9-expose-is_deprecated
d7fa06b 
Expose is_deprecated in index.json
@tdruez
Upgrade ScanCode-toolkit to latest version
3069232
@tdruez
Upgrade ScanCode-toolkit to latest version
5b0f829
@tdruez
Upgrade ScanCode-toolkit to latest version
1d81315
@tdruez
Upgrade ScanCode-toolkit to latest version
838f364
@AyanSinhaMahapatra
Rename .yml files to .RULE.new
8d234ca 
Signed-off-by: Ayan Sinha Mahapatra <[email protected]>
@AyanSinhaMahapatra
Rename the .RULE files to .RULE.new
0156b3a 
Signed-off-by: Ayan Sinha Mahapatra <[email protected]>
@AyanSinhaMahapatra
Merge branch '3049-too-many-files-rename' into 3049-too-many-files
e755274
@AyanSinhaMahapatra
Rename .RULE.new files back to .RULE #3049
e71edc1 
This commit ends the process of merging .RULE and .yml files into
a single .RULE file which has YAML frontmatter storing the rule
metadata present in the .yml file previously.

This renaming and merging has been done to preserve line-history
for both the files.

Reference: #3049
Signed-off-by: Ayan Sinha Mahapatra <[email protected]>
@AyanSinhaMahapatra
Use modified functions and classes from frontmatter
bd3545c 
* Import and use functions/classes from frontmatter
* Copy and modify class to use saneyaml
* Copy and modify load/parse functions
* Add .ABOUT and LICENSE files.

Signed-off-by: AyanSinhaMahapatra <[email protected]>
@AyanSinhaMahapatra AyanSinhaMahapatra mentioned this pull request Sep 28, 2022
Merged
2 tasks
@pombredanne pombredanne mentioned this pull request Oct 10, 2022
Open
@AyanSinhaMahapatra
Fix binary gibberish bug on extra newlines
8a8fba0 
Regen test expectations for this and also fix another test.

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>
@pombredanne pombredanne mentioned this pull request Oct 28, 2022
Closed
pombredanne
pombredanne approved these changes Oct 28, 2022
View reviewed changes
Copy link
Member

@pombredanne pombredanne left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM... Please do one last check wrt. spaces:

from licensedcode.models import *
gpl=db["gpl-2.0"]
bottom = gpl.text.strip("\n") + "\n"
top = saneyaml.dump(gpl.to_dict())
fm=f"---\n{top}\n---\n{bottom}"

IMHO redumping all licenses as frontmatter this way should be to help that we are not damaging space-based formatting. In particualr watch out for the GPL -2.0.

src/licensedcode/models.py Outdated Show resolved Hide resolved
@AyanSinhaMahapatra
Reinstate whitespace in license text
8b2bbfe 
Signed-off-by: Ayan Sinha Mahapatra <[email protected]>
@AyanSinhaMahapatra AyanSinhaMahapatra force-pushed the add-license-dump-option branch 2 times, most recently from 8b1d58e to 093b8b4 Compare November 1, 2022 11:09
@AyanSinhaMahapatra
Update changelog
c857fa4 
Signed-off-by: Ayan Sinha Mahapatra <[email protected]>
@AyanSinhaMahapatra
Copy link
Member Author

@pombredanne post your approval I've pushed the following changes:

  1. Fix whitespace in License texts
  2. Merge the latest changes in develop including external licenses work and make tests pass.
  3. Update changelog

Should I go ahead and merge?

@AyanSinhaMahapatra AyanSinhaMahapatra mentioned this pull request Nov 1, 2022
Open
@AyanSinhaMahapatra
Copy link
Member Author

AyanSinhaMahapatra commented Nov 1, 2022 •
edited
Loading

Some scripts need update but I opened an issue: #3138 to track that separately to not block this anymore.

@AyanSinhaMahapatra AyanSinhaMahapatra mentioned this pull request Nov 1, 2022
Open
7 tasks
@AyanSinhaMahapatra AyanSinhaMahapatra merged commit 9f91bf5 into develop Nov 2, 2022
@pombredanne pombredanne deleted the add-license-dump-option branch November 14, 2022 16:53
@AyanSinhaMahapatra
Copy link
Member Author

Note that the command line option was later renamed from --get-license-data to --dump-license-data later.

sschuberth added a commit to oss-review-toolkit/ort that referenced this pull request Nov 16, 2023
@sschuberth
fix: Remove any YAML front matter from ScanCode license files
c2d89f1 
ScanCode 32.0.0 started to prepend its `*.LICENSE` files with
YAML-encoded metadata, see [1]. This is a hot fix to remove this header,
if present, from the license files. A better solution will be
implemented later as part of a larger refactoring of license providers.

Different ScanCode versions also differ in whether license files come
with a final newline or not. Align on not having a final newline to make
tests pass either way.

[1]: aboutcode-org/scancode-toolkit#3100

Signed-off-by: Sebastian Schuberth <[email protected]>
@sschuberth sschuberth mentioned this pull request Nov 16, 2023
Merged
sschuberth added a commit to oss-review-toolkit/ort that referenced this pull request Nov 16, 2023
@sschuberth
fix: Remove any YAML front matter from ScanCode license files
b0ac800 
ScanCode 32.0.0 started to prepend its `*.LICENSE` files with
YAML-encoded metadata, see [1]. This is a hot fix to remove this header,
if present, from the license files. A better solution will be
implemented later as part of a larger refactoring of license providers.

Different ScanCode versions also differ in whether license files come
with a final newline or not. Align on not having a final newline to make
tests pass either way.

[1]: aboutcode-org/scancode-toolkit#3100

Signed-off-by: Sebastian Schuberth <[email protected]>
sschuberth added a commit to oss-review-toolkit/ort that referenced this pull request Nov 17, 2023
@sschuberth
fix: Remove any YAML front matter from ScanCode license files
2000181 
ScanCode 32.0.0 started to prepend its `*.LICENSE` files with
YAML-encoded metadata, see [1]. This is a hot fix to remove this header,
if present, from the license files. A better solution will be
implemented later as part of a larger refactoring of license providers.

Different ScanCode versions also differ in whether license files come
with a final newline or not. Align on not having a final newline to make
tests pass either way.

[1]: aboutcode-org/scancode-toolkit#3100

Signed-off-by: Sebastian Schuberth <[email protected]>
sschuberth added a commit to oss-review-toolkit/ort that referenced this pull request Nov 17, 2023
@sschuberth
fix: Remove any YAML front matter from ScanCode license files
b9c038e 
ScanCode 32.0.0 started to prepend its `*.LICENSE` files with
YAML-encoded metadata, see [1]. This is a hot fix to remove this header,
if present, from the license files. A better solution will be
implemented later as part of a larger refactoring of license providers.

Different ScanCode versions also differ in whether license files come
with a final newline or not. Align on not having a final newline to make
tests pass either way.

[1]: aboutcode-org/scancode-toolkit#3100

Signed-off-by: Sebastian Schuberth <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@pombredanne pombredanne pombredanne approved these changes

@JonoYang JonoYang Awaiting requested review from JonoYang

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@AyanSinhaMahapatra @pombredanne @tdruez @JonoYang