License updates from scancode not working as intended

[sschuberth]

During a license text import in ORT, I noticed that the following LICENSE files seem to be missing:

https://scancode-licensedb.aboutcode.org/adacore-doc.LICENSE
https://scancode-licensedb.aboutcode.org/bsd-advertising-acknowledgement.LICENSE
https://scancode-licensedb.aboutcode.org/cc-by-nc-sa-2.0-de.LICENSE
https://scancode-licensedb.aboutcode.org/cfitsio.LICENSE
https://scancode-licensedb.aboutcode.org/cornell-lossless-jpeg.LICENSE
https://scancode-licensedb.aboutcode.org/hpnd-export-us.LICENSE
https://scancode-licensedb.aboutcode.org/hpnd-sell-variant-mit-disclaimer.LICENSE
https://scancode-licensedb.aboutcode.org/iec-code-components-eula.LICENSE
https://scancode-licensedb.aboutcode.org/ijg-short.LICENSE
https://scancode-licensedb.aboutcode.org/jpl-image.LICENSE
https://scancode-licensedb.aboutcode.org/kazlib.LICENSE
https://scancode-licensedb.aboutcode.org/loop.LICENSE
https://scancode-licensedb.aboutcode.org/offis.LICENSE
https://scancode-licensedb.aboutcode.org/qpl-1.0-inria-2004.LICENSE
https://scancode-licensedb.aboutcode.org/symlinks.LICENSE
https://scancode-licensedb.aboutcode.org/tpdl.LICENSE
https://scancode-licensedb.aboutcode.org/ttwl.LICENSE
https://scancode-licensedb.aboutcode.org/ucar.LICENSE
https://scancode-licensedb.aboutcode.org/w3m.LICENSE

[AyanSinhaMahapatra]

@sschuberth weird, looks like the CI job did not work as intended 🤔 looking into this

[AyanSinhaMahapatra]

@sschuberth I've added a temporary fix, these files are now available, let's still keep this open as I need to investigate and fix why the CI did not work as intended, even though running the scripts locally worked. 🤔

[AyanSinhaMahapatra]

The github action run: https://github.com/nexB/scancode-licensedb/actions/runs/4356107800 did update to the latest scancode-toolkit develop with the license updates in there, but the licensedb html update failed, maybe some reindexing issues?

[AyanSinhaMahapatra]

@sschuberth Btw also

During a license text import in ORT, I noticed that the following LICENSE files seem to be missing

How do you perform this import (is there a script in ORT you can point me to), and do you think using the scancode command directly to generate the licensedb and dump all the data (as API) is better than using scancode-licensedb.org? See aboutcode-org/scancode-toolkit#2738 btw, which is why the command was added to scancode-toolkit.

[sschuberth]

How do you perform this import (is there a script in ORT you can point me to)

The import is implemented as a Gradle task in the spdx-utils module and can be run from the root of the ORT source tree like ./gradlew :utils:spdx-utils:generateSpdxEnums.

See aboutcode-org/scancode-toolkit#2738 btw, which is why the command was added to scancode-toolkit.

I believe that issue has a slightly different use-case in mind. There are two places where ORT makes use of the ScanCode LicenseDB: For one, there's ORT's built-in list of SPDX (core) license IDs and associated texts. While we could exclusively query SPDX for this, we also query the ScanCode LicenseDB as it has the better formatted license text for SPDX licenses. Secondly, we also need the texts for ScanCode-specific licenses / LicenseRefs findings after running ScanCode, and these we currently get from the ScanCode installation directory. That's what aboutcode-org/scancode-toolkit#2738 is about.

For the former use-case of getting license texts for SPDX licenses, I'd prefer to keep querying the remote ScanCode LicenseDB as that also works if ScanCode is not installed.

[AyanSinhaMahapatra]

Thanks for explaining @sschuberth 👍

[armijnhemel]

Is this the reason why I am seeing for example:

Showing 2,147 changed files with 2,147 additions and 2,147 deletions.

in Git commits (for example: 355f106 )?

It seems that the only thing that is updated is the version number. Adding @pombredanne

[AyanSinhaMahapatra]

@armijnhemel no, that issue is tracked in #31, this issue was fixed by the PR above.