Define a "namespace" for SPDX LicenseRef- for ScanCode licenses that are NOT known to SPDX (yet)

[pombredanne]

ScanCode tracks many more licenses that SPDX does. As a result LicenseRef-xxx needs to be created (e.g. e37fec9 )

For sanity and simplicity and clarity we should define a namespace for these non-SPDX refs.
For instance LicenseRef-scancode-<license-key> ... I am not sure there is a convention that exist for this in SPDX though we could suggest it.

@sschuberth input welcome!

[sschuberth]

I was just wondering whether there are cases where it's uncontroversial how <license-key> should look like, e.g. because such a key / id is already provided by the creator of the license, and if for such cases we deliberately should not use a namespace to create a de-facto standard how the LicenseRef should look like for easier interoperability with other tools?

[mjherzog]

@pombredanne I recommend contacting Gary and/or Mark G. about the namespace idea. We have had many discussions and good ideas on this topic in the SPDX Tech group, but I cannot remember the details and life is too short to reread the emails.

[pombredanne]

@goneall @MarkGisi ping!

[goneall]

Currently, license-refs are local to a document. We have discussed different license lists with different namespaces in the past, but so far have not standardized an approach. The current convention is to use the license name field and comment field. Probably something we should revisit in the legal and tech teams

[pombredanne]

@goneall thanks. So for now I am inclined to namespace all the non-SPDX scancode licenses with LicenseRef-scancode-<license-key> ... This would help adopters and provide a clean reminder that the license is available and where. that would still keep the ref logically local to a doc, but conveys better info (and certainly better than LicenseRef-08 ;) )

[pombredanne]

@sschuberth you wrote:

I was just wondering whether there are cases where it's uncontroversial how <license-key> should look like, e.g. because such a key / id is already provided by the creator of the license, and if for such cases we deliberately should not use a namespace to create a de-facto standard how the LicenseRef should look like for easier interoperability with other tools?

That's a possibility, but would you have any such example?
One thing that could be done in such cases if any would be to pre-assign a license with an spdx_license_key: LicenseRef-XYZ this way it would not be pre-computed from the scancode key ?
And this would require no code change at all. In fact I was thinking to actually make no code change here but only add an spdx_license_key LicenseRef to every license that do not have one.

[sschuberth]

No, I don't have a concrete example. It was just something that came to my mind. Populating the spdx_license_key field in any case sounds like a reasonable idea to me.

[goneall]

@pombredanne Seems like a reasonable and valid approach to generating the license-ref's in a readable manner. When we created namespaces for the documents, we spent quite a bit of energy making sure there were no collisions and versions were properly handled. From a spec perspective, these license refs will still be treated as local to the SPDX document. Perhaps we should (re)discuss expanding the spec to include external license lists and just work out the details for a complete solution,

Another solution would be to submit any missing licenses that fit the SPDX listed license guidelines to the spdx listed license page and use that as a primary list of licenses.

[sschuberth]

Another solution would be to submit any missing licenses [...]

Addressing the issue upstream by doing so should indeed be the first choice, IMO, but we certainly need a fallback if that's not feasible for whatever reason.

[pombredanne]

@goneall @sschuberth

Another solution would be to submit any missing licenses that fit the SPDX listed license guidelines

This is surely the best approach, but think about this: ScanCode has about 600+ more licenses than SPDX. Even if 200 of these could be set aside if I could determine quickly that they do not clearly fit the guidelines (hint: I cannot do this easily), then the work required to add to SPDX the 400 remaining licenses is immense!

I can float the idea on the spdx-legal ML for sure. But @jlovejoy @DennisClark @MarkGisi what would be your initial take/reaction/feedback on this?

[pombredanne]

@goneall note that when I speak of "namespace" I am NOT speaking of XML/RDF namespaces at all. I am just thinking of a convention to clearly identify extra non-SPDX-listed licenses in simplified/conventional way.

[pombredanne]

Unless there is an objection I would like to start using this in SPDX outputs: LicenseRef-scancode-<license-key>

[sschuberth]

No objections from my side!

[jlovejoy]

@pombredanne - using LicenseRef-scancode- for licenses you have in Scancode, but that are not on the SPDX License List sounds reasonable to me. Is there any reason not to just use:
LicenseRef- ? Is there any concern that use of the same identifier might wander beyond Scancode, and then cause confusion that the license itself is related to Scancode? just a thought.

As to adding a large number of licenses to the SPDX License List - it would be a lot of work, but I think it's worthwhile to have a conversation at an upcoming legal call. we are starting to have better processes for adding licenses, so... (and don't we still need to get some of the license found in the kernel added? I digress...) Next SPDX legal call is Nov 1 - hope you can join

[pombredanne]

@jlovejoy thank you++ for your feedback!

Is there any reason not to just use: LicenseRef- ?

The idea here is to provide some hint that this is a ScanCode-standard, curated license by prefixing the ref with -scancode.

And also, there are cases where we have conflicting right hand parts used in LicenseRef between ScanCode and Fossology for instance and these may or may not be the same license... but the point is that there should be no ambiguity.

As to adding a large number of licenses to the SPDX License List...

I will try to join the call at https://wiki.spdx.org/view/Legal_Team for sure!

don't we still need to get some of the license found in the kernel added?

We quite surely need that and I kinda dropped a bit the ball for this.

[furuholm]

Really like the idea of using a scancode- prefix for non-spdx licenses.

@pombredanne One comment regarding your statement above:

In fact I was thinking to actually make no code change here but only add an spdx_license_key LicenseRef to every license that do not have one.

If the only thing to do is to add spdx_license_key to the license config files then this is something I could probably write a script to do and provide the result as a PR. But without having looked at the code this struck me as something that might not work. If an spdx_license_key is simply added then what will trigger the creation of the Other Licensing Information Detected fields (LicenseID, LicenseRef-"[idstring], ExtractedText, ...)? Or is this not a problem?

[pombredanne]

If the only thing to do is to add spdx_license_key to the license config files then this is something I could probably write a script to do and provide the result as a PR.

This could work OK, but then this would introduce a subtle issue: the spdx_license_key License attribute is supposed to be set only if this is a license present on the SPDX license list.

Because of this and for a start I would likely only touch this one line here:
https://github.com/nexB/scancode-toolkit/blob/8699983d6044625a906a3896d3a5c049acac63cd/src/formattedcode/output_spdx.py#L241

Then I would do a replace of regen=False by regen=True in https://github.com/nexB/scancode-toolkit/blob/8699983d6044625a906a3896d3a5c049acac63cd/tests/formattedcode/test_output_spdx.py#L104 and in https://github.com/nexB/scancode-toolkit/blob/8699983d6044625a906a3896d3a5c049acac63cd/tests/formattedcode/test_output_spdx.py#L136 and run all the tests with py.test -vvs tests/formattedcode/test_output_spdx.py

Then I would undo this replace.

This would regenerate all tests expectations: I would then review these with diffs as the outgoing git changes.

This is likely all there is to this at first

[furuholm]

@pombredanne #1307 implements the proposed changes. Thanks to your excellent instructions this was very straight forward. I have verified this on a few projects. Think it may be good if you do the same.

[pombredanne]

This has been implemented by @furuholm (thank you ++ ) and is now available in the develop branch.

[pombredanne]

There is some discussions around namespacing @ SPDX so let me reopen this ticket since we have a good history here.

[pombredanne]

This has been completely implemented and we now assign ALWAYS an SPDX id to every license we have. See also https://scancode-licensedb.aboutcode.org/ for all the LicenseRefs

[sschuberth]

A somewhat related question: Has ScanCode already bothered to submit a license namespace at https://tools.spdx.org/app/?

[pombredanne]

@sschuberth

A somewhat related question: Has ScanCode already bothered to submit a license namespace at https://tools.spdx.org/app/?

Of course! a couple times but this is still failing for now. See spdx/spdx-online-tools#292

[pombredanne]

@sschuberth this is now registered at https://tools.spdx.org/app/license_namespace_requests/2/ thanks to @goneall help.