[Question] About the parameter of writeToFile method and readFile method

[shi9qiu]

Hello,

My team is working on enhancing the functionalities of SW360 (https://github.com/eclipse/sw360) - a software component catalog application. We plan to replace the old library used in SW360 with this library. But we are facing some problems so I come here to see if I can get some help.

We are using writeToFile method (https://github.com/spdx/Spdx-Java-Library/blob/master/src/main/java/org/spdx/library/Read.java#L124) and readFile method (https://github.com/spdx/Spdx-Java-Library/blob/master/src/main/java/org/spdx/library/Write.java#L97) to implement the Import/Export functions of SW360. However, these methods need one parameter modelStore, but as we understand, this interface was not implemented yet.
Ref:

Spdx-Java-Library/src/main/java/org/spdx/storage/ISerializableModelStore.java

Line 31 in 42fcd5b

public interface ISerializableModelStore extends IModelStore {

Could you tell me if our understanding is correct or not?

Another question is, we found this message in the README.md:
The default storage interface is an in-memory Map which should be sufficient for light weight usage of the library.

Does it mean we cannot use this library to read or write the large size file?

Thank you in advance!

[goneall]

@shi9qiu I'm quite happy to help support the use of this library in SW360.

However, these methods need one parameter modelStore, but as we understand, this interface was not implemented yet.

There are 3 implementations of this interface, but they are in different Github repos - you can add them as dependencies to your project. The 3 implementations represent 3 different serialization formats for SPDX:

• SPDX Jackson Store supports JSON, YAML and XML formats
• SPDX RDF Store supports various RDF format (e.g. RDF/XML)
• SPDX tag/value Store supports the SPDX tag/value file format

Does it mean we cannot use this library to read or write the large size file?

I've actually used this store for some very large files without issue - it does store everything in memory, however, so I'm sure there will be a limit. I have a prototype RDF graph database implementation for very large RDF graphs. I've thought about writing an SQL based store, but haven't found the need yet.

[goneall]

@shi9qiu If you would like an example using this library with the above mentioned store, check out the SPDX Converter utility

[shi9qiu]

Thank you so much for your speedy reply! We will try it.

[nam-np]

Hello, @goneall
I used this function createSpdxDocument() for make SpdxDocument object from RDF file input, but it is seem not work with large file. In library Spdx-Java-Library has same function can work with large file?

[nam-np]

And a sample RDF large file https://github.com/eclipse/sw360/files/5984378/SPDX2_linux-4.18.tar.xz_1535128294-spdx.rdf.zip

[nam-np]

I re-checked the library spdx-tools with function Verify and it used same function createSpdxDocument for making SpdxDocument.
I tested with RDF large file https://github.com/eclipse/sw360/files/5984378/SPDX2_linux-4.18.tar.xz_1535128294-spdx.rdf.zip (~80MB) and it can work fine.

namnp@namnp:~/tools$ java -jar target/spdx-tools-2.2.6-SNAPSHOT-jar-with-dependencies.jar Verify SPDX2_linux-4.18.tar.xz_1535128294-spdx.rdf/SPDX2_linux-4.18.tar.xz_1535128294-spdx.rdf 
--------Start verify: [SPDX2_linux-4.18.tar.xz_1535128294-spdx.rdf/SPDX2_linux-4.18.tar.xz_1535128294-spdx.rdf]
WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console. Set system property 'log4j2.debug' to show Log4j2 internal initialization logging.
15:55:40.523 [main] ERROR org.spdx.rdfparser.license.ListedLicenses - I/O error opening Json TOC URL, using local TOC file
spdx.org
spdx.org
spdx.org
...
--------getSpecVersion: SPDX-2.1
--------getSpecName:   /srv/fossology/repository/report
This SPDX Document is valid.

So, I will re-check when using function with SW360.

[goneall]

@nam-np Were you able to re-check with SW360? I'm working on a minor release for this library and would like to work in fixes for this if needed.

[KoukiHama]

@goneall @nam-np
Some members in SW360 community tried new library,

We got error again.
eclipse-sw360/sw360#1171 (comment)
eclipse-sw360/sw360#1171 (comment)

But we found a root problem.
eclipse-sw360/sw360#1171 (comment)

[goneall]

@KoukiHama Thanks for the update - I left a comment in the SW360 issue.

Let me know if you'd like any help migrating from the old SPDX tools library to this one (I've done it a few times).

Since the issue is an incompatibility between the old library and this library, I'm going to close out this issue since I can't think of any way to resolve it without re-introducing a security vulnerability. If you disagree or know of a way to resolve it with a change to this issue, please add a comment.