Don't require Knora and Sipi to share a directory for image upload

[subotic]

At the moment, the file upload through the webapi requires that there is a shared directory with sipi. This limits the way how the webapi and sipi can be deployed. Either they need to be running on the same server or both have access to the same network share.

[tobiasschweizer]

Yes, that is correct.

I did it this way because I did want not send the file to Sipi again after is has been already submitted to Knora (non GUI case). Do you think a network share would be feasible? I think it would make sense to run Sipi and Knora on different machines (performance).

[subotic]

Network shares require to either share directories between two machines or to have a third machine providing the share. I don't think that they are a good way to go. It would only make the installation more complicated and fragile. Just think of the network shares that we use for our backups. We would need to make sure that the network share is accessible from webapi and sipi.

The cleanest solution would be to send the file to sipi instead of storing it locally. This way, only sipi is responsible for files and doesn't require any additional setup.

[loicjaouen]

@subotic

The cleanest solution would be to send the file to sipi instead of storing it locally. This way, only sipi is responsible for files and doesn't require any additional setup.

that's the way Salsah 1.5 works, and I kept import scripts to behave alike because none of our installs (at Lausanne) do share a file system.

[benjamingeer]

The cleanest solution would be to send the file to sipi instead of storing it locally. This way, only sipi is responsible for files and doesn't require any additional setup.

I suggested doing this for the bulk import, but @tobiasschweizer pointed out that there's a problem: The Knora API server is supposed to store the original MIME type of the file. If the user tells webapi what the original MIME type is, webapi should check that this information is correct. Currently the way it works is that webapi gives Sipi the path to the original file, asking Sipi to import the file and return its original MIME type.

If the user submitted the file to Sipi first, and then just submitted the metadata (not the file) to webapi, this wouldn't be possible, because Sipi doesn't store the original MIME type. The user could say that the original file was a TIFF, when actually it was a JPEG, and webapi would have no way of knowing.

Therefore @lrosenth said that the bulk import XML should contain the filesystem path to each file to be imported, and that both webapi and sipi would need access to that path. So that's what I implemented in #491.

[tobiasschweizer]

Please note that are actually two ways of uploading images (GUI case and non GUI case), please see http://www.knora.org/documentation/manual/rst/sipi/sipi-and-knora.html

[subotic]

I think that this is not the case anymore.

[loicjaouen]

I am running late, sorry about that, so it means that the shared directory is not required anymore?

Even for the bulk import?

[benjamingeer]

If I'm not mistaken, the shared directory is still required, both for bulk import and for the 'non-GUI case' of a file upload to Knora. SipiResponderConversionPathRequestV1 contains an absolute file path:

https://github.com/dhlab-basel/Knora/blob/develop/webapi/src/main/scala/org/knora/webapi/messages/v1/responder/sipimessages/SipiMessagesV1.scala#L65

This is sent to Sipi's /convert_from_binaries route, which reads the file from disk:

https://github.com/dhlab-basel/Knora/blob/develop/sipi/scripts/convert_from_binaries.lua#L114

[benjamingeer]

What's left to be done here:

• Improve bulk import and separate from API v1 Improve bulk import and separate from API v1 #1110
• Eliminate the 'non-GUI case' in API v1

[subotic]

@tobiasschweizer is using the non-gui case. He would need some help in changing his import script.

[benjamingeer]

Yes, it shouldn't be too hard to change it to use API v2. Anyway, we don't need to do this until after we redo the bulk import.