Issues installing YAML ontologies in 3.4.1

[ghost]

I'm having issues installing the example YAML ontology provided by the SimDomeWrapper repository using OSP Core 3.4.1 (2824724). This is using Python 3.8.5 on Windows.

Note that the same YAML file installed correctly under OSP Core 3.3.8.

INFO 2020-09-16 12:11:12,695 [osp.core.ontology.yml.yml_parser]: You can now use "from osp.core.namespaces import simdome".
INFO 2020-09-16 12:11:12,695 [osp.core.ontology.parser]: Loaded 102 ontology triples in total
ERROR 2020-09-16 12:11:12,705 [osp.core.pico]: An Exception occurred during installation.
Traceback (most recent call last):
  File "C:\Users\Michael\AppData\Local\Programs\Python\Python38\lib\shutil.py", line 788, in move
    os.rename(src, real_dst)
FileNotFoundError: [WinError 3] The system cannot find the path specified: "C:\\Users\\Michael\\AppData\\Local\\Temp\\tmpofq23_pr" -> "\\Users\\Michael\\.osp_ontologies\\graph.xml"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\users\michael\appdata\python-venvs\simdome\lib\site-packages\osp\core\pico.py", line 60, in install_from_terminal
    ontology_installer.install(*args.files)
  File "c:\users\michael\appdata\python-venvs\simdome\lib\site-packages\osp\core\ontology\installation.py", line 23, in install
    self._install(files, self._get_new_packages, False)
  File "c:\users\michael\appdata\python-venvs\simdome\lib\site-packages\osp\core\ontology\installation.py", line 139, in _install
    self.namespace_registry.store(self.path)
  File "c:\users\michael\appdata\python-venvs\simdome\lib\site-packages\osp\core\ontology\namespace_registry.py", line 133, in store
    self._graph.serialize(destination=path_graph, format="xml")
  File "c:\users\michael\appdata\python-venvs\simdome\lib\site-packages\rdflib\graph.py", line 982, in serialize
    shutil.move(name, path)
  File "C:\Users\Michael\AppData\Local\Programs\Python\Python38\lib\shutil.py", line 802, in move
    copy_function(src, real_dst)
  File "C:\Users\Michael\AppData\Local\Programs\Python\Python38\lib\shutil.py", line 432, in copy2
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "C:\Users\Michael\AppData\Local\Programs\Python\Python38\lib\shutil.py", line 261, in copyfile
    with open(src, "rb") as fsrc, open(dst, "wb") as fdst:
FileNotFoundError: [Errno 2] No such file or directory: "\\Users\\Michael\\.osp_ontologies\\graph.xml"
ERROR 2020-09-16 12:11:12,705 [osp.core.pico]: Consider running "pico --log-level debug install ..."

YAML file used: simdome.ontology.yml.txt

[pablo-de-andres]

have you tried running pico --log-level debug install <path/to/ontology.yml>?
what is the output?

[pablo-de-andres]

I am not able to reproduce the error. Are you using Windows?

My suggestion:

• First check the output of --log-level debug
• That path looks weird. What happens if you give the full path to the yml file?
• Can you try with a clean installation of OSP-core?

Note: It looks like we forgot to increase the patch number in the release, I believe the version you are using is in fact 3.4.2

[ghost]

• Running with debug logging produces the exact same stack trace
• Using the absolute path to the yml file seems to make no change
• Removing OSP core and PICO from pip then rebuilding them also produces no change

[pablo-de-andres]

Could you open a python command line and run the following, please?

import os; os.path.join(os.path.expanduser("~"), ".osp_ontologies")

I don't really understand why yours is missing the C:

[ghost]

This gives the expected (correct) response.

"C:\Users\Michael\.osp_ontologies"

If I have some time this week I'll try to debug through the code, looking for anything that edits that path.

[pablo-de-andres]

I believe @urbanmatthias (in charge of this code) is back tomorrow, so he should be able to give us a better insight!

[urbanmatthias]

Hi,

thanks for reporting this issue. We are using the serialize method of rdflib to store the installed ontologies.
It looks like they internally store the graph in a temporary directory first C:\\Users\\Michael\\AppData\\Local\\Temp\\tmpofq23_pr and then they move it to the desired directory \\Users\\Michael\\.osp_ontologies\\graph.xml.

Obviously the second path is wrong - It should start with C://.

The path is constructed in osp/core/namespaces.py in line 10:

https://github.com/simphony/osp-core/blob/28247243b2d02cfd85b14d4ef12cf50cdd14432a/osp/core/namespaces.py#L10-L13

which is the command you executed and which returned the correct path.

Can you check if you have an outdated version of rdflib installed?

PS> pip freeze | findstr "rdflib"
rdflib==5.0.0

Can you print the path_graph at line 133 of osp/core/ontology/namespace_registry.py?

https://github.com/simphony/osp-core/blob/28247243b2d02cfd85b14d4ef12cf50cdd14432a/osp/core/ontology/namespace_registry.py#L133

Can you print _path at line 14 of osp/core/namespaces.py?

https://github.com/simphony/osp-core/blob/28247243b2d02cfd85b14d4ef12cf50cdd14432a/osp/core/namespaces.py#L14

[urbanmatthias]

Just as a side note, enabling debug log messages will almost always not change the traceback - but it might give us more information that can help us debugging. I'm afraid that in this case it wouldn't help much, because I didn't add any log messages that print the path. It probably would make sense to add some at the places I specified above. Anyway, maybe we have luck and there is something helpful in the logs, so could you please rerun with --log-level debug and paste the entire output here.

[urbanmatthias]

Okay I just tried to reproduce on my windows machine but couldn't.

[ghost]

I get the same version of rdflib as you (5.0.0), prints of the requested variables are below.

path_graph from namespace_registry.py:
C:\Users\Michael\.osp_ontologies\graph.xml

_path from namespaces.py:
C:\Users\Michael\.osp_ontologies

I think I've found the line that's stripping the drive from the path though; rdflib's graph.serialize() function calls urlparse() (from a version of the urllib library packaged within six I think), this appears to remove the drive. I can reproduce this with the following test code:

from urllib.parse import urlparse
import os

path = os.path.join(os.path.expanduser("~"), ".osp_ontologies")
print(path)

scheme, netloc, newpath, params, _query, fragment = urlparse(path)
print(newpath)

Which outputs:

C:\Users\Michael\.osp_ontologies
\Users\Michael\.osp_ontologies

[urbanmatthias]

You are right, I see the line in rdflib and I have the same result when I execute the code you posted.

But it seems like leaving out the C: is not an issue on my machine. The following works:

shutil.move("setup.py", "\\Users\\urba\\.osp_ontologies\\setup.py")

Maybe you have multiple drives, and therefore shutil wants the user to specify it?
Can you check if the above works for you and whether the directory C:\Users\Michael\.osp_ontologies exists.

[urbanmatthias]

I created a rdflib issue. Feel free to add anything you like: RDFLib/rdflib#1170

[ghost]

I do have multiple drives, this probably the likely cause.

The shutil.move() call does fail if no drive identifier is provided.

[urbanmatthias]

Okay let's see whether the rdflib guys answer, if not we'll either fork the repo or subclass the graph.

[urbanmatthias]

Can you check if it works with this branch: https://github.com/urbanmatthias/rdflib/tree/patch-1

This is the change I did: urbanmatthias/rdflib@81d0f99

Will create a PR if it works.

[ghost]

This version of rdflib works like a charm, I can install my ontology successfully now.

Many thanks @urbanmatthias!

[urbanmatthias]

I close this issue, as the rdflib devs merged my pr: RDFLib/rdflib#1172

[dln22]

Hello, I have just done a clean "osp-core" and "simcmclkinetics" wrapper install. It seems this issue is not fixed as I got the same error as Michael when I tried to install cmcl ontology via pico. As I understand the rdflib devs indicated that the issue is fixed on their side, maybe they forgot to merge it or we have a wrong rdflib version number set in the osp-core requirements file? When I re-installed everything I made sure that I purged conda and pip caches so that during installation pip was forced to download the rdflib from the remote PyPI repository rather than using the cached version. This did not help either. Could you please advise?

[urbanmatthias]

Hey, have you tried to install the latest rdflib master from source? I think the latest release does not contain the fix.

I agree that this should be documented on our side in an error message (or have another fix on our side that works with the latest rdflib release)

[dln22]

Thanks Matthias, installing rdflib directly from its github source fixed the issue.