Uncomplete installation

[alexisber]

Hi community,

I'm a new user of pywafo and have met difficulties to install it. It seems to be a classical issue for windows users (#43).

Following instructions given by @morbult in #43, I progress trough steps of the installation but it doesn't seem to work.

Commands "import wafo" and "import wafo.data as wd" work without issue

BUT commands :
"import wafo.spectrum.models as wsm" returns ModuleNotFoundError: No module named 'wafo.spectrum'
"import wafo.objects as wo" returns ModuleNotFoundError: No module named 'wafo.objects'
"import wafo.misc as wm" returns ModuleNotFoundError: No module named 'wafo.misc'
"import wafo.stats as ws" returns ModuleNotFoundError: No module named 'wafo.stats'
"import wafo as wf
wf.test(coverage=True, doctests=True)" returns AttributeError: module 'wafo' has no attribute 'test'

What's wrong with my installation? How can I fix it?

Alexis

[morbult]

Hi!

Yes, it can be a bit tricky. It seems like you have managed to install it since you are able to import wafo but that the installation is corrupt in some way.

I would start to look in the wafo installation and compare it to the files/folders here at github. All .py files and all folders except source here at github should be in your wafo installation. Run import wafo and then just wafo to see the path to your wafo installation directory, for example 'C:\\Anaconda3\\lib\\site-packages\\wafo\\__init__.py'.

Could you give some more information about how exactly you installed it? Did you create a wheel? Which Python version do you use?

[alexisber]

I use python 3.7.6 with anaconda... but i can move to another python if needed.

I did some progress restarting from the beginning (I use to have a wafo folder inside of my site-package/wafo folder. Now it's better.

But still meet problems with wheel.

First command python setup.py bdist_wheel "failed with intfcmod.o"
Second command python setup.py bdist_wheel "failed with rind71mod.o"
And the third command python setup.py bdist_wheel goes to the end

And when I test it, i get error messages (see the test notebook). https://www.dropbox.com/s/lzgyaf6guyy136m/test.ipynb?dl=0

It might come from the use of gfortran ?

[morbult]

I get about the same number of errors and failures when running wf.test(coverage=True, doctests=True) and I don't know why, I know to little about the package in whole. But the few bits I use work fine. So even though many tests fail, wafo might still work for you now depending on what parts you need to use.

[davidovitch]

In #48 a revised installation and compilation process is being worked on.

[rscorrea1]

Dears,
Is pywafo mac os compatible?
I first tried to create the binary wheel and got a fortran compile error.
Then I installed gfortran and I get the following error:

gfortran: fatal error: cannot execute 'f951': execvp: No such file or directory compilation terminated.

If pywafo is macOs compatible could you suggest me a fortran compiler?
Rodrigo

[pbrod]

I have struggled to compile the fortran files in wafo with gfortran on windows for many years.
Just recently I managed to compile it on windows by setting up the build environment in the same way as described here (see Building with numpy.distutils section):
https://pav.iki.fi/blog/2017-10-08/pywingfortran.html

For mac-os I have no idea if pywafo compiles. In theory it should.

Maybe you can find some pointers here:
https://www.macinchem.org/reviews/fortran/fortran.php

[morbult]

@pbrod Do you need to build and include OpenBLAS to make the fortran files to compile?

After cloning the pywafo master, I followed that guide you linked to (almost, the differences are marked in bold):

0. numpy>=1.14
1. Install MSVC build tools corresponding to my Python version (use version 14.X for Python 3.5, 3.6, 3.7, 3.8), https://www.visualstudio.com/downloads/#build-tools-for-visual-studio-2019, requires setuptools>=34.4.0
2. Install mingw-w64 (with gfortran) from http://www.msys2.org/. I didn't get this to work so I installed tdm-gcc 64-bit from https://jmeubank.github.io/tdm-gcc/ instead.
3. Update packages pip setuptools wheel. I used conda like so conda update pip setuptools wheel
4. "If you need LAPACK/BLAS, build OpenBLAS with the Mingw64 toolchain into a static openblas.a library."
   I skipped this step since I don't know if wafo needs OpenBLAS and have no idea how to build it.
5. Add gfortran.exe tp PATH (done automaticcaly when installing tdm-gcc)
   python setup.py bdist_wheel

The final build command still needs to be run 3 times. After that however a wheel is created under dist, and these DLLs are placed in the .libs folder in the wafo installation/wheel, so it seems the fortran files where compiled.

But the DLLs can't be found by wafo (import wafo.mvn gives ImportError: DLL load failed: The specified module could not be found.) so I think the following needs to be added to the beginning of __init__.py, before any fortran extensions are imported

import os
_dll_dir = os.path.join(os.path.dirname(__file__), '.libs')
if os.path.isdir(_dll_dir):
    os.environ['PATH'] += os.pathsep + _dll_dir

Adding that, building again, reinstalling wafo. import wafo.mvn then causes Python to crash. So I think the DLLs are found but they are obviously corrupt. Maybe I used the wrong gfortran version? Or I need OpenBLAS? Or something else I missed?

[pbrod]

I don't think you neeed to build OpenBLAS. For me it works without it.
After installing mingw-w64 (with gfortran) from http://www.msys2.org/,
I opened a conda powershell prompt and added the path to mingw by the following command:

 $env:Path += ";c:\msys64\mingw64\bin"      

To build the wheel I just do (in the same powershell window and at the pywafo root):

python setup.py bdist_wheel

Then move to the dist subfolder and

pip install wafo-1.0.0-cp37-cp37m-win_amd64.whl   

After the install, all the "XXXX.pyd"-files should be located in the
Anaconda\Lib\site-packages\wafo\ folder. If not, you must copy them to this folder.
There is no point to add the path to the dlls, it will not work because the other py-files expects them to be in the root folder of wafo.

To test that everything works you could open up an ipython window and do:

>>> import wafo
>>> wafo.test() 

The output should give 186 tests passed, 1 skipped and 149 warnings.

[morbult]

Thanks for the walk-through!

I think I installed mingw-64 (with gfortran) properly after realizing I should run this in the MING64 prompt (which was not at all obvious to me)

pacman -S mingw-w64-x86_64-toolchain

When running python setup.py bdist_wheel I now get this gfortran error 5 times and the build stops, no wheel is built.

gfortran.exe: error: unrecognized command-line option '-mno-cygwin'; did you mean '-mno-clwb'?

I don't understand where this option is set but I will continue my attempts when I get more time.

With the wheel I built and installed before (using tdm-gcc 64 instead of mingw-w64) I do get the 5 .pyd files in the installation root Anaconda3\Lib\site-packages\wafo\ (c_library, mvn, mvnprdmod, rindmod and cov2mod) and the dlls in Anaconda3\Lib\site-packages\wafo\.libs\ . At import wafo I get the 4 warnings due to ImportError of mvn, mvnprdmod, rindmod and cov2mod. And wafo.test() gives 5 failed, 181 passed, 1 skipped, 152 warnings. 5 fails all due to the fortran extensions.

[pbrod]

I think I used pacman -S mingw64/mingw-w64-x86_64-gcc-fortran on my installation that works.

My gcc and gfortran versions are:
gcc.exe (Rev2, Built by MSYS2 project) 9.3.0
GNU Fortran (Rev2, Built by MSYS2 project) 9.3.0

You may also check out these pages on how to compile fortran on windows:
https://python-at-risoe.pages.windenergy.dtu.dk/compiling-on-windows/index.html
https://www.scivision.dev/f2py-fortran-python-windows/

[justaPCWingo]

Hi all,
Thanks for the instructions, @pbrod . I've been attempting to build pywafo on Win10 under Anaconda3. Following your walkthrough I've successfully begun execution of setup.py bdist_wheel, with gfortran being located and utilized. However, when the compilation process hits the mvnprd package I receive the following error: could not find library '_mvnprdmod' in directories [], followed by: relocation truncated to fit: IMAGE_REL_AMD64_REL32 against undefined symbol 'strtoflt128'.

The former is odd, since the temp directory is added to the library search paths, and _mvnprdmod.lib is present. For the latter, I've seen some suggestion that this is due to a missing link to libquadmath.

Any suggestions as to where to investigate next?

I'm using the following:
gcc/gfortran version: 10.2.0 (Rev6, Built by MSYS2 project)
target: x86_64-w64-mingw32

[morbult]

I just tried to build wafo again and got the C and Fortran extensions to work on Windows 10.
Tested for Python 3.6 up to 3.10.
gcc/gfortran version: (Rev2, Built by MSYS2 project) 12.1.0

Once again I downloaded MSVC++ build tools from https://visualstudio.microsoft.com/visual-cpp-build-tools/ and mingw64 (with gfortran) from https://www.msys2.org/. I added C:\msys64\mingw64\bin to the environment variable PATH.
And once again I cloned the master branch, ran python setup.py bdist_wheel and got this error gfortran.exe: error: unrecognized command-line option '-mno-cygwin'; did you mean '-mno-clwb'?.
But after some googling I found that the problem is in numpy.distutils and by trial and error I found that it's been solved in numpy>=1.19.1, so make sure you fulfil this in your build environment.

I create a fresh environment using conda like this

conda create -n wafobuild38 python=3.8 numpy=1.19.1
conda activate wafobuild38

Then I navigate to wafo's setup.py and run python setup.py bdist_wheel 3 times to get the wheel file which can be installed with pip.

One remaining installation problem is that the dll files containing the compiled Fortran code are not placed in the root of wafo, but rather in wafo/.libs so they can not be found by Python.
Following the fix in this guide suggested by @pbrod (add the code below at the very top of __init__.py) works perfectly well with Python 3.6 up to 3.9.

import os
_dll_dir = os.path.join(os.path.dirname(__file__), '.libs')
if os.path.isdir(_dll_dir):
    os.environ['PATH'] += os.pathsep + _dll_dir

However with Python 3.10 I still get warnings that all Fortran extensions can not be found. I have tried to build again with the exact same set of packages and versions (numpy 1.21.2 btw) in both Python 3.9.12 and 3.10.4. In 3.9 it works and in 3.10 it doesn't...
A dirty fix is to manually move the dlls to the root of wafo after installing.

[morbult]

Forgot to mention that I included the fixes for importing newer scipy (f63f28b) and collections in Python 3.10 (7fc9cca) in #54.

[morbult]

However with Python 3.10 I still get warnings that all Fortran extensions can not be found.

Solved.

Searching 'dll' in Python's changelog lead me to os.add_dll_directory which was added in Python 3.8 and is the correct way to add a path to Windows' DLL search path.

Adding it to %PATH% with os.environ['PATH'] is apparently broken, but as far as I can tell it's the only option for Python<3.8, it may or may not work, for me it worked.

Read more here if you're interested
python/cpython#80266