(numpy) svd segfault with some big matrices

[ChristophHaag]

Openblas 0.2.6 is built with USE_OPENMP=1 NO_LAPACK=1 NUM_THREADS=4
numpy for python3 1.7.1, the same happens with python2.

Test case:

#!/usr/bin/env python3 
import numpy as np
U, s, VT = np.linalg.svd(np.ones(shape=(51, 64000)), full_matrices=False)

I was not really investigative with the size of the matrix, but this is about the smallest area in which it segfaults.
If you make it bigger in either one or both directions, it continues to segfault.
If you make it smaller in either dimension it seems to work, e.g. (50, 1000000) and (1000, 50000) work fine for me, but I haven't really tested much more.

Full backtrace: https://gist.github.com/ChristophHaag/5659298

BUT: My testcase above works when I run it with OMP_NUM_THREADS=1 ./openblas_svd.py. Any more threads than that and it segfaults.

[xianyi]

Thank you for the feedback
I think this is a bug related to DGEMM multi-threaded implementation.

[ViralBShah]

This could possibly also be the reason for some of the failures we see in julia's linear algebra test suite (multi-threaded), which have been hard to track down.

[xianyi]

Hi @ChristophHaag ,

Could you give me any tip about which LAPACK & BLAS function numpy svd calls?

Then, I can write the test code in C.

Xianyi

[ChristophHaag]

Sorry, I have no idea. I have tried looking through the numpy code, but it's confusing.

I think it should be there somewhere, but I am really not familiar with all that.
https://github.com/numpy/numpy/blob/a02457f1d76dc6727f2118f2d129ce3a5261c253/numpy/linalg/umath_linalg.c.src#L3026

Maybe the numpy people can help more?

[ghost]

I also had problems with 0.2.6 and SVD. Sometimes SVD returns completely incorrect answer with lot of NANs. I tracked it down to this http://icl.cs.utk.edu/lapack-forum/viewtopic.php?f=13&t=4250.

It turns out LAPACK 3.4.2 has many code changes and is buggy. Reverting to LAPACK 3.4.1 (git revert 08c177c) in OpenBLAS solved my problem. Maybe this is a different bug, however.

[xianyi]

Hi all,

I will try to link OpenBLAS with LAPACK 3.4.1. I hope it can fix both NANs and segfault bugs :)

Xianyi

[xianyi]

Hi @ChristophHaag ,

I think I fixed this SEGFAULT bug on develop branch. Could you test it?

Xianyi

[ChristophHaag]

Yes, segfault is gone. Thanks.