python 3.6.2 on older 64-bit Linux fails with "Illegal instruction"

[benmwebb]

Since the most recent conda update, python (and thus pretty much everything using Python, including conda itself) fails with Illegal instruction on older 64-bit Linux machines running CentOS 6. Works OK on some newer 64-bit machines though, so my guess is your most recent Python build is just pulling in a newer set of instructions (I see you rebuilt it with a newer gcc).

On the affected machine:

$ grep 'model name' /proc/cpuinfo | head -1
model name	: AMD Opteron(tm) Processor 242
$ conda info
Illegal instruction
$ python
Illegal instruction
# grep opcode /var/log/messages|tail -1
Oct  4 11:08:25 mymachine kernel: python[15066] trap invalid opcode ip:7f40774b27b3 sp:7ffe3504f490 error:0 in python3.6[7f407734a000+2be000]

While on the unaffected machine:

$ grep 'model name' /proc/cpuinfo | head -1
model name	: Intel(R) Xeon(R) CPU           E5520  @ 2.27GHz
$ conda info
               platform : linux-64
          conda version : 4.3.27
       conda is private : False
      conda-env version : 4.3.27
    conda-build version : not installed
         python version : 3.6.2.final.0
       requests version : 2.18.4
              user-agent : conda/4.3.27 requests/2.18.4 CPython/3.6.2 Linux/2.6.32-696.10.2.el6.x86_64 CentOS/6.9 glibc/2.12    
$ python
Python 3.6.2 |Anaconda custom (64-bit)| (default, Sep 30 2017, 18:42:57) [GCC 7.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 

If I force a downgrade to Python 3.6.1 with conda install python=3.6.1 then everything works OK on both machines - but I'd rather not have to do that!

[mingwandroid]

We require at least Nocona on 86_64 and Pentium 4 (Prescott) on i686. Basically SSE3 is needed. I'll bring it up with my team but we're unlikely to be able to do much about it at this stage.

From my understanding, over 99.99% of Intel computers have SSE3 nowadays.

[benmwebb]

Well, that's unfortunate. My users would be quite upset if told them I was going to decommission their Linux cluster just because Anaconda doesn't work on it (everything else in CentOS 6 does). So I guess we'll just have to stick with an older version of Anaconda here.

[msarahan]

I'm very sorry that your cluster doesn't have this feature, but I don't think we can rebuild all of our software for the next notch down at this point.

Your CPU is 130 nm (http://www.cpu-world.com/CPUs/K8/AMD-Opteron%20242%20-%20OSA242CCO5AH.html), while SSE3 was added to opterons at the 90nm process level: https://en.wikipedia.org/wiki/Opteron

The first release of the 90nm process was February 2005. I think it is reasonable to require a CPU from the past 12 years. It would probably cost us comparably in time to do those rebuilds than it would cost you to replace your cluster with a smaller number of new, more powerful nodes to have equivalent computing capability. Your energy bill would be lower, too. If you really need this cluster to stay alive, yeah, stick with older anaconda, build updated packages yourself, or contact [email protected] to get us to build it for you.

[benmwebb]

You don't need to tell me when we purchased this cluster - I remember ;)

We are academics so our energy bill is zero - grant money doesn't work the same way as real money.

We will work around this for now, but it would be nice not to be surprised by things like this again in future - where are changes like this announced? I didn't see this on the Anaconda blog, for example.

[mingwandroid]

There was no announcement, we really did not expect anyone to do serious computing on such outdated hardware!

[benmwebb]

The problem is other impoverished academics use hardware even older than ours, then complain to us when the conda packages of our software don't work on their machines. I can't point them to your requirements page, because it just says they need CentOS 6+, not that they also need newer hardware.

[bryevdv]

@msarahan perhaps we can open a discussion internally about publishing a specific hardware baseline

[mingwandroid]

We can probably dial back a bit the CFLAGS and as packages get rebuilt, the issue would go away slowly (at the cost of a slower distro).

Can you point to what older hardware people are using? AFAICT, you are using just about the oldest possible 64-bit Intel (architecture-wise) CPU in existence.

@bryevdv that sounds like a good idea.

[benmwebb]

Can you point to what older hardware people are using

On the 64-bit side, yes, I think we have the oldest. But people still run our software on 500MHz Pentium III machines...

[msarahan]

Yes, @bryevdv, we need that. Had it existed before, Ray and I would have known to make a more cautious choice about any changes. Because of the implications, we need to make this choice as a company, not as a couple of people.

[bryevdv]

Because of the implications, we need to make this choice as a company, not as a couple of people.

That was my intended meaning with "open a discussion internally"