Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CMake(Python): Installing OR Tools for Python on a M1 Processor Mac #2332

Closed
NatanMish opened this issue Jan 12, 2021 · 60 comments
Closed

CMake(Python): Installing OR Tools for Python on a M1 Processor Mac #2332

NatanMish opened this issue Jan 12, 2021 · 60 comments
Assignees
@Mizux
Labels
Build: CMake CMake based build issue Feature Request Missing Feature/Wrapper Lang: C++ Native implementation issue Lang: Python Python wrapper issue OS: Mac MacOS
Milestone
v9.4

Comments

@NatanMish
Copy link

What version of OR-tools and what language are you using?
Version: Trying to install OR Tools on a Mac OSX 11.1 with a M1 chip. No specific version.
Language: Python

What operating system (Linux, Windows, ...) and version?
Mac Big Sur 11.1 Apple M1 chip

I am using a conda environment with Python 3.9.1. What I've tried:

  1. Installing using the binary distribution as instructed here: https://developers.google.com/optimization/install/python/mac
    I receive the error message "No matching distribution found for ortools" which makes sense as from how it looks there is no version for this stack: https://pypi.org/project/ortools/#files

  2. Installing from source as instructed here: https://developers.google.com/optimization/install/python/source_mac

My guess is maybe there are special adjustments needed to be done and a specific distribution to be released for this processor? or maybe I am missing something? Anyone had any experience with this kind of stack?

Any help would be appreciated!
Natan.
@aschn
Copy link

aschn commented Jan 13, 2021

+1 to the issue. I have the same problem, and no solution yet.

@lperron
Copy link
Collaborator

lperron commented Jan 13, 2021 via email

@NatanMish
Copy link
Author

Are there any plans for building a distribution for this hardware? This is probably going to be a problem for many more users in the future since Apple are double downing on this chip.

I offer my help with testing etc. if needed!

Thanks.

@Mizux Mizux self-assigned this Jan 14, 2021
@Mizux Mizux added Feature Request Missing Feature/Wrapper Lang: C++ Native implementation issue OS: Mac MacOS labels Jan 14, 2021
@Mizux Mizux added this to the v8.3 milestone Jan 14, 2021
@Mizux Mizux added the Build: CMake CMake based build issue label Jan 14, 2021
@Mizux
Copy link
Collaborator

Mizux commented Jan 14, 2021 •
edited
Loading

My understanding:

  1. I can't find any Apple M1 native python package related discussion on pypa

  2. Apple M1 should have roseta2 to support x86_64 emulation so using the macos x86_64 python package should work

    • now we need to understand why Python3 pip disallow to install it (any help/link welcome)

  3. Using the CMake based build, we now support the cross compilation for aarch64
    So, if Apple provide us a M1 cmake cross toolchain, we may be able to build a M1 native ortools lib, then we'll need to find a way to generate a M1 python package from it and publish it to pypi.org...
    ref: https://github.com/google/or-tools/blob/master/cmake/docker/arm/or-tools.Dockerfile

@Mizux
Copy link
Collaborator

Mizux commented Jan 14, 2021

The M1 is in the same class of processors as the A-series chips used in iPhones and iPads so I don't think it should be a problem. They already said the M1 computers will be able to run all iPhone apps natively

ref: react-native-community/discussions-and-proposals#295 (comment)

@DanielChuDC
Copy link

DanielChuDC commented Jan 14, 2021 •
edited
Loading

What version of OR-tools and what language are you using?
Version: Trying to install OR Tools on a Mac OSX 11.1 with a M1 chip. No specific version.
Language: Python

What operating system (Linux, Windows, ...) and version?
Mac Big Sur 11.1 Apple M1 chip

I am using a conda environment with Python 3.9.1. What I've tried:

  1. Installing using the binary distribution as instructed here: https://developers.google.com/optimization/install/python/mac
    I receive the error message "No matching distribution found for ortools" which makes sense as from how it looks there is no version for this stack: https://pypi.org/project/ortools/#files
  2. Installing from source as instructed here: https://developers.google.com/optimization/install/python/source_mac

My guess is maybe there are special adjustments needed to be done and a specific distribution to be released for this processor? or maybe I am missing something? Anyone had any experience with this kind of stack?

Any help would be appreciated!
Natan.

Hi @NatanMish, do you mind downgrading your python into 3.8? I using 3.91 also faced the no found error. but 3.8 can. FYI, my mac is not m1 mac. Just a suggestion.

@Mizux
Copy link
Collaborator

Mizux commented Jan 14, 2021

@NatanMish could you run this python command so we can know how python3 see a M1 hardware:

import os
import platform

print(f'os name: {os.name}')

print(f'processor: {platform.processor()}')
print(f'machine: {platform.machine()}')
print(f'system: {platform.system()}')
print(f'release: {platform.release()}')

print(f'uname: {platform.uname()}')

@Mizux
Copy link
Collaborator

Mizux commented Jan 14, 2021

Installer news
3.9.1 is the first version of Python to support macOS 11 Big Sur. With Xcode 11 and later it is now possible to build “Universal 2” binaries which work on Apple Silicon. We are providing such an installer as the macos11.0 variant. This installer can be deployed back to older versions, tested down to OS X 10.9. As we are waiting for an updated version of pip, please consider the macos11.0 installer experimental.

ref: https://www.python.org/downloads/release/python-391/

@NatanMish
Copy link
Author

Thanks @Mizux !
os name: posix
processor: arm
machine: arm64
system: Darwin
release: 20.2.0
uname: uname_result(system='Darwin', node='Natans-MBP', release='20.2.0', version='Darwin Kernel Version 20.2.0: Wed Dec 2 20:40:21 PST 2020; root:xnu-7195.60.75~1/RELEASE_ARM64_T8101', machine='arm64', processor='arm')

@NatanMish
Copy link
Author

What version of OR-tools and what language are you using?
Version: Trying to install OR Tools on a Mac OSX 11.1 with a M1 chip. No specific version.
Language: Python
What operating system (Linux, Windows, ...) and version?
Mac Big Sur 11.1 Apple M1 chip
I am using a conda environment with Python 3.9.1. What I've tried:

  1. Installing using the binary distribution as instructed here: https://developers.google.com/optimization/install/python/mac
    I receive the error message "No matching distribution found for ortools" which makes sense as from how it looks there is no version for this stack: https://pypi.org/project/ortools/#files
  2. Installing from source as instructed here: https://developers.google.com/optimization/install/python/source_mac

My guess is maybe there are special adjustments needed to be done and a specific distribution to be released for this processor? or maybe I am missing something? Anyone had any experience with this kind of stack?
Any help would be appreciated!
Natan.

Hi @NatanMish, do you mind downgrading your python into 3.8? I using 3.91 also faced the no found error. but 3.8 can. FYI, my mac is not m1 mac. Just a suggestion.

Thanks @DanielChuDC but I already tried and couldn't find a distribution either.

@NatanMish NatanMish changed the title Installing OR Tools for Python on an M1 Processor Mac Installing OR Tools for Python on a M1 Processor Mac Jan 14, 2021
@Mizux
Copy link
Collaborator

Mizux commented Jan 14, 2021 •
edited
Loading

@NatanMish many thanks !
it seems wheel need/use the machine field to tag a package...

I'll try to see, if using the linaro aarch64 cross compile toolchain, I can build an arm64 version of ortools...

@Mizux Mizux added the Lang: Python Python wrapper issue label Jan 14, 2021
@DanielChuDC
Copy link

DanielChuDC commented Jan 14, 2021 •
edited
Loading

@NatanMish can check it out.

I using pyenv and let it manage python version to 3.8.6. And update the pip version using .pyenv/versions/3.8.6/bin/python -m pip install --upgrade pip into 20.3.3 and it can find the or-tools


// log

$ `.pyenv/versions/3.8.6/bin/python -m pip install --upgrade pip`
Collecting pip
  Downloading pip-20.3.3-py2.py3-none-any.whl (1.5 MB)
     |████████████████████████████████| 1.5 MB 10.7 MB/s 
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 20.2.1
    Uninstalling pip-20.2.1:
      Successfully uninstalled pip-20.2.1
Successfully installed pip-20.3.3


$ `python -m pip install --upgrade --user ortools`                           
Collecting ortools
  WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Read timed out. (read timeout=15)")': /packages/95/5c/2bd6de6d3f23ef6771dca18b1e6efd8c6b6216f3871cb5bd102e161ada85/ortools-8.1.8487-cp38-cp38-macosx_10_9_x86_64.whl
  WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Read timed out. (read timeout=15)")': /packages/95/5c/2bd6de6d3f23ef6771dca18b1e6efd8c6b6216f3871cb5bd102e161ada85/ortools-8.1.8487-cp38-cp38-macosx_10_9_x86_64.whl
  Downloading ortools-8.1.8487-cp38-cp38-macosx_10_9_x86_64.whl (11.9 MB)
     |████████████████████████████████| 11.9 MB 7.5 MB/s 
Collecting absl-py>=0.11
  Downloading absl_py-0.11.0-py3-none-any.whl (127 kB)
     |████████████████████████████████| 127 kB 10.7 MB/s 
Collecting protobuf>=3.14.0
  Downloading protobuf-3.14.0-cp38-cp38-macosx_10_9_x86_64.whl (1.0 MB)
     |████████████████████████████████| 1.0 MB 11.6 MB/s 
Collecting six
  Using cached six-1.15.0-py2.py3-none-any.whl (10 kB)
Installing collected packages: six, protobuf, absl-py, ortools
Successfully installed absl-py-0.11.0 ortools-8.1.8487 protobuf-3.14.0 six-1.15.0

@NatanMish
Copy link
Author

Hey @DanielChuDC yeah I was using pip 20.3.3 too, that is not the issue.
Probably because what Mizux mentioned above.

@Mizux
Copy link
Collaborator

Mizux commented Jan 14, 2021 •
edited
Loading

could be related to pypa/packaging#319

note to myself: python/cpython#21246
https://developer.apple.com/documentation/xcode/building_a_universal_macos_binary

@gboudreau
Copy link

Trying to compile from source on M1, you'll hit issues during make third_party where a few third-party dependencies don't compile, complaining about configure: error: cannot guess build type; you must specify one (I'm guessing that is because they do not know about aarch64 as a possible architecture.)
There is also a library filename that is hard-coded to the x86_64 variant, causing an error during the linking of libortools.so.

I was able to compile everything after making those changes to the makefile: Netlift@bbfcaa5

or-tools itself compiles without problems on M1 after that.

Trivia: make third_party takes 417s (7 minutes), and make python takes 476s (8 minutes), on my Macbook Pro M1.

@NatanMish
Copy link
Author

@gboudreau Thanks for sharing!
but unfortunately make third party still doesn't work. Here is the output:

src/nlpi/exprinterpret_cppad.cpp:102:38: error: expected ';' after top level declarator
static std::atomic_size_t ncurthreads{0};
                                     ^
                                     ;
src/nlpi/exprinterpret_cppad.cpp:103:8: error: unknown type name 'thread_local'
static thread_local int thread_number{-1};
       ^
src/nlpi/exprinterpret_cppad.cpp:103:38: error: expected ';' after top level declarator
static thread_local int thread_number{-1};
                                     ^
                                     ;
3 errors generated.
make[2]: *** [obj/static/O.darwin.arm.gnu.opt/lib/nlpi/exprinterpret_cppad.o] Error 1
make[1]: *** [libnlpi] Error 2
make: *** [dependencies/install/lib/libscip.a] Error 2

@gboudreau
Copy link

I should have said that I am working in a Linux docker container, so the fixes I made were for Linux aarch64 really (but they also fix architecture issues when building on/for Mac).

The error you show above doesn't seem related to the M1 architecture; looks like a syntax problem, which could happen if the toolchain used to compile doesn't support the right version of the syntax used by the source (the SCIP third-party library). My guess would be an incompatibility between the latest Xcode build tools and that library.

To validate, someone on Big Sur and using an Intel processor, and latest Xcode could try compiling or-tools from source.

@lperron
Copy link
Collaborator

lperron commented Jan 23, 2021 via email

@NatanMish
Copy link
Author

You can pass USE_SCIP=OFF to cmake, or define it in Makefile.local if using make to disable SCIP. Laurent Perron | Operations Research | [email protected] | (33) 1 42 68 53 00 Le sam. 23 janv. 2021 à 20:36, NatanMish [email protected] a écrit :
…
@gboudreau https://github.com/gboudreau Thanks for sharing! but unfortunately make third party still doesn't work. Here is the output: src/nlpi/exprinterpret_cppad.cpp:102:38: error: expected ';' after top level declarator static std::atomic_size_t ncurthreads{0}; ^ ; src/nlpi/exprinterpret_cppad.cpp:103:8: error: unknown type name 'thread_local' static thread_local int thread_number{-1}; ^ src/nlpi/exprinterpret_cppad.cpp:103:38: error: expected ';' after top level declarator static thread_local int thread_number{-1}; ^ ; 3 errors generated. make[2]: *** [obj/static/O.darwin.arm.gnu.opt/lib/nlpi/exprinterpret_cppad.o] Error 1 make[1]: *** [libnlpi] Error 2 make: *** [dependencies/install/lib/libscip.a] Error 2 — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#2332 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACUPL3IXVIHQBGLDUWKRK2TS3MQMBANCNFSM4V7PMD4A .

Unfortunately this doesn't work because of #2333

@gregy4
Copy link

gregy4 commented Feb 25, 2021

Maybe off topic but what part of or-tools do you plan to use ? I know that cp-sat solver was on notebook with only 2 real cores pretty useless. M1 is not x86 architecture, has two types of cores (performance and slow) which is normal in arm world but not in x86 ... highly probably some time will be needed to tweak performance for this architecture. Not to mention that it is probably better to wait for more powerful variant (desktop, more cores) of M1 to justify the effort for new architecture primarily in multicore tasks.

@lperron
Copy link
Collaborator

lperron commented Jan 12, 2022 via email

@IgnaceBleukx
Copy link

Yes, indeed. Install ortools using pip works on a rosetta emulated terminal. But you can expect some serious slowdowns...
https://osxdaily.com/2020/11/18/how-run-homebrew-x86-terminal-apple-silicon-mac/

@bjackson
Copy link

The binaries in the universal2 wheels are only for x86_64.

You can verify this by downloading the wheel and running

➜ lipo -archs linear_solver/_pywraplp.so
x86_64

@hazelweakly hazelweakly mentioned this issue Jan 21, 2022
Merged
@vitormhenrique
Copy link

@machinaut or @jcpaik would you guys mind sharing your compiled wheel packages?

@Mizux Mizux modified the milestones: v9.3, v10.0 Feb 11, 2022
@joaquinjsb
Copy link

any news on this issue? I use the dylib with Java and it doesn't work

@lperron
Copy link
Collaborator

lperron commented Mar 5, 2022 via email

@Mizux
Copy link
Collaborator

Mizux commented Mar 5, 2022 •
edited
Loading

You should be able to build a python M1 package
please verify you are using a m1 python environment and not a rosetta2's x86_64 one
e.g; anaconda provide a x86_64 python on m1...

to check:

python -c "import platform as p;print(p.machine(), p.platform())"

Then to build:

cmake -S. -Bbuild -DBUILD_PYTHON=ON -DPython3_ROOT_DIR="path_of_your_python_install"
cmake --build build -j4 -v

you'll have the wheel package in build/python/dist

@pioz
Copy link

pioz commented Mar 7, 2022

My 2 cents to compile and install or-tools with M1 mac using Python 3.9 installed with Homebrew:

git clone https://github.com/google/or-tools.git
cd ./or-tools
cmake -S. -Bbuild -DBUILD_PYTHON=ON -DPython3_ROOT_DIR="/opt/homebrew/Cellar/[email protected]/3.9.10/"
cmake --build build -j8 -v  
pip3 install ./build/python/dist/ortools-9.2.9974-cp39-cp39-macosx_12_0_arm64.whl

@lperron
Copy link
Collaborator

lperron commented Mar 7, 2022 via email

@pioz
Copy link

pioz commented Mar 7, 2022

I believe you need 3.10 to generate a native M1 lib. Le lun. 7 mars 2022, 18:25, Enrico @.***> a écrit :

On my mac, it works well with Python 3.9.

@lperron
Copy link
Collaborator

lperron commented Mar 7, 2022 via email

@pioz
Copy link

pioz commented Mar 7, 2022 •
edited by Mizux
Loading

It works, through the Rosetta emulation. Le lun. 7 mars 2022, 20:57, Enrico @.***> a écrit :

mmm, I'm not sure, but I think I'm not under Rosetta: 🧐

[~]$ arch
arm64
[~]$ which python3                                                     
/opt/homebrew/bin/python3
[~]$ python3 -c "import platform as p;print(p.machine(), p.platform())"

arm64 macOS-12.2.1-arm64-arm-64bit
[~]$ python3                                                           
Python 3.9.10 (main, Jan 15 2022, 11:40:53) 
[Clang 13.0.0 (clang-1300.0.29.3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from ortools.linear_solver import pywraplp
>>> solver = pywraplp.Solver('Cart optimizer', pywraplp.Solver.CBC_MIXED_INTEGER_PROGRAMMING)
>>> solver.__class__
<class 'ortools.linear_solver.pywraplp.Solver'>
>>>

@lperron
Copy link
Collaborator

lperron commented Mar 7, 2022 via email

@gverbist gverbist mentioned this issue Mar 8, 2022
Open
@Mizux Mizux modified the milestones: v10.0, v9.4 Mar 17, 2022
@Mizux Mizux closed this as completed Mar 17, 2022
@artur79
Copy link

artur79 commented Apr 1, 2022 •
edited
Loading

My 2 cents to compile and install or-tools with M1 mac using Python 3.9 installed with Homebrew:

git clone https://github.com/google/or-tools.git
cd ./or-tools
cmake -S. -Bbuild -DBUILD_PYTHON=ON -DPython3_ROOT_DIR="/opt/homebrew/Cellar/[email protected]/3.9.10/"
cmake --build build -j8 -v  
pip3 install ./build/python/dist/ortools-9.2.9974-cp39-cp39-macosx_12_0_arm64.whl

it failed for me on compile step (prelast line) that may be the reason:

--mypy_out: protoc-gen-mypy: Plugin failed with status code 1.

last line is:

[ 88%] Built target flatzinc
make: *** [all] Error 2

I'v got mypy-protobuf 3.2.0

any ideas ?

@joe-warrington
Copy link

I also have this issue.

@lperron
Copy link
Collaborator

lperron commented May 9, 2022 via email

@roytman roytman mentioned this issue Jun 8, 2022
Closed
@BenBrostoff BenBrostoff mentioned this issue Sep 13, 2022
Closed
Closed
@lperron
Copy link
Collaborator

lperron commented Oct 11, 2022 via email

@BenBrostoff BenBrostoff mentioned this issue Aug 31, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Mizux Mizux

Labels
Build: CMake CMake based build issue Feature Request Missing Feature/Wrapper Lang: C++ Native implementation issue Lang: Python Python wrapper issue OS: Mac MacOS
Projects
None yet
Milestone
v9.4
Development

No branches or pull requests