Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Win build + install python bindings on Unix #8

Merged
merged 56 commits into from
Aug 9, 2022

Conversation

Tobias-Fischer
Copy link
Contributor

Checklist

  • Used a personal fork of the feedstock to propose changes
  • Bumped the build number (if the version is unchanged)
  • Reset the build number to 0 (if the version changed)
  • Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
  • Ensured the license file is being packaged.

Sorry, something went wrong.

Verified

This commit was signed with the committer’s verified signature.
crazy-max CrazyMax
@conda-forge-linter
Copy link

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

@Tobias-Fischer
Copy link
Contributor Author

@conda-forge-admin, please rerender

@Tobias-Fischer
Copy link
Contributor Author

Hi @traversaro @wolfv - any clue how to overcome this issue? I think it's something basic but I'm not a Windows expert ..

2022-07-28T02:58:21.9878572Z LINK : fatal error LNK1104: cannot open file 'lib\metis-gtsamDebug.lib'

Note that prior to that:

2022-07-28T02:51:29.6173860Z [62/285] Linking C shared library bin\metis-gtsamDebug.dll

So the dll gets created but not the lib.

@Tobias-Fischer
Copy link
Contributor Author

@conda-forge-admin, please rerender

@traversaro
Copy link
Contributor

So the dll gets created but not the lib.

Not sure if you solved the problem, but this tipically happens if no symbols are exported.

@Tobias-Fischer
Copy link
Contributor Author

I'm trying without Python and without the gtsam-unstable for now just to see if that succeeds. I also limited the number of cores on the win build, as half the time the compiler went out of heap space.

@Tobias-Fischer
Copy link
Contributor Author

So it seems like the python buildings on Windows are terribly broken, with

python\gtsam\gtsam.cp37-win_amd64.pyd : fatal error LNK1120: 105 unresolved externals

Just the first few lines of errors:


LINK: command "C:\PROGRA~2\MICROS~1\2019\ENTERP~1\VC\Tools\MSVC\1429~1.301\bin\Hostx64\x64\link.exe /nologo python\CMakeFiles\gtsam_py.dir\gtsam.cpp.obj python\CMakeFiles\gtsam_py.dir\base.cpp.obj python\CMakeFiles\gtsam_py.dir\geometry.cpp.obj python\CMakeFiles\gtsam_py.dir\linear.cpp.obj python\CMakeFiles\gtsam_py.dir\nonlinear.cpp.obj python\CMakeFiles\gtsam_py.dir\symbolic.cpp.obj python\CMakeFiles\gtsam_py.dir\sam.cpp.obj python\CMakeFiles\gtsam_py.dir\slam.cpp.obj python\CMakeFiles\gtsam_py.dir\sfm.cpp.obj python\CMakeFiles\gtsam_py.dir\navigation.cpp.obj python\CMakeFiles\gtsam_py.dir\basis.cpp.obj /out:python\gtsam\gtsam.cp37-win_amd64.pyd /implib:lib\python_gtsam.lib /pdb:python\gtsam\gtsam.pdb /dll /version:0.0 /machine:x64 /INCREMENTAL:NO -LTCG lib\gtsam.lib C:\hostedtoolcache\windows\Python\3.7.9\x64\libs\python37.lib %PREFIX%\Library\lib\boost_serialization.lib %PREFIX%\Library\lib\boost_system.lib %PREFIX%\Library\lib\boost_filesystem.lib %PREFIX%\Library\lib\boost_thread.lib %PREFIX%\Library\lib\boost_atomic.lib %PREFIX%\Library\lib\boost_date_time.lib %PREFIX%\Library\lib\boost_regex.lib %PREFIX%\Library\lib\boost_timer.lib %PREFIX%\Library\lib\boost_chrono.lib tbb.lib tbbmalloc.lib %PREFIX%\Library\lib\metis.lib kernel32.lib user32.lib gdi32.lib winspool.lib shell32.lib ole32.lib oleaut32.lib uuid.lib comdlg32.lib advapi32.lib /MANIFEST /MANIFESTFILE:python\gtsam\gtsam.cp37-win_amd64.pyd.manifest" failed (exit code 1120) with the following output:
   Creating library lib\python_gtsam.lib and object lib\python_gtsam.exp
geometry.cpp.obj : error LNK2001: unresolved external symbol "public: class Eigen::Matrix<double,-1,1,0,-1,1> __cdecl gtsam::SO<-1>::vec(class gtsam::OptionalJacobian<-1,-1>)const " (?vec@?$SO@$0?0@gtsam@@QEBA?AV?$Matrix@N$0?0$00$0A@$0?0$00@Eigen@@V?$OptionalJacobian@$0?0$0?0@2@@Z)
nonlinear.cpp.obj : error LNK2001: unresolved external symbol "public: virtual class boost::shared_ptr<class gtsam::NonlinearFactor> __cdecl gtsam::LinearContainerFactor::rekey(class std::map<unsigned __int64,unsigned __int64,struct std::less<unsigned __int64>,class std::allocator<struct std::pair<unsigned __int64 const ,unsigned __int64> > > const &)const " (?rekey@LinearContainerFactor@gtsam@@UEBA?AV?$shared_ptr@VNonlinearFactor@gtsam@@@boost@@AEBV?$map@_K_KU?$less@_K@std@@V?$allocator@U?$pair@$$CB_K_K@std@@@2@@std@@@Z)
nonlinear.cpp.obj : error LNK2001: unresolved external symbol "public: virtual class boost::shared_ptr<class gtsam::NonlinearFactor> __cdecl gtsam::LinearContainerFactor::rekey(class std::vector<unsigned __int64,class tbb::detail::d1::tbb_allocator<unsigned __int64> > const &)const " (?rekey@LinearContainerFactor@gtsam@@UEBA?AV?$shared_ptr@VNonlinearFactor@gtsam@@@boost@@AEBV?$vector@_KV?$tbb_allocator@_K@d1@detail@tbb@@@std@@@Z)
nonlinear.cpp.obj : error LNK2001: unresolved external symbol "public: virtual void __cdecl gtsam::CustomFactor::print(class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> > const &,class std::function<class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> > __cdecl(unsigned __int64)> const &)const " (?print@CustomFactor@gtsam@@UEBAXAEBV?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@AEBV?$function@$$A6A?AV?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@_K@Z@4@@Z)
nonlinear.cpp.obj : error LNK2001: unresolved external symbol "public: virtual class Eigen::Matrix<double,-1,1,0,-1,1> __cdecl gtsam::CustomFactor::unwhitenedError(class gtsam::Values const &,class boost::optional<class std::vector<class Eigen::Matrix<double,-1,-1,0,-1,-1>,class std::allocator<class Eigen::Matrix<double,-1,-1,0,-1,-1> > > &>)const " (?unwhitenedError@CustomFactor@gtsam@@UEBA?AV?$Matrix@N$0?0$00$0A@$0?0$00@Eigen@@AEBVValues@2@V?$optional@AEAV?$vector@V?$Matrix@N$0?0$0?0$0A@$0?0$0?0@Eigen@@V?$allocator@V?$Matrix@N$0?0$0?0$0A@$0?0$0?0@Eigen@@@std@@@std@@@boost@@@Z)
slam.cpp.obj : error LNK2001: unresolved external symbol "__declspec(dllimport) public: __cdecl gtsam::FrobeniusFactor<class gtsam::SO<3> >::FrobeniusFactor<class gtsam::SO<3> >(unsigned __int64,unsigned __int64,class boost::shared_ptr<class gtsam::noiseModel::Base> const &)" (__imp_??0?$FrobeniusFactor@V?$SO@$02@gtsam@@@gtsam@@QEAA@_K0AEBV?$shared_ptr@VBase@noiseModel@gtsam@@@boost@@@Z)
slam.cpp.obj : error LNK2001: unresolved external symbol "__declspec(dllimport) public: __cdecl gtsam::FrobeniusFactor<class gtsam::SO<4> >::FrobeniusFactor<class gtsam::SO<4> >(unsigned __int64,unsigned __int64,class boost::shared_ptr<class gtsam::noiseModel::Base> const &)" (__imp_??0?$FrobeniusFactor@V?$SO@$03@gtsam@@@gtsam@@QEAA@_K0AEBV?$shared_ptr@VBase@noiseModel@gtsam@@@boost@@@Z)
slam.cpp.obj : error LNK2001: unresolved external symbol "__declspec(dllimport) public: __cdecl gtsam::FrobeniusBetweenFactor<class gtsam::SO<4> >::FrobeniusBetweenFactor<class gtsam::SO<4> >(unsigned __int64,unsigned __int64,class gtsam::SO<4> const &,class boost::shared_ptr<class gtsam::noiseModel::Base> const &)" (__imp_??0?$FrobeniusBetweenFactor@V?$SO@$03@gtsam@@@gtsam@@QEAA@_K0AEBV?$SO@$03@1@AEBV?$shared_ptr@VBase@noiseModel@gtsam@@@boost@@@Z)
  Hint on symbols that are defined and could potentially match:
    "__declspec(dllimport) public: __cdecl gtsam::FrobeniusBetweenFactor<class gtsam::SO<3> >::FrobeniusBetweenFactor<class gtsam::SO<3> >(unsigned __int64,unsigned __int64,class gtsam::SO<3> const &,class boost::shared_ptr<class gtsam::noiseModel::Base> const &)" (__imp_??0?$FrobeniusBetweenFactor@V?$SO@$02@gtsam@@@gtsam@@QEAA@_K0AEBV?$SO@$02@1@AEBV?$shared_ptr@VBase@noiseModel@gtsam@@@boost@@@Z)
slam.cpp.obj : error LNK2001: unresolved external symbol "__declspec(dllimport) public: virtual __cdecl gtsam::FrobeniusBetweenFactor<class gtsam::SO<4> >::~FrobeniusBetweenFactor<class gtsam::SO<4> >(void)" (__imp_??1?$FrobeniusBetweenFactor@V?$SO@$03@gtsam@@@gtsam@@UEAA@XZ)
  Hint on symbols that are defined and could potentially match:
    "__declspec(dllimport) public: virtual __cdecl gtsam::FrobeniusBetweenFactor<class gtsam::SO<3> >::~FrobeniusBetweenFactor<class gtsam::SO<3> >(void)" (__imp_??1?$FrobeniusBetweenFactor@V?$SO@$02@gtsam@@@gtsam@@UEAA@XZ)

Therefore, I suggest disabling python bindings on Windows for now. We can open an upstream issue to see if they are interested in fixing it?

@Tobias-Fischer Tobias-Fischer changed the title Try win build Win build + install python bindings on Unix Jul 30, 2022
@Tobias-Fischer
Copy link
Contributor Author

I also noticed that previous builds didn't install the python bindings (they were built but not installed).

On linux-64 everything builds+installs fine, but on osx something goes wrong with the relocations. The python bindings are looking for the libs in the build folder, which don't exist at test time:

import: 'gtsam'
Traceback (most recent call last):
  File "/Users/runner/miniforge3/conda-bld/gtsam_1659143045054/test_tmp/run_test.py", line 2, in <module>
    import gtsam
  File "/Users/runner/miniforge3/conda-bld/gtsam_1659143045054/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/lib/python3.8/site-packages/gtsam/__init__.py", line 7, in <module>
    from gtsam import gtsam, utils
ImportError: dlopen(/Users/runner/miniforge3/conda-bld/gtsam_1659143045054/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/lib/python3.8/site-packages/gtsam/gtsam.cpython-38-darwin.so, 2): Library not loaded: /Users/runner/miniforge3/conda-bld/gtsam_1659143045054/work/build/gtsam/libgtsam.4.dylib
  Referenced from: /Users/runner/miniforge3/conda-bld/gtsam_1659143045054/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/lib/python3.8/site-packages/gtsam/gtsam.cpython-38-darwin.so
  Reason: image not found

Also see related warnings:

WARNING (gtsam,lib/python3.8/site-packages/gtsam_unstable/gtsam_unstable.cpython-38-darwin.so): /Users/runner/miniforge3/conda-bld/gtsam_1659143045054/work/build/gtsam_unstable/libgtsam_unstable.4.dylib not found in packages, sysroot(s) nor the missing_dso_whitelist.
.. is this binary repackaging?
WARNING (gtsam,lib/python3.8/site-packages/gtsam_unstable/gtsam_unstable.cpython-38-darwin.so): /Users/runner/miniforge3/conda-bld/gtsam_1659143045054/work/build/gtsam/libgtsam.4.dylib not found in packages, sysroot(s) nor the missing_dso_whitelist.
.. is this binary repackaging?
...
WARNING (gtsam,lib/python3.8/site-packages/gtsam/gtsam.cpython-38-darwin.so): /Users/runner/miniforge3/conda-bld/gtsam_1659143045054/work/build/gtsam/libgtsam.4.dylib not found in packages, sysroot(s) nor the missing_dso_whitelist.
.. is this binary repackaging?

Any ideas how to resolve this @wolfv @traversaro? Weird that it works on linux but not osx.

@Tobias-Fischer
Copy link
Contributor Author

I fixed above error by setting CMAKE_SKIP_RPATH=TRUE

In this case conda-build is smart enough to do a proper relocation. However this does not happen till the packaging step, so I'm skipping ninja check on osx for now.

I think this is ready to go, unless there are any objections?

The only missing bit are the windows python bindings - I'll open an upstream issue. I'll also try and get some patches upstreamed.

@Tobias-Fischer
Copy link
Contributor Author

@conda-forge-admin, please rerender

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants