Substantial increase in memory required to build between v2.2 and v2.3

[wxmerkt]

I've noticed this on my notebook with 16 GB before and it's currently failing on the ROS buildfarm (i.e., releases beyond 2.2.1 may not work):

virtual memory exhausted: Operation not permitted

The changes between 2.2.1 and 2.3.0 increased memory usage substantially. E.g., with 16 GB memory I can only build with a maximum of 2 threads without exhausting the built-in memory. Using 16 threads I needed ~40-50 GB memory (for Crocoddyl).

Do you have any insights what caused the increase in memory requirement to build between 2.2 and 2.3 and is there the possibility to reduce it?

[jcarpent]

There is no apparent reason I would say, except for a change in Boost. This is happening for the Python bindings or the unit tests?
One possibility is the serialization of Data and Model, which requires to stear the whole mechanism behind Boost.Variant.

[jcarpent]

On your side, can you compile the Python bindings without exposition of Data and Model?

[jcarpent]

Or by just commenting the lines:

pinocchio/bindings/python/multibody/model.hpp

Line 331 in 7765909

.def_pickle(PickleModel<Model>())

and:

pinocchio/bindings/python/multibody/model.hpp

Line 328 in 7765909

.def(SerializableVisitor<Model>())

[wxmerkt]

I am using the default CMake settings as if you were to run mkdir build ; cd build ; cmake .. ; make. I can compile (on two cores max without excessive swap) on my notebook but I have no possibility to debug the buildfarm directly.

[jcarpent]

By the way, the bindings of Crocoddyl are not done correctly. The exposition is not made in individual files, requiring a lot of memory usage.

[wxmerkt]

Sorry, I should have been clearer:

a) We see the increase in compiling just Pinocchio with its default CMake settings. This is what causes the ROS buildfarm to fail. It also makes the notebook ache/swap a lot.

b) In Crocoddyl, we see peaks of 40-50GB RAM usage when using 16 threads to compile.

I am mainly interested in solving (a) right now.

[jcarpent]

I think this is coming then from the serialization. I may find a trick to avoid this burden in memory.

[wxmerkt]

This is an example output in case it helps debugging:

/tmp/binarydeb/ros-kinetic-pinocchio-2.3.0/obj-arm-linux-gnueabihf/include/pinocchio/multibody/fcl.hpp:107:55: note: declared here
In file included from /tmp/binarydeb/ros-kinetic-pinocchio-2.3.0/obj-arm-linux-gnueabihf/include/pinocchio/multibody/geometry.hpp:8:0,
                 from /tmp/binarydeb/ros-kinetic-pinocchio-2.3.0/obj-arm-linux-gnueabihf/include/pinocchio/parsers/urdf.hpp:10,
                 from /tmp/binarydeb/ros-kinetic-pinocchio-2.3.0/unittest/multiprecision.cpp:5:
/tmp/binarydeb/ros-kinetic-pinocchio-2.3.0/obj-arm-linux-gnueabihf/include/pinocchio/multibody/fcl.hpp:201:17: warning: ‘pinocchio::GeometryObject::fcl’ is deprecated [-Wdeprecated-declarations]
   : fcl(geometry)
                 ^
In file included from /tmp/binarydeb/ros-kinetic-pinocchio-2.3.0/obj-arm-linux-gnueabihf/include/pinocchio/multibody/geometry.hpp:8:0,
                 from /tmp/binarydeb/ros-kinetic-pinocchio-2.3.0/obj-arm-linux-gnueabihf/include/pinocchio/parsers/urdf.hpp:10,
                 from /tmp/binarydeb/ros-kinetic-pinocchio-2.3.0/unittest/multiprecision.cpp:5:
/tmp/binarydeb/ros-kinetic-pinocchio-2.3.0/obj-arm-linux-gnueabihf/include/pinocchio/multibody/fcl.hpp:107:55: note: declared here

**cc1plus: out of memory allocating 5468152 bytes after a total of 67739648 bytes**
unittest/CMakeFiles/test-cpp-multiprecision.dir/build.make:65: recipe for target 'unittest/CMakeFiles/test-cpp-multiprecision.dir/multiprecision.cpp.o' failed
make[4]: *** [unittest/CMakeFiles/test-cpp-multiprecision.dir/multiprecision.cpp.o] Error 1
make[4]: Leaving directory '/tmp/binarydeb/ros-kinetic-pinocchio-2.3.0/obj-arm-linux-gnueabihf'
CMakeFiles/Makefile2:2331: recipe for target 'unittest/CMakeFiles/test-cpp-multiprecision.dir/all' failed
make[3]: *** [unittest/CMakeFiles/test-cpp-multiprecision.dir/all] Error 2
make[3]: Leaving directory '/tmp/binarydeb/ros-kinetic-pinocchio-2.3.0/obj-arm-linux-gnueabihf'
Makefile:141: recipe for target 'all' failed
make[2]: *** [all] Error 2

[jcarpent]

Yes, I will deactivate this test by default.

[jcarpent]

This is due to Boost.Multiprecision rootings.

[jcarpent]

#1079 partially solves the issue. The last missing point is the visitor test.

[wxmerkt]

Thank you for the record-time resolution of this :-)

[jcarpent]

@wxmerkt Do we need a new release to get the latest change on the ROS build farm?

[wxmerkt]

Yes, please.

[jcarpent]

Solved by #1079 and #1077.