numpy include path not discovered on non-Ubuntu distributions

[richmattes]

Bug report

Required Info:

• Operating System:
  • Fedora 30 x86_64
• Installation type:
  • Source
• Version or commit hash:
  • 155dacf (0.7.7)
• DDS implementation:
  • Fast-RTPS
• Client library (if applicable):
  • N/A

Steps to reproduce issue

Follow dashing installation instructions, with the Fedora-specific sub-page for package installation (with some edits I will PR once I get a complete build). The process can be automated using the following Dockerfile and commands.

Dockerfile:

FROM fedora:30

RUN useradd -m -s /bin/bash -G wheel ros
RUN echo "ros ALL=(root) NOPASSWD:ALL" > /etc/sudoers.d/ros && \
    chmod 0440 /etc/sudoers.d/ros
USER ros

RUN sudo dnf install -y cppcheck cmake gcc-c++ redhat-rpm-config \
    make patch liblsan wget libXaw-devel opencv-devel poco-devel \
    poco-foundation python3-empy python3-devel python3-nose python3-pip \
    python3-pyparsing python3-pytest python3-pytest-cov \
    python3-pytest-runner python3-setuptools python3-yaml tinyxml-devel \
    eigen3-devel python3-pydocstyle python3-pyflakes python3-coverage python3-mock \
    python3-pep8 uncrustify python3-argcomplete python3-flake8 \
    python3-flake8-import-order asio-devel tinyxml2-devel libyaml-devel \
    python3-lxml python3-vcstool python3-lark-parser python3-rosdep

RUN pip3 install --user --upgrade colcon-common-extensions "pytest>=3.10"

# Download ROS sources
RUN mkdir -p /home/ros/ros2_ws/src

WORKDIR /home/ros/ros2_ws

RUN wget https://raw.githubusercontent.com/ros2/ros2/release-latest/ros2.repos && \
    vcs import src < ros2.repos

RUN sudo rosdep init && \
    rosdep update && \
    rosdep install --from-paths src --ignore-src --rosdistro dashing -y --skip-keys "console_bridge fastcdr fastrtps libopensplice67 libopensplice69 rti-connext-dds-5.3.1 urdfdom_headers"

ENV PATH=$PATH:/home/ros/.local/bin
ENV PYTHONPATH=$PYTHONPATH:/home/ros/.local/lib/python3.7/site-packages

RUN colcon build --symlink-install

Commands

docker build -f Dockerfile .
# fails during colcon build --symlink install
# Can get an interactive shell to the environment by saving an image of the failed container
# Get ID of failed container (should be newest one)
docker container ls -a
# Commit the image
docker commit <container_id> f30_numpy
# Run bash in a new container, build tree is in ./ros2_ws
docker --rm -it f30_numpy /bin/bash

Expected behavior

colcon build completes without error

Actual behavior

rosidl_generator_py fails with warnings about missing numpy includes:

/home/ros/ros2_ws/build/rosidl_generator_py/rosidl_generator_py/rosidl_generator_py/msg/_empty_s.c:11:10: fatal error: numpy/ndarrayobject.h: No such file or directory
   11 | #include "numpy/ndarrayobject.h"
      |          ^~~~~~~~~~~~~~~~~~~~~~~
/home/ros/ros2_ws/build/rosidl_generator_py/rosidl_generator_py/rosidl_generator_py/msg/_constants_s.c:11:10: fatal error: numpy/ndarrayobject.h: No such file or directory
   11 | #include "numpy/ndarrayobject.h"
      |          ^~~~~~~~~~~~~~~~~~~~~~~
/home/ros/ros2_ws/build/rosidl_generator_py/rosidl_generator_py/rosidl_generator_py/msg/_nested_s.c:11:10: fatal error: numpy/ndarrayobject.h: No such file or directory
   11 | #include "numpy/ndarrayobject.h"
      |          ^~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
compilation terminated.
/home/ros/ros2_ws/build/rosidl_generator_py/rosidl_generator_py/rosidl_generator_py/msg/_primitives_s.c:11:10: fatal error: numpy/ndarrayobject.h: No such file or directory
   11 | #include "numpy/ndarrayobject.h"
      |          ^~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
/home/ros/ros2_ws/build/rosidl_generator_py/rosidl_generator_py/rosidl_generator_py/msg/_strings_s.c:11:10: fatal error: numpy/ndarrayobject.h: No such file or directory
   11 | #include "numpy/ndarrayobject.h"
      |          ^~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.

Additional information

The issue is in rosidl_generator_py/cmake/rosidl_generator_py_generate_interfaces.cmake

On Fedora, the numpy headers are installed to /usr/lib64/pythonx.y/site-packages/numpy/core/include/numpy/, and the python2-numpy package installs a symlink to the system include path: /usr/include/numpy -> /usr/lib64/python2.7/site-packages/numpy/core/include/numpy/. On systems without the python2 version of numpy, the include path can't be resolved, and the above errors occur.

It appears Debian is either installing or providing a symlink to numpy's include directory under /usr/include/pythonx.y, which is being added to the include path via the PythonExtra module in python_cmake_module, which may be why the problem doesn't occur with Debian/Ubuntu.

There is some CMake code in the referenced CMake module that runs the python interpreter to get the numpy include path on APPLE and WIN32 systems.

There are a few ways to solve this. In order of my preference:

• Use the existing numpy include path logic to discover the numpy include dir
  • Unconditionally
  • Conditionally if numpy/numpyconfig.h can't be found with find_file given the hint ${PythonExtra_INCLUDE_DIRS}
• Ask the Fedora numpy maintainers to install a symlink in /usr/include/pythonx.y
• Require that python2-numpy be installed on Fedora systems (the headers don't seem to be different between the python2 and python3 versions, but python2 is losing support very quickly)

The following diff works for conditional numpy include discovery. It prints the Using NUMPY_INCLUDE_DIR message with the above configuration, and doesn't print it when I symlink the numpy include dir to /usr/include/numpy or /usr/include/python3.7m/numpy

diff --git a/rosidl_generator_py/cmake/rosidl_generator_py_generate_interfaces.cmake b/rosidl_generator_py/cmake/rosidl_generator_py_generate_interfaces.cmake
index 4a28cbb..fa7f48f 100644
--- a/rosidl_generator_py/cmake/rosidl_generator_py_generate_interfaces.cmake
+++ b/rosidl_generator_py/cmake/rosidl_generator_py_generate_interfaces.cmake
@@ -190,7 +190,14 @@ target_include_directories(${_target_name_lib}
   ${CMAKE_CURRENT_BINARY_DIR}/rosidl_generator_py
   ${PythonExtra_INCLUDE_DIRS}
 )
-if(APPLE OR WIN32)
+
+# Check if numpy is in the include path
+find_file(_numpy_h numpy/numpyconfig.h
+  PATHS ${PythonExtra_INCLUDE_DIRS}
+  ${INCLUDE_INSTALL_DIR}
+)
+
+if(APPLE OR WIN32 OR NOT _numpy_h )
   # add include directory for numpy headers
   set(_python_code
     "import numpy"
@@ -208,6 +215,8 @@ if(APPLE OR WIN32)
       "error code ${_result}")
   endif()
   target_include_directories(${_target_name_lib} PUBLIC "${_output}")
+
+  message(STATUS "Using NUMPY_INCLUDE_DIR: ${_output}")
 endif()
 
 rosidl_target_interfaces(${_target_name_lib}

[calvertdw]

This is also the case on Arch with the package https://www.archlinux.org/packages/extra/x86_64/python-numpy/.

It places the headers in /usr/lib/python3.7/site-packages/numpy/core/include/.

I temporarily fixed this by creating the symbolic link /usr/include/python3.7m/numpy to the path above.

@richmattes Maybe you can update the title to reflect that this affects multiple distros?

[mjcarroll]

@richmattes I definitely agree with approach 1, do you think you could get that into a PR? I would be happy to review.

[richmattes]

Sure, but it won't be until this weekend. Do you prefer the patch posted, or using python to query the include path unconditionally?

…

On Tue, Jul 9, 2019, 3:40 PM Michael Carroll ***@***.***> wrote: @richmattes <https://github.com/richmattes> I definitely agree with approach 1, do you think you could get that into a PR? I would be happy to review. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#66?email_source=notifications&email_token=AACJWLDYDDZHSSWFOWEFIVDP6TSURA5CNFSM4H3XFPRKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZRJ4UQ#issuecomment-509779538>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AACJWLAG64RPE6FKGY5ZLBDP6TSURANCNFSM4H3XFPRA> .