Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DEVOPS-135 Fix capnproto object/code mismatch in manylinux build #1030

Merged
merged 11 commits into from
Aug 12, 2016
Merged

DEVOPS-135 Fix capnproto object/code mismatch in manylinux build #1030

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
b78a074
DEVOPS-135 Remove capnproto from python extensions shared libs in ord…
Aug 3, 2016
d69de99
DEVOPS-135 Implement a hacky work-around by preloading pycapnp's exte…
Aug 5, 2016
91042e4
Use library utils to correctly combine multiple static libraries into…
Aug 9, 2016
3de24d3
issue-1034 Reorganized build to link nupic_core static library tests …
Aug 9, 2016
9138342
Removed nupic_core_solo from installed targets, since it's only an in…
Aug 9, 2016
6477a1f
Try building nupic_core_solo intermediate static library without spec…
Aug 9, 2016
f48a84d
No need for custom target in CREATE_CAPNPC_COMMAND function, since
Aug 10, 2016
89f3696
Represent external build of capnproto as single static library with t…
Aug 10, 2016
56e8045
Completed wrapping of external static libs in `add_library` targets
Aug 10, 2016
f800e70
Work around issue in cmake 2.8.7: have external project AprUtil1Stati…
Aug 11, 2016
fd33070
Clean up in response to Scott's request to make another pass over my …
Aug 11, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .travis.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,7 @@ install:
- "mkdir -p $TRAVIS_BUILD_DIR/build/scripts"
- "cd $TRAVIS_BUILD_DIR/build/scripts"
- "mkdir -p $TRAVIS_BUILD_DIR/build/release"
- "cmake --version"
# Enable test coverage for travis-ci build
- "cmake $TRAVIS_BUILD_DIR -DNTA_COV_ENABLED=ON -DCMAKE_INSTALL_PREFIX=${TRAVIS_BUILD_DIR}/build/release -DPY_EXTENSIONS_DIR=$TRAVIS_BUILD_DIR/bindings/py/nupic/bindings"
# Configure bash such that a failure in any link in the pipe causes the whole pipe to fail
Expand Down
19 changes: 6 additions & 13 deletions CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,9 +46,12 @@ endif()
#
# Global NuPIC CMake options
#
option(NUPIC_BUILD_PYEXT_MODULES "Turn on building of python extension modules for nupic.bindings"
option(NUPIC_BUILD_PYEXT_MODULES
"Turn on building of python extension modules for nupic.bindings; turn off to build only static nupic_core lib with full symbol visibility."
ON)

message(STATUS "NUPIC_BUILD_PYEXT_MODULES = ${NUPIC_BUILD_PYEXT_MODULES}")

#
# Identify build type - local or deployment (Travis)
# the variable NUPIC_DEPLOYMENT_BUILD must be set in travis CI scripts
Expand Down Expand Up @@ -77,18 +80,8 @@ set(EP_BASE ${CMAKE_BINARY_DIR}/ThirdParty)
# Determine common toolchain settings, compiler and link flags
include(CommonCompilerConfig)

# Set up external dependencies.
#
# This will try find_library and find_program and if that fails use
# ExternalProject to fetch and build the libraries.
#
# Sets the following variables:
# CAPNP_LINK_LIBRARIES
# EXTERNAL_INCLUDE_DIRS
#
# Defines the follow functions:
# create_capnpc_target - Creates target that generates C++ files from
# .capnp schema files.
# Set up builds of external dependencies and get their exports.
# (see individual external/*.cmake modules for exported settings and functions)
add_subdirectory(external)
include_directories(SYSTEM ${EXTERNAL_INCLUDE_DIRS})
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why remove this info? It seems useful to mention here

Copy link
Member Author

@vitaly-krugl vitaly-krugl Aug 10, 2016 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The list was substantially incomplete and duplicates information documented elsewhere. When making changes in the underlying cmake modules, it's not obvious that you need to come back here and update the duplicated info. I will add a comment that will help the reader discover what those settings are.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The problem is that things defined in one subdirectory (external) need to be used in another (src). That is a difficult relationship to figure out and it is enable in this file so I thought it would be nice to explicitly call out the items defined in external that we are intentionally exposed to src. And we don't have to include everything, just the things that we want src to know about. Perhaps this explanation could be converted into a better comment?


Expand Down
1 change: 1 addition & 0 deletions CommonCompilerConfig.cmake
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -316,6 +316,7 @@ IF(UNIX AND CMAKE_COMPILER_IS_GNUCXX AND (NOT "${CMAKE_BUILD_TYPE}" STREQUAL "De
list(APPEND EXTERNAL_STATICLIB_CMAKE_DEFINITIONS_OPTIMIZED
-DCMAKE_AR:PATH=gcc-ar
-DCMAKE_RANLIB:PATH=gcc-ranlib)
# And ditto for externals that use the configure-based build system
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure what ditto is referring to.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto makes sense if you expand the code and look at the preceding statement's comment in the code.

list(APPEND EXTERNAL_STATICLIB_CONFIGURE_DEFINITIONS_OPTIMIZED
AR=gcc-ar
RANLIB=gcc-ranlib)
Expand Down
1 change: 0 additions & 1 deletion appveyor.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -166,7 +166,6 @@ install:
-DPY_EXTENSIONS_DIR=%NUPIC_CORE%\bindings\py\nupic\bindings
%NUPIC_CORE%


build_script:
- cd %NUPIC_CORE%\build\scripts

Expand Down
45 changes: 32 additions & 13 deletions external/Apr1Lib.cmake
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,18 +22,28 @@
# Creates ExternalProject for building the apr-1 static library
# (apache public runtime)
#
# Exports:
# LIB_STATIC_APR1_INC_DIR: directory of installed apr-1 lib headers
# LIB_STATIC_APR1_LOC: path to installed static apr-1 lib
# OUTPUT VARIABLES:
#
# APR1_STATIC_LIB_TARGET: name of static library target that contains all
# of the apr-1 library objects.
# APR1_STATIC_LIB_INC_DIR: directory of installed apr-1 lib headers

include(../src/NupicLibraryUtils) # for MERGE_STATIC_LIBRARIES


# Output static library target for linking and dependencies
set(APR1_STATIC_LIB_TARGET apr-1-bundle)


set(aprlib_install_prefix "${EP_BASE}/Install/Apr1StaticLib")
set(aprlib_install_lib_dir "${aprlib_install_prefix}/lib")

# Export directory of installed apr-1 lib headers
set(LIB_STATIC_APR1_INC_DIR "${aprlib_install_prefix}/include")
set(APR1_STATIC_LIB_INC_DIR "${aprlib_install_prefix}/include")

# Export path to installed static apr-1 lib
set(LIB_STATIC_APR1_LOC "${aprlib_install_lib_dir}/${STATIC_PRE}apr-1${STATIC_SUF}")
# Path to static apr-1 lib installed by external project
set(aprlib_built_archive_file
"${aprlib_install_lib_dir}/${STATIC_PRE}apr-1${STATIC_SUF}")

# NOTE -DCOM_NO_WINDOWS_H fixes a bunch of OLE-related build errors on Win32
# (reference: https://bz.apache.org/bugzilla/show_bug.cgi?id=56342)
Expand Down Expand Up @@ -109,18 +119,21 @@ else()
ALWAYS 0
#LOG 1

# Move the installed ${LIB_STATIC_APR1_INC_DIR}/*.h to
# ${LIB_STATIC_APR1_INC_DIR}/apr-1
# Move the installed ${APR1_STATIC_LIB_INC_DIR}/*.h to
# ${APR1_STATIC_LIB_INC_DIR}/apr-1
COMMAND
${CMAKE_COMMAND} -DGLOBBING_EXPR=${LIB_STATIC_APR1_INC_DIR}/*.h
-DDEST_DIR_PATH=${LIB_STATIC_APR1_INC_DIR}/apr-1
${CMAKE_COMMAND} -DGLOBBING_EXPR=${APR1_STATIC_LIB_INC_DIR}/*.h
-DDEST_DIR_PATH=${APR1_STATIC_LIB_INC_DIR}/apr-1
-P ${CMAKE_SOURCE_DIR}/external/MoveFilesToNewDir.cmake
# Copy <SOURCE_DIR>/include/arch to ${LIB_STATIC_APR1_INC_DIR}/apr-1 as
# Copy <SOURCE_DIR>/include/arch to ${APR1_STATIC_LIB_INC_DIR}/apr-1 as
# expected by nupic.core
COMMAND
${CMAKE_COMMAND} -E make_directory ${LIB_STATIC_APR1_INC_DIR}/apr-1/arch
${CMAKE_COMMAND} -E make_directory
${APR1_STATIC_LIB_INC_DIR}/apr-1/arch
COMMAND
${CMAKE_COMMAND} -E copy_directory <SOURCE_DIR>/include/arch ${LIB_STATIC_APR1_INC_DIR}/apr-1/arch
${CMAKE_COMMAND} -E copy_directory
<SOURCE_DIR>/include/arch
${APR1_STATIC_LIB_INC_DIR}/apr-1/arch
)
endif()

Expand All @@ -143,3 +156,9 @@ ExternalProject_Add_Step(Apr1StaticLib patch_sources
COMMAND
patch -f -p1 --directory=<SOURCE_DIR> --input=${aprlib_patch_file}
)


# Wrap external project-generated static library in an `add_library` target.
merge_static_libraries(${APR1_STATIC_LIB_TARGET}
${aprlib_built_archive_file})
add_dependencies(${APR1_STATIC_LIB_TARGET} Apr1StaticLib)
44 changes: 30 additions & 14 deletions external/AprUtil1Lib.cmake
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,30 +22,40 @@
# Creates ExternalProject for building the aprutil-1 static library
# (apache public runtime utilities)
#
# Exports:
# LIB_STATIC_APRUTIL1_INC_DIR: directory of installed aprutil-1 lib headers
# LIB_STATIC_APRUTIL1_LOC: path to installed static aprutil-1 lib
# OUTPUT VARIABLES:
#
# APRUTIL1_STATIC_LIB_TARGET: name of static library target that contains all
# of the aprutil-1 library objects.
# APRUTIL1_STATIC_LIB_INC_DIR: directory of installed aprutil-1 lib headers

include(../src/NupicLibraryUtils) # for MERGE_STATIC_LIBRARIES


# Output static library target for linking and dependencies
set(APRUTIL1_STATIC_LIB_TARGET aprutil-1-bundle)


set(aprutillib_install_prefix "${EP_BASE}/Install/AprUtil1StaticLib")
set(aprutillib_install_lib_dir "${aprutillib_install_prefix}/lib")

# Export directory of installed aprutil-1 lib headers to parent
set(LIB_STATIC_APRUTIL1_INC_DIR "${aprutillib_install_prefix}/include")
set(APRUTIL1_STATIC_LIB_INC_DIR "${aprutillib_install_prefix}/include")

# Export path to installed static aprutil-1 lib to parent
set(LIB_STATIC_APRUTIL1_LOC "${aprutillib_install_lib_dir}/${STATIC_PRE}aprutil-1${STATIC_SUF}")
# Path to static aprutil-1 lib installed by external project
set(aprutillib_built_archive_file
"${aprutillib_install_lib_dir}/${STATIC_PRE}aprutil-1${STATIC_SUF}")

# NOTE -DCOM_NO_WINDOWS_H fixes a bunch of OLE-related build errors in apr-1
# on Win32 (reference: https://bz.apache.org/bugzilla/show_bug.cgi?id=56342)
set(aprutillib_cflags "-DCOM_NO_WINDOWS_H -DAPR_DECLARE_STATIC")
set(aprutillib_cflags "${aprutillib_cflags} -I${LIB_STATIC_APR1_INC_DIR}/apr-1")
set(aprutillib_cflags "${aprutillib_cflags} -I${APR1_STATIC_LIB_INC_DIR}/apr-1")
set(aprutillib_cflags "${EXTERNAL_C_FLAGS_OPTIMIZED} ${COMMON_COMPILER_DEFINITIONS_STR} ${aprutillib_cflags}")

set(aprutillib_url "${REPOSITORY_DIR}/external/common/share/apr-util/unix/apr-util-1.5.4.tar.gz")

if (UNIX)
set(aprutillib_config_options
--disable-util-dso --with-apr=${LIB_STATIC_APR1_INC_DIR}/..)
--disable-util-dso --with-apr=${APR1_STATIC_LIB_INC_DIR}/..)

ExternalProject_Add(AprUtil1StaticLib
DEPENDS Apr1StaticLib
Expand Down Expand Up @@ -91,8 +101,8 @@ else()
-DCMAKE_INSTALL_PREFIX=${aprutillib_install_prefix}
-DAPR_HAS_LDAP=OFF
-DAPU_HAVE_ODBC=OFF
-DAPR_INCLUDE_DIR=${LIB_STATIC_APR1_INC_DIR}/apr-1
-DAPR_LIBRARIES=${LIB_STATIC_APR1_LOC}
-DAPR_INCLUDE_DIR=${APR1_STATIC_LIB_INC_DIR}/apr-1
-DAPR_LIBRARIES=
-DINSTALL_PDB=OFF

#LOG_INSTALL 1
Expand All @@ -112,11 +122,11 @@ else()
ALWAYS 0
#LOG 1

# Move the installed ${LIB_STATIC_APRUTIL1_INC_DIR}/*.h to
# ${LIB_STATIC_APRUTIL1_INC_DIR}/apr-1
# Move the installed ${APRUTIL1_STATIC_LIB_INC_DIR}/*.h to
# ${APRUTIL1_STATIC_LIB_INC_DIR}/apr-1
COMMAND
${CMAKE_COMMAND} -DGLOBBING_EXPR=${LIB_STATIC_APRUTIL1_INC_DIR}/*.h
-DDEST_DIR_PATH=${LIB_STATIC_APRUTIL1_INC_DIR}/apr-1
${CMAKE_COMMAND} -DGLOBBING_EXPR=${APRUTIL1_STATIC_LIB_INC_DIR}/*.h
-DDEST_DIR_PATH=${APRUTIL1_STATIC_LIB_INC_DIR}/apr-1
-P ${CMAKE_SOURCE_DIR}/external/MoveFilesToNewDir.cmake

)
Expand All @@ -141,3 +151,9 @@ ExternalProject_Add_Step(AprUtil1StaticLib patch_sources
COMMAND
patch -f -p1 --directory=<SOURCE_DIR> --input=${aprutillib_patch_file}
)


# Wrap external project-generated static library in an `add_library` target.
merge_static_libraries(${APRUTIL1_STATIC_LIB_TARGET}
${aprutillib_built_archive_file})
add_dependencies(${APRUTIL1_STATIC_LIB_TARGET} AprUtil1StaticLib)
28 changes: 14 additions & 14 deletions external/CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,32 +45,32 @@ include(CapnProto)
include(Swig)

include(Apr1Lib)
set(LIB_STATIC_APR1_LOC ${LIB_STATIC_APR1_LOC} PARENT_SCOPE)
set(LIB_STATIC_APR1_INC_DIR ${LIB_STATIC_APR1_INC_DIR} PARENT_SCOPE)
set(APR1_STATIC_LIB_TARGET ${APR1_STATIC_LIB_TARGET} PARENT_SCOPE)
set(APR1_STATIC_LIB_INC_DIR ${APR1_STATIC_LIB_INC_DIR} PARENT_SCOPE)
list(APPEND EXTERNAL_INCLUDE_DIRS
${LIB_STATIC_APR1_INC_DIR}
"${LIB_STATIC_APR1_INC_DIR}/apr-1"
${APR1_STATIC_LIB_INC_DIR}
"${APR1_STATIC_LIB_INC_DIR}/apr-1"
)

# AprUtil1Lib MUST be imported after Apr1Lib due to depencies on exported
# properties
include(AprUtil1Lib)
set(LIB_STATIC_APRUTIL1_LOC ${LIB_STATIC_APRUTIL1_LOC} PARENT_SCOPE)
set(LIB_STATIC_APRUTIL1_INC_DIR ${LIB_STATIC_APRUTIL1_INC_DIR} PARENT_SCOPE)
list(APPEND EXTERNAL_INCLUDE_DIRS ${LIB_STATIC_APRUTIL1_INC_DIR})
set(APRUTIL1_STATIC_LIB_TARGET ${APRUTIL1_STATIC_LIB_TARGET} PARENT_SCOPE)
set(APRUTIL1_STATIC_LIB_INC_DIR ${APRUTIL1_STATIC_LIB_INC_DIR} PARENT_SCOPE)
list(APPEND EXTERNAL_INCLUDE_DIRS ${APRUTIL1_STATIC_LIB_INC_DIR})

include(YamlLib)
set(LIB_STATIC_YAML_LOC ${LIB_STATIC_YAML_LOC} PARENT_SCOPE)
set(YAML_STATIC_LIB_TARGET ${YAML_STATIC_LIB_TARGET} PARENT_SCOPE)

include(YamlCppLib)
set(LIB_STATIC_YAML_CPP_LOC ${LIB_STATIC_YAML_CPP_LOC} PARENT_SCOPE)
set(LIB_STATIC_YAML_CPP_INC_DIR ${LIB_STATIC_YAML_CPP_INC_DIR} PARENT_SCOPE)
list(APPEND EXTERNAL_INCLUDE_DIRS ${LIB_STATIC_YAML_CPP_INC_DIR})
set(YAML_CPP_STATIC_LIB_TARGET ${YAML_CPP_STATIC_LIB_TARGET} PARENT_SCOPE)
set(YAML_CPP_STATIC_LIB_INC_DIR ${YAML_CPP_STATIC_LIB_INC_DIR} PARENT_SCOPE)
list(APPEND EXTERNAL_INCLUDE_DIRS ${YAML_CPP_STATIC_LIB_INC_DIR})

include(Zlib)
set(LIB_STATIC_Z_LOC ${LIB_STATIC_Z_LOC} PARENT_SCOPE)
set(LIB_STATIC_Z_INC_DIR ${LIB_STATIC_Z_INC_DIR} PARENT_SCOPE)
list(APPEND EXTERNAL_INCLUDE_DIRS ${LIB_STATIC_Z_INC_DIR})
set(Z_STATIC_LIB_TARGET ${Z_STATIC_LIB_TARGET} PARENT_SCOPE)
set(Z_STATIC_LIB_INC_DIR ${Z_STATIC_LIB_INC_DIR} PARENT_SCOPE)
list(APPEND EXTERNAL_INCLUDE_DIRS ${Z_STATIC_LIB_INC_DIR})

# Add vendored prebuilt library include paths.
list(APPEND EXTERNAL_INCLUDE_DIRS
Expand Down
Loading