cmake px4 + sitl package #9309

dagar · 2018-04-14T05:14:42Z

TODO

- add gazebo libs and models
- test ~~jmavsim and~~ gazebo running from both package and build dir
- Jenkins use package instead of rebuilding many times

lamping7

Shaves ~2 minutes off each test.

lamping7 · 2018-04-30T20:41:53Z

lamping7

This works with Gazebo + Jenkins. I added some comments to explain some of the details, but It looks good to go.

lamping7 · 2018-05-02T14:32:22Z

Jenkinsfile

+            unstash 'px4_sitl_package'
+            sh 'unzip build/posix_sitl_default/px4-posix_sitl_default*.zip'
+            sh 'mv px4-posix_sitl_default*/bin/px4 px4-posix_sitl_default*/share/px4/'
+            sh 'chmod +x px4-posix_sitl_default*/share/px4/px4'


Not entirely happy about having to do this -- it looks messy. Creating a procedure like the one for build nodes will clean it up.

lamping7 · 2018-05-02T14:33:43Z

Jenkinsfile

-            sh './Tools/ecl_ekf/process_logdata_ekf.py `find . -name *.ulg -print -quit`'
+            unstash 'px4_sitl_package'
+            sh 'unzip build/posix_sitl_default/px4-posix_sitl_default*.zip'
+            sh 'mv px4-posix_sitl_default*/bin/px4 px4-posix_sitl_default*/share/px4/'


This is a result of cpack using different bin and share dirs. It needs to be moved for ROS.

lamping7 · 2018-05-02T14:34:15Z

Tools/setup_gazebo.bash

    return 1
 fi

 SRC_DIR=$1
 BUILD_DIR=$2

 # setup Gazebo env and update package path
-export GAZEBO_PLUGIN_PATH=${BUILD_DIR}/build_gazebo:${GAZEBO_PLUGIN_PATH}
-export GAZEBO_MODEL_PATH=${GAZEBO_MODEL_PATH}:${SRC_DIR}/Tools/sitl_gazebo/models
-# Disabling the remote model download seems only necessary with Gazebo 6


Gazebo 6 isn't supported anymore.

lamping7 · 2018-05-02T14:35:11Z

Tools/setup_gazebo.bash

-export GAZEBO_MODEL_PATH=${GAZEBO_MODEL_PATH}:${SRC_DIR}/Tools/sitl_gazebo/models
-# Disabling the remote model download seems only necessary with Gazebo 6
-#export GAZEBO_MODEL_DATABASE_URI=""
-export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:${SRC_DIR}/Tools/sitl_gazebo/Build/msgs/:${BUILD_DIR}/build_gazebo


Build/msgs doesn't exist and msgs aren't needed to be in the path variable

lamping7 · 2018-05-03T11:40:29Z

@dagar So far we have everything we need for use by Jenkins. Do we want to consider use by users?

Do we want jmavsim in the package?
Do we want this to work more easily for local use by users? Possible required changes:
- Involves changing setup scripts
- Not treating px4 as a ROS package (i.e. not relying on a cmakelists and package file)
- px4 wouldn't launch from .launch, but rather some other script that does and then calls roslaunch for gazebo and mavros launching
- Above is only for Ubuntu + ROS. How about different OS support for the package?

dagar · 2018-05-04T05:46:50Z

platforms/posix/CMakeLists.txt

+	DIRECTORY
+		${PROJECT_SOURCE_DIR}/build/posix_sitl_default/build_gazebo
+	DESTINATION
+		${CMAKE_INSTALL_DATADIR}/${PROJECT_NAME}/build/posix_sitl_default


I need to think about how to do this independant of directory.

We could create a list of the files and use DIRECTORY with FILES_MATCHING to reduce the number of install commands. I originally did it this way to be clear about where everything is coming from while I was determining the pieces that we needed. I'm not attached to this implementation.

dagar · 2018-05-04T05:47:30Z

Unless someone feels strongly about it I'd like to skip jmavsim.
Eventually yes, but this is still a huge improvement.

dagar · 2018-05-04T05:49:52Z

If we do a tarball instead of zip it will preserve the permissions.

lamping7 · 2018-05-07T17:06:37Z

Permissions are good. I'm happier with this. Let me know if you want this all squashed and/or if this is OK.

FYI, the tar.bz2 is close to 90MB. The size of the extracted contents is ~270MB. The large size comes from the 50 models in sitl_gazebo that takes up ~240MB. Jenkins seems to be handling this fine, but this will eat space on master.

From the docs:
Note that the stash and unstash steps are designed for use with small files. For large data transfers, use the External Workspace Manager plugin, or use an external repository manager such as Nexus or Artifactory. This is because stashed files are archived in a compressed TAR, and with large files this demands considerable on-master resources, particularly CPU time. There's not a hard stash size limit, but between 5-100 MB you should probably consider alternatives.

dagar · 2018-05-23T05:52:06Z

Jenkinsfile

-            sh './Tools/ecl_ekf/process_logdata_ekf.py `find . -name *.ulg -print -quit`'
+            unstash 'px4_sitl_package'
+            sh 'tar -xjpf build/posix_sitl_default/px4-posix_sitl_default*.bz2'
+            sh 'mv px4-posix_sitl_default*/bin/px4 px4-posix_sitl_default*/share/px4/'


Why do we need the move?

For ROS to find px4 -- for use with the launch file -- px4 needs to be next to the package.xml and CMakeLists.txt. This makes it think it is a ROS package, allowing it to be used in the launch file.

Alternatively, we can launch everything differently with a different set of scripts, but for now, this resembles the current structure.

I just found it a bit weird that the package contents need to be massaged to work.

Agree. It is weird. To avoid this we can either 1) change cpack target destination for px4 -- don't use bin and share dirs or 2) don't launch px4 in roslaunch anymore -- script the launch instead

I would do whatever's easier and simpler at this point. Keep in mind CI is effectively the only consumer of the package at the moment, so it might as well be made to fit.
I don't have an opinion about dropping roslaunch yet. What's better long term?

At this point, where the pkg is only for CI, I'd rather change the CPack destination dirs. Option 2 is a more invasive change (new launch file and script, more testing, and so on). Long term, if users are to sudo dpkg -i px4.deb, this would be the more appropriate approach.

dagar · 2018-05-23T05:55:21Z

This looks great. I have a couple minor questions and things to verify about the paths, but I think we're nearly good to go.

What do you think about having a separate sitl_gazebo package in the future?

dagar · 2018-05-23T06:26:21Z

Can we skip the checkout per mission build?

lamping7 · 2018-05-23T12:32:43Z

What do you think about having a separate sitl_gazebo package in the future?

Yes. If we need it, we can build it as a separate component in cmake, but it would be nicer of we could simply pull the already-built lib files and models we need from a "released" version of sitl_gazebo (OSRF hosted or even a separate Jenkins job that would build and keep versions on S3) for use with Jenkins. I also think the idea here is to move in the direction where users don't have to build anything to run SITL sims.

Can we skip the checkout per mission build?

Yes. I think so. Let me check this out later today.

dagar · 2018-05-25T05:32:42Z

Nice improvement!

dagar requested a review from lamping7 April 14, 2018 05:15

dagar force-pushed the pr-cmake_package branch 3 times, most recently from e5f984f to 3f8a59b Compare April 14, 2018 06:04

lamping7 self-assigned this Apr 30, 2018

lamping7 force-pushed the pr-cmake_package branch 8 times, most recently from 6f1a44b to de255a3 Compare April 30, 2018 20:19

lamping7 previously approved these changes Apr 30, 2018

View reviewed changes

lamping7 dismissed their stale review via e57ad17 May 2, 2018 13:48

lamping7 force-pushed the pr-cmake_package branch 2 times, most recently from e57ad17 to c26ce65 Compare May 2, 2018 14:05

lamping7 previously approved these changes May 2, 2018

View reviewed changes

lamping7 dismissed their stale review via aadaa50 May 2, 2018 18:51

lamping7 force-pushed the pr-cmake_package branch from c26ce65 to aadaa50 Compare May 2, 2018 18:51

dagar commented May 4, 2018

View reviewed changes

dagar self-assigned this May 4, 2018

lamping7 force-pushed the pr-cmake_package branch 3 times, most recently from 3de5843 to 5e803c3 Compare May 7, 2018 16:39

dagar added this to the Release v1.8.0 milestone May 23, 2018

dagar commented May 23, 2018

View reviewed changes

dagar changed the title ~~[WIP] cmake px4 + sitl package~~ cmake px4 + sitl package May 23, 2018

dagar added the Admin: Enhancement (improvement) 💡 label May 23, 2018

lamping7 previously approved these changes May 24, 2018

View reviewed changes

lamping7 added 5 commits May 24, 2018 17:48

cmake: package px4 with sitl_gazebo plugins for jenkins ROS tests

e8b3408

tools: cleanup setup for gazebo env vars

57bf5cd

cmake: package create tarball

115e714

cmake: package don't use bin and share dirs

2aec846

jenkins: don't do checkout step for ROS tests with package

b792699

lamping7 dismissed their stale review via b792699 May 24, 2018 21:52

lamping7 force-pushed the pr-cmake_package branch from d28d86b to b792699 Compare May 24, 2018 21:52

lamping7 previously approved these changes May 24, 2018

View reviewed changes

dagar dismissed lamping7’s stale review via 1a815be May 25, 2018 00:53

dagar force-pushed the pr-cmake_package branch from 1a815be to fe501ea Compare May 25, 2018 04:29

Jenkins don't distclean without checkout

68aaac9

dagar force-pushed the pr-cmake_package branch from fe501ea to 68aaac9 Compare May 25, 2018 04:44

dagar merged commit a9275d6 into master May 25, 2018

dagar deleted the pr-cmake_package branch May 25, 2018 05:32

mrpollo mentioned this pull request Jun 5, 2018

v1.8 Changelog scratchpad #9593

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cmake px4 + sitl package #9309

cmake px4 + sitl package #9309

dagar commented Apr 14, 2018 •

edited by AuterionWrikeBot

Loading

lamping7 left a comment •

edited

Loading

lamping7 commented Apr 30, 2018

lamping7 left a comment

lamping7 May 2, 2018

lamping7 May 2, 2018

lamping7 May 2, 2018

lamping7 May 2, 2018 •

edited

Loading

lamping7 commented May 3, 2018

dagar May 4, 2018

lamping7 May 7, 2018

dagar commented May 4, 2018

dagar commented May 4, 2018

lamping7 commented May 7, 2018 •

edited

Loading

dagar May 23, 2018

lamping7 May 23, 2018

dagar May 23, 2018

lamping7 May 23, 2018

dagar May 23, 2018

lamping7 May 23, 2018

dagar commented May 23, 2018

dagar commented May 23, 2018

lamping7 commented May 23, 2018

dagar commented May 25, 2018

cmake px4 + sitl package #9309

cmake px4 + sitl package #9309

Conversation

dagar commented Apr 14, 2018 • edited by AuterionWrikeBot Loading

lamping7 left a comment • edited Loading

Choose a reason for hiding this comment

lamping7 commented Apr 30, 2018

lamping7 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lamping7 May 2, 2018 • edited Loading

Choose a reason for hiding this comment

lamping7 commented May 3, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dagar commented May 4, 2018

dagar commented May 4, 2018

lamping7 commented May 7, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dagar commented May 23, 2018

dagar commented May 23, 2018

lamping7 commented May 23, 2018

dagar commented May 25, 2018

dagar commented Apr 14, 2018 •

edited by AuterionWrikeBot

Loading

lamping7 left a comment •

edited

Loading

lamping7 May 2, 2018 •

edited

Loading

lamping7 commented May 7, 2018 •

edited

Loading