BuildProjects to automatically compile Modelica binaries

[modelica-trac-importer]

Modified by dietmarw on 27 Feb 2015 12:01 UTC
Since some time we have had a directory Modelica/Resources/BuildProjects to make it easier to compile the binary libraries used by the package. This makes maintenance easier since you do not need to compile and keep svn/git updated with new versions of the library all the time (since the user/tool provider can easily compile the source code).

I propose that this is specified and made automatic by Modelica tools (that support external C). I even made a prototype implementation in OpenModelica.

The following uses the library name ModelicaExternalC2 instead of ModelicaExternalC because OpenModelica itself has ModelicaExternalC pre-compiled:

model M
function f
   input Real time;
   output Real r;
external "C" annotation(Library="ModelicaExternalC2");
end f;

   Real x = f(time);
end M;

The prototype works by looking in the usual library directories for a file libmylib.a or libmylib.so (as well as OPENMODELICAHOME/lib/omc OPENMODELICAHOME/lib and ~/.openmodelica/binaries/PACKAGENAME). If no library is found, it tries to look for modelica://M/Resources/BuildProjects/*/autogen.sh. It then tries to run that project using the command below.

The generated library for ModelicaTables is linked against hdf5 if the system has hdf5 installed. If it is not installed, it asks the OS if the hdf5 is available in the package manager, and if it is it asks for admin rights to install it.

[[Image(hdf5-install.png, 100%)]]

[1] 13:32:08 Translation Notification
[/home/marsj/tmp/a.mo: 2:1-6:6]: Removed directory /tmp/omc_compile_ModelicaExternalC2_7UJy2p

[2] 13:32:08 Translation Notification
[/home/marsj/tmp/a.mo: 2:1-6:6]: Compiled /home/marsj/.openmodelica/binaries/M/libModelicaExternalC2.so by running build project /home/marsj/tmp/Resources/BuildProjects/autotools

[3] 13:32:08 Translation Notification
[/home/marsj/tmp/a.mo: 2:1-6:6]: Succeeded with compilation and installation of the library using: command: sh ./autogen.sh && ./configure --libdir='/home/marsj/.openmodelica/binaries/M' && make && make install
[...]
Reading package lists... Building dependency tree... Reading state information... Suggested packages: libhdf5-doc The following NEW packages will be installed: libhdf5-dev 0 upgraded, 1 newly installed, 0 to remove and 2 not upgraded. Need to get 0 B/4616 kB of archives. After this operation, 41.8 MB of additional disk space will be used. Selecting previously unselected package libhdf5-dev. (Reading database ... 755658 files and directories currently installed.) Preparing to unpack .../libhdf5-dev_1.8.12+docs-1.1ubuntu1_amd64.deb ... Unpacking libhdf5-dev (1.8.12+docs-1.1ubuntu1) ... Setting up libhdf5-dev (1.8.12+docs-1.1ubuntu1) ...
[...]
Libraries have been installed in: /home/marsj/.openmodelica/binaries/M
[...]

[4] 13:32:08 Translation Notification
[/home/marsj/tmp/a.mo: 2:1-6:6]: Changed directory to /tmp/omc_compile_ModelicaExternalC2_7UJy2p/BuildProjects/autotools

[5] 13:32:08 Translation Notification
[/home/marsj/tmp/a.mo: 2:1-6:6]: cp -a "/home/marsj/tmp/Resources"/* "/tmp/omc_compile_ModelicaExternalC2_7UJy2p"

[6] 13:32:08 Translation Notification
[/home/marsj/tmp/a.mo: 2:1-6:6]: Created directory /tmp/omc_compile_ModelicaExternalC2_7UJy2p

---

Reported by sjoelund.se on 26 Feb 2015 12:41 UTC
Since some time we have had a directory Modelica/Resources/BuildProjects to make it easier to compile the binary libraries used by the package. This makes maintenance easier since you do not need to compile and keep svn/git updated with new versions of the library all the time (since the user/tool provider can easily compile the source code).

I propose that this is specified and made automatic by Modelica tools (that support external C). I even made a prototype implementation in OpenModelica.

The following uses the library name ModelicaExternalC2 instead of ModelicaExternalC because OpenModelica itself has ModelicaExternalC pre-compiled:

model M
function f
   input Real time;
   output Real r;
external "C" annotation(Library="ModelicaExternalC2");
end f;

   Real x = f(time);
end M;

The prototype works by looking in the usual library directories for a file libmylib.a or libmylib.so (as well as OPENMODELICAHOME/lib/omc OPENMODELICAHOME/lib and ~/.openmodelica/binaries/PACKAGENAME). If no library is found, it tries to look for modelica://M/Resources/BuildProjects/*/autogen.sh. It then tries to run that project using the command below.

The generated library for ModelicaTables is linked against hdf5 if the system has hdf5 installed. If it is not installed, it asks the OS if the hdf5 is available in the package manager, and if it is it asks for admin rights to install it.

[1] 13:32:08 Translation Notification
[/home/marsj/tmp/a.mo: 2:1-6:6]: Removed directory /tmp/omc_compile_ModelicaExternalC2_7UJy2p

[2] 13:32:08 Translation Notification
[/home/marsj/tmp/a.mo: 2:1-6:6]: Compiled /home/marsj/.openmodelica/binaries/M/libModelicaExternalC2.so by running build project /home/marsj/tmp/Resources/BuildProjects/autotools

[3] 13:32:08 Translation Notification
[/home/marsj/tmp/a.mo: 2:1-6:6]: Succeeded with compilation and installation of the library using: command: sh ./autogen.sh && ./configure --libdir='/home/marsj/.openmodelica/binaries/M' && make && make install
[...]
Reading package lists... Building dependency tree... Reading state information... Suggested packages: libhdf5-doc The following NEW packages will be installed: libhdf5-dev 0 upgraded, 1 newly installed, 0 to remove and 2 not upgraded. Need to get 0 B/4616 kB of archives. After this operation, 41.8 MB of additional disk space will be used. Selecting previously unselected package libhdf5-dev. (Reading database ... 755658 files and directories currently installed.) Preparing to unpack .../libhdf5-dev_1.8.12+docs-1.1ubuntu1_amd64.deb ... Unpacking libhdf5-dev (1.8.12+docs-1.1ubuntu1) ... Setting up libhdf5-dev (1.8.12+docs-1.1ubuntu1) ...
[...]
Libraries have been installed in: /home/marsj/.openmodelica/binaries/M
[...]

[4] 13:32:08 Translation Notification
[/home/marsj/tmp/a.mo: 2:1-6:6]: Changed directory to /tmp/omc_compile_ModelicaExternalC2_7UJy2p/BuildProjects/autotools

[5] 13:32:08 Translation Notification
[/home/marsj/tmp/a.mo: 2:1-6:6]: cp -a "/home/marsj/tmp/Resources"/* "/tmp/omc_compile_ModelicaExternalC2_7UJy2p"

[6] 13:32:08 Translation Notification
[/home/marsj/tmp/a.mo: 2:1-6:6]: Created directory /tmp/omc_compile_ModelicaExternalC2_7UJy2p

---

Migrated-From: https://trac.modelica.org/Modelica/ticket/1668

[modelica-trac-importer]

Modified by dietmarw on 27 Feb 2015 12:01 UTC

[modelica-trac-importer]

Comment by sjoelund.se on 27 Feb 2015 20:29 UTC
Talking to @Thiele, we came to the conclusion that it would be good if impact could provide links to zip-files containing the binaries by platform.

1. This makes the download size smaller (as you only download the binaries you need)
2. This works as long as the index generator knows how to automatically compile binaries for the most common platforms, or if someone provides these binaries using github releases (possibly automated by a 3rd-party build service)

This solves the issues that some Modelica tools only partially support the Modelica language (you can include C code in the model, so a C compiler can be assumed to exist).

[modelica-trac-importer]

Comment by sjoelund.se on 27 Feb 2015 20:35 UTC
Some issues exist for how to specify Library="..." annotations. It would be good if the Modelica tool could know which libraries are provided by the Modelica library, and which ones are system libraries. A proposal would be something like:

Library={"system:dbus-1"} // For things that require dbus, regardless of platform
Library={"win32|win64:shlwapi","linux:dl"} // Make the thing before the : a regular expression matching the platform name x86_64-linux or linux64 for example

This allows you to specify libraries that are different on each platform. Currently, the spec says you are recommended to link all dependencies. But these are often platform-specific and best practice is to only depend on a single statically linked library instead.

[modelica-trac-importer]

Comment by dietmarw on 15 Apr 2015 14:33 UTC
Ticket retargeted after milestone closed

[modelica-trac-importer]

Comment by beutlich on 18 Jan 2017 21:26 UTC
See also #2104 (or close it as duplicate).

[modelica-trac-importer]

Comment by hansolsson on 25 Oct 2017 07:41 UTC
And conclusions in #2145

[HansOlsson]

There is use case for specifying the source code (directories) that we had not considered earlier.

The case is for generating FMUs containing source code, for models that use external functions. It might be that this should be part of a re-opened #2145 - separate from this ticket. However, they need to be considered together. In this case we need the source code so that we can copy it to the FMU.

One proposal to solve this is to have an annotation SourceDirectory (similary as LibraryDirectory and IncludeDirectory) possibly with default "modelica://LibraryName/Resources/Source". That can then be used when building a source code FMU by first copying the contents of that that source-directory to the FMU.

It seems best to also copy the build-project to the source code FMU (I assume the build-projects are general enough that they are useful also for the target-system of the FMU), which requires one of the following:

• SourceDirectory should for MSL specify both C-sources and build-projects (possibly the annotation should have a better name). Note that similarly as the other annotations in 12.9.4 it should be possible to specify an array of directories for this.
• The build-projects directories should be placed in source-directory
• Another annotation for specifying build-projects

[HansOlsson]

Design meeting:

Seems that straightforward to standardize on CMake - and have some CMake-variable for ModelicaUtilities.h from tools.

[HansOlsson]

Shall we push this?

1. Provide CMakeList.txt for MSL - even without anything in the specification, with a readme (create issue for it?)
2. Tools can then support cmake
3. If that happens we can standardize the existing behavior in tools in the specification

(And the cmake-variable for ModelicaUtilities.h)

[casella]

@beutlich just completed modelica/ModelicaStandardLibrary#4093, now a proof of concept is available in the MSL master branch.