Local repo with include files and libs

[cypof]

C.f. details on javacpp PR

[saudet]

Looks interesting, thanks! Would we have to change all the presets in a similar fashion or can it fall back automatically?

BTW, I'm not sure I understand what problem we are trying to solve here. If we're looking at rebuilding the binaries, wouldn't we need to build the native libraries from scratch anyway?

[cypof]

We can change the other presets over time, it's opt-in. The thing it brings is that you only need to rebuild you own library, not the dependencies. E.g. for Caffe, you avoid opencv, and if we split the other dependencies in separate jar(s?) we can remove them too.

Medium term, I imagine projects built on top on opencv or Caffe which would have small pieces of native code, e.g. cuda kernels, that now can be built without any setup, possibly at runtime!

[saudet]

Yes, I see, it's interesting, but I feel we need a better use case before prioritizing this. NVIDIA won't let us bundle CUDA's files anyway, OpenCL doesn't need any extra files to compile kernels, OpenCV compiles much faster than Caffe itself, and it's other depenecies are not useful to any other presets, at the moment... But I get your point... Any other ideas?

[cypof]

Yes, I can think of several points. First, for us the use case is obvious because we are building a library on top of OpenCV and Caffe. It contains both C++ and JVM languages, so having all dependencies in Maven with the include jar means we can work on our lib without installing or building anything.

More generally, it's not only about simplifying distribution. I think the main thing is to have a single version of everything on your box, it's a lot less error prone. Otherwise when you prototype things, you have one version in Maven, one in cppbuild. And if the setup does not work out of the box, people try to fix it. So they will yum install a missing lib, or for us use yahoo packages, and now their library references half Maven, half something else and it turns into a mess.

I understand your concerns about adding some complexity. So a few points on that, first I would be happy to help on this, and iterate until you are comfortable with it. That's a big point for our current project. Also I believe it will actually simplify JavaCPP, because the preset libraries are now
self-contained. You can remove the links between pom files, and people can work on their own libs purely using Maven references, instead of hardcoding paths like ../javacpp-presets/...

I suspect our use case will become more and more common, as people realize they can easily use C++ inside their java app. E.g. Caffe offers a set of features, but you often end up needing to add things, or customize, which usually require some C++. And then you are back to downloading javacpp from github, running cppbuild, but something fails at some point and you have to dig in etc. At this point you have already lost 90% of your potential users.

[saudet]

Yes you're right, when developing code in both C++ and Java, we need to have the files required to compile the C++ code as well. Very good. 👍 Now, we should make this portable. One thing we need to keep in mind is that on Windows, we usually have two sets of file, one set to link (.lib, .a, or .dll.a) in the "lib" directory, and the DLLs themselves, usually placed in the "bin" directory. We need to support that as well...

[cypof]

It will definitely require some work, but I think it is worth it. Within a couple weeks we should have something good enough, and it's only cleanup from there as the underlying runtimes almost never change.

For Linux, I ran into issues with so file versioning. Apparently even if a so.x version is renamed to .so, it still contains it's version. So if you link an executable against it, and try to load it, it complains it doesn't find .so.x. Also Caffe requires protoc, which links to libprotoc.so.9. That makes me think that the right thing would be to package the versions symlinks too. I tested this, maven can package symlinks if they are part of the target folder so should be fine. I can test more and update the PR.

[saudet]

A couple of weeks full time maybe, but I don't have that kind of time :)
But there is definitely demand out there:
https://blogs.oracle.com/geertjan/entry/mixed_java_and_c_c
I really hope that we'll be able to convince the right people at some point...

Yes under Linux and Mac OS X we would need something like that. The maven-jar-plugin supports symlinks, really? Yes please update your code, and let's see what other "little details" pop up. As long as the current functionality on all platforms isn't broken, getting this working just on Linux is a good start, so please do not give up! :) Thanks

[saudet]

I've finally started to work on that! If we set the copyResources parameter at

javacpp-presets/pom.xml

Line 250 in 37ddac1

<!-- <copyResources>true</copyResources>-->

there are 2 phases to what happens next:

1. After compiling the JNI library, the associated resources (for example, "include", "lib", "sdk", and "share" in the case of opencv_core) are located in the specified resourcePath and copied into the package of that class.
2. When building some other presets that depends on OpenCV, such as Caffe, these resources are extracted and cached, as specified in the includeResource and linkResource parameters. The paths are passed as if they were additional includePath and linkPath.

Notes:

• The resources are not currently packaged into another JAR file. That's something you could adapt from your PR and include in the parent pom.xml file, but it's not clear how to generalize that to all types of resources, as per the next point below.
• The reason it's copying "lib" is because there are resources other than symbolic links that dependent libraries need (such as pkg-config files, CMake config files, dummy .lib files for Visual Studio, etc). If we eventually start distributing artifacts for those that way, we'll need to think a bit more about what to do for this.
• The resources are not currently available to the cppbuild.sh scripts. We could include an additional buildResource parameter or something, and pass them to the script via environment variables, but we need to know which path corresponds to OpenCV, which other corresponds to OpenBLAS, etc as required by the native build tools. Maybe something like OPENCV_CORE_RESOURCE, OPENBLAS_RESOURCE, etc containing the extracted paths?

[saudet]

The second part of this is in as commit 778b586 where the specified buildResource are extracted and passed with the BUILD_PATH environment variable to the build script!

As mentioned above, since we have other types of resources than just header files, it currently just bundles everything in the same JAR file when the copyResources parameter (or the javacpp.copyResources property) is set to true. If you have better ideas about how to package this, please open another PR! Thanks a lot for your time on this.