Fix GDAL and bump version of DGGRID7 #6740

danlooo · 2023-05-11T12:25:28Z

This PR aims to fix a version mismatch of DGGRID7_jll and GDAL_jll by updating DGGRID7_jll to the latest official release 7.7 of DGGRID that works with BinaryBuilder. @allixender what do you think?

Error

Executing in docker image julia:1.8.5-bullseye :

using Pkg
Pkg.add("DGGRID7_jll")

using DGGRID7_jll

download("https://raw.githubusercontent.com/sahrk/DGGRID/4f24e9de46f9e669af4cba94d9d148b557a1de11/examples/icosahedron/icosahedron.meta", "meta.txt")
mkdir("outputfiles")

DGGRID7_jll.dggrid() do dggrid_path
    run(`$dggrid_path meta.txt`)
end

results in error

/root/.julia/artifacts/3381b9f883af7e984906191545c4fa939e07e91b/bin/dggrid: error while loading shared libraries: libgdal.so.28: cannot open shared object file: No such file or directory

Indeed, dggrid requires libgdal.so.28, but package GDAL_jll provides libgdal.so.32.3.6.2:

root@6e8b0f893490:~/.julia/artifacts# find ~/.julia/artifacts/6f97fbb4bedf23ccd1d7757586cb7659222a0465/lib
/root/.julia/artifacts/6f97fbb4bedf23ccd1d7757586cb7659222a0465/lib
/root/.julia/artifacts/6f97fbb4bedf23ccd1d7757586cb7659222a0465/lib/pkgconfig
/root/.julia/artifacts/6f97fbb4bedf23ccd1d7757586cb7659222a0465/lib/pkgconfig/gdal.pc
/root/.julia/artifacts/6f97fbb4bedf23ccd1d7757586cb7659222a0465/lib/libgdal.so.32
/root/.julia/artifacts/6f97fbb4bedf23ccd1d7757586cb7659222a0465/lib/gdalplugins
/root/.julia/artifacts/6f97fbb4bedf23ccd1d7757586cb7659222a0465/lib/gdalplugins/drivers.ini
/root/.julia/artifacts/6f97fbb4bedf23ccd1d7757586cb7659222a0465/lib/libgdal.so
/root/.julia/artifacts/6f97fbb4bedf23ccd1d7757586cb7659222a0465/lib/cmake
/root/.julia/artifacts/6f97fbb4bedf23ccd1d7757586cb7659222a0465/lib/cmake/gdal
/root/.julia/artifacts/6f97fbb4bedf23ccd1d7757586cb7659222a0465/lib/cmake/gdal/GDALConfig.cmake
/root/.julia/artifacts/6f97fbb4bedf23ccd1d7757586cb7659222a0465/lib/cmake/gdal/GDALConfigVersion.cmake
/root/.julia/artifacts/6f97fbb4bedf23ccd1d7757586cb7659222a0465/lib/cmake/gdal/GDAL-targets.cmake
/root/.julia/artifacts/6f97fbb4bedf23ccd1d7757586cb7659222a0465/lib/cmake/gdal/GDAL-targets-release.cmake
/root/.julia/artifacts/6f97fbb4bedf23ccd1d7757586cb7659222a0465/lib/libgdal.so.32.3.6.2

Fix

This PR uses the latest official release 7.7 of DGGRID that works with BinaryBuilder.

Version 7.8 of DGGRID results in error /workspace/srcdir/DGGRID/src/lib/dglib/lib/DgZOrderRF.cpp:96:24: error: expected ‘)’ before ‘PRIx64‘

Test

julia build_tarballs.jl successed to run on a local Linux machine.
Furthermore, products/DGGRID7.v0.9.0.x86_64-linux-gnu-cxx11.tar.gz was tested by integration test doexamples.sh. Exit code was 0, but output files still contain these errors:

$ find ~/repos/Yggdrasil/D/DGGRID7/DGGRID/examples -type f | grep outputfiles | xargs -i grep -H ERROR {}

Yggdrasil/D/DGGRID7/DGGRID/examples/planetRiskGridGen/outputfiles/planetRiskGridGen.txt:FATAL ERROR: Invalid parameter data in parameter:
Yggdrasil/D/DGGRID7/DGGRID/examples/planetRiskClipHiRes/outputfiles/planetRiskClipHiRes.txt:FATAL ERROR: Invalid parameter data in parameter:
Yggdrasil/D/DGGRID7/DGGRID/examples/gdalCollection/outputfiles/gdalCollection.txt:FATAL ERROR: DgParamList::setParam() unknown parameter collection_output_gdal_format
Yggdrasil/D/DGGRID7/DGGRID/examples/gdalExample/outputfiles/gdalExample.txt:FATAL ERROR: Invalid parameter data in parameter:
Yggdrasil/D/DGGRID7/DGGRID/examples/holes/outputfiles/holes.txt:FATAL ERROR: Invalid parameter data in parameter:
Yggdrasil/D/DGGRID7/DGGRID/examples/planetRiskClipMulti/outputfiles/planetRiskClipMulti.txt:FATAL ERROR: Invalid parameter data in parameter:
Yggdrasil/D/DGGRID7/DGGRID/examples/planetRiskGridNoWrap/outputfiles/planetRiskGridNoWrap.txt:FATAL ERROR: Invalid parameter data in parameter:

giordano · 2023-05-11T15:36:56Z

D/DGGRID7/build_tarballs.jl

+cd $WORKSPACE/srcdir/DGGRID
+mkdir build
+cd build
+cmake -DCMAKE_BUILD_TYPE=Release ..


https://docs.binarybuilder.org/stable/build_tips/#CMake-builds

D/DGGRID7/build_tarballs.jl

giordano · 2023-05-11T15:37:57Z

D/DGGRID7/build_tarballs.jl

@@ -37,4 +41,4 @@ dependencies = [
 ]

 # Build the tarballs, and possibly a `build.jl` as well.
-build_tarballs(ARGS, name, version, sources, script, platforms, products, dependencies; preferred_gcc_version = v"8.1.0")
+build_tarballs(ARGS, name, version, sources, script, platforms, products, dependencies; preferred_gcc_version=v"8.1.0", julia_compat="v1.6.0")


To change the julia_compat you must change the version number: #2763

This has still to be addressed 🙂

julia_compat="v1.6.0" fixed in 9c5c12d

The version number of the packag, as explained in the non-negotiable requirements in issue #2763 linked above

ok. See f9bf39b

giordano · 2023-05-11T15:39:19Z

results in error

Sounds like you have to specify the compat bound with GDAL.

allixender · 2023-05-12T05:10:37Z

Hi, thanks for the preps, much appreciated. Updating DGGRID would be great. They changed the build to cmake, and a GDAL bump to at least 3.3+ would also be great. The errors in the examples might point to linking to an outdated GDAL version. How can I help with something?

Co-authored-by: Mosè Giordano <[email protected]>

danlooo · 2023-05-12T08:59:19Z

Thanks for the comments! I just set GDAL version to v301.600.20, because file libgdal.so.32 is present in the package

D/DGGRID7/build_tarballs.jl

giordano · 2023-05-12T10:04:31Z

D/DGGRID7/build_tarballs.jl

 if [[ "${target}" == x86_64-linux-musl ]]; then
    # Remove libexpat to avoid it being picked up by mistake
    rm /usr/lib/libexpat.so*
 fi
 make -j${nproc} CCOMP="${CC}" CPPCOMP="${CXX}"
-cp "apps/dggrid/dggrid${exeext}" "${bindir}/."
+install -Dvm 755 "src/apps/dggrid/dggrid${exeext}" "${bindir}/dggrid${exeext}"


I presume

Suggested change

install -Dvm 755 "src/apps/dggrid/dggrid${exeext}" "${bindir}/dggrid${exeext}"

make install

would it now that there's a proper build system? I haven't tried it though

I think there is no make install.

Yes:

/usr/bin/cmake -E cmake_progress_start /workspace/srcdir/DGGRID/build/CMakeFiles 0 make: *** No rule to make target 'install'. Stop.

D/DGGRID7/build_tarballs.jl

giordano · 2023-05-12T10:04:58Z

D/DGGRID7/build_tarballs.jl

@@ -37,4 +41,4 @@ dependencies = [
 ]

 # Build the tarballs, and possibly a `build.jl` as well.
-build_tarballs(ARGS, name, version, sources, script, platforms, products, dependencies; preferred_gcc_version = v"8.1.0")
+build_tarballs(ARGS, name, version, sources, script, platforms, products, dependencies; preferred_gcc_version=v"8.1.0", julia_compat="v1.6.0")


This has still to be addressed 🙂

Co-authored-by: Mosè Giordano <[email protected]>

allixender · 2023-05-12T11:50:52Z

Hi @danlooo , Kevin Sahr has updated the latest DGGRID in his source tree a lot and it is great to now use his GtiHub. I saw your updates. Maybe try already version 7.8 (https://github.com/sahrk/DGGRID/releases/tag/v7.8), I am in contact with him occasionally, I am also trying to submit build a conda package and provide the occasional pull request to the DGGRID repo to improve these types of making it build on different distribution systems. Great to see your activity.

If it helps:

https://github.com/allixender/dggrid4py is a Python wrapper, that creates the metafile and runs DGGRID as a separate process, it looks like DGGS.jl wants to go this way
https://github.com/am2222/pydggrid was initial prototype to bind Python directly against DGGRID C++ functions via pybind11, but didn't succeed/wasn't completed
https://github.com/r-barnes/dggridR/ use the R->C++ code to link directly against DGGRID C++ functions and calls the grid functions directly from R to the "dglib", but contains a complete restructured fork of DGGRIDv6 or early v7, and is standalone

I am working with Kevin to make the DGGRID native original version to be usable via C++ bindings eventually, that might also be an effort that could be valuable for DGGS.jl.

danlooo · 2023-05-12T12:50:31Z

Unfortunately, DGGRID v7.8 does not compile:

/workspace/srcdir/DGGRID/src/lib/dglib/lib/DgZOrderRF.cpp:96:24: error: expected ‘)’ before ‘PRIx64’
    if (!sscanf(tok, "%" PRIx64, &val))
               ~        ^~~~~~~

See dggrid_v7.8.log and this branch for details. I encountered similar erros while working on julia bindings in project ddgrid-julia.
However, resolving this looks like a new issue.

allixender · 2023-05-12T12:53:37Z

Unfortunately, DGGRID v7.8 does not compile:
/workspace/srcdir/DGGRID/src/lib/dglib/lib/DgZOrderRF.cpp:96:24: error: expected ‘)’ before ‘PRIx64’
    if (!sscanf(tok, "%" PRIx64, &val))
               ~        ^~~~~~~
See dggrid_v7.8.log and this branch for details. I encountered similar erros while working on julia bindings in project ddgrid-julia. However, resolving this looks like a new issue.

Whoa, you are already on the way, amazing. I'll see if I can reproduce this and I will send something to Kevin.

Fix GDAL and bump version of DGGRID7

5e5d5da

danlooo force-pushed the master branch from dbc1ca0 to 5e5d5da Compare May 11, 2023 12:39

giordano reviewed May 11, 2023

View reviewed changes

D/DGGRID7/build_tarballs.jl Outdated Show resolved Hide resolved

giordano reviewed May 11, 2023

View reviewed changes

danlooo and others added 2 commits May 12, 2023 10:40

Update install in build_tarballs.jl

f757006

Co-authored-by: Mosè Giordano <[email protected]>

Fix GDAL version

9c5c12d

giordano reviewed May 12, 2023

View reviewed changes

danlooo and others added 3 commits May 12, 2023 13:26

Update make in D/DGGRID7/build_tarballs.jl

587f47d

Co-authored-by: Mosè Giordano <[email protected]>

Fix DGGRID7 version see JuliaPackaging#2763

f9bf39b

Update D/DGGRID7/build_tarballs.jl

d29f8f9

Co-authored-by: Mosè Giordano <[email protected]>

giordano merged commit 3631a62 into JuliaPackaging:master May 12, 2023

This was referenced Jun 7, 2023

JLL dggrid can not find GDAL danlooo/DGGS.jl#1

Closed

Do we need to ship DGGS.js with DGGRID? danlooo/DGGS.jl#2

Closed

This was referenced Nov 27, 2023

Fix gdal iversion in DGGRID7 #7720

Closed

Fix version of GDAL in DGGRID7 #7721

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix GDAL and bump version of DGGRID7 #6740

Fix GDAL and bump version of DGGRID7 #6740

danlooo commented May 11, 2023

giordano May 11, 2023

giordano May 11, 2023

giordano May 12, 2023

danlooo May 12, 2023

giordano May 12, 2023

danlooo May 12, 2023

giordano commented May 11, 2023

allixender commented May 12, 2023

danlooo commented May 12, 2023

giordano May 12, 2023

allixender May 12, 2023

danlooo May 12, 2023

giordano May 12, 2023

allixender commented May 12, 2023 •

edited

Loading

danlooo commented May 12, 2023

allixender commented May 12, 2023

	install -Dvm 755 "src/apps/dggrid/dggrid${exeext}" "${bindir}/dggrid${exeext}"
	make install

Fix GDAL and bump version of DGGRID7 #6740

Fix GDAL and bump version of DGGRID7 #6740

Conversation

danlooo commented May 11, 2023

Error

Fix

Test

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

giordano commented May 11, 2023

allixender commented May 12, 2023

danlooo commented May 12, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

allixender commented May 12, 2023 • edited Loading

danlooo commented May 12, 2023

allixender commented May 12, 2023

allixender commented May 12, 2023 •

edited

Loading