docker/ubuntu-large/build.sh fails with libarrow-dataset-dev : Depends: libarrow-acero-dev dependency error

[PerMildner]

Expected behavior and actual behavior.

I expect the docker build to succeed, doing:
git checkout v3.8.3 && cd docker/ubuntu-full && /bin/sh -x ./build.sh

Steps to reproduce the problem.

git clone https://github.com/OSGeo/gdal.git
cd gdal/
git checkout v3.8.3
cd docker/ubuntu-small/
/bin/sh -x ./build.sh
cd ../ubuntu-full/
/bin/sh -x ./build.sh

The docker/ubuntu-small/ build succeeds.

The docker/ubuntu-full/ build fails with:

Setting up libparquet-dev:amd64 (14.0.1-1) ...
Reading package lists...
Building dependency tree...
Reading state information...
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 libarrow-dataset-dev : Depends: libarrow-acero-dev (= 14.0.1-1) but it is not going to be installed
E: Unable to correct problems, you have held broken packages.
The command '/bin/sh -c . /buildscripts/bh-set-envvars.sh     && apt-get update -y     && DEBIAN_FRONTEND=noninteractive apt-get install -y -V ca-certificates lsb-release wget     && wget https://apache.jfrog.io/artifactory/arrow/$(lsb_release --id --short | tr 'A-Z' 'a-z')/apache-arrow-apt-source-latest-$(lsb_release --codename --short).deb     && DEBIAN_FRONTEND=noninteractive apt-get install -y -V ./apache-arrow-apt-source-latest-$(lsb_release --codename --short).deb     && apt-get update     && DEBIAN_FRONTEND=noninteractive apt-get install -y -V libarrow${ARROW_SOVERSION}${APT_ARCH_SUFFIX}     && DEBIAN_FRONTEND=noninteractive apt-get install -y -V libparquet${ARROW_SOVERSION}${APT_ARCH_SUFFIX}     && DEBIAN_FRONTEND=noninteractive apt-get install -y -V libarrow-dataset${ARROW_SOVERSION}${APT_ARCH_SUFFIX}     && DEBIAN_FRONTEND=noninteractive apt-get install -y -V libarrow-dev${APT_ARCH_SUFFIX}=${ARROW_VERSION}     && DEBIAN_FRONTEND=noninteractive apt-get install -y -V libparquet-dev${APT_ARCH_SUFFIX}=${ARROW_VERSION}     && DEBIAN_FRONTEND=noninteractive apt-get install -y -V libarrow-dataset-dev${APT_ARCH_SUFFIX}=${ARROW_VERSION}     && rm -rf /var/lib/apt/lists/*' returned a non-zero code: 100
$ 

Full transcript:
gdal_docker_build_problem.txt

Possibly related: Apache Arrow released version 15.0.0 a few days ago and when I tried to reproduce the Dockerfile RUN steps in a fresh ubuntu 22, it failed with the following, which mentions versions 14 and 15 (the following incomplete transcript is not the problem I am reporting, but it might provide some hints):

$ docker run --interactive --tty --rm ubuntu:22.04
....
root@28b06df59024:/# DEBIAN_FRONTEND=noninteractive apt-get install -y -V libarrow-dataset-dev${APT_ARCH_SUFFIX}=${ARROW_VERSION}
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 libarrow-dataset-dev : Depends: libarrow-acero-dev (= 14.0.1-1) but 15.0.0-1 is to be installed
                        Depends: libparquet-dev (= 14.0.1-1) but 15.0.0-1 is to be installed
E: Unable to correct problems, you have held broken packages.
root@28b06df59024:/#

Operating system

Docker Desktop on macOS Sonoma 14.3 Intel (presumably irrelevant)

GDAL version and provenance

v.3.8.3 from https://github.com/OSGeo/gdal.git

[rouault]

fixed per ba34c9f and 6fb04d9
You'll have to checkout "release/3.8" branch

[PerMildner]

Thanks for the quick reply.

Will this stop working again, when/if Apache Arrow releases version 16 in a few month (they seem to release new major version every few months)?

[rouault]

Will this stop working again, when/if Apache Arrow releases version 16 in a few month

likely

[PerMildner]

So, if I understand correctly, this issue is not permanently fixed with the above commits.

[rouault]

no, I can't think of a permanent fix. The alternative of not pinning to a particular version caused issues because of the mutli-stage design of the Dockerfile and with caching: you could potential refresh to Arrow 15 in the building stage, but the "production" stage could still used Arrow 14 if refreshing from a previous build...

[PerMildner]

It would be unfortunate if this comes back.
My guess is that this is a bug in how libarrow-dataset-dev specifies its dependencies, and until that is fixed this problem will come back every few months and make the then current version of GDAL unbuildable for reasons outside GDAL's control.

I tried the following, that just ensures the correct version of ibarrow-acero-dev is installed before installing libarrow-dataset-dev:

$ git diff
diff --git a/docker/ubuntu-full/Dockerfile b/docker/ubuntu-full/Dockerfile
index a597e9db72..70cf261156 100644
--- a/docker/ubuntu-full/Dockerfile
+++ b/docker/ubuntu-full/Dockerfile
@@ -241,6 +241,7 @@ RUN . /buildscripts/bh-set-envvars.sh \
     && DEBIAN_FRONTEND=noninteractive apt-get install -y -V libarrow-dataset${ARROW_SOVERSION}${APT_ARCH_SUFFIX} \
     && DEBIAN_FRONTEND=noninteractive apt-get install -y -V libarrow-dev${APT_ARCH_SUFFIX}=${ARROW_VERSION} \
     && DEBIAN_FRONTEND=noninteractive apt-get install -y -V libparquet-dev${APT_ARCH_SUFFIX}=${ARROW_VERSION} \
+    && DEBIAN_FRONTEND=noninteractive apt-get install -y -V libarrow-acero-dev${APT_ARCH_SUFFIX}=${ARROW_VERSION} \
     && DEBIAN_FRONTEND=noninteractive apt-get install -y -V libarrow-dataset-dev${APT_ARCH_SUFFIX}=${ARROW_VERSION} \
     && rm -rf /var/lib/apt/lists/*

With this change I was able to successfully build tag v3.8.3, see transcript.

An advantage with this proposed fix, compared to trying to have ARROW_VERSION play catch up with changes in the Apache Arrow releases, is that a particular version of GDAL will continue to be built, and be buildable, using the version of Apache Arrow that was used when the GDAL version was released.

gdal_docker_build_success.txt

[rouault]

With this change I was able to successfully build tag v3.8.3, see transcript.

ah that's nice. Can you issue a PR with that change?

[PerMildner]

I will issue a PR. Running a test build now on my slow machine...

[PerMildner]

I created the PR (#9186) based on the commit right before you bumped ARROW_VERSION since bumping Arrow version is no longer required. I did not roll back the version bump since I do not know whether there are unrelated reasons to use version 15 instead of version 14 of Apache Arrow.