was cloud-lifesciences taken off gcr.io before the deadline?

[purajit]

https://cloud.google.com/life-sciences/docs/how-tos/variant-transforms says

Cloud Life Sciences is deprecated and will no longer be available on Google Cloud after July 8, 2025

However,

$ docker pull gcr.io/cloud-lifesciences/gcp-variant-transforms:latest
Error response from daemon: Head "https://gcr.io/v2/cloud-lifesciences/gcp-variant-transforms/manifests/latest": denied: Unauthenticated request. Unauthenticated requests do not have permission "artifactregistry.repositories.downloadArtifacts" on resource "projects/cloud-lifesciences/locations/us/repositories/gcr.io" (or it may not exist)

While this is for the API, I would have assumed this would apply to the Docker
images as well. Was this taken down 6 months before the publicly shared deadline?
Was there an announcement we missed?

[bolianyin]

The old images have too many security vulnerabilities for public access. Do you think you can build the image yourself and host it privately in your project? If it does not work, we can try to think of other ways.

[pgrosu]

There seem to be two on DockerHub:

• https://hub.docker.com/r/samesense/gcp-variant-transforms

• https://hub.docker.com/r/avabhyankar2/gcp-variant-transforms

[purajit]

The old images have too many security vulnerabilities for public access. Do you think you can build the image yourself and host it privately in your project? If it does not work, we can try to think of other ways.

I know haha - I'd love to get rid of it, but one of the teams was using it with a plan to deprecate it
before the deadline; I would love for us to not depend on it (we're pretty much fully on python3.12,
even readying up for 3.13, except for this).

Sadly, there are too many images involved - the process within gcp-variant-transforms pulls and
runs other images that are also not available anymore, and can't be configured via variables either.
The entire cloud-lifesciences has been taken down of what I can tell.

There seem to be two on DockerHub

I did notice that, but I'm looking for official images, and there's also the handful of other images
needed.

[bolianyin]

Could you send the service accounts that need access to the images to [email protected]? We might be able to give access to some users, but not to the public.

[pgrosu]

Hi Bolian (@bolianyin),

I think Purajit (@purajit) is going to have trouble creating the Docker image using the provided Dockerfile, as it will produce the following errors:

Get:47 http://deb.debian.org/debian bullseye/main amd64 python3-wheel all 0.34.2-1 [24.0kB]
Get:48 http://deb.debian.org/debian bullseye/main amd64 python3-pip all 20.3.4-4+deb11u1[337 kB]
Get:49 http://deb.debian.org/debian bullseye/main amd64 python3-venv amd64 3.9.2-3 [1188B]
Fetched 138 MB in 3min 33s (648 kB/s)
E: Failed to fetch http://security.debian.org/debian-security/pool/updates/main/p/perl/libperl5.32_5.32.1-4%2bdeb11u4_amd64.deb  Connection timed out [IP: 151.101.2.132 80]
E: Failed to fetch http://security.debian.org/debian-security/pool/updates/main/p/perl/perl_5.32.1-4%2bdeb11u4_amd64.deb  Connection timed out [IP: 151.101.2.132 80]
E: Failed to fetch http://security.debian.org/debian-security/pool/updates/main/p/perl/perl-modules-5.32_5.32.1-4%2bdeb11u4_all.deb  Connection timed out [IP: 151.101.2.132 80]
E: Failed to fetch http://security.debian.org/debian-security/pool/updates/main/o/openssl/libssl1.1_1.1.1w-0%2bdeb11u2_amd64.deb  Connection timed out [IP: 151.101.2.132 80]
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
The command '/bin/sh -c apt-get update && apt-get install -y     apt-transport-https    autoconf     automake     g++     gcc     gnupg     google-cloud-sdk     libbz2-dev     libcurl4-openssl-dev     liblzma-dev     libssl-dev     make     perl     zlib1g-dev     python3-pip     python3-venv' returned a non-zero code: 100
paul:~/tests/docker_tests/gcp-variant-transforms$

Basically, Debian 11 "Bullseye" reached the end of its regular support phase on August 14, 2024, and is now in LTS (Long Term Support). That would probably mean updating the repositories in the base image of the second stage (apache/beam_python3.8_sdk:2.37.0) to ensure newer versions are accessible, but that would also require regression testing of GCP Variant Transforms to make sure it is still giving the same results for all the tests.

Hope it helps,
~p

[purajit]

My teammate actually did manage to build an image (even before I filed this issue), but the problem
is that during the running process of gcp-variant-transforms, it pulls other cloud-lifesciences
images to run them, and they're all unavailable now, and there's no way to configure where those images
are fetched from, meaning we would not only have to build each image, but also make changes to internals
to be able to pull from our own container repo. That's when I filed the issue.

I believe the team is figuring out what service accounts we need for this and responding to that email.

[pgrosu]

Hi Purajit (@purajit),

That's cool that you tried that! So if we look back at the original error, it indicated a denied: Unauthenticated request...[for the path]...repositories/gcr.io response. So if you search the gcp-variant-transforms codebase, the cloud-lifesciences images are all part of Google's gcr.io repository. After you are given access for the service account, any launches under that account should just inherit those authenticated permissions for any program launches (giving you access to all those cloud-lifesciences images).

Hope it helps,
~p

[vadzimra]

Could you send the service accounts that need access to the images to [email protected]? We might be able to give access to some users, but not to the public.

Hi @bolianyin . I am @purajit 's teammate. We've created the services accounts and I'm trying to email them to [email protected] but I'm getting

We're writing to let you know that the group you tried to contact (gcp-batch-preview) may not exist, or you may not have permission to post messages to the group.

[pgrosu]

@vadzimra I'm sure it was received, as it already recognized this as a group email and not as something else. The warning you see is just a permissions setting that needs to be updated for the gcp-batch-preview group by the Google folks, so that anyone can post to the group rather than just specific members.

Hope it helps,
~p

[vadzimra]

thank you, @pgrosu

[bolianyin]

Sorry, folks. That group email did not work. Do you have an account contact who can relay the information to engineering team?

[purajit]

@pgrosu I've tried to be cordial, and I'm sorry to be blunt now, but you don't have any value or context to add in this thread - can you please leave it to those who do and not mislead people?

@bolianyin do you mean provide you with an email that you can contact us on? Is there a way to provide it without putting it on a public thread?

[bolianyin]

@purajit: Do you have a Goolge support contact who can forward the info to the engineering team?

[bolianyin]

Or could you provide your company name and we can see who is your contact?

[vadzimra]

@bolianyin it's Color Health, Inc.

[bolianyin]

Could you also share your project number? At the same time, it will be easier and faster if you can find your account representative and let them contact the Cloud Lifesciences engineering team.

[CoeColor]

@bolianyin Could you please summarize the ask? Is it sufficient to provide the project #s? And, if so, which project number do you mean? The one where the service accounts reside?
(there is a difference between the service accounts project and the BigQuery projects that they have access to)

We can submit a Google Support request, but we need clarity on what you mean by "forward the info to the engineering team". What should we request from Google Support?

[bolianyin]

@CoeColor , we were trying to use a project number and company name to figure out who might be your support representative so that we can ask them to contact you for the information.

Submitting a Google Support request is better and the support team will forward the request to the engineering team. In the request, please provide the service account that will be used to access the docker containers and mention that the request should be forward to the Cloud Lifesciences engineering team.

Hope that helps.