[Failing Test]: MLTransform IT test is failing

[tvalentyn]

What happened?

apache_beam/examples/ml_transform/ml_transform_it_test.py::LargeMovieReviewDatasetProcessTest::test_process_large_movie_review_dataset is perma-failing:

FileNotFoundError: [Errno 2] No such file or directory: 'gs://temp-storage-for-end-to-end-tests/tft/8825e9c2853143f289c2673feb3bc64b'

Sample run: https://github.com/apache/beam/actions/runs/7732452652/job/21082406012#step:29:2783

cc: @AnandInguva

Issue Failure

Failure: Test is continually failing

Issue Priority

Priority: 1 (unhealthy code / failing or flaky postcommit so we cannot be sure the product is healthy)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[tvalentyn]

gsutil ls gs://temp-storage-for-end-to-end-tests/tft/8825e9c2853143f289c2673feb3bc64b
gs://temp-storage-for-end-to-end-tests/tft/8825e9c2853143f289c2673feb3bc64b/attributes.json
gs://temp-storage-for-end-to-end-tests/tft/8825e9c2853143f289c2673feb3bc64b/c4f900/

[tvalentyn]

interestingly, files are there, so maybe it's a permission issue?

[tvalentyn]

Repro:

pip install tensorflow-transform apache-beam

from apache_beam.ml.transforms.utils import ArtifactsFetcher
artifact_location='gs://temp-storage-for-end-to-end-tests/tft/8825e9c2853143f289c2673feb3bc64b'
artifacts_fetcher = ArtifactsFetcher(artifact_location=artifact_location)
actual_vocab_list = artifacts_fetcher.get_vocab_list()

FileNotFoundError: [Errno 2] No such file or directory: 'gs://temp-storage-for-end-to-end-tests/tft/8825e9c2853143f289c2673feb3bc64b'

[tvalentyn]

Probably failing after #29564.

We should use Beam FileSystem in these cases:

beam/sdks/python/apache_beam/ml/transforms/utils.py

Line 33 in 574e7bf

files = os.listdir(artifact_location)

[tvalentyn]

It might be a regression in the currently cut release.

[tvalentyn]

Actually, not a regression in 2.54.0 since #30191 was already in 2.53.0-RC1. But I think it limits usability of MLTransform.