datahub docker quickstart breaks with latest Docker Compose on Mac

[nicholasjng]

Describe the bug
docker/compose#10099 was merged last week, which enables parsing the target platform for building a container from the value of DOCKER_DEFAULT_PLATFORM.

Currently, the DOCKER_DEFAULT_PLATFORM is set to the value of uname -m by datahub docker quickstart here:

datahub/metadata-ingestion/src/datahub/cli/docker_cli.py

Lines 75 to 79 in cb6314c

	DOCKER_COMPOSE_PLATFORM: Optional[str] = (
	subprocess.run(["uname", "-m"], stdout=subprocess.PIPE)
	.stdout.decode("utf-8")
	.rstrip()
	)

Docker's internal platform parsing requires the OS type as well, and will infer it from the host machine if not provided, as is the case here. On my machine, an Apple M1 (where uname -m = arm64), this sets the Docker platform to darwin/arm64, which results in build failures because none of the available images are made for darwin/arm64.

To Reproduce
In a Python virtual environment, on an M1 Mac with docker-compose >= 2.14.2:

pip install --upgrade acryl-datahub
datahub docker quickstart

Expected behavior
Everything starts up as expected.

Screenshots
Console output (only the last line matters to illustrate the issue):

✗ datahub docker quickstart
Detected M1 machine
No Datahub Neo4j volume found, starting with elasticsearch as graph service.
To use neo4j as a graph backend, run
`datahub docker quickstart --quickstart-compose-file ./docker/quickstart/docker-compose.quickstart.yml`
from the root of the datahub repo

Fetching docker-compose file https://raw.githubusercontent.com/datahub-project/datahub/master/docker/quickstart/docker-compose-without-neo4j-m1.quickstart.yml from GitHub
Pulling docker images...
Finished pulling docker images!

[+] Running 0/3
 ⠦ Container mysql          Recreate                                                                                                                                                                   1.6s
 ⠦ Container zookeeper      Recreate                                                                                                                                                                   1.6s
 ⠦ Container elasticsearch  Recreate                                                                                                                                                                   1.6s
Error response from daemon: image with reference confluentinc/cp-zookeeper:7.2.2 was found but does not match the specified platform: wanted darwin/arm64, actual: linux/arm64

Desktop (please complete the following information):

• OS: macOS 12.6.2
• Browser: ---
• version: 0.9.3.2

Additional context
The fix is in theory as easy as prepending linux/ to the value of uname -m in _docker_subprocess_env, since most containers used by DataHub are only available for Linux anyway:

import platform

@functools.lru_cache()
def _docker_subprocess_env() -> Dict[str, str]:
    try:
        DOCKER_COMPOSE_PLATFORM = "linux/" + platform.machine()  # platform.machine() saves a subprocess here and works on windows, too
    ...

Looking forward to hearing your thoughts.

[pedro93]

Hello @nicholasjng

Thank you for reporting this issue. I've validated that this is an issue and will open a PR to fix it using your suggestion.
Appreciate the highly detailed report!! 🙇

[nicholasjng]

Muito obrigado!