[#75]: add CI check for playground #111

unknowntpo · 2024-12-01T22:49:13Z

Resolves #75

…vices

…atalog_hive

…on.sql.out

unknowntpo · 2025-02-27T13:42:28Z

@danhuawang Would you like to review this PR ?
I enabled ci test in my repo, and it works.

danhuawang · 2025-03-01T03:01:02Z

init/common/init_metalake_catalog.sh

@@ -35,9 +38,12 @@ if echo "$response" | grep -q "\"code\":0"; then
  true
 else
  # Create Hive catalog for experience Gravitino service
-  response=$(curl -X POST -H "Content-Type: application/json" -d '{"name":"catalog_hive","type":"RELATIONAL", "provider":"hive", "comment":"comment","properties":{"metastore.uris":"thrift://'${HIVE_HOST_IP}':9083" }}' http://gravitino:8090/api/metalakes/metalake_demo/catalogs)
+  response=$(curl -X POST -H "Content-Type: application/json" -d '{"name":"catalog_hive","type":"RELATIONAL", "provider":"hive", "comment":"comment","properties":{"metastore.uris":"thrift://hive:9083" }}' http://gravitino:8090/api/metalakes/metalake_demo/catalogs)


Why replace the environment variable ${HIVE_HOST_IP}? The Trino in docker-compose.yaml defined the environment variables.

trino: image: apache/gravitino-playground:trino-435-gravitino-0.8.0-incubating ports: - "18080:8080" container_name: playground-trino environment: - HADOOP_USER_NAME=root - GRAVITINO_HOST_IP=gravitino - GRAVITINO_HOST_PORT=8090 - GRAVITINO_METALAKE_NAME=metalake_demo - HIVE_HOST_IP=hive - MYSQL_HOST_IP=mysql - POSTGRES_HOST_IP=postgresql entrypoint: /bin/bash /tmp/trino/init.sh volumes: - ./init/trino:/tmp/trino - ./init/common:/tmp/common - ./healthcheck:/tmp/healthcheck

Uh, I hard-coded this because in this script, we also hard-coded host name of gravitino, so i want to make the style consistent.

http://gravitino:8090/api/metalakes/metalake_demo/catalogs

danhuawang · 2025-03-01T03:05:03Z

init/common/init_metalake_catalog.sh

  if echo "$response" | grep -q "\"code\":0"; then
    true # Placeholder, do nothing
+  elif echo "$response" | grep -q "\"type\":\"CatalogAlreadyExistsException\""; then
+    echo "Catalog catalog_hive already exists"
+    true # Placeholder, do nothing


If there's old catalog data in gravitino, will it cause the test cases run failed ? The test case output data may not match.

Because we use init_metalake_catalog.sh in both trino and spark,

For example, during spark container initialization, even if we send GET request to make sure catalog_hive not exist, between first and second request, trino container may also find catalog_hive not exist and create catalog_hive, this cause problem.

response=$(curl http://gravitino:8090/api/metalakes/metalake_demo/catalogs/catalog_hive) if echo "$response" | grep -q "\"code\":0"; then true else # Create Hive catalog for experience Gravitino service response=$(curl -X POST -H "Content-Type: application/json" -d '{"name":"catalog_hive","type":"RELATIONAL", "provider":"hive", "comment":"comment","properties":{"metastore.uris":"thrift://hive:9083" }}' http://gravitino:8090/api/metalakes/metalake_demo/catalogs) if echo "$response" | grep -q "\"code\":0"; then true # Placeholder, do nothing else echo "catalog_hive create failed" exit 1 fi fi

danhuawang · 2025-03-01T03:05:22Z

init/common/init_metalake_catalog.sh

@@ -52,6 +58,9 @@ else
  response=$(curl -X POST -H "Accept: application/vnd.gravitino.v1+json" -H "Content-Type: application/json" -d '{ "name":"catalog_postgres", "type":"RELATIONAL", "provider":"jdbc-postgresql", "comment":"comment", "properties":{ "jdbc-url":"jdbc:postgresql://postgresql/db", "jdbc-user":"postgres", "jdbc-password":"postgres", "jdbc-database":"db", "jdbc-driver": "org.postgresql.Driver" } }' http://gravitino:8090/api/metalakes/metalake_demo/catalogs)
  if echo "$response" | grep -q "\"code\":0"; then
    true # Placeholder, do nothing
+  elif echo "$response" | grep -q "\"type\":\"CatalogAlreadyExistsException\""; then


If there's old catalog data in gravitino, will it cause the test cases run failed ?

danhuawang · 2025-03-01T03:06:11Z

init/common/init_metalake_catalog.sh

  if echo "$response" | grep -q "catalog_mysql"; then
    true # Placeholder, do nothing
+  elif echo "$response" | grep -q "\"type\":\"CatalogAlreadyExistsException\""; then
+    echo "Catalog catalog_mysql already exists"
+    true # Placeholder, do nothing


If there's old catalog data in gravitino, will it cause the test cases run failed ?

danhuawang · 2025-03-01T03:06:21Z

init/common/init_metalake_catalog.sh

  if echo "$response" | grep -q "\"code\":0"; then
    true # Placeholder, do nothing
+  elif echo "$response" | grep -q "\"type\":\"CatalogAlreadyExistsException\""; then
+    echo "Catalog catalog_iceberg already exists"
+    true # Placeholder, do nothing


If there's old catalog data in gravitino, will it cause the test cases run failed ?

danhuawang · 2025-03-01T03:07:03Z

init/common/init_metalake_catalog.sh

@@ -78,9 +90,12 @@ if echo "$response" | grep -q "\"code\":0"; then
  true
 else
  # Create Iceberg catalog for experience Gravitino service
-  response=$(curl -X POST -H "Accept: application/vnd.gravitino.v1+json" -H "Content-Type: application/json" -d '{ "name":"catalog_iceberg", "type":"RELATIONAL", "provider":"lakehouse-iceberg", "comment":"comment", "properties":{ "uri":"jdbc:mysql://'${MYSQL_HOST_IP}':3306/db", "catalog-backend":"jdbc", "warehouse":"hdfs://'${HIVE_HOST_IP}':9000/user/iceberg/warehouse/", "jdbc-user":"mysql", "jdbc-password":"mysql", "jdbc-driver":"com.mysql.cj.jdbc.Driver"} }' http://gravitino:8090/api/metalakes/metalake_demo/catalogs)
+  response=$(curl -X POST -H "Accept: application/vnd.gravitino.v1+json" -H "Content-Type: application/json" -d '{ "name":"catalog_iceberg", "type":"RELATIONAL", "provider":"lakehouse-iceberg", "comment":"comment", "properties":{ "uri":"jdbc:mysql://mysql:3306/db", "catalog-backend":"jdbc", "warehouse":"hdfs://hive:9000/user/iceberg/warehouse/", "jdbc-user":"mysql", "jdbc-password":"mysql", "jdbc-driver":"com.mysql.cj.jdbc.Driver"} }' http://gravitino:8090/api/metalakes/metalake_demo/catalogs)


Why replace the environment variable ${HIVE_HOST_IP},MYSQL_HOST_IP? The Trino in docker-compose.yaml defined the environment variables.

trino: image: apache/gravitino-playground:trino-435-gravitino-0.8.0-incubating ports: - "18080:8080" container_name: playground-trino environment: - HADOOP_USER_NAME=root - GRAVITINO_HOST_IP=gravitino - GRAVITINO_HOST_PORT=8090 - GRAVITINO_METALAKE_NAME=metalake_demo - HIVE_HOST_IP=hive - MYSQL_HOST_IP=mysql - POSTGRES_HOST_IP=postgresql entrypoint: /bin/bash /tmp/trino/init.sh volumes: - ./init/trino:/tmp/trino - ./init/common:/tmp/common - ./healthcheck:/tmp/healthcheck

Uh, I hard-coded this because in this script, we also hard-coded host name of gravitino, so i want to make the style consistent.

http://gravitino:8090/api/metalakes/metalake_demo/catalogs

danhuawang · 2025-03-01T03:07:48Z

init/jupyter/init.sh

-  cp -r /tmp/gravitino/*.ipynb /home/jovyan
-else
-  cp -r /tmp/gravitino/authorization/*.ipynb /home/jovyan
-fi


Why remove the condition for $RANGER_ENABLE

Because I found something strange. Why can't we use jupyter notebook like gravitino-trino-example.ipynb while enabling ranger ?

danhuawang · 2025-03-01T03:13:13Z

@danhuawang Would you like to review this PR ? I enabled ci test in my repo, and it works.

@unknowntpo I have some comments.

…alog.sh

unknowntpo force-pushed the feat-ci branch 2 times, most recently from e839f0d to f2210c8 Compare December 24, 2024 00:43

unknowntpo changed the title ~~feat(ci): add CI check for playground~~ [#75]: add CI check for playground Dec 24, 2024

unknowntpo force-pushed the feat-ci branch 3 times, most recently from e67031e to c4b4d9f Compare December 24, 2024 02:13

unknowntpo force-pushed the feat-ci branch 4 times, most recently from 85ab134 to d1b177f Compare January 19, 2025 12:01

unknowntpo force-pushed the feat-ci branch from 272f609 to 8656710 Compare February 26, 2025 07:10

unknowntpo added 19 commits February 27, 2025 20:15

feat(ci): init

ab66086

feat(ci): let ci run for every push

8f42d35

feat(ci): push workflow file to correct location

201ec29

refactor(ci): remove unnecessary workflows

ad85682

feat(shfmt): add .editorconfig

e63aaaf

feat(shfmt): format .sh files

9b09f42

refactor(ci): tweak shfmt command

04df1cc

refactor(ci): tweak workflow name

964d041

refactor(ci): remove redundant checkout

92aa7d0

fix(ci): change cd dir

1c60043

refactor(ci): split steps into two jobs

4105c1a

refactor(ci): curl in quiet mode to wait for gravitino playground ser…

fe66924

…vices

refactor(ci): use docker compose ps to show status of services

9ba9fef

refactor(ci): add artifacts at test job

b8eb3dd

refactor(ci): add --enable-ranger option

5bebda0

fix(ci): fix wrong playground command

b9e4726

fix(ci): move logs to /tmp/playground-log dir

268533f

fix(ci): disable ranger temporary, FIXME: should revert this

73c05e0

refactor(docker-compose.yml): increase healthcheck timeout

062df26

unknowntpo added 11 commits February 27, 2025 20:15

refactor(ci-test.yml): fix incorrect folder in Test sql step

6561e8b

refactor(docker-compose.yml): ranger: tweak interval

64e1304

refactor(tests/trino-cross-catalog.sql): rename catalog gt_hive2 to c…

5ceccad

…atalog_hive

fix: try to fix spark sql failed problem

054b0d5

fix(ci): try to fix spark problem

f4d4af1

refactor(ci): reduce requiredDiskSpaceGB to 24G

76533ee

refactor(ci): reduce timeout to 10 min

7b18736

fix(ci): runSQLOnPlaygroud.sh: do not remove jar files

73752e0

refactor(ci): runSQLOnPlaygroud.sh: rename union-trino.sql.out to uni…

f7336ae

…on.sql.out

refactor(ci): runSQLOnPlaygroud.sh: minor tweaks

bb78cdf

refactor(ci): minor tweaks

1cbc218

unknowntpo force-pushed the feat-ci branch from 5b4a392 to 1cbc218 Compare February 27, 2025 12:15

styles: minor tweaks

6214c84

unknowntpo marked this pull request as ready for review February 27, 2025 13:39

danhuawang reviewed Mar 1, 2025

View reviewed changes

refactor(init/trino): add check for return value of init_metalake_cat…

8af0b9e

…alog.sh

unknowntpo force-pushed the feat-ci branch 2 times, most recently from 28cda30 to 8af0b9e Compare March 1, 2025 13:43

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[#75]: add CI check for playground #111

[#75]: add CI check for playground #111

unknowntpo commented Dec 1, 2024

unknowntpo commented Feb 27, 2025

danhuawang Mar 1, 2025

unknowntpo Mar 1, 2025

danhuawang Mar 1, 2025

unknowntpo Mar 1, 2025

danhuawang Mar 1, 2025

danhuawang Mar 1, 2025

danhuawang Mar 1, 2025

danhuawang Mar 1, 2025

unknowntpo Mar 1, 2025

danhuawang Mar 1, 2025

unknowntpo Mar 1, 2025

danhuawang commented Mar 1, 2025

[#75]: add CI check for playground #111

Are you sure you want to change the base?

[#75]: add CI check for playground #111

Conversation

unknowntpo commented Dec 1, 2024

unknowntpo commented Feb 27, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

danhuawang commented Mar 1, 2025