Failed to run run_spark_sql.sh

[flyrain]

Describe the bug

The fixed token(principal:root;realm:default-realm) for regression test doesn't work any more. It worked before the Quarkus migration. cc @jbonofre @adutra @dimas-b @eric-maynard

There are a bunch of places relies on this credential:

1. polaris/regtests/run_spark_sql.sh

   Line 58 in ee26660

   SPARK_BEARER_TOKEN="${REGTEST_ROOT_BEARER_TOKEN:-principal:root;realm:default-realm}"

2. https://github.com/polaris-catalog/polaris/blob/ee26660f08041b5d2ef08c6a490d88f2a846e4ba/regtests/t_spark_sql/src/spark_sql_views.sh#L22-L22
3. https://github.com/polaris-catalog/polaris/blob/ee26660f08041b5d2ef08c6a490d88f2a846e4ba/getting-started/trino/create-polaris-catalog.sh#L20-L20
4. https://github.com/polaris-catalog/polaris/blob/ee26660f08041b5d2ef08c6a490d88f2a846e4ba/getting-started/trino/trino-config/catalog/iceberg.properties#L24-L24

All these places relies on this credential will fail.

I'd suggest to restore this.

[HonahX]

Seems we need to add some additional env to make the fixed token work
I followed the docker compose yaml and adding the following env to getting-started trino example

polaris/regtests/docker-compose.yml

Lines 34 to 42 in ee26660

	polaris.persistence.type: in-memory
	polaris.authentication.authenticator.type: test
	polaris.authentication.token-service.type: test
	polaris.authentication.token-broker.type: symmetric-key
	polaris.authentication.token-broker.symmetric-key.secret: polaris
	polaris.features.defaults."SUPPORTED_CATALOG_STORAGE_TYPES": '["FILE","S3","GCS","AZURE"]'
	polaris.realm-context.realms: default-realm,realm1
	quarkus.log.file.enable: false
	quarkus.otel.sdk.disabled: "true"

and I am able to create the catalog via token principal:root;realm:default-realm

[flyrain]

Thanks for the finding, @HonahX! It works without changing the credential as well, see #866. I think the class TestInlineBearerTokenPolarisAuthenticator is the one not checking the token, which is set by polaris.authentication.authenticator.type: test.

[MonkeyCanCode]

So here is this one which @adutra created (not yet ready for review): https://github.com/apache/polaris/pull/804/files

It used the following snippet to pass around token for the test:

if ! output=$(curl -X POST -H "Polaris-Realm: POLARIS" "http://${POLARIS_HOST:-localhost}:8181/api/catalog/v1/oauth/tokens" \
  -d "grant_type=client_credentials" \
  -d "client_id=root" \
  -d "client_secret=secret" \
  -d "scope=PRINCIPAL_ROLE:ALL"); then
  logred "Error: Failed to retrieve bearer token"
  exit 1
fi

token=$(echo "$output" | awk -F\" '{print $4}')

export REGTEST_ROOT_BEARER_TOKEN=$token

[flyrain]

I'm OK if we decided to go that direction, that means all places with the token principal:root;realm:default-realm needed to be replaced with the token generation script, and we probably don't need the class TestInlineBearerTokenPolarisAuthenticator any more.

[MonkeyCanCode]

Yes, I am still reading through recent changes. @adutra what do u think? Should we proceed with #804 as docker change is merged.

[adutra]

Hi @MonkeyCanCode @flyrain @HonahX sorry for creating so much confusion here.

Here is my long term vision:

• I think the test profile (used in automated tests) is too different from the prod profile. It is always good to make those converge as much as possible. That's why I opened Cleanup test profile and make it converge with prod profile #786. I think it would be good to merge that one quickly, I will do it today.
• I also think that the regression tests should strive to use the same config as in production. This is not the case today, as you noticed, which is why I think you opened this issue. I started a PR (in draft) here: Make regression tests use default authentication #804 where the authenticator for regression tests is changed to use the production one.
• As much as possible, I think that the test authenticator + token broker should only be used for prototyping, quickstart guides and demos, but not for tests.

Does that make sense?

If that's OK with you, let's proceed as follows:

• Get Cleanup test profile and make it converge with prod profile #786 merged
• Get Make regression tests use default authentication #804 reviewed and merged
• Revisit the docs and fix all the places where the regression tests are not launched with the appropriate parameters.

[flyrain]

+1 on consolidating to default authenticator.

With a setting like this, there seems no need to keep the test authenticator around. PoC, demos and quickstart can also use the default authenticator. Let me know If I missed something.

  jvmArgs = listOf("-Dpolaris.bootstrap.credentials=POLARIS,root,secret")

[MonkeyCanCode]

This is verified with the current main branch.