Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-38803][K8S][TESTS] Add minio tag for K8s IT #36085

Closed
wants to merge 1 commit into from

Conversation

Yikun
Copy link
Member

@Yikun Yikun commented Apr 6, 2022

What changes were proposed in this pull request?

This PR aims to add the minio tag. Skip minio related test only when specified -Dtest.exclude.tags=minio

Why are the changes needed?

In some cases (such as resource limited case), we don't want to execute minio related test.

Does this PR introduce any user-facing change?

No, test only

How was this patch tested?

IT passsed

$  build/sbt -Phadoop3 -Pvolcano -Pkubernetes -Pkubernetes-integration-tests -Dtest.include.tags=minio -Dspark.kubernetes.test.deployMode=minikube -Dspark.kubernetes.test.namespace=default "kubernetes-integration-tests/testOnly"
[info] KubernetesSuite:
[info] - Launcher client dependencies (47 seconds, 498 milliseconds)
[info] - SPARK-33615: Launcher client archives (50 seconds, 562 milliseconds)
[info] - SPARK-33748: Launcher python client respecting PYSPARK_PYTHON (52 seconds, 375 milliseconds)
[info] - SPARK-33748: Launcher python client respecting spark.pyspark.python and spark.pyspark.driver.python (54 seconds, 755 milliseconds)
[info] - Launcher python client dependencies using a zip file (52 seconds, 44 milliseconds)
[info] VolcanoSuite:
[info] - Launcher client dependencies (1 minute, 13 seconds)
[info] - SPARK-33615: Launcher client archives (49 seconds, 703 milliseconds)
[info] - SPARK-33748: Launcher python client respecting PYSPARK_PYTHON (51 seconds, 952 milliseconds)
[info] - SPARK-33748: Launcher python client respecting spark.pyspark.python and spark.pyspark.driver.python (1 minute, 3 seconds)
[info] - Launcher python client dependencies using a zip file (2 minutes, 10 seconds)

@dcoliversun
Copy link
Contributor

Sorry, Yikun. -1 for this intention. Tests about minio is important for usage with hadoop. You could refer this issue https://issues.apache.org/jira/browse/SPARK-38652

@dcoliversun
Copy link
Contributor

+1, LGTM. I have a discussion with @Yikun offline. Tag minikube can exclude DepsTestSuite and PVTestsSuite, and tag minio can exclude DepsTestSuite. DepsTestSuite works or not are shown below.

DepsTestSuite tag: minikube tag: minio
not work excluded excluded
work not excluded not excluded
not work excluded not excluded
not work not excluded excluded

@Yikun Yikun marked this pull request as ready for review April 7, 2022 06:35
@Yikun
Copy link
Member Author

Yikun commented Apr 7, 2022

@dcoliversun Thanks, this pr only add a tag, skip minio related test only when specified -Dtest.exclude.tags=minio

@Yikun
Copy link
Member Author

Yikun commented Apr 7, 2022

Replaced by #36096

@Yikun Yikun closed this Apr 7, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants