[POC][K8S][INFRA] Enable resource limited Spark on K8S integration tests in Github Action #35830

Yikun · 2022-03-13T03:07:28Z

What changes were proposed in this pull request?

Enable K8S integration tests in Github Action.
Note that this PR didn't add Volcano test due to limited resource of github action.

For current implementations, we enable 2 style k8s ci:

on push: validation on each commits pushed (PR tirgger) and merged (commits post validation): test push demo
workflow_dispatch for manually PR/Branch/tags validation:
- validate pr demo
- validate branch-3.3 failed demo (because 3.3 IT didn't support test.driver.cpu yet)

Why are the changes needed?

This will also improve the efficiency of K8S development and guarantee the quality of spark on K8S in some level.

Why setting driver 0.5 cpu, executor 0.2 cpu?
Github-hosted runner hardware limited: 2U7G
IT Job available CPU = 2U - 0.85U(K8S deploy) = 1.15U
There are 1.15 cpu left after k8s installation, to meet the requirement of K8S tests (one driver + max to 3 executors).
Time cost info:
- 14 mins to compile related code.
- 3 mins to build docker images.
- 20-30 mins to test
- Total: about 30-40 mins
Stablity:
I also do a stablity validation based on scheduled job:
https://github.com/spark-volcano/spark/actions/workflows/k8s_integration_test.yml
All test passed always passed at each half hours since enable.

Does this PR introduce any user-facing change?

No

How was this patch tested?

CI passed

Yikun · 2022-03-13T03:08:40Z

cc @dongjoon-hyun @HyukjinKwon @holdenk @yaooqinn @attilapiros

WDYT? Do you think this is a work that can go on?

.github/workflows/build_and_test.yml

Yikun · 2022-03-13T03:37:09Z

[info] Run completed in 22 minutes, 38 seconds.
[info] Total number of tests run: 23
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 19, failed 4, canceled 0, ignored 0, pending 0
[info] *** 4 TESTS FAILED ***

As expected.

.github/workflows/build_and_test.yml

dongjoon-hyun · 2022-03-13T05:30:19Z

...ation-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/KubernetesSuite.scala

@@ -191,6 +191,8 @@ class KubernetesSuite extends SparkFunSuite
      .set("spark.kubernetes.driver.pod.name", driverPodName)
      .set("spark.kubernetes.driver.label.spark-app-locator", appLocator)
      .set("spark.kubernetes.executor.label.spark-app-locator", appLocator)
+      .set("spark.kubernetes.executor.request.cores", "0.5")
+      .set("spark.kubernetes.driver.request.cores", "0.5")


Although this is only for tests, I'd like not to recommend this way. The driver has many threads and the flakiness is increasing when the driver CPU are capped.

dongjoon-hyun

Thank you for making a PR. Although I understand your intention, I'm a little negative for this approach for now.

Yikun · 2022-03-13T05:36:47Z

 - Run SparkRemoteFileTest using a remote data file *** FAILED ***
[info] Run completed in 19 minutes, 18 seconds.
[info] Total number of tests run: 23
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 22, failed 1, canceled 0, ignored 0, pending 0
[info] *** 1 TEST FAILED ***

https://github.com/Yikun/spark/runs/5525624508?check_suite_focus=true

With driver 0.5 CPU and executors 0.2 CPU only SparkRemoteFileTest failed.

Kubernetes is using 0.85 CPU 0.25G memeory, so only 1.15 CPU left, we can give 0.5 for driver, 0.2 for others executor (3 is max limited).

dongjoon-hyun · 2022-03-13T05:42:13Z

@Yikun . Fine tuning is just to aim to launch the tests, not for guaranteeing the stability. The minimum requirement of CI is stability. As you see, we are already hitting many flakiness issues in GitHub Action due to the memory issues and the committers are manually mitigating them by re-triggering . Unless we can get a bigger VM machine, this addition will add another complexity and instability to Apache Spark GitHub Action.

Kubernetes is using 0.85 CPU 0.25G memeory, so only 1.15 CPU left, we can give 0.5 for driver, 0.2 for others executor (3 is max limited).

Yikun · 2022-03-13T05:52:35Z

@dongjoon-hyun Thanks for your comments, this is only an initial exploring and share to enable K8S github action IT, I also think it is very important to ensure stability if we want to get this merged.

dongjoon-hyun · 2022-03-13T17:03:50Z

If ASF allows the minikube job, we may revisit this and your Volcano test resource reducing PR at Apache Spark 3.4 timeline with a longer monitoring period.

Yikun · 2022-03-18T14:29:29Z

I think recent k8s ci breaks have proved that this CI is useful, it helps found 2 [1] [2] master bugs within 3 days, and also help some on regression test.

@dongjoon-hyun I also contact with ASF, minikube is allowed in Apache projects, see: INFRA-23000.

I also update the PR description to show demo

Yikun · 2022-03-18T14:45:18Z

cc @dongjoon-hyun @HyukjinKwon @holdenk @martin-g

Would you mind take a look? I think now it's ready for review.

Yikun · 2022-04-02T23:43:51Z

@dongjoon-hyun Sure, next 3 days is day off in China, but I will try to some time address this.

btw, other concerns are

should we enable this in every pr ci?
should we keep IT in a separate file (it's for easy to enable workflow dispatch, but if we enable the every pr ci I guess this might no need)

dongjoon-hyun · 2022-04-02T23:46:33Z

Sure. This isn't urgent. Take your time.

We didn't decide this main PR about new CI. We will discuss new spinoff PRs first.

Yikun · 2022-04-07T12:00:01Z

Split two PRs as independent ones.

Support spark.kubernetes.test.(driver|executor)RequestCores
~~Support spark.kubernetes.test.minioRequestCores~~ Set minio cpu to 250m (0.25) in K8s IT

also enable minio test with 250m cpu, that means all kubernetes tests (execpt volcano batch suite) are running in current PR.

…ver|executor)RequestCores` ### What changes were proposed in this pull request? This patch adds support for `spark.kubernetes.test.(driver|executor)RequestCores`, this help devs set specific cores info for (driver|executor)RequestCores. ### Why are the changes needed? In some cases (such as resource limited case), we want to set specifc `RequestCores`. See also: #35830 (review) ### Does this PR introduce _any_ user-facing change? No, test only ### How was this patch tested? IT passed Closes #36087 from Yikun/SPARK-38802. Authored-by: Yikun Jiang <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

…s IT ### What changes were proposed in this pull request? This PR aims to set minio request cpu to `250m` (0.25). - This value also recommand in [link](https://docs.gitlab.com/charts/charts/minio/#installation-command-line-options). - There are [no cpu request limitation](https://github.com/minio/minio/blob/a3e317773a2b90a433136e1ff2a8394bc5017c75/helm/minio/values.yaml#L251) on current minio. ### Why are the changes needed? In some cases (such as resource limited case), we reduce request cpu of minio. See also: #35830 (review) ### Does this PR introduce _any_ user-facing change? No, test only ### How was this patch tested? IT passsed Closes #36096 from Yikun/minioRequestCores. Authored-by: Yikun Jiang <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

Yikun · 2022-04-16T11:38:00Z

Here is the complete spark on k8s test on Github Action and all test passed (also include volcano IT)：
https://github.com/Yikun/spark/runs/6046699187?check_suite_focus=true

It depends two patches:

Reafactor volcano test to use fabaric-io/k8s-cli: [SPARK-38921][K8S][Tests] Use k8s-client to create queue resource in Volcano IT #36219
Make volcano test can set cpu and set queue capacity according specified cpu dynamically: Yikun@55af9d8

dongjoon-hyun · 2022-07-21T04:48:28Z

Sorry for being late revisiting. Since we start to discuss Apache Spark 3.4 planning, shall we restart this for Apache Spark 3.4, @Yikun ?

dongjoon-hyun · 2022-07-21T04:49:05Z

Also, cc @HyukjinKwon , too

Yikun · 2022-07-21T07:28:12Z

@dongjoon-hyun Thanks for reminder! It's definate YES for me. I will revisit it soon to make it work with latest CI infra.

Given that we already have the docker image as part of the spark release, there is no doubt that it will not only help spark on k8s and also the docker image stable.

Yikun · 2022-07-21T12:10:04Z

To avoid the redundant of historical information, I submitted a new PR: #37244

…ver|executor)RequestCores` ### What changes were proposed in this pull request? This patch adds support for `spark.kubernetes.test.(driver|executor)RequestCores`, this help devs set specific cores info for (driver|executor)RequestCores. ### Why are the changes needed? In some cases (such as resource limited case), we want to set specifc `RequestCores`. See also: apache#35830 (review) ### Does this PR introduce _any_ user-facing change? No, test only ### How was this patch tested? IT passed Closes apache#36087 from Yikun/SPARK-38802. Authored-by: Yikun Jiang <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

…ver|executor)RequestCores` ### What changes were proposed in this pull request? This patch adds support for `spark.kubernetes.test.(driver|executor)RequestCores`, this help devs set specific cores info for (driver|executor)RequestCores. ### Why are the changes needed? In some cases (such as resource limited case), we want to set specifc `RequestCores`. See also: #35830 (review) ### Does this PR introduce _any_ user-facing change? No, test only ### How was this patch tested? IT passed Closes #36087 from Yikun/SPARK-38802. Authored-by: Yikun Jiang <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]> (cherry picked from commit 8396382) Signed-off-by: Dongjoon Hyun <[email protected]>

…s IT ### What changes were proposed in this pull request? This PR aims to set minio request cpu to `250m` (0.25). - This value also recommand in [link](https://docs.gitlab.com/charts/charts/minio/#installation-command-line-options). - There are [no cpu request limitation](https://github.com/minio/minio/blob/a3e317773a2b90a433136e1ff2a8394bc5017c75/helm/minio/values.yaml#L251) on current minio. ### Why are the changes needed? In some cases (such as resource limited case), we reduce request cpu of minio. See also: #35830 (review) ### Does this PR introduce _any_ user-facing change? No, test only ### How was this patch tested? IT passsed Closes #36096 from Yikun/minioRequestCores. Authored-by: Yikun Jiang <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]> (cherry picked from commit 5ea2b38) Signed-off-by: Dongjoon Hyun <[email protected]>

github-actions bot added INFRA KUBERNETES labels Mar 13, 2022

Yikun commented Mar 13, 2022

View reviewed changes

.github/workflows/build_and_test.yml Outdated Show resolved Hide resolved

dongjoon-hyun reviewed Mar 13, 2022

View reviewed changes

.github/workflows/build_and_test.yml Outdated Show resolved Hide resolved

dongjoon-hyun reviewed Mar 13, 2022

View reviewed changes

dongjoon-hyun previously requested changes Mar 13, 2022

View reviewed changes

Yikun force-pushed the k8s-it branch from ba682e6 to 72e5f44 Compare March 13, 2022 12:49

Yikun mentioned this pull request Mar 15, 2022

[SPARK-38544][BUILD] Upgrade log4j2 to 2.17.2 #35839

Closed

Yikun force-pushed the k8s-it branch from 72e5f44 to 1f50ec4 Compare March 18, 2022 02:46

github-actions bot added the DOCS label Mar 18, 2022

Yikun force-pushed the k8s-it branch 7 times, most recently from 69c4993 to 65815bf Compare March 18, 2022 09:22

Yikun closed this Mar 18, 2022

Yikun reopened this Mar 18, 2022

Yikun changed the title ~~[WIP][K8S][INFRA] Enable K8S integration tests in Github Action~~ [SPARK-38597][K8S][INFRA] Enable K8S integration tests in Github Action Mar 18, 2022

Yikun marked this pull request as ready for review March 18, 2022 14:37

Yikun force-pushed the k8s-it branch from 65815bf to 31e9f0e Compare March 18, 2022 14:37

Yikun force-pushed the k8s-it branch 7 times, most recently from 6dc006c to ef07343 Compare April 7, 2022 09:10

Yikun added 2 commits April 8, 2022 07:54

Add resource limited k8s it

d5a4fdd

set minio cpu to 250m

8c96cdd

Yikun force-pushed the k8s-it branch from ef07343 to 8c96cdd Compare April 7, 2022 23:56

Yikun changed the title ~~[SPARK-38597][K8S][INFRA] Enable resource limited Spark on K8S integration tests in Github Action~~ [POC][K8S][INFRA] Enable resource limited Spark on K8S integration tests in Github Action Jul 21, 2022

dongjoon-hyun mentioned this pull request Jul 21, 2022

[SPARK-38597][K8S][INFRA] Enable Spark on K8S integration tests #37244

Closed

dongjoon-hyun closed this in 5e6aab4 Jul 22, 2022

dongjoon-hyun removed their assignment Apr 14, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[POC][K8S][INFRA] Enable resource limited Spark on K8S integration tests in Github Action #35830

[POC][K8S][INFRA] Enable resource limited Spark on K8S integration tests in Github Action #35830

Yikun commented Mar 13, 2022 •

edited

Loading

Yikun commented Mar 13, 2022

Yikun commented Mar 13, 2022

dongjoon-hyun Mar 13, 2022

dongjoon-hyun left a comment

Yikun commented Mar 13, 2022

dongjoon-hyun commented Mar 13, 2022 •

edited

Loading

Yikun commented Mar 13, 2022 •

edited

Loading

dongjoon-hyun commented Mar 13, 2022

Yikun commented Mar 18, 2022 •

edited

Loading

Yikun commented Mar 18, 2022 •

edited

Loading

Yikun commented Apr 2, 2022

dongjoon-hyun commented Apr 2, 2022

Yikun commented Apr 7, 2022 •

edited

Loading

Yikun commented Apr 16, 2022

dongjoon-hyun commented Jul 21, 2022

dongjoon-hyun commented Jul 21, 2022

Yikun commented Jul 21, 2022

Yikun commented Jul 21, 2022

[POC][K8S][INFRA] Enable resource limited Spark on K8S integration tests in Github Action #35830

[POC][K8S][INFRA] Enable resource limited Spark on K8S integration tests in Github Action #35830

Conversation

Yikun commented Mar 13, 2022 • edited Loading

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Yikun commented Mar 13, 2022

Yikun commented Mar 13, 2022

dongjoon-hyun Mar 13, 2022

Choose a reason for hiding this comment

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Yikun commented Mar 13, 2022

dongjoon-hyun commented Mar 13, 2022 • edited Loading

Yikun commented Mar 13, 2022 • edited Loading

dongjoon-hyun commented Mar 13, 2022

Yikun commented Mar 18, 2022 • edited Loading

Yikun commented Mar 18, 2022 • edited Loading

Yikun commented Apr 2, 2022

dongjoon-hyun commented Apr 2, 2022

Yikun commented Apr 7, 2022 • edited Loading

Yikun commented Apr 16, 2022

dongjoon-hyun commented Jul 21, 2022

dongjoon-hyun commented Jul 21, 2022

Yikun commented Jul 21, 2022

Yikun commented Jul 21, 2022

Yikun commented Mar 13, 2022 •

edited

Loading

dongjoon-hyun commented Mar 13, 2022 •

edited

Loading

Yikun commented Mar 13, 2022 •

edited

Loading

Yikun commented Mar 18, 2022 •

edited

Loading

Yikun commented Mar 18, 2022 •

edited

Loading

Yikun commented Apr 7, 2022 •

edited

Loading