[SPARK-46692][BUILD] Load `inputs.envs` always in `Python` and `Upload`-related steps #44698

panbingkun · 2024-01-12T02:31:07Z

What changes were proposed in this pull request?

The pr aims to fix potential environmental variable(eg: PYTHON_TO_TEST.) transfer issue in pyspark job GA

Why are the changes needed?

https://github.com/apache/spark/actions/workflows/build_python.yml
Let's take a successful build_ python Using as an example, https://github.com/apache/spark/actions/runs/7476792796/job/20348090667

1.python 3.10
Obviously, it is meaningless to enumerate the packages of Python 3.9 when building PySpark based on Python 3.10.
https://github.com/apache/spark/actions/runs/7476792796/job/20348079659

2.python 3.11
https://github.com/apache/spark/actions/runs/7476792796/job/20348091081

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Pass GA.

Was this patch authored or co-authored using generative AI tooling?

No.

…ariable transmission `PYTHON_TO_TEST`

panbingkun · 2024-01-12T02:35:58Z

cc @HyukjinKwon @dongjoon-hyun @zhengruifeng @LuciferYang

…ariable transmission `PYTHON_TO_TEST`

panbingkun · 2024-01-12T03:05:12Z

For greater stability, we can fix this issue first, and then I will observe whether the environment variables PYTHON_TO_TEST for each step of build_python tomorrow are effective. If they are effective, we will bring https://github.com/apache/spark/pull/44662 back.

panbingkun · 2024-01-12T03:09:52Z

I found that in step PySpark Run test, it is used as follows:

spark/.github/workflows/build_and_test.yml

Line 446 in efa891c

env: ${{ fromJSON(inputs.envs) }}

panbingkun · 2024-01-12T08:40:29Z

After this PR is merged, I will observe whether the value of the environment variable 'PYTHON_TO_TEST' in steps List Python packages (${{ env.PYTHON_TO_TEST }}), Upload test results to report and Upload unit tests log files in build_python is normal at tomorrow morning.

dongjoon-hyun

Ya, this looks like the root causes.

Are these all instances?
Given that the change affects not only PYTHON_TO_TEST, but also all environment variables, please revise the PR title.

Fix potential issues with environment variable transmission PYTHON_TO_TEST

panbingkun · 2024-01-12T10:31:24Z

Ya, this looks like the root causes.

Are these all instances?

Given that the change affects not only PYTHON_TO_TEST, but also all environment variables, please revise the PR title.

Fix potential issues with environment variable transmission PYTHON_TO_TEST

@dongjoon-hyun
Yes, in ' pyspark job GA', I checked and found that currently only these 3 steps are affected by this issue.
After this PR is merged, I will continue to observe.

dongjoon-hyun

+1, LGTM.

panbingkun · 2024-01-12T10:55:48Z

+1, LGTM.

@dongjoon-hyun
Thank you very much for giving this PR a suitable title. 😄

If tomorrow's scheduled task meets expectations, I will provide a PR to bring https://github.com/apache/spark/pull/44662 back.

dongjoon-hyun · 2024-01-12T11:15:40Z

Ya, +1 for the plan. Thank you.

panbingkun · 2024-01-12T15:50:41Z

@dongjoon-hyun
Has the scheduling started yet? I have observed some errors in the pypy3 environment, such as missing package: grpc. Is this expected?
https://github.com/apache/spark/actions/runs/7503864095/job/20429679584

I have observed that the environment variables PYTHON_TO_TEST are set correctly, but I am not sure about the error above

Perhaps we need to install the grpc package for Python 3.8 (pypy3)?

spark/dev/infra/Dockerfile

Line 95 in a3266b4

    
           RUN pypy3 -m pip install numpy 'six==1.16.0' 'pandas<=2.1.4' scipy coverage matplotlib lxml

HyukjinKwon · 2024-01-13T00:18:26Z

We should but I think that's not available. If grpc is not installed it should skip the tests properly.

HyukjinKwon · 2024-01-13T00:24:49Z

#44715

panbingkun · 2024-01-13T02:02:29Z

We should but I think that's not available. If grpc is not installed it should skip the tests properly.

Okay, let's continue to observe.

…logics out ### What changes were proposed in this pull request? This PR factor Connect/non-Connect specific logics out into dedicated test classes. This PR is a followup of #40785 ### Why are the changes needed? In order to avoid test failure such as #44698 (comment) by missing dependencies ### Does this PR introduce _any_ user-facing change? No, test-only. ### How was this patch tested? CI in this PR should verify it. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #44715 from HyukjinKwon/SPARK-42960-followup. Authored-by: Hyukjin Kwon <[email protected]> Signed-off-by: Hyukjin Kwon <[email protected]>

panbingkun · 2024-01-15T01:10:56Z

@HyukjinKwon
I have observed the following errors in the environment pypy3.
https://github.com/apache/spark/actions/runs/7519959346/job/20474204069

Do we need to install pyarrow in the environment? Or do we ignore some UT when we lack pyarrow again?

HyukjinKwon · 2024-01-15T01:14:12Z

pyarrow in PyPy isn't available up to my best knowledge.

panbingkun · 2024-01-15T01:22:16Z

pyarrow in PyPy isn't available up to my best knowledge.

I tried to install 'pyarrow' in pypy3 over the weekend, but one UT failed.
https://github.com/panbingkun/spark/actions/runs/7519505235/job/20468699700

So should we ignore it?

HyukjinKwon · 2024-01-15T01:24:56Z

For them, let's skip it for now, and file a JIRA for each tests.

panbingkun · 2024-01-15T01:26:45Z

For them, let's skip it for now, and file a JIRA for each tests.

Okay.

### What changes were proposed in this pull request? The pr aims to upgrade upload-artifact action from v3 to v4. After PR #44698, our environment variable(`PYTHON_TO_TEST`) is correctly passed and assigned value. We will bring back this PR: #44662 ### Why are the changes needed? - v4.0.0 release notes: https://github.com/actions/upload-artifact/releases/tag/v4.0.0 They have numerous performance and behavioral improvements. - v3 VS v4: actions/upload-artifact@v3...v4.0.0 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass GA. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #44728 from panbingkun/SPARK-46474_GO_AHEAD. Authored-by: panbingkun <[email protected]> Signed-off-by: Hyukjin Kwon <[email protected]>

[SPARK-46692][BUILD][PYSPARK] Fix potential issues with environment v…

0367eb3

…ariable transmission `PYTHON_TO_TEST`

github-actions bot added SQL INFRA PYTHON labels Jan 12, 2024

panbingkun marked this pull request as ready for review January 12, 2024 02:35

[SPARK-46692][BUILD][PYSPARK] Fix potential issues with environment v…

76b8a2b

…ariable transmission `PYTHON_TO_TEST`

github-actions bot removed SQL PYTHON labels Jan 12, 2024

panbingkun changed the title ~~[SPARK-46692][BUILD][PYSPARK] Fix potential issues with environment variable transmission PYTHON_TO_TEST~~ [SPARK-46692][BUILD] Fix potential issues with environment variable transmission PYTHON_TO_TEST Jan 12, 2024

panbingkun mentioned this pull request Jan 12, 2024

[SPARK-46474][INFRA] Upgrade upload-artifact action to v4 #44662

Closed

zhengruifeng approved these changes Jan 12, 2024

View reviewed changes

dongjoon-hyun reviewed Jan 12, 2024

View reviewed changes

panbingkun changed the title ~~[SPARK-46692][BUILD] Fix potential issues with environment variable transmission PYTHON_TO_TEST~~ [SPARK-46692][BUILD] Fix potential environmental variable transfer issue in pyspark job GA Jan 12, 2024

panbingkun requested a review from dongjoon-hyun January 12, 2024 10:31

dongjoon-hyun approved these changes Jan 12, 2024

View reviewed changes

dongjoon-hyun changed the title ~~[SPARK-46692][BUILD] Fix potential environmental variable transfer issue in pyspark job GA~~ [SPARK-46692][BUILD] Load inputs.envs always in Python and Upload steps Jan 12, 2024

dongjoon-hyun changed the title ~~[SPARK-46692][BUILD] Load inputs.envs always in Python and Upload steps~~ [SPARK-46692][BUILD] Load inputs.envs always in Python and Upload-related steps Jan 12, 2024

dongjoon-hyun closed this in 9f061b0 Jan 12, 2024

HyukjinKwon mentioned this pull request Jan 13, 2024

[SPARK-42960][PYTHON][SS][TESTS] Factor Connect/non-Connect specific logics out #44715

Closed

panbingkun mentioned this pull request Jan 15, 2024

[SPARK-46474][INFRA] Upgrade upload-artifact action to v4 #44728

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-46692][BUILD] Load `inputs.envs` always in `Python` and `Upload`-related steps #44698

[SPARK-46692][BUILD] Load `inputs.envs` always in `Python` and `Upload`-related steps #44698

panbingkun commented Jan 12, 2024 •

edited

Loading

panbingkun commented Jan 12, 2024 •

edited

Loading

panbingkun commented Jan 12, 2024 •

edited

Loading

panbingkun commented Jan 12, 2024 •

edited

Loading

panbingkun commented Jan 12, 2024

dongjoon-hyun left a comment •

edited

Loading

panbingkun commented Jan 12, 2024

dongjoon-hyun left a comment

panbingkun commented Jan 12, 2024

dongjoon-hyun commented Jan 12, 2024

panbingkun commented Jan 12, 2024 •

edited

Loading

HyukjinKwon commented Jan 13, 2024

HyukjinKwon commented Jan 13, 2024

panbingkun commented Jan 13, 2024

panbingkun commented Jan 15, 2024

HyukjinKwon commented Jan 15, 2024

panbingkun commented Jan 15, 2024

HyukjinKwon commented Jan 15, 2024

panbingkun commented Jan 15, 2024

[SPARK-46692][BUILD] Load inputs.envs always in Python and Upload-related steps #44698

[SPARK-46692][BUILD] Load inputs.envs always in Python and Upload-related steps #44698

Conversation

panbingkun commented Jan 12, 2024 • edited Loading

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

panbingkun commented Jan 12, 2024 • edited Loading

panbingkun commented Jan 12, 2024 • edited Loading

panbingkun commented Jan 12, 2024 • edited Loading

panbingkun commented Jan 12, 2024

dongjoon-hyun left a comment • edited Loading

Choose a reason for hiding this comment

panbingkun commented Jan 12, 2024

dongjoon-hyun left a comment

Choose a reason for hiding this comment

panbingkun commented Jan 12, 2024

dongjoon-hyun commented Jan 12, 2024

panbingkun commented Jan 12, 2024 • edited Loading

HyukjinKwon commented Jan 13, 2024

HyukjinKwon commented Jan 13, 2024

panbingkun commented Jan 13, 2024

panbingkun commented Jan 15, 2024

HyukjinKwon commented Jan 15, 2024

panbingkun commented Jan 15, 2024

HyukjinKwon commented Jan 15, 2024

panbingkun commented Jan 15, 2024

[SPARK-46692][BUILD] Load `inputs.envs` always in `Python` and `Upload`-related steps #44698

[SPARK-46692][BUILD] Load `inputs.envs` always in `Python` and `Upload`-related steps #44698

panbingkun commented Jan 12, 2024 •

edited

Loading

panbingkun commented Jan 12, 2024 •

edited

Loading

panbingkun commented Jan 12, 2024 •

edited

Loading

panbingkun commented Jan 12, 2024 •

edited

Loading

dongjoon-hyun left a comment •

edited

Loading

panbingkun commented Jan 12, 2024 •

edited

Loading