[SPARK-32419][PYTHON][BUILD] Avoid using subshell for Conda env (de)activation in pip packaging test

[HyukjinKwon]

What changes were proposed in this pull request?

This PR proposes to avoid using subshell when it activates Conda environment. Looks like it ends up with activating the env within the subshell even if you use conda command.

Why are the changes needed?

If you take a close look for GitHub Actions log:

 Installing dist into virtual env
Processing ./python/dist/pyspark-3.1.0.dev0.tar.gz
Collecting py4j==0.10.9
 Downloading py4j-0.10.9-py2.py3-none-any.whl (198 kB)
Using legacy setup.py install for pyspark, since package 'wheel' is not installed.
Installing collected packages: py4j, pyspark
 Running setup.py install for pyspark: started
 Running setup.py install for pyspark: finished with status 'done'
Successfully installed py4j-0.10.9 pyspark-3.1.0.dev0

...

Installing dist into virtual env
Obtaining file:///home/runner/work/spark/spark/python
Collecting py4j==0.10.9
 Downloading py4j-0.10.9-py2.py3-none-any.whl (198 kB)
Installing collected packages: py4j, pyspark
 Attempting uninstall: py4j
 Found existing installation: py4j 0.10.9
 Uninstalling py4j-0.10.9:
 Successfully uninstalled py4j-0.10.9
 Attempting uninstall: pyspark
 Found existing installation: pyspark 3.1.0.dev0
 Uninstalling pyspark-3.1.0.dev0:
 Successfully uninstalled pyspark-3.1.0.dev0
 Running setup.py develop for pyspark
Successfully installed py4j-0.10.9 pyspark

It looks not properly using Conda as it removes the previously installed one when it reinstalls again.
We should ideally test it with Conda environment as it's intended.

Does this PR introduce any user-facing change?

No, dev-only.

How was this patch tested?

GitHub Actions will test. I also manually tested in my local.

[HyukjinKwon]

Okay, seems it properly uses the activated Conda env.

Collecting py4j==0.10.9
  Downloading py4j-0.10.9-py2.py3-none-any.whl (198 kB)
Building wheels for collected packages: pyspark
  Building wheel for pyspark (setup.py): started
  Building wheel for pyspark (setup.py): finished with status 'done'
  Created wheel for pyspark: filename=pyspark-3.1.0.dev0-py2.py3-none-any.whl size=301489400 sha256=d5ddda7599a48b31099641a4bc71ace4838cfc4f615cb133c413b67bf0eb6304
  Stored in directory: /tmp/pip-ephem-wheel-cache-5pwgtgf2/wheels/42/ef/92/48cc19770d32e29bd58811e58bc68ee45511755f8cc987fcfa
Successfully built pyspark
...

Installing dist into virtual env
Obtaining file:///home/runner/work/spark/spark/python
Collecting py4j==0.10.9
  Downloading py4j-0.10.9-py2.py3-none-any.whl (198 kB)
Installing collected packages: py4j, pyspark
  Running setup.py develop for pyspark
Successfully installed py4j-0.10.9 pyspark

@dongjoon-hyun, sorry I missed this part and I believe this is the last touch for pip packaging test for now. Could you take a look please?

[HyukjinKwon]

BTW, this shouldn't affect Jenkins build but we can run a couple of times more just to make doubly sure.

[HyukjinKwon]

I tested as below in my local:

➜  spark git:(master) python
Python 3.7.7 (default, Mar 10 2020, 15:43:03)
[Clang 11.0.0 (clang-1100.0.33.17)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> exit(0)

➜  spark git:(master) source activate koalas-dev-env || (echo "a" && conda activate koalas-dev-env)
source: no such file or directory: activate
a

➜  spark git:(master) python
Python 3.7.7 (default, Mar 10 2020, 15:43:03)
[Clang 11.0.0 (clang-1100.0.33.17)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> exit(0)

➜  spark git:(master) source activate koalas-dev-env || conda activate koalas-dev-env
source: no such file or directory: activate

➜  spark git:(master) python
Python 3.6.10 |Anaconda, Inc.| (default, May  7 2020, 23:06:31)
[GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE_401/final)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> exit(0)

➜  spark git:(master) source deactivate || (echo "a" && conda deactivate)
source: no such file or directory: deactivate
a

➜  spark git:(master) python
Python 3.6.10 |Anaconda, Inc.| (default, May  7 2020, 23:06:31)
[GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE_401/final)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> exit(0)

➜  spark git:(master) source deactivate || conda deactivate
source: no such file or directory: deactivate

➜  spark git:(master) python
Python 3.7.7 (default, Mar 10 2020, 15:43:03)
[Clang 11.0.0 (clang-1100.0.33.17)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> exit(0)

[SparkQA]

Test build #126447 has finished for PR 29212 at commit 71193f1.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[HyukjinKwon]

retest this please

[HyukjinKwon]

retest this please

[SparkQA]

Test build #126474 has finished for PR 29212 at commit 71193f1.

• This patch fails due to an unknown error code, -9.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #126469 has finished for PR 29212 at commit 71193f1.

• This patch fails due to an unknown error code, -9.
• This patch merges cleanly.
• This patch adds no public classes.

[HyukjinKwon]

retest this please

[HyukjinKwon]

retest this please

[SparkQA]

Test build #126485 has finished for PR 29212 at commit 71193f1.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[HyukjinKwon]

retest this please

[SparkQA]

Test build #126493 has finished for PR 29212 at commit 71193f1.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #126495 has finished for PR 29212 at commit 71193f1.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[HyukjinKwon]

@dongjoon-hyun and @srowen, let me merge this to make GitHub Actions correctly tests pip packaging. At least it fixes the issue as shown above and I am pretty sure this fix is correct. This is also test-only and rather minor.

[HyukjinKwon]

Merged to master.

[dongjoon-hyun]

+1, late LGTM.