Skip to content

Commit

Permalink
[SPARK-32491][INFRA] Do not install SparkR in test-only mode in testi…
Browse files Browse the repository at this point in the history
…ng script

This PR proposes to skip SparkR installation that is to run R linters (see SPARK-8505) in the test-only mode at `dev/run-tests.py` script.

As of SPARK-32292, the test-only mode in `dev/run-tests.py` was introduced, for example:

```
dev/run-tests.py --modules sql,core
```

which only runs the relevant tests and does not run other tests such as linters. Therefore, we don't need to install SparkR when `--modules` are specified.

GitHub Actions build is currently failed as below:

```
ERROR: this R is version 3.4.4, package 'SparkR' requires R >= 3.5
[error] running /home/runner/work/spark/spark/R/install-dev.sh ; received return code 1
```

For some reasons, looks GitHub Actions started to have R 3.4.4 installed by default; however, R 3.4 was dropped as of SPARK-32073.  When SparkR tests are not needed, GitHub Actions still builds SparkR with a low R version and it causes the test failure.

This PR partially fixes it by avoid the installation of SparkR.

No, dev-only.

GitHub Actions tests should run to confirm this fix is correct.

Closes apache#29300 from HyukjinKwon/install-r.

Authored-by: HyukjinKwon <[email protected]>
Signed-off-by: HyukjinKwon <[email protected]>
  • Loading branch information
HyukjinKwon committed Aug 19, 2020
1 parent d22539e commit a045944
Showing 1 changed file with 8 additions and 8 deletions.
16 changes: 8 additions & 8 deletions dev/run-tests.py
Original file line number Diff line number Diff line change
Expand Up @@ -559,13 +559,14 @@ def main():
" install one and retry.")
sys.exit(2)

java_version = determine_java_version(java_exe)

# install SparkR
if which("R"):
run_cmd([os.path.join(SPARK_HOME, "R", "install-dev.sh")])
else:
print("Cannot install SparkR as R was not found in PATH")
# Install SparkR
should_only_test_modules = opts.modules is not None
if not should_only_test_modules:
# If tests modules are specified, we will not run R linter.
if which("R"):
run_cmd([os.path.join(SPARK_HOME, "R", "install-dev.sh")])
else:
print("Cannot install SparkR as R was not found in PATH")

if os.environ.get("AMPLAB_JENKINS"):
# if we're on the Amplab Jenkins build servers setup variables
Expand All @@ -590,7 +591,6 @@ def main():
changed_modules = None
test_modules = None
changed_files = None
should_only_test_modules = opts.modules is not None
included_tags = []
excluded_tags = []
if should_only_test_modules:
Expand Down

0 comments on commit a045944

Please sign in to comment.