[SPARK-7017][HOTFIX][Project Infra]: Refactor dev/run-tests into Python #6865
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ode as docs will be built with 'jekyll' and not any calls through 'sbt'
| @@ -231,7 +231,7 @@ def get_hadoop_profiles(hadoop_version): | |||
| sys.exit(int(os.environ.get("CURRENT_BLOCK", 255))) | |||
|
|
|||
|
|
|||
| def get_build_profiles(hadoop_version="hadoop2.3", | |||
| def get_build_profiles(hadoop_version, | |||
| enable_base_profiles=True, | |||
| enable_hive_profiles=False, | |||
| enable_doc_profiles=False): | |||
There was a problem hiding this comment.
If you want to remove the doc_profiles then you'll also have to remove this since it's now unused.
|
Hey @brennonyork, I'm about to submit a large PR of my own to do a bunch of refactoring / modularization in this file, so I'll be sure to incorporate these changes. |
|
Test build #35067 has finished for PR 6865 at commit
|
|
I think we can close this since it's subsumed by #6866. |
|
Closing in favor of #6866 |
JoshRosen
added a commit
to JoshRosen/spark
that referenced
this pull request
Jun 18, 2015
dbtsai
pushed a commit
to dbtsai/spark
that referenced
this pull request
Jun 20, 2015
…run-tests This patch builds upon apache#5694 to add a 'module' abstraction to the `dev/run-tests` script which groups together the per-module test logic, including the mapping from file paths to modules, the mapping from modules to test goals and build profiles, and the dependencies / relationships between modules. This refactoring makes it much easier to increase the granularity of test modules, which will let us skip even more tests. It's also a prerequisite for other changes that will reduce test time, such as running subsets of the Python tests based on which files / modules have changed. This patch also adds doctests for the new graph traversal / change mapping code. Author: Josh Rosen <[email protected]> Closes apache#6866 from JoshRosen/more-dev-run-tests-refactoring and squashes the following commits: 75de450 [Josh Rosen] Use module system to determine which build profiles to enable. 4224da5 [Josh Rosen] Add documentation to Module. a86a953 [Josh Rosen] Clean up modules; add new modules for streaming external projects e46539f [Josh Rosen] Fix camel-cased endswith() 35a3052 [Josh Rosen] Enable Hive tests when running all tests df10e23 [Josh Rosen] update to reflect fact that no module depends on root 3670d50 [Josh Rosen] mllib should depend on streaming dc6f1c6 [Josh Rosen] Use changed files' extensions to decide whether to run style checks 7092d3e [Josh Rosen] Skip SBT tests if no test goals are specified 43a0ced [Josh Rosen] Minor fixes 3371441 [Josh Rosen] Test everything if nothing has changed (needed for non-PRB builds) 37f3fb3 [Josh Rosen] Remove doc profiles option, since it's not actually needed (see apache#6865) f53864b [Josh Rosen] Finish integrating module changes f0249bd [Josh Rosen] WIP
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixed minor nits from the previous PR and removed unnecessary doc build code as docs will be built with 'jekyll' and not any calls through 'sbt' (i.e. the
get_build_profilesfunction)./cc @JoshRosen