[SPARK-10446][SQL] Support to specify join type when calling join with usingColumns #8600
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Test build #41997 has finished for PR 8600 at commit
|
|
Test build #42001 has finished for PR 8600 at commit
|
| * @group dfops | ||
| * @since 1.4.0 | ||
| */ | ||
| def join(right: DataFrame, usingColumns: Seq[String]): DataFrame = { | ||
| def join(right: DataFrame, usingColumns: Seq[String], joinType: String = "inner"): DataFrame = { |
There was a problem hiding this comment.
We cannot use default parameter values in order to maintain compatibility with Java. You can add an extra method.
|
Test build #42058 has finished for PR 8600 at commit
|
|
ping @rxin |
|
Thanks - I've merged this. |
ghost
pushed a commit
to dbtsai/spark
that referenced
this pull request
Dec 28, 2015
…i-Join After reading the JIRA https://issues.apache.org/jira/browse/SPARK-12520, I double checked the code. For example, users can do the Equi-Join like ```df.join(df2, 'name', 'outer').select('name', 'height').collect()``` - There exists a bug in 1.5 and 1.4. The code just ignores the third parameter (join type) users pass. However, the join type we called is `Inner`, even if the user-specified type is the other type (e.g., `Outer`). - After a PR: apache#8600, the 1.6 does not have such an issue, but the description has not been updated. Plan to submit another PR to fix 1.5 and issue an error message if users specify a non-inner join type when using Equi-Join. Author: gatorsmile <[email protected]> Closes apache#10477 from gatorsmile/pyOuterJoin.
asfgit
pushed a commit
that referenced
this pull request
Dec 28, 2015
…i-Join After reading the JIRA https://issues.apache.org/jira/browse/SPARK-12520, I double checked the code. For example, users can do the Equi-Join like ```df.join(df2, 'name', 'outer').select('name', 'height').collect()``` - There exists a bug in 1.5 and 1.4. The code just ignores the third parameter (join type) users pass. However, the join type we called is `Inner`, even if the user-specified type is the other type (e.g., `Outer`). - After a PR: #8600, the 1.6 does not have such an issue, but the description has not been updated. Plan to submit another PR to fix 1.5 and issue an error message if users specify a non-inner join type when using Equi-Join. Author: gatorsmile <[email protected]> Closes #10477 from gatorsmile/pyOuterJoin.
|
How can we combine two columns with different values? |
|
nvm. USING join can support outer join types, but we are unable to treat them as actual outer join. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
JIRA: https://issues.apache.org/jira/browse/SPARK-10446
Currently the method
join(right: DataFrame, usingColumns: Seq[String])only supports inner join. It is more convenient to have it support other join types.