Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-17104][SQL] LogicalRelation.newInstance should follow the semantics of MultiInstanceRelation #14682

Closed
wants to merge 3 commits into from
Closed

[SPARK-17104][SQL] LogicalRelation.newInstance should follow the semantics of MultiInstanceRelation #14682

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
f9a8c76
Fix LogicalRelation.newInstance.
viirya Aug 17, 2016
e243323
Address comment.
viirya Aug 19, 2016
e7fe68b
Update the doc.
viirya Aug 20, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 9 additions & 2 deletions sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/LogicalRelation.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,11 +79,18 @@ case class LogicalRelation(
/** Used to lookup original attribute capitalization */
val attributeMap: AttributeMap[AttributeReference] = AttributeMap(output.map(o => (o, o)))

def newInstance(): this.type =
/**
* Returns a new instance of this LogicalRelation. According to the semantics of
* MultiInstanceRelation, this method returns a copy of this object with
* unique expression ids. We respect the `expectedOutputAttributes` and create
Copy link
Contributor

@cloud-fan cloud-fan Aug 19, 2016 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

doesn't expectedOutputAttributes.map(_.map(_.newInstance())) work?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Once expectedOutputAttributes is None, we still need to have new instances of relation's output. output already covers both relation's output and expectedOutputAttributes.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if expectedOutputAttributes is None, output is defined as relation.schema.toAttributes, we will get different expr ids.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh, yeah, you are right. It should work. Both are ok for me. I will update it.

* new instances of attributes in it.
*/
override def newInstance(): this.type = {
LogicalRelation(
relation,
expectedOutputAttributes,
expectedOutputAttributes.map(_.map(_.newInstance())),
metastoreTableIdentifier).asInstanceOf[this.type]
}

override def refresh(): Unit = relation match {
case fs: HadoopFsRelation => fs.refresh()
Expand Down
7 changes: 7 additions & 0 deletions sql/hive/src/test/scala/org/apache/spark/sql/hive/parquetSuites.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -589,6 +589,13 @@ class ParquetMetastoreSuite extends ParquetPartitioningTest {
}
}
}

test("self-join") {
val table = spark.table("normal_parquet")
val selfJoin = table.as("t1").join(table.as("t2"))
checkAnswer(selfJoin,
sql("SELECT * FROM normal_parquet x JOIN normal_parquet y"))
}
}

/**
Expand Down