Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Refactor GpuFilteredRDD and FilteredChunkItrator #14

Closed
meisam opened this issue Oct 23, 2014 · 0 comments
Closed

Refactor GpuFilteredRDD and FilteredChunkItrator #14

meisam opened this issue Oct 23, 2014 · 0 comments
Assignees
@meisam
Labels
bug enhancement
Milestone
Baseline

Comments

@meisam
Copy link
Owner

meisam commented Oct 23, 2014

No description provided.

@meisam meisam self-assigned this Oct 23, 2014
@meisam meisam added this to the Baseline milestone Oct 23, 2014
meisam added a commit that referenced this issue Oct 28, 2014
@meisam
#14: changing access modifier of currentPosition and currentChunk
369a9b9 
To refactor GpuFilteredRDD and FilteredChunkItrator

Task-Url: http://github.com/meisam/spark/issues/issue/14
meisam added a commit that referenced this issue Oct 28, 2014
@meisam
Extracting a new function for refilling currentChunk.
c18ee19 
This change make it possible to override guaranteeFill in subclasses

#14: Refactor GpuFilteredRDD and FilteredChunkItrator 

Task-Url: http://github.com/meisam/spark/issues/issue/14
meisam added a commit that referenced this issue Oct 28, 2014
@meisam
Adding a chunk cpacity parameter to GpuFilteredRDD
58e3dea 
#14: Refactor GpuFilteredRDD and FilteredChunkItrator 

Task-Url: http://github.com/meisam/spark/issues/issue/14
meisam added a commit that referenced this issue Oct 28, 2014
@meisam
Unstable intermidiate changes
bc00a5d 
A lot of renaming and moving.


#14: Refactor GpuFilteredRDD and FilteredChunkItrator 

Task-Url: http://github.com/meisam/spark/issues/issue/14
meisam added a commit that referenced this issue Oct 28, 2014
@meisam
[Unstable] organizing imports after moving classes and renaming them
afd0f48 
#14: Refactor GpuFilteredRDD and FilteredChunkItrator 

Task-Url: http://github.com/meisam/spark/issues/issue/14
meisam added a commit that referenced this issue Oct 28, 2014
@meisam
Using one class-wide OpenClContext for all methods in GpuPartition
448adf7 
#14: Refactor GpuFilteredRDD and FilteredChunkItrator 

Task-Url: http://github.com/meisam/spark/issues/issue/14
meisam added a commit that referenced this issue Oct 28, 2014
@meisam
Making GpuPartition functions genteric
b9bf7a0 
Dynamically find the type of an array based on T

#14: Refactor GpuFilteredRDD and FilteredChunkItrator 

Task-Url: http://github.com/meisam/spark/issues/issue/14
meisam added a commit that referenced this issue Oct 28, 2014
@meisam
[Unstable] Minor fixes in GpuFilteredRDDSuit
62a5919 
#14: Refactor GpuFilteredRDD and FilteredChunkItrator 

Task-Url: http://github.com/meisam/spark/issues/issue/14
meisam added a commit that referenced this issue Oct 28, 2014
@meisam
Uing Scala's new reflection API to create type aware pointers to buffers
808dfc8 
#14: Refactor GpuFilteredRDD and FilteredChunkItrator 

Task-Url: http://github.com/meisam/spark/issues/issue/14
meisam added a commit that referenced this issue Oct 28, 2014
@meisam
Removing the pointer generator for primitive types
9add613 
Having two generic methods with the same name causes compiler errors.
Type erasure in scala removes information about generic types.

#14: Refactor GpuFilteredRDD and FilteredChunkItrator 

Task-Url: http://github.com/meisam/spark/issues/issue/14
meisam added a commit that referenced this issue Oct 28, 2014
@meisam
Fixing two test cases in GpuFilteredRDDSuit after refactoring.
aaee138 
Major Gpu operations are now in GpuPartition and not in GpuFilteredRDD
anymore. So all the test cases should be adjusted accordingly.


#14: Refactor GpuFilteredRDD and FilteredChunkItrator 

Task-Url: http://github.com/meisam/spark/issues/issue/14
meisam added a commit that referenced this issue Oct 28, 2014
@meisam
Updating a few test cases for GpuFilteredRDD
ce6fb21 
#14: Refactor GpuFilteredRDD and FilteredChunkItrator 

Task-Url: http://github.com/meisam/spark/issues/issue/14
meisam added a commit that referenced this issue Oct 28, 2014
@meisam
fixing bus in one of the testcases
89f905f 
#14: Refactor GpuFilteredRDD and FilteredChunkItrator 

Task-Url: http://github.com/meisam/spark/issues/issue/14
meisam added a commit that referenced this issue Oct 28, 2014
@meisam
Adding a note for fixing String column data later
474b729 
#14: Refactor GpuFilteredRDD and FilteredChunkItrator 

Task-Url: http://github.com/meisam/spark/issues/issue/14
meisam added a commit that referenced this issue Oct 28, 2014
@meisam
Adding a function to return the appropriate column based on its index
1916803 
#14: Refactor GpuFilteredRDD and FilteredChunkItrator 

Task-Url: http://github.com/meisam/spark/issues/issue/14
meisam added a commit that referenced this issue Oct 28, 2014
@meisam
Adding a class and an enum for comparissons in selections
b976f53 
#14: Refactor GpuFilteredRDD and FilteredChunkItrator 

Task-Url: http://github.com/meisam/spark/issues/issue/14
meisam added a commit that referenced this issue Oct 28, 2014
@meisam
[unstable] Adding a function do generic filter operation on data.
a02d30d 
#14: Refactor GpuFilteredRDD and FilteredChunkItrator 

Task-Url: http://github.com/meisam/spark/issues/issue/14
meisam added a commit that referenced this issue Oct 29, 2014
@meisam
Adding a method to find the size of an element.
b85519e 
#14: Refactor GpuFilteredRDD and FilteredChunkItrator 

Task-Url: http://github.com/meisam/spark/issues/issue/14
meisam added a commit that referenced this issue Oct 29, 2014
@meisam
Adding a function to return the name of a primitive type.
cf532d3 
The name of a type is used for fonding the appropriate opencl kernel
name.

#14: Refactor GpuFilteredRDD and FilteredChunkItrator 

Task-Url: http://github.com/meisam/spark/issues/issue/14
meisam added a commit that referenced this issue Oct 29, 2014
@meisam
Clean up: inlining a local variable and deleting it.
ea21cf0 
#14: Refactor GpuFilteredRDD and FilteredChunkItrator 

Task-Url: http://github.com/meisam/spark/issues/issue/14
meisam added a commit that referenced this issue Oct 29, 2014
@meisam
Rewriting the filter method.
915265b 
The old implementation was slow. This rewrite makes it a lot faster.

The new version is based on Yuan's implementation, which itself is based
on Cuda's implementation.

#14: Refactor GpuFilteredRDD and FilteredChunkItrator 

Task-Url: http://github.com/meisam/spark/issues/issue/14
meisam added a commit that referenced this issue Oct 29, 2014
@meisam
Fixing test cases based on the latest changes in GpuPartition
0fb0044 
#14: Refactor GpuFilteredRDD and FilteredChunkItrator 

Task-Url: http://github.com/meisam/spark/issues/issue/14
@meisam meisam closed this as completed Oct 30, 2014
meisam added a commit that referenced this issue Oct 30, 2014
@meisam
Sending the correct offset to the file.
dbee6e8 
#14: Refactor GpuFilteredRDD and FilteredChunkItrator 

Task-Url: http://github.com/meisam/spark/issues/issue/14
meisam added a commit that referenced this issue Oct 30, 2014
@meisam
Fixing the offset passed to deviceToHostCopy
045cd1c 
#14: Refactor GpuFilteredRDD and FilteredChunkItrator 

Task-Url: http://github.com/meisam/spark/issues/issue/14
meisam added a commit that referenced this issue Oct 30, 2014
@meisam
Updating test cases for GpuFilteredPartition after refactoring
4208fb4 
#14: Refactor GpuFilteredRDD and FilteredChunkItrator 

Task-Url: http://github.com/meisam/spark/issues/issue/14
meisam added a commit that referenced this issue Jan 15, 2015
@meisam
changing one missed "eq" comparison to "eql" comparison
58ed654 
#14: Refactor GpuFilteredRDD and FilteredChunkItrator

Task-Url: http://github.com/meisam/spark/issues/issue/14
meisam pushed a commit that referenced this issue Apr 16, 2015
@YanTangZhai @marmbrus
[SPARK-4692] [SQL] Support ! boolean logic operator like NOT
0ca51cc 
Support ! boolean logic operator like NOT in sql as follows
select * from for_test where !(col1 > col2)

Author: YanTangZhai <[email protected]>
Author: Michael Armbrust <[email protected]>

Closes apache#3555 from YanTangZhai/SPARK-4692 and squashes the following commits:

1a9f605 [YanTangZhai] Update HiveQuerySuite.scala
7c03c68 [YanTangZhai] Merge pull request #23 from apache/master
992046e [YanTangZhai] Update HiveQuerySuite.scala
ea618f4 [YanTangZhai] Update HiveQuerySuite.scala
192411d [YanTangZhai] Merge pull request #17 from YanTangZhai/master
e4c2c0a [YanTangZhai] Merge pull request #15 from apache/master
1e1ebb4 [YanTangZhai] Update HiveQuerySuite.scala
efc4210 [YanTangZhai] Update HiveQuerySuite.scala
bd2c444 [YanTangZhai] Update HiveQuerySuite.scala
1893956 [YanTangZhai] Merge pull request #14 from marmbrus/pr/3555
59e4de9 [Michael Armbrust] make hive test
718afeb [YanTangZhai] Merge pull request #12 from apache/master
950b21e [YanTangZhai] Update HiveQuerySuite.scala
74175b4 [YanTangZhai] Update HiveQuerySuite.scala
92242c7 [YanTangZhai] Update HiveQl.scala
6e643f8 [YanTangZhai] Merge pull request #11 from apache/master
e249846 [YanTangZhai] Merge pull request #10 from apache/master
d26d982 [YanTangZhai] Merge pull request #9 from apache/master
76d4027 [YanTangZhai] Merge pull request #8 from apache/master
03b62b0 [YanTangZhai] Merge pull request #7 from apache/master
8a00106 [YanTangZhai] Merge pull request #6 from apache/master
cbcba66 [YanTangZhai] Merge pull request #3 from apache/master
cdef539 [YanTangZhai] Merge pull request #1 from apache/master
meisam added a commit that referenced this issue Apr 16, 2015
@meisam
#14: changing access modifier of currentPosition and currentChunk
960ce31 
To refactor GpuFilteredRDD and FilteredChunkItrator

Task-Url: http://github.com/meisam/spark/issues/issue/14
meisam added a commit that referenced this issue Apr 16, 2015
@meisam
Extracting a new function for refilling currentChunk.
dd87b47 
This change make it possible to override guaranteeFill in subclasses

#14: Refactor GpuFilteredRDD and FilteredChunkItrator

Task-Url: http://github.com/meisam/spark/issues/issue/14
meisam added a commit that referenced this issue Apr 16, 2015
@meisam
Adding a chunk cpacity parameter to GpuFilteredRDD
5b81b99 
#14: Refactor GpuFilteredRDD and FilteredChunkItrator

Task-Url: http://github.com/meisam/spark/issues/issue/14
meisam added a commit that referenced this issue Apr 16, 2015
@meisam
Unstable intermidiate changes
1ba693e 
A lot of renaming and moving.

#14: Refactor GpuFilteredRDD and FilteredChunkItrator

Task-Url: http://github.com/meisam/spark/issues/issue/14
meisam added a commit that referenced this issue Apr 16, 2015
@meisam
[Unstable] organizing imports after moving classes and renaming them
3d894e8 
#14: Refactor GpuFilteredRDD and FilteredChunkItrator

Task-Url: http://github.com/meisam/spark/issues/issue/14
meisam added a commit that referenced this issue Apr 16, 2015
@meisam
Using one class-wide OpenClContext for all methods in GpuPartition
cd98567 
#14: Refactor GpuFilteredRDD and FilteredChunkItrator

Task-Url: http://github.com/meisam/spark/issues/issue/14
meisam added a commit that referenced this issue Apr 16, 2015
@meisam
Making GpuPartition functions genteric
48bdd31 
Dynamically find the type of an array based on T

#14: Refactor GpuFilteredRDD and FilteredChunkItrator

Task-Url: http://github.com/meisam/spark/issues/issue/14
meisam added a commit that referenced this issue Apr 16, 2015
@meisam
[Unstable] Minor fixes in GpuFilteredRDDSuit
2fbca3f 
#14: Refactor GpuFilteredRDD and FilteredChunkItrator

Task-Url: http://github.com/meisam/spark/issues/issue/14
meisam added a commit that referenced this issue Apr 16, 2015
@meisam
Uing Scala's new reflection API to create type aware pointers to buffers
4e21b61 
#14: Refactor GpuFilteredRDD and FilteredChunkItrator

Task-Url: http://github.com/meisam/spark/issues/issue/14
meisam added a commit that referenced this issue Apr 16, 2015
@meisam
Removing the pointer generator for primitive types
4cc91a3 
Having two generic methods with the same name causes compiler errors.
Type erasure in scala removes information about generic types.

#14: Refactor GpuFilteredRDD and FilteredChunkItrator

Task-Url: http://github.com/meisam/spark/issues/issue/14
meisam added a commit that referenced this issue Apr 16, 2015
@meisam
Fixing two test cases in GpuFilteredRDDSuit after refactoring.
5be5b00 
Major Gpu operations are now in GpuPartition and not in GpuFilteredRDD
anymore. So all the test cases should be adjusted accordingly.

#14: Refactor GpuFilteredRDD and FilteredChunkItrator

Task-Url: http://github.com/meisam/spark/issues/issue/14
meisam added a commit that referenced this issue Apr 16, 2015
@meisam
Updating a few test cases for GpuFilteredRDD
ff24286 
#14: Refactor GpuFilteredRDD and FilteredChunkItrator

Task-Url: http://github.com/meisam/spark/issues/issue/14
meisam added a commit that referenced this issue Apr 16, 2015
@meisam
fixing bus in one of the testcases
e99ccac 
#14: Refactor GpuFilteredRDD and FilteredChunkItrator

Task-Url: http://github.com/meisam/spark/issues/issue/14
meisam added a commit that referenced this issue Apr 16, 2015
@meisam
Adding a note for fixing String column data later
82a6a37 
#14: Refactor GpuFilteredRDD and FilteredChunkItrator

Task-Url: http://github.com/meisam/spark/issues/issue/14
meisam added a commit that referenced this issue Apr 16, 2015
@meisam
Adding a function to return the appropriate column based on its index
8dc6f24 
#14: Refactor GpuFilteredRDD and FilteredChunkItrator

Task-Url: http://github.com/meisam/spark/issues/issue/14
meisam added a commit that referenced this issue Apr 16, 2015
@meisam
Adding a class and an enum for comparissons in selections
6b48141 
#14: Refactor GpuFilteredRDD and FilteredChunkItrator

Task-Url: http://github.com/meisam/spark/issues/issue/14
meisam added a commit that referenced this issue Apr 16, 2015
@meisam
[unstable] Adding a function do generic filter operation on data.
9831884 
#14: Refactor GpuFilteredRDD and FilteredChunkItrator

Task-Url: http://github.com/meisam/spark/issues/issue/14
meisam added a commit that referenced this issue Apr 16, 2015
@meisam
Adding a method to find the size of an element.
6466c4c 
#14: Refactor GpuFilteredRDD and FilteredChunkItrator

Task-Url: http://github.com/meisam/spark/issues/issue/14
meisam added a commit that referenced this issue Apr 16, 2015
@meisam
Adding a function to return the name of a primitive type.
26a5acf 
The name of a type is used for fonding the appropriate opencl kernel
name.

#14: Refactor GpuFilteredRDD and FilteredChunkItrator

Task-Url: http://github.com/meisam/spark/issues/issue/14
meisam added a commit that referenced this issue Apr 16, 2015
@meisam
Clean up: inlining a local variable and deleting it.
a82aef5 
#14: Refactor GpuFilteredRDD and FilteredChunkItrator

Task-Url: http://github.com/meisam/spark/issues/issue/14
meisam added a commit that referenced this issue Apr 16, 2015
@meisam
Rewriting the filter method.
c9284cf 
The old implementation was slow. This rewrite makes it a lot faster.

The new version is based on Yuan's implementation, which itself is based
on Cuda's implementation.

#14: Refactor GpuFilteredRDD and FilteredChunkItrator

Task-Url: http://github.com/meisam/spark/issues/issue/14
meisam added a commit that referenced this issue Apr 16, 2015
@meisam
Fixing test cases based on the latest changes in GpuPartition
085e2d6 
#14: Refactor GpuFilteredRDD and FilteredChunkItrator

Task-Url: http://github.com/meisam/spark/issues/issue/14
meisam added a commit that referenced this issue Apr 16, 2015
@meisam
Sending the correct offset to the file.
318a9c2 
#14: Refactor GpuFilteredRDD and FilteredChunkItrator

Task-Url: http://github.com/meisam/spark/issues/issue/14
meisam added a commit that referenced this issue Apr 16, 2015
@meisam
Fixing the offset passed to deviceToHostCopy
4a26cc5 
#14: Refactor GpuFilteredRDD and FilteredChunkItrator

Task-Url: http://github.com/meisam/spark/issues/issue/14
meisam added a commit that referenced this issue Apr 16, 2015
@meisam
Updating test cases for GpuFilteredPartition after refactoring
62d8aa9 
#14: Refactor GpuFilteredRDD and FilteredChunkItrator

Task-Url: http://github.com/meisam/spark/issues/issue/14
meisam added a commit that referenced this issue Apr 16, 2015
@meisam
rewriting gpu filtered partition
c141e40 
#14: Refactor GpuFilteredRDD and FilteredChunkItrator

Task-Url: http://github.com/meisam/spark/issues/issue/14
meisam added a commit that referenced this issue Apr 16, 2015
@meisam
changing one missed "eq" comparison to "eql" comparison
b1eba46 
#14: Refactor GpuFilteredRDD and FilteredChunkItrator

Task-Url: http://github.com/meisam/spark/issues/issue/14
meisam pushed a commit that referenced this issue Jun 25, 2015
@pzzs @marmbrus
[SPARK-7824] [SQL] Collapse operator reordering and constant folding …
7914c72 
…into a single batch.

SQL
```
select * from tableA join tableB on (a > 3 and b = d) or (a > 3 and b = e)
```
Plan before modify
```
== Optimized Logical Plan ==
Project [a#293,b#294,c#295,d#296,e#297]
 Join Inner, Some(((a#293 > 3) && ((b#294 = d#296) || (b#294 = e#297))))
  MetastoreRelation default, tablea, None
  MetastoreRelation default, tableb, None
```
Plan after modify
```
== Optimized Logical Plan ==
Project [a#293,b#294,c#295,d#296,e#297]
 Join Inner, Some(((b#294 = d#296) || (b#294 = e#297)))
  Filter (a#293 > 3)
   MetastoreRelation default, tablea, None
  MetastoreRelation default, tableb, None
```

CombineLimits ==> Limit(If(LessThan(ne, le), ne, le), grandChild) and LessThan is in BooleanSimplification ,  so CombineLimits  must before BooleanSimplification and BooleanSimplification must before PushPredicateThroughJoin.

Author: Zhongshuai Pei <[email protected]>
Author: DoingDone9 <[email protected]>

Closes apache#6351 from DoingDone9/master and squashes the following commits:

20de7be [Zhongshuai Pei] Update Optimizer.scala
7bc7d28 [Zhongshuai Pei] Merge pull request #17 from apache/master
0ba5f42 [Zhongshuai Pei] Update Optimizer.scala
f8b9314 [Zhongshuai Pei] Update FilterPushdownSuite.scala
c529d9f [Zhongshuai Pei] Update FilterPushdownSuite.scala
ae3af6d [Zhongshuai Pei] Update FilterPushdownSuite.scala
a04ffae [Zhongshuai Pei] Update Optimizer.scala
11beb61 [Zhongshuai Pei] Update FilterPushdownSuite.scala
f2ee5fe [Zhongshuai Pei] Update Optimizer.scala
be6b1d5 [Zhongshuai Pei] Update Optimizer.scala
b01e622 [Zhongshuai Pei] Merge pull request #15 from apache/master
8df716a [Zhongshuai Pei] Update FilterPushdownSuite.scala
d98bc35 [Zhongshuai Pei] Update FilterPushdownSuite.scala
fa65718 [Zhongshuai Pei] Update Optimizer.scala
ab8e9a6 [Zhongshuai Pei] Merge pull request #14 from apache/master
14952e2 [Zhongshuai Pei] Merge pull request #13 from apache/master
f03fe7f [Zhongshuai Pei] Merge pull request #12 from apache/master
f12fa50 [Zhongshuai Pei] Merge pull request #10 from apache/master
f61210c [Zhongshuai Pei] Merge pull request #9 from apache/master
34b1a9a [Zhongshuai Pei] Merge pull request #8 from apache/master
802261c [DoingDone9] Merge pull request #7 from apache/master
d00303b [DoingDone9] Merge pull request #6 from apache/master
98b134f [DoingDone9] Merge pull request #5 from apache/master
161cae3 [DoingDone9] Merge pull request #4 from apache/master
c87e8b6 [DoingDone9] Merge pull request #3 from apache/master
cb1852d [DoingDone9] Merge pull request #2 from apache/master
c3f046f [DoingDone9] Merge pull request #1 from apache/master
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@meisam meisam

Labels
bug enhancement
Projects
None yet
Milestone
Baseline
Development

No branches or pull requests
1 participant
@meisam