Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Find the optimal size for global and local number of threads #15

Closed
meisam opened this issue Oct 23, 2014 · 0 comments
Closed

Find the optimal size for global and local number of threads #15

meisam opened this issue Oct 23, 2014 · 0 comments
Assignees
@meisam
Labels
bug
Milestone
Baseline

Comments

@meisam
Copy link
Owner

meisam commented Oct 23, 2014

No description provided.

@meisam meisam self-assigned this Oct 23, 2014
@meisam meisam added this to the Baseline milestone Oct 23, 2014
@meisam meisam added the bug label Oct 29, 2014
meisam added a commit that referenced this issue Oct 29, 2014
@meisam
Adding a method to find the best number of threads based on input size
95f6953 
#15: Find the optimal size for global and local number of threads 

Task-Url: http://github.com/meisam/spark/issues/issue/15
meisam added a commit that referenced this issue Oct 29, 2014
@meisam
#15: Find the optimal size for global and local number of threads
a778eb3 
This commit fixes #15
Task-Url: http://github.com/meisam/spark/issues/issue/15
@meisam meisam closed this as completed Oct 29, 2014
meisam added a commit that referenced this issue Oct 29, 2014
@meisam
Eliminate passing the global and local thread group size to GpuPartition
375baea 
The best global and local size is infered basd on input size.
In future, we should infer it based on available hardware and input
size.

#15: Find the optimal size for global and local number of threads 

Task-Url: http://github.com/meisam/spark/issues/issue/15
meisam added a commit that referenced this issue Jan 15, 2015
@meisam
Adding a method to find the best number of threads based on input size
879a498 
#15: Find the optimal size for global and local number of threads

Task-Url: http://github.com/meisam/spark/issues/issue/15
meisam added a commit that referenced this issue Jan 15, 2015
@meisam
#15: Find the optimal size for global and local number of threads
7639006 
This commit fixes #15
Task-Url: http://github.com/meisam/spark/issues/issue/15
meisam added a commit that referenced this issue Jan 15, 2015
@meisam
Eliminate passing the global and local thread group size to GpuPartition
23c7a59 
The best global and local size is infered basd on input size.
In future, we should infer it based on available hardware and input
size.

#15: Find the optimal size for global and local number of threads

Task-Url: http://github.com/meisam/spark/issues/issue/15
meisam pushed a commit that referenced this issue Apr 16, 2015
@YanTangZhai @JoshRosen
[SPARK-4946] [CORE] Using AkkaUtils.askWithReply in MapOutputTracker.…
815de54 
…askTracker to reduce the chance of the communicating problem

Using AkkaUtils.askWithReply in MapOutputTracker.askTracker to reduce the chance of the communicating problem

Author: YanTangZhai <[email protected]>
Author: yantangzhai <[email protected]>

Closes apache#3785 from YanTangZhai/SPARK-4946 and squashes the following commits:

9ca6541 [yantangzhai] [SPARK-4946] [CORE] Using AkkaUtils.askWithReply in MapOutputTracker.askTracker to reduce the chance of the communicating problem
e4c2c0a [YanTangZhai] Merge pull request #15 from apache/master
718afeb [YanTangZhai] Merge pull request #12 from apache/master
6e643f8 [YanTangZhai] Merge pull request #11 from apache/master
e249846 [YanTangZhai] Merge pull request #10 from apache/master
d26d982 [YanTangZhai] Merge pull request #9 from apache/master
76d4027 [YanTangZhai] Merge pull request #8 from apache/master
03b62b0 [YanTangZhai] Merge pull request #7 from apache/master
8a00106 [YanTangZhai] Merge pull request #6 from apache/master
cbcba66 [YanTangZhai] Merge pull request #3 from apache/master
cdef539 [YanTangZhai] Merge pull request #1 from apache/master
meisam pushed a commit that referenced this issue Apr 16, 2015
@YanTangZhai @marmbrus
[SPARK-4692] [SQL] Support ! boolean logic operator like NOT
0ca51cc 
Support ! boolean logic operator like NOT in sql as follows
select * from for_test where !(col1 > col2)

Author: YanTangZhai <[email protected]>
Author: Michael Armbrust <[email protected]>

Closes apache#3555 from YanTangZhai/SPARK-4692 and squashes the following commits:

1a9f605 [YanTangZhai] Update HiveQuerySuite.scala
7c03c68 [YanTangZhai] Merge pull request #23 from apache/master
992046e [YanTangZhai] Update HiveQuerySuite.scala
ea618f4 [YanTangZhai] Update HiveQuerySuite.scala
192411d [YanTangZhai] Merge pull request #17 from YanTangZhai/master
e4c2c0a [YanTangZhai] Merge pull request #15 from apache/master
1e1ebb4 [YanTangZhai] Update HiveQuerySuite.scala
efc4210 [YanTangZhai] Update HiveQuerySuite.scala
bd2c444 [YanTangZhai] Update HiveQuerySuite.scala
1893956 [YanTangZhai] Merge pull request #14 from marmbrus/pr/3555
59e4de9 [Michael Armbrust] make hive test
718afeb [YanTangZhai] Merge pull request #12 from apache/master
950b21e [YanTangZhai] Update HiveQuerySuite.scala
74175b4 [YanTangZhai] Update HiveQuerySuite.scala
92242c7 [YanTangZhai] Update HiveQl.scala
6e643f8 [YanTangZhai] Merge pull request #11 from apache/master
e249846 [YanTangZhai] Merge pull request #10 from apache/master
d26d982 [YanTangZhai] Merge pull request #9 from apache/master
76d4027 [YanTangZhai] Merge pull request #8 from apache/master
03b62b0 [YanTangZhai] Merge pull request #7 from apache/master
8a00106 [YanTangZhai] Merge pull request #6 from apache/master
cbcba66 [YanTangZhai] Merge pull request #3 from apache/master
cdef539 [YanTangZhai] Merge pull request #1 from apache/master
meisam pushed a commit that referenced this issue Apr 16, 2015
@marmbrus
[SQL] Improve error messages
e3a975d 
Author: Michael Armbrust <[email protected]>
Author: wangfei <[email protected]>

Closes apache#4558 from marmbrus/errorMessages and squashes the following commits:

5e5ab50 [Michael Armbrust] Merge pull request #15 from scwf/errorMessages
fa38881 [wangfei] fix for grouping__id
f279a71 [wangfei] make right references for ScriptTransformation
d29fbde [Michael Armbrust] extra case
1a797b4 [Michael Armbrust] comments
d4e9015 [Michael Armbrust] add comment
af9e668 [Michael Armbrust] no braces
34eb3a4 [Michael Armbrust] more work
6197cd5 [Michael Armbrust] [SQL] Better error messages for analysis failures

(cherry picked from commit aa4ca8b)
Signed-off-by: Michael Armbrust <[email protected]>
meisam added a commit that referenced this issue Apr 16, 2015
@meisam
Adding a method to find the best number of threads based on input size
177d082 
#15: Find the optimal size for global and local number of threads

Task-Url: http://github.com/meisam/spark/issues/issue/15
meisam added a commit that referenced this issue Apr 16, 2015
@meisam
#15: Find the optimal size for global and local number of threads
bf05b9c 
This commit fixes #15
Task-Url: http://github.com/meisam/spark/issues/issue/15
meisam added a commit that referenced this issue Apr 16, 2015
@meisam
Eliminate passing the global and local thread group size to GpuPartition
0275197 
The best global and local size is infered basd on input size.
In future, we should infer it based on available hardware and input
size.

#15: Find the optimal size for global and local number of threads

Task-Url: http://github.com/meisam/spark/issues/issue/15
meisam pushed a commit that referenced this issue Jun 25, 2015
@rxin
[SPARK-7462] By default retain group by columns in aggregate
0a4844f 
Updated Java, Scala, Python, and R.

Author: Reynold Xin <[email protected]>
Author: Shivaram Venkataraman <[email protected]>

Closes apache#5996 from rxin/groupby-retain and squashes the following commits:

aac7119 [Reynold Xin] Merge branch 'groupby-retain' of github.com:rxin/spark into groupby-retain
f6858f6 [Reynold Xin] Merge branch 'master' into groupby-retain
5f923c0 [Reynold Xin] Merge pull request #15 from shivaram/sparkr-groupby-retrain
c1de670 [Shivaram Venkataraman] Revert workaround in SparkR to retain grouped cols Based on reverting code added in commit amplab-extras@9a6be74
b8b87e1 [Reynold Xin] Fixed DataFrameJoinSuite.
d910141 [Reynold Xin] Updated rest of the files
1e6e666 [Reynold Xin] [SPARK-7462] By default retain group by columns in aggregate
meisam pushed a commit that referenced this issue Jun 25, 2015
@pzzs @marmbrus
[SPARK-7824] [SQL] Collapse operator reordering and constant folding …
7914c72 
…into a single batch.

SQL
```
select * from tableA join tableB on (a > 3 and b = d) or (a > 3 and b = e)
```
Plan before modify
```
== Optimized Logical Plan ==
Project [a#293,b#294,c#295,d#296,e#297]
 Join Inner, Some(((a#293 > 3) && ((b#294 = d#296) || (b#294 = e#297))))
  MetastoreRelation default, tablea, None
  MetastoreRelation default, tableb, None
```
Plan after modify
```
== Optimized Logical Plan ==
Project [a#293,b#294,c#295,d#296,e#297]
 Join Inner, Some(((b#294 = d#296) || (b#294 = e#297)))
  Filter (a#293 > 3)
   MetastoreRelation default, tablea, None
  MetastoreRelation default, tableb, None
```

CombineLimits ==> Limit(If(LessThan(ne, le), ne, le), grandChild) and LessThan is in BooleanSimplification ,  so CombineLimits  must before BooleanSimplification and BooleanSimplification must before PushPredicateThroughJoin.

Author: Zhongshuai Pei <[email protected]>
Author: DoingDone9 <[email protected]>

Closes apache#6351 from DoingDone9/master and squashes the following commits:

20de7be [Zhongshuai Pei] Update Optimizer.scala
7bc7d28 [Zhongshuai Pei] Merge pull request #17 from apache/master
0ba5f42 [Zhongshuai Pei] Update Optimizer.scala
f8b9314 [Zhongshuai Pei] Update FilterPushdownSuite.scala
c529d9f [Zhongshuai Pei] Update FilterPushdownSuite.scala
ae3af6d [Zhongshuai Pei] Update FilterPushdownSuite.scala
a04ffae [Zhongshuai Pei] Update Optimizer.scala
11beb61 [Zhongshuai Pei] Update FilterPushdownSuite.scala
f2ee5fe [Zhongshuai Pei] Update Optimizer.scala
be6b1d5 [Zhongshuai Pei] Update Optimizer.scala
b01e622 [Zhongshuai Pei] Merge pull request #15 from apache/master
8df716a [Zhongshuai Pei] Update FilterPushdownSuite.scala
d98bc35 [Zhongshuai Pei] Update FilterPushdownSuite.scala
fa65718 [Zhongshuai Pei] Update Optimizer.scala
ab8e9a6 [Zhongshuai Pei] Merge pull request #14 from apache/master
14952e2 [Zhongshuai Pei] Merge pull request #13 from apache/master
f03fe7f [Zhongshuai Pei] Merge pull request #12 from apache/master
f12fa50 [Zhongshuai Pei] Merge pull request #10 from apache/master
f61210c [Zhongshuai Pei] Merge pull request #9 from apache/master
34b1a9a [Zhongshuai Pei] Merge pull request #8 from apache/master
802261c [DoingDone9] Merge pull request #7 from apache/master
d00303b [DoingDone9] Merge pull request #6 from apache/master
98b134f [DoingDone9] Merge pull request #5 from apache/master
161cae3 [DoingDone9] Merge pull request #4 from apache/master
c87e8b6 [DoingDone9] Merge pull request #3 from apache/master
cb1852d [DoingDone9] Merge pull request #2 from apache/master
c3f046f [DoingDone9] Merge pull request #1 from apache/master
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@meisam meisam

Labels
bug
Projects
None yet
Milestone
Baseline
Development

No branches or pull requests
1 participant
@meisam