Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

updated ec2 instance types #6014

Closed
wants to merge 6 commits into from
Closed

updated ec2 instance types #6014

wants to merge 6 commits into from

Conversation

brendancol
Copy link

I needed to run some d2 instances, so I updated the spark_ec2.py accordingly

@AmplabJenkins
Copy link

Can one of the admins verify this patch?

@srowen
Copy link
Member

srowen commented May 8, 2015

CC @nchammas and @shivaram who might know whether these instances should work
This PR loses the nice lineup of numbers and changes a bunch of lines unnecessarily. Can you undo the whitespace-only changes?

Brendan Collins added 3 commits May 8, 2015 15:20
vertically aligned whitespace
85c6236
vertically aligned whitespace
80c2aa6
updated ec2 instance types
fc2931f 
vertically aligned whitespace

vertically aligned whitespace
shivaram
shivaram reviewed May 8, 2015
View reviewed changes
ec2/spark_ec2.py
"c3.2xlarge": 2,
"c3.4xlarge": 2,
"c3.8xlarge": 2,
"c3.large": 2,
"c3.xlarge": 2,
"cc1.4xlarge": 2,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like cc1.4xlarge has been removed - Is that intentional ?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah so I was going off of:

http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/InstanceStorage.html

Its absense may be an error, or intentional from AWS. I've seen:

EC2 is steering support away from cc1.4xlarge instances and toward
cc2.8xlarge instances. The cc2.8xlarge instance supports additional
features such as VPC and availability in all regions. When creating
instances in the RightScale Dashboard, use cc2.8xlarge instances instead of
cc1.4xlarge.

On Fri, May 8, 2015 at 3:36 PM, Shivaram Venkataraman <
[email protected]> wrote:

In ec2/spark_ec2.py
#6014 (comment):

     "c3.2xlarge":  2,
     "c3.4xlarge":  2,
     "c3.8xlarge":  2,

  •    "c3.large":    2,

  •    "c3.xlarge":   2,

  •    "cc1.4xlarge": 2,

Looks like cc1.4xlarge has been removed - Is that intentional ?

—
Reply to this email directly or view it on GitHub
https://github.com/apache/spark/pull/6014/files#r29969797.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm - I can still choose cc1.4xlarge as an instance type in the launch wizard in us-east-1. While its not recommended there might be some users using it -- Lets keep it around for some more time.

Brendan Collins added 2 commits May 8, 2015 15:42
added back cc1.4xlarge
6366c45
Merge branch 'ec2-instance-types-update' of github.com:brendancol/spa…
d7b4191 
…rk into ec2-instance-types-update

Conflicts:
	ec2/spark_ec2.py
@shivaram
Copy link
Contributor

shivaram commented May 8, 2015

Thanks @brendancol - LGTM

@shivaram
Copy link
Contributor

shivaram commented May 8, 2015

Jenkins, ok to test

@AmplabJenkins
Copy link

Merged build triggered.

@AmplabJenkins
Copy link

Merged build started.

@SparkQA
Copy link

SparkQA commented May 8, 2015

Test build #32260 has started for PR 6014 at commit d7b4191.

@SparkQA
Copy link

SparkQA commented May 8, 2015

Test build #32260 has finished for PR 6014 at commit d7b4191.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

Merged build finished. Test PASSed.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32260/
Test PASSed.

@nchammas
Copy link
Contributor

nchammas commented May 8, 2015

I wish there was a way to simplify this code.

As it is, we are still prone to silent duplicate entries in these dictionaries (something which has happened in the past; see t1.micro), and having to update these dictionaries as Amazon changes stuff is no fun.

Anyway, I'm just whining... Don't have a solution to offer at the moment. ¯_(ツ)_/¯

@shivaram
Copy link
Contributor

shivaram commented May 8, 2015

Alright I'm merging this. @nchammas we can brainstorm more solutions to this in a new JIRA if we have better ideas

asfgit pushed a commit that referenced this pull request May 8, 2015
@shivaram
updated ec2 instance types
6e35cb5 
I needed to run some d2 instances, so I updated the spark_ec2.py accordingly

Author: Brendan Collins <[email protected]>

Closes #6014 from brendancol/ec2-instance-types-update and squashes the following commits:

d7b4191 [Brendan Collins] Merge branch 'ec2-instance-types-update' of github.com:brendancol/spark into ec2-instance-types-update
6366c45 [Brendan Collins] added back cc1.4xlarge
fc2931f [Brendan Collins] updated ec2 instance types
80c2aa6 [Brendan Collins] vertically aligned whitespace
85c6236 [Brendan Collins] vertically aligned whitespace
1657c26 [Brendan Collins] updated ec2 instance types

(cherry picked from commit 1c78f68)
Signed-off-by: Shivaram Venkataraman <[email protected]>
@asfgit asfgit closed this in 1c78f68 May 8, 2015
@shivaram shivaram mentioned this pull request May 9, 2015
Closed
@jnordling
Copy link

Nice, #6020

jeanlyn pushed a commit to jeanlyn/spark that referenced this pull request May 28, 2015
@jeanlyn
updated ec2 instance types
d591eb1 
I needed to run some d2 instances, so I updated the spark_ec2.py accordingly

Author: Brendan Collins <[email protected]>

Closes apache#6014 from brendancol/ec2-instance-types-update and squashes the following commits:

d7b4191 [Brendan Collins] Merge branch 'ec2-instance-types-update' of github.com:brendancol/spark into ec2-instance-types-update
6366c45 [Brendan Collins] added back cc1.4xlarge
fc2931f [Brendan Collins] updated ec2 instance types
80c2aa6 [Brendan Collins] vertically aligned whitespace
85c6236 [Brendan Collins] vertically aligned whitespace
1657c26 [Brendan Collins] updated ec2 instance types
jeanlyn pushed a commit to jeanlyn/spark that referenced this pull request Jun 12, 2015
@jeanlyn
updated ec2 instance types
bc27a34 
I needed to run some d2 instances, so I updated the spark_ec2.py accordingly

Author: Brendan Collins <[email protected]>

Closes apache#6014 from brendancol/ec2-instance-types-update and squashes the following commits:

d7b4191 [Brendan Collins] Merge branch 'ec2-instance-types-update' of github.com:brendancol/spark into ec2-instance-types-update
6366c45 [Brendan Collins] added back cc1.4xlarge
fc2931f [Brendan Collins] updated ec2 instance types
80c2aa6 [Brendan Collins] vertically aligned whitespace
85c6236 [Brendan Collins] vertically aligned whitespace
1657c26 [Brendan Collins] updated ec2 instance types
nemccarthy pushed a commit to nemccarthy/spark that referenced this pull request Jun 19, 2015
@nemccarthy
updated ec2 instance types
03970a8 
I needed to run some d2 instances, so I updated the spark_ec2.py accordingly

Author: Brendan Collins <[email protected]>

Closes apache#6014 from brendancol/ec2-instance-types-update and squashes the following commits:

d7b4191 [Brendan Collins] Merge branch 'ec2-instance-types-update' of github.com:brendancol/spark into ec2-instance-types-update
6366c45 [Brendan Collins] added back cc1.4xlarge
fc2931f [Brendan Collins] updated ec2 instance types
80c2aa6 [Brendan Collins] vertically aligned whitespace
85c6236 [Brendan Collins] vertically aligned whitespace
1657c26 [Brendan Collins] updated ec2 instance types
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
7 participants
@brendancol @AmplabJenkins @srowen @shivaram @SparkQA @nchammas @jnordling