-
Notifications
You must be signed in to change notification settings - Fork 28.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Bug fix for SPARK-5242: "ec2/spark_ec2.py lauch" does not work with VPC if no public DNS or IP is available #4038
Conversation
Can one of the admins verify this patch? |
ok to test. @nchammas |
Test build #25585 has started for PR 4038 at commit
|
Test build #25585 has finished for PR 4038 at commit
|
Test PASSed. |
Reported bug confirmed. Now testing fix. |
Hmm, it seems that setting up an environment to test this fix is non-trivial. @voukka Could you share a minimal VPC and subnet setup that I can duplicate to test this fix? Is it something similar to the scenario described here? |
@nchammas yes, http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Scenario2.html should be sufficient to test this bug. After you followed that guide, start new instance in VPC, ssh to it (though NAT server) and launch cluster as in https://issues.apache.org/jira/browse/SPARK-5242. The setup where I strive to launch the cluster and found this bug is closer to http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Scenario3.html, which differs from Scenario 2 by VPN connection to corporate network. |
cc @shivaram I haven't had a chance to look at this more closely yet, and likely won't until next weekend. |
@voukka @nchammas - This high level goal looks fine to me. However I the function get_hostname is being called on all instances (its inside a loop) in many cases. I wonder if we can do something more lightweight by exploiting the fact that you typically want to use the same kind of resolution for all machines. What this will mean is that for the very first machine we will try all four options and then just save which field was used -- Then the function just picks the appropriate field going forward. Will this solve your use case ? Or are there use cases where we need to do this for every instance ? |
Is there any plan to merge this? We also need this feature. |
I will try to update/resolve conflicts today/tomorrow. |
Test PASSed. |
Superseded by #5244 |
The fix for https://issues.apache.org/jira/browse/SPARK-5242
This pull request is my original work and I contribute it under project's open source license.