Spark-5246 resolving hostname #91
Conversation
…y (eg: magic DNS instance-data.ec2.internal is not supported in eu-west-1 Ireland). Please see https://forums.aws.amazon.com/message.jspa?messageID=536813#536813 and http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html
| @@ -0,0 +1,32 @@ | |||
| #!/bin/bash | |||
There was a problem hiding this comment.
Lets rename this to something like resolve-hostname.sh or setup-hostname.sh in the top-level directory
|
This PR is related to: apache/spark#4038 |
| @@ -14,7 +14,7 @@ source ec2-variables.sh | |||
|
|
|||
| # Set hostname based on EC2 private DNS name, so that it is set correctly | |||
| # even if the instance is restarted with a different private DNS name | |||
| PRIVATE_DNS=`wget -q -O - http://instance-data.ec2.internal/latest/meta-data/local-hostname` | |||
| PRIVATE_DNS=`wget -q -O - http://169.254.169.254/latest/meta-data/local-hostname` | |||
There was a problem hiding this comment.
We need to invoke the new setup-hostname script here -- You can assume that the spark-ec2 directory exists, so just adding a line like bash /root/spark-ec2/setup-hostname.sh should be sufficient
There was a problem hiding this comment.
I thought about creation of "module" and putting it in the list of modules in spark-ec2.py before "spark" module.
It might be better to do it your way.
There was a problem hiding this comment.
Yeah - modules are more appropriate for new packages or something like that. For things like fixing hostnames we can just put it in the top level directory
There was a problem hiding this comment.
I added it to setup-slave.sh as you suggested. Should it also be invoked from setup.sh?
There was a problem hiding this comment.
No - setup-slave.sh is invoked on all machines (including the master node). So this should be enough
|
Thanks @voukka -- I left some inline comments |
|
@nchammas thank you! I forgot to mention that these two bugs relate to each other :) Also |
|
Thanks for the update. I want to try this on a cluster once before merging. Unfortunately I am out traveling today, tomm -- so it might be Saturday by the time I get a chance. |
| # Are we in VPC? | ||
| MAC=`wget -q -O - http://169.254.169.254/latest/meta-data/mac` | ||
| VCP_ID=`wget -q -O - http://169.254.169.254/latest/meta-data/network/interfaces/macs/${MAC}/vpc-id` | ||
| if [ -n "${VCP_ID}" ]; then |
There was a problem hiding this comment.
@voukka Shouldn't this be if [ -z "${VPC_ID}"] ? -n is true if $VPC_ID is not empty and we want to exit if the string is empty ?
There was a problem hiding this comment.
that's correct. I pushed fixed version.
| PRIVATE_IP=`wget -q -O - http://169.254.169.254/latest/meta-data/local-ipv4` | ||
|
|
||
| # do changes only if short hostname does not resolve | ||
| if ( ! ping -c 1 -q "${SHORT_HOSTNAME}" > /dev/null 2>&1 ); then |
There was a problem hiding this comment.
Minor style comment: We use [[ or [ with if statements in all our scripts. Could you change this and line 26 to match that ?
There was a problem hiding this comment.
fixed
On 18 Jan 2015, at 07:35, Shivaram Venkataraman [email protected] wrote:
In resolve-hostname.sh:
+#
+
+# Are we in VPC?
+MAC=wget -q -O - http://169.254.169.254/latest/meta-data/mac
+VCP_ID=wget -q -O - http://169.254.169.254/latest/meta-data/network/interfaces/macs/${MAC}/vpc-id
+if [ -n "${VCP_ID}" ]; then
echo "nothing to do - instance is not in VPC"
- exit 0
+fi
+SHORT_HOSTNAME=
hostname
+
+PRIVATE_IP=wget -q -O - http://169.254.169.254/latest/meta-data/local-ipv4
+
+# do changes only if short hostname does not resolve
+if ( ! ping -c 1 -q "${SHORT_HOSTNAME}" > /dev/null 2>&1 ); then
Minor style comment: We use [[ or [ with if statements in all our scripts. Could you change this and line 26 to match that ?—
Reply to this email directly or view it on GitHub.
|
Thanks @voukka - LGTM. Merging this |
Spark-5246 resolving hostname
Please see https://issues.apache.org/jira/browse/SPARK-5246
Fix for local hostname problem in VPC