[bad error message] net=host can't handle adding extra hosts

[pbecotte]

Am trying to run a multi-container service with net=host. The main container communicates (currently) with the other services by dns name through links. To try and duplicate the existing behavior (instead of changing the code to look for localhost) I tried to add the hostnames to the extraHosts command. Adding entries here causes the container to fail to be created with

Status reason	CannotCreateContainerError: container already exists
Extra hosts	[{"hostname":"redis","ipAddress":"127.0.0.1"}]

I created a minimal task definition that demonstrates the problem

Amazon ECS
Clusters
Task Definitions
Repositories
Task Definitions  nethosttest  3
Task Definition: nethosttest:3
View detailed information for your task definition. To modify the task definition, you need to create a new revision and then make the required changes to the task definition
Create new revision 
Actions
Builder
JSON
 
{
  "attributes": null,
  "requiresAttributes": [
    {
      "value": null,
      "name": "com.amazonaws.ecs.capability.docker-remote-api.1.17",
      "targetId": null,
      "targetType": null
    },
    {
      "value": null,
      "name": "com.amazonaws.ecs.capability.docker-remote-api.1.18",
      "targetId": null,
      "targetType": null
    }
  ],
  "taskDefinitionArn": "arn:aws:ecs:us-east-1:609866929576:task-definition/nethosttest:3",
  "networkMode": "host",
  "status": "ACTIVE",
  "revision": 3,
  "taskRoleArn": null,
  "containerDefinitions": [
    {
      "volumesFrom": [],
      "memory": 128,
      "extraHosts": null,
      "dnsServers": null,
      "disableNetworking": null,
      "dnsSearchDomains": null,
      "portMappings": [],
      "hostname": null,
      "essential": true,
      "entryPoint": null,
      "mountPoints": [],
      "name": "redis",
      "ulimits": null,
      "dockerSecurityOptions": null,
      "environment": [],
      "links": null,
      "workingDirectory": null,
      "readonlyRootFilesystem": null,
      "image": "redis",
      "command": null,
      "user": null,
      "dockerLabels": null,
      "logConfiguration": null,
      "cpu": 0,
      "privileged": null,
      "memoryReservation": null
    },
    {
      "volumesFrom": [],
      "memory": 128,
      "extraHosts": [
        {
          "hostname": "redis",
          "ipAddress": "127.0.0.1"
        }
      ],
      "dnsServers": null,
      "disableNetworking": null,
      "dnsSearchDomains": null,
      "portMappings": [
        {
          "hostPort": 80,
          "containerPort": 80,
          "protocol": "tcp"
        }
      ],
      "hostname": null,
      "essential": true,
      "entryPoint": null,
      "mountPoints": [],
      "name": "nginx",
      "ulimits": null,
      "dockerSecurityOptions": null,
      "environment": [],
      "links": null,
      "workingDirectory": null,
      "readonlyRootFilesystem": null,
      "image": "nginx",
      "command": null,
      "user": null,
      "dockerLabels": null,
      "logConfiguration": null,
      "cpu": 0,
      "privileged": null,
      "memoryReservation": null
    }
  ],
  "placementConstraints": [],
  "volumes": [],
  "family": "nethosttest"
}

Just to be sure, running the following commands from a docker client (which should be pretty close to the above implementation) does work to demonstrate that the daemon can handle add-host combined with net=host...

docker run -d --net=host redis
docker run --net=host --add-host redis:127.0.0.1 redis redis-cli -h redis set basekey 1

[samuelkarp]

@pbecotte Thanks for reporting. What Docker versions are you using with ECS and locally to test? It looks like --net=host and --add-host are mutually exclusive in Docker versions prior to 1.12.0; moby/moby#22408 added support for using --net=host and --add-host together, which was released with Docker 1.12.0.

For example, when I try to run --net=host and --add-host with Docker 1.11.2 (which is the current version shipped in Amazon Linux) I get the following output:

[ec2-user@ip-172-31-21-237 ~]$ docker run --net=host --add-host redis:127.0.0.1 nginx
docker: Error response from daemon: Conflicting options: custom host-to-IP mapping and the network mode.

[pbecotte]

I am sorry, I must have gotten my terminal windows confused and run that on my laptop. I tried it on my ECS machine with 1.11.2 and got the same result you did. So- this would be a big with version 1.12, but is not a bug with 1.11.2 (though the error message is horrid...). I am not sure how to handle this issue :)

[samuelkarp]

though the error message is horrid...

Yeah, it's a pretty terrible error message. The "container already exists" message comes from the Docker API returning a 409 Conflict HTTP error rather than an error code indicating that there's actually something wrong with the options (400 Bad Request seems more appropriate).

As another approach to fixing this, we can:

• add API 1.24 support here (which requires making sure that all the fields we care about in go-dockerclient are actually present)
• start requiring API 1.24 (Docker 1.12.x) when trying to use "networkMode":"host" along with "extraHosts".

I'm going to leave this open until we've addressed this by either making the error message better or fixing the capabilities required for task definitions specified with those fields.

[adnxn]

closing this issue since the suggested fix for adding API 1.24 is already in the current release.

[samuelkarp]

@adnxn Did we adjust the attribute inference logic such that "networkMode":"host" + "extraHosts" now implies com.amazonaws.ecs.capability.docker-remote-api.1.24?

[adnxn]

Did we adjust the attribute inference logic such that "networkMode":"host" + "extraHosts" now implies com.amazonaws.ecs.capability.docker-remote-api.1.24?

@samuelkarp, nope. Reopening to track service side changes.

[poojamaiya]

We have updated the attribute inference logic on the service side to infer com.amazonaws.ecs.capability.docker-remote-api.1.24 for task definitions with "networkMode":"host" + "extraHosts". Closing this issue, please reopen if you have any issues.