docker_container with action :run not idempotent

[bplunkert]

The docker_container resource successfully starts up a container on Ubuntu 14.04, but upon a second converge, the container resource attempts to execute docker run despite the container already running. The issue is reproducible with the following code:

node.set['docker']['package']['repo_url'] = 'https://get.docker.io/ubuntu'
include_recipe 'docker'

docker_container 'redis-cache' do
  image 'dockerfile/redis'
  container_name 'redis-cache'
  detach true
  cmd_timeout 1200
  action :run
end

docker_container 'registry-s3' do
  image 'registry'
  link 'redis-cache:db'
  container_name 'registry-s3'
  volume '/etc/docker_registry/:/docker-registry/config/:rw'
  port '80:5000'
  env ['SETTINGS_FLAVOR=prod', 'DOCKER_REGISTRY_CONFIG=/docker-registry/config.yml']
  detach true
  cmd_timeout 1200
  action :run
end

The error produced is the following:

  * docker_container[redis-cache] action run[2014-07-03T18:23:28+00:00] INFO: Processing docker_container[redis-cache] action run (runnable_docker::registry line 28)       

================================================================================       
Error executing action `run` on resource 'docker_container[redis-cache]'       
================================================================================       


Mixlib::ShellOut::ShellCommandFailed       
------------------------------------       
Expected process to exit with [0], but received '1'       
       ---- Begin output of docker run  --detach=true --name="redis-cache" dockerfile/redis  ----
       STDOUT: 
       STDERR: 2014/07/03 18:23:28 Error response from daemon: Conflict, The name redis-cache is already assigned to 33e6769d23af. You have to delete (or rename) that container to be able to assign redis-cache to a container again.
       ---- End output of docker run  --detach=true --name="redis-cache" dockerfile/redis  ----
       Ran docker run  --detach=true --name="redis-cache" dockerfile/redis  returned 1

Possibly there is a bug with the running? method here: https://github.com/bflad/chef-docker/blob/master/providers/container.rb#L355-L357

[bplunkert]

It's worth noting that the link parameter causes the resulting container name to be different than the name parameter.

See here for example:

root@registry-ubuntu-1404:~# docker ps
CONTAINER ID        IMAGE                     COMMAND                CREATED             STATUS              PORTS               NAMES
33e6769d23af        dockerfile/redis:latest   redis-server /etc/re   2 hours ago         Up 2 hours          6379/tcp            redis-cache,registry-s3/db   

On these grounds I am looking at this section of the code for the source of the bug: https://github.com/bflad/chef-docker/blob/master/providers/container.rb#L152-L168

[mikexstudios]

I'm running into this issue too, and it gave me some grief. It stems from the :run action creating a new service. In the docs: "By default, this will handle creating a service for the container when action is run or start." In other words, an init script is created. Because an init script is created, the container_name must be constant, so the container_name is pre-populated for the docker run command to be a constant, see:

https://github.com/bflad/chef-docker/blob/master/providers/container.rb#L162

But we can turn that off by: "set['docker']['container_init_type'] = false or add init_type false for LWRP to disable this behavior." (from the docs)

That solved the issue for me.

I think this behavior is unexpected and that ['docker']['container_init_type'] = true should be the default.

[bplunkert]

@mikexstudios I think we might be describing different issues. In my case, an upstart service is desired, and the error I'm experiencing on second converge is coming from Chef, not upstart, trying to run docker run. Now that you mention it, I am not sure why Chef would run this itself, instead of attempting to start the service via the already-created Upstart script (from the first converge).

I think the init script behavior you describe may be a separate bug, because when I manually run service redis-cache restart it works fine, even though the converge is failing because it tries to start a brand new container (when it fails to see the old one running).

PR #185 prevents Chef from trying to run the docker run again because it allows the cookbook to detect that the container is already running, preserving idempotence when the names are modified by linking other containers.

[mikexstudios]

@bplunkert Ah, I see. Thanks for clarifying.

[bflad]

Should be fixed via #185, created #191 to add test cases. @mikexstudios please create another issue if it makes sense for your problem. Thanks!

[benlangfeld]

I'm still experiencing this problem. I've thrown a demonstration up at https://github.com/benlangfeld/docker_bug_demo. Every time the image is updated on the registry, I get an error that looks like this:

[2014-09-23T01:30:50+00:00] INFO: docker_image[quay.io/mojolingo/adam_xmpp_server] sending redeploy action to docker_container[adam_xmpp_server] (immediate)
         * docker_container[adam_xmpp_server] action redeploy[2014-09-23T01:30:50+00:00] INFO: Processing docker_container[adam_xmpp_server] action redeploy (adam_snark_rabbit::xmpp line 9)


    ================================================================================
    Error executing action `redeploy` on resource 'docker_container[adam_xmpp_server]'
    ================================================================================

    Mixlib::ShellOut::ShellCommandFailed
    ------------------------------------
           Expected process to exit with [0], but received '1'
           ---- Begin output of docker run  --detach=true --env="ERL_OPTIONS="-noshell"" --env="XMPP_DOMAIN=default-ubuntu-1404.vagrantup.com" --env="MEMORY_BASE_URL=http://localhost" --env="INTERNAL_USERNAME=internal" --env="INTERNAL_PASSWORD=abc123" --name="adam_xmpp_server" --publish="5222:5222" --publish="5269:5269" --publish="5280:5280" quay.io/mojolingo/adam_xmpp_server  ----
           STDOUT:

       STDERR: 2014/09/23 01:30:50 Error response from daemon: Conflict, The name adam_xmpp_server is already assigned to 6438440113cb. You have to delete (or rename) that container to be able to assign adam_xmpp_server to a container again.
           ---- End output of docker run  --detach=true --env="ERL_OPTIONS="-noshell"" --env="XMPP_DOMAIN=default-ubuntu-1404.vagrantup.com" --env="MEMORY_BASE_URL=http://localhost" --env="INTERNAL_USERNAME=internal" --env="INTERNAL_PASSWORD=abc123" --name="adam_xmpp_server" --publish="5222:5222" --publish="5269:5269" --publish="5280:5280" quay.io/mojolingo/adam_xmpp_server  ----
           Ran docker run  --detach=true --env="ERL_OPTIONS="-noshell"" --env="XMPP_DOMAIN=default-ubuntu-1404.vagrantup.com" --env="MEMORY_BASE_URL=http://localhost" --env="INTERNAL_USERNAME=internal" --env="INTERNAL_PASSWORD=abc123" --name="adam_xmpp_server" --publish="5222:5222" --publish="5269:5269" --publish="5280:5280" quay.io/mojolingo/adam_xmpp_server  returned 1

           Cookbook Trace:

           ---------------
           /tmp/kitchen/cookbooks/docker/libraries/helpers.rb:248:in `execute_cmd!'
           /tmp/kitchen/cookbooks/docker/libraries/helpers.rb:241:in `docker_cmd!'
           /tmp/kitchen/cookbooks/docker/providers/container.rb:355:in `run'

/tmp/kitchen/cookbooks/docker/providers/container.rb:49:in `block in class_from_file'

           Resource Declaration:
           ---------------------

           # In /tmp/kitchen/cookbooks/adam_snark_rabbit/recipes/xmpp.rb

             9: docker_container 'adam_xmpp_server' do
            10:   image 'quay.io/mojolingo/adam_xmpp_server'
            11:   detach true

            12:   port [
            13:     "5222:5222",
            14:     "5269:5269",
            15:     "5280:5280",
            16:   ]

            17:   env [
            18:     'ERL_OPTIONS="-noshell"',
            19:     "XMPP_DOMAIN=#{node['adam']['root_domain']}",
            20:     "MEMORY_BASE_URL=#{node['adam']['memory_base_url']}",


            21:     "INTERNAL_USERNAME=#{node['adam']['memory']['internal_username']}",
            22:     "INTERNAL_PASSWORD=#{node['adam']['memory']['internal_password']}",
            23:   ]
            24: end


           Compiled Resource:
           ------------------
           # Declared in /tmp/kitchen/cookbooks/adam_snark_rabbit/recipes/xmpp.rb:9:in `from_file'

           docker_container("adam_xmpp_server") do
             action :run
             retries 0
             retry_delay 2
             guard_interpreter :default
             cookbook_name :adam_snark_rabbit
             recipe_name "xmpp"
             image "quay.io/mojolingo/adam_xmpp_server"

             detach true

             port ["5222:5222", "5269:5269", "5280:5280"]
             env ["ERL_OPTIONS=\"-noshell\"", "XMPP_DOMAIN=default-ubuntu-1404.vagrantup.com", "MEMORY_BASE_URL=http://localhost", "INTERNAL_USERNAME=internal", "INTERNAL_PASSWORD=abc123"]
             cmd_timeout 60
             init_type "upstart"

           end

       [2014-09-23T01:30:51+00:00] INFO: Running queued delayed notifications before re-raising exception


Running handlers:
       [2014-09-23T01:30:51+00:00] ERROR: Running exception handlers
       Running handlers complete

       [2014-09-23T01:30:51+00:00] ERROR: Exception handlers complete
[2014-09-23T01:30:51+00:00] FATAL: Stacktrace dumped to /tmp/kitchen/cache/chef-stacktrace.out
Chef Client failed. 2 resources updated in 54.987546746 seconds
[2014-09-23T01:30:51+00:00] ERROR: docker_container[adam_xmpp_server] (adam_snark_rabbit::xmpp line 9) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of docker run  --detach=true --env="ERL_OPTIONS="-noshell"" --env="XMPP_DOMAIN=default-ubuntu-1404.vagrantup.com" --env="MEMORY_BASE_URL=http://localhost" --env="INTERNAL_USERNAME=internal" --env="INTERNAL_PASSWORD=abc123" --name="adam_xmpp_server" --publish="5222:5222" --publish="5269:5269" --publish="5280:5280" quay.io/mojolingo/adam_xmpp_server  ----
STDOUT:
STDERR: 2014/09/23 01:30:50 Error response from daemon: Conflict, The name adam_xmpp_server is already assigned to 6438440113cb. You have to delete (or rename) that container to be able to assign adam_xmpp_server to a container again.
---- End output of docker run  --detach=true --env="ERL_OPTIONS="-noshell"" --env="XMPP_DOMAIN=default-ubuntu-1404.vagrantup.com" --env="MEMORY_BASE_URL=http://localhost" --env="INTERNAL_USERNAME=internal" --env="INTERNAL_PASSWORD=abc123" --name="adam_xmpp_server" --publish="5222:5222" --publish="5269:5269" --publish="5280:5280" quay.io/mojolingo/adam_xmpp_server  ----
Ran docker run  --detach=true --env="ERL_OPTIONS="-noshell"" --env="XMPP_DOMAIN=default-ubuntu-1404.vagrantup.com" --env="MEMORY_BASE_URL=http://localhost" --env="INTERNAL_USERNAME=internal" --env="INTERNAL_PASSWORD=abc123" --name="adam_xmpp_server" --publish="5222:5222" --publish="5269:5269" --publish="5280:5280" quay.io/mojolingo/adam_xmpp_server  returned 1
[2014-09-23T01:30:51+00:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)

It seems that nothing is stopping/deleting the existing container. Does anyone know why this might be?

[benlangfeld]

I guess what I'm seeing is the same as #194 (comment)

[benlangfeld]

It seems to be necessary to set the container_name attribute explicitly on a docker_container resource in order to achieve idempotence. The matching here should use the same default value as when creating the container here. I'll submit a PR for this.