x509: certificate is valid for 192.168.99.103, not 192.168.99.100

[stephenlawrence]

After having my machine running on the .103 IP I rebooted my Mac. Upon reboot my docker-machine switched to the .100 address. When I tried to perform any docker commands on my machine I get things like this:

docker exec -it mycontainer bash:
FATA[0000] An error occurred trying to connect: Post https://192.168.99.100:2376/v1.16/containers/mycontainer/exec: x509: certificate is valid for 192.168.99.103, not 192.168.99.100

[ehazlett]

Can you give some more detail on the issue?

• What version of Machine?
• What driver (I'm assuming vbox based upon the IP)?
• Command line used to create machine

Thanks

[stephenlawrence]

docker-machine version 0.1.0
virtualbox
docker-machine create -d virtualbox --virtualbox-disk-size '40000' --virtualbox-memory '4096' devbox

I also added an --insecure-registry to the daemon so I could talk to our private registry (if that matters).

[ehazlett]

Would you mind testing from master (you can use https://docker-machine-builds.evanhazlett.com/latest/ if you don't want to build locally). I have tested this with VMs changing IPs and everything works as expected:

ehazlett machine> docker-machine ls
NAME        ACTIVE   DRIVER       STATE     URL
test-vbox   *        virtualbox   Running   tcp://192.168.99.100:2376
ehazlett machine> docker-machine stop test-vbox
ehazlett machine> docker-machine create -d virtualbox test-vbox-2
...
ehazlett machine> docker-machine ls
NAME          ACTIVE   DRIVER       STATE     URL
test-vbox              virtualbox   Stopped   
test-vbox-2   *        virtualbox   Running   tcp://192.168.99.100:2376
ehazlett machine> docker-machine start test-vbox
...      
ehazlett machine> docker-machine ls
NAME          ACTIVE   DRIVER       STATE     URL
test-vbox              virtualbox   Running   tcp://192.168.99.101:2376
test-vbox-2   *        virtualbox   Running   tcp://192.168.99.100:2376            
ehazlett machine> docker $(docker-machine config test-vbox) ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

You can see that test-vbox switched IPs.

[stephenlawrence]

Ok, here is what I did:

Stop the current machine
Start using the new machine bin
"machine ls" with new machine was ok and shows:
"devbox * virtualbox Running tcp://192.168.99.100:2376
devbox2 virtualbox Stopped"

"docker info" produces:
FATA[0000] An error occurred trying to connect: Get https://192.168.99.100:2376/v1.16/info: x509: certificate is valid for 192.168.99.103, not 192.168.99.100

[ehazlett]

thanks. did you try with the master build above? I ran the same procedure you did and I didn't have any cert issues.

[stephenlawrence]

I downloaded that "latest" binary.

[ehazlett]

Hmm ok. @sthulb can you try to see if you can reproduce this?

• Create a virtualbox machine
• Stop first machine
• Create second virtualbox machine
• Start first machine
• Check that the first machine IP changed
• Check if you can access the docker daemon on both

@stephenlawrence is that process correct?

Thanks

[stephenlawrence]

@ehazlett Technically this is not how it started. It started when I rebooted my Mac and the DM came up with a different IP on the reboot. I don't know if that is different than just doing what you describe. But now, any attempts at accessing that previously-working machine result in the x509 message.

[ehazlett]

@stephenlawrence i cannot re-create that as my IP always changes and the VM stays. I simply emulated the VM getting a different IP which should create the same thing I believe. Would you mind trying the above with the new binary and a fresh set of VMs? I'm wondering if the certs were created using an old build not using the correct process or something.

[cpuguy83]

@stephenlawrence Are you sure you don't have something in your bashrc that is messing with your settings, like DOCKER_CERT_PATH?

[md5]

Since I didn't see anyone else reference the relevant code, I'll just point out that machine is creating a certificate with the IP address in it here:

machine/host.go

Line 123 in 973b267

if err := utils.GenerateCert([]string{ip}, serverCertPath, serverKeyPath, h.CaCertPath, h.PrivateKeyPath, org, bits); err != nil {

If the IP address changes, that cert will no longer work.

In the b2d init scripts, there is code that regenerates the server certificates if the relevant IP addresses change: https://github.com/boot2docker/boot2docker/blob/5db7efbb4e0557f6efefdb56cb0263f80ed55834/rootfs/rootfs/usr/local/etc/init.d/docker#L46

I'm not entirely sure why that bit of b2d code isn't triggered in the machine case since I thought you guys were using the b2d ISO.

[ehazlett]

Yeah thanks, I know this code is there however, when the IP changed (as
shown above) there were no issues when testing so I'm trying to at least
get this re-producable to test.

On Sat, Feb 14, 2015 at 12:58 AM, Mike Dillon [email protected]
wrote:

Since I didn't see anyone else reference the relevant code, I'll just
point out that machine is creating a certificate with the IP address in
it here:

machine/host.go

Line 123 in 973b267

if err := utils.GenerateCert([]string{ip}, serverCertPath, serverKeyPath, h.CaCertPath, h.PrivateKeyPath, org, bits); err != nil {

If the IP address changes, that cert will no longer work.

In the b2d init scripts, there is code that regenerates the server
certificates if the relevant IP addresses change:
https://github.com/boot2docker/boot2docker/blob/5db7efbb4e0557f6efefdb56cb0263f80ed55834/rootfs/rootfs/usr/local/etc/init.d/docker#L46

I'm not entirely sure why that bit of b2d code isn't triggered in the
machine case since I thought you guys were using the b2d ISO.

—
Reply to this email directly or view it on GitHub
#531 (comment).

[ehazlett]

@md5 @stephenlawrence would you mind checking to see if you have anything that would be using the b2d certs by chance? after debugging, it looks like the b2d certs are being used with machine.

[md5]

@ehazlett I'm not sure what you're asking me to check.

[ehazlett]

@md5 sorry i should have been more clear. check the environment variables DOCKER_CERT_PATH and DOCKER_HOST to see if they are pointed at the correct machine. It sounds like the the DOCKER_CERT_PATH may be configured to the b2d certs when accessing a machine or vice versa.

[md5]

@ehazlett if this were a case of an incorrect DOCKER_CERT_PATH, it would be a different error. The problem in that situation would either be that ca.pem is not a trust root for the cert presented by the Docker server on the machine-controlled instance or that the cert.pem client certificate is not accepted by the server.

The error that we're seeing here is clearly that the client (i.e. docker exec) is not accepting the server's certificate. It's attempting to hit the server at 192.168.99.100, but the server is presenting a certificate that is only valid for 192.168.99.103. This can only mean that a server that is responding at 192.168.99.100 on port 2376 is configured to use a certificate that was created for 192.168.99.103.

[ehazlett]

@md5 that makes sense. how do i re-create? as you can see above i've changed the IP of my virtualbox instances and don't see this issue.

[md5]

That's a good question. I haven't been able to reproduce it myself either.

[ehazlett]

Closing as we cannot seem to reproduce. My suspicion is that the environment variables are getting mixed for the boot2docker VM and the machine instance. I would recommend to check your .bashrc etc for anything that may be setting them.

[stephenlawrence]

I have this issue happening again. I had a docker-machine running on 192.168.99.101, and after a re-boot of my Mac, that machine is now running on 192.168.99.100.

This seems as if its a docker issue since I can do "docker-compose run" into a container OK, but trying any docker commands throws this error:

$docker ps
FATA[0000] An error occurred trying to connect: Get https://192.168.99.100:2376/v1.17/containers/json: x509: certificate is valid for 192.168.99.101, not 192.168.99.100

[stephenlawrence]

Current env:
DOCKER_CERT_PATH=/Users/myusername/.docker/machines/.client
DOCKER_TLS_VERIFY=yes
DOCKER_HOST=tcp://192.168.99.100:2376

$ ls -l ~/.docker/machines/.client/
total 48
-rw-r--r-- 1 myusername staff 1054 Feb 11 10:21 ca.pem
-rw-r--r-- 1 myusername staff 1054 Jan 30 08:53 ca.pem.bak
-rw-r--r-- 1 myusername staff 1094 Feb 11 10:21 cert.pem
-rw-r--r-- 1 myusername staff 1094 Jan 30 08:53 cert.pem.bak
-rw------- 1 myusername staff 1675 Feb 11 10:21 key.pem
-rw------- 1 myusername staff 1679 Jan 30 08:53 key.pem.bak

[ehazlett]

@stephenlawrence ugh ok. would you be using b2d as well by chance? the only time i've seen this was something between b2d and machine. i can't personally reproduce but it had to do with the certs and env vars.

maybe we need that "regenerate-certs" command sooner rather than later :)

[stephenlawrence]

Well, I had been using b2d alone before I switch to DM. DM uses b2d as well no?

[razic]

@stephenlawrence just use machine to manage b2d's. delete your original b2d installation and VM

[parshap]

I've had this happen to me as well. I used machine to create the virtual machine, not b2d directly. A command to regenerate certs sounds helpful.

[stephenlawrence]

Removing b2d doesn't necessarily resolve my issue, but I have removed it.

I would add that this is making docker-machine unusable for me at the moment. I just created a new machine hoping to get rid of that issue, but I have the same issue on a new machine.

[onnimonni]

I'm using OS X and tired of using these hacks and having them broken eventually.

Vagrant has option to set ip address in configuration. I can't understand why this kind of parameter or flag is not already existing in docker-machine (Just comparing to vagrant because both use virtualbox)

[cpuguy83]

@onnimonni: this has been resolved for a long long time (almost a year). Machine includes a command to regenerate certs in case the ip changed.

[lancehunt]

I just ran into this issue for the first time and checked everything in this thread with any of the following options successfully fixing the issue: (pick one)

• docker-machine regenerate-certs default2
• docker-machine kill default2 then docker-machine create default2 ...
• docker-machine upgrade default2 (if not already upgraded)

To repro this issue on OSX:

1. Install Docker Toolbox (v1.10)
2. Create a default machine which gets 192.168.99.100
3. Create a default2 machine which gets 192.168.99.101
4. Shutdown both (sometimes requires a reboot too)
5. docker-compose start default2 which gets 192.168.99.100 (and x509 cert conflict)

[MikeTLive]

so it comes down to multiple machines started in different order may cause a different IP which breaks the certificate which requires searching github for this issue to find the resolution of creating a new certificate to allow you to use them again ... is there a way to PREVENT this happening? is it so unusual for folks to flip between multiple docker machines?

[ivancarrancho]

$alias docker="docker --tlsverify=false" Work for me!

[nathanleclaire]

@ivancarrancho Why not docker-machine regenerate-certs -f and keep TLS enabled?

[ivancarrancho]

@nathanleclaire yes, that's better than "--tlsveify" many many thanks

[l15k4]

@nathanleclaire because it takes ~ 4 minutes ... Imagine you have a cluster of 6+ nodes... I will have to write a parallel certificate regenerator thingy to not spend whole day by regenerating some certificates.... Moreover it restarts all docker containers on the target machine ...

This requirement to regenerate certificates after IP address change is a huge pain on aws cloud... Public IPs change all the time. The only solution known to me is creating new instances from an ec2 instance, but for some reason it doesn't work #1453 (comment)

Btw any idea if docker-machine create --amazonec2-use-private-address is supposed to create an ec2 instance from another ec2 instance?

This is the only way I'm aware of how to avoid constant certificate regeneration...

[nathanleclaire]

Btw any idea if docker-machine create --amazonec2-use-private-address is supposed to create an ec2 instance from another ec2 instance?

Yes, unless there is some other way you have to access the private IP from the creation node (e.g. proxy).

We are considering a variety of solutions, e.g. Elastic IP, to potentially resolve this issue.

[l15k4]

yeah,but as I mentioned,it is hanging on ssh access even with this option
enabled
On May 2, 2016 20:26, "Nathan LeClaire" [email protected] wrote:

Btw any idea if docker-machine create --amazonec2-use-private-address is
supposed to create an ec2 instance from another ec2 instance?

Yes, unless there is some other way you have to access the private IP from
the creation node (e.g. proxy).

We are considering a variety of solutions, e.g. Elastic IP, to potentially
resolve this issue.

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#531 (comment)

[nathanleclaire]

Maybe the security group is not configured properly? Currently, if using --amazonec2-use-private-ip-address, it's assumed that user is willing to ensure that these sorts of network access details are configured properly beforehand.

[ShashankaNataraj]

I dont understand why I have to run that ugly $eval statement everytime docker needs to be started via a terminal.

I dont understand why this issue even exists.

Docker is seeming more and more like a horribly broken product that a lot of people got behind because it was the "in" thing.

[twang2218]

I'm using the latest version of docker and docker-machine, and I still have the same issue, I created several virtualbox docker host by

    docker-machine create -d virtualbox xxxx \
        --engine-opt="cluster-store=${KVSTORE}" \
        --engine-opt="cluster-advertise=eth1:2376" \
        ${NAME}
...

and almost every time I reboot the VMs, or reboot my Mac, I will face the error like certificate is valid for 192.168.99.100, not 192.168.99.101.

My versions are:

$ docker --version
Docker version 1.12.0-rc4, build e4a0dbc, experimental
$ docker-machine -v
docker-machine version 0.8.0-rc2, build 4ca1b85
$ vboxmanage -v
5.1.0r108711

Is there any solution to prevent those error to happen? or create a virtualbox host with a fixed IP?

[pyhedgehog]

Workaround: docker-machine provision.

[qixiuying]

Fixed the issue by docker-machine regenerate-certs xxx

$ docker-machine ls
NAME        ACTIVE   DRIVER       STATE     URL                         SWARM   DOCKER    ERRORS
xxx   -        virtualbox   Running   tcp://192.168.99.100:2376           Unknown   Unable to query docker version: Get https://192.168.99.100:2376/v1.15/version: x509: certificate is valid for 192.168.99.101, not 192.168.99.100
~$ docker-machine regenerate-certs xxx
Regenerate TLS machine certs?  Warning: this is irreversible. (y/n): y
Regenerating TLS certificates
Waiting for SSH to be available...
Detecting the provisioner...
Copying certs to the local machine directory...
Copying certs to the remote machine...
Setting Docker configuration on the remote daemon...
~$ docker-machine ls
NAME        ACTIVE   DRIVER       STATE     URL                         SWARM   DOCKER    ERRORS
xxx   *        virtualbox   Running   tcp://192.168.99.100:2376           v1.12.6

[themsay]

Following @VonC link I came across docker-machine-ipconfig https://github.com/fivestars/docker-machine-ipconfig

$ cd ~/.docker/
$ git clone https://github.com/fivestars/docker-machine-ipconfig.git
# Add to ~/.profile
$ echo 'alias docker-machine-ipconfig=~/.docker/docker-machine-ipconfig/docker-machine-ipconfig' >> ~/.profile 
$ source ~/.profile

E.g: Assign machine-name a static IP:

$ docker-machine-ipconfig static machine-name
# or specify implicit IP
$ docker-machine-ipconfig static machine-name 192.168.99.110

This eliminate need of docker-machine regenerate-certs -f machine-name on every restart. An easy way to set machine static IP running on virtualbox.

[pyhedgehog]

Does it supports Windows? I mean both "for" and "on".

[z0r1k]

it looks like it is still an issue. any way to fix it?

[mvma]

@johnmccabe,
thanks.
works fine. i just restarted my toolbox and everything is back in the game.

[alaagalal]

how can i have static account of container , it enables me to change the account every time i restart the docker-machine ?

[tannerchung]

I see the workaround with docker-machine-ipconfig but just for documentation sake I'm reporting that this happens with Windows 10 Docker Machine and Hyper-V as well

[rdp]

Might want to tweak this message "Started machines may have new IP addresses. You may need to re-run the docker-machine env command." to mention something like "and also do docker-machine regenerate-certs ..." FWIW...

[jotaoncode]

@rdp nice catch, I had this problem was half an hour looking what had happened, and after trying to do some things with kubernetes, installing and uninstalling things... running
docker-machine env
the IP matched in my envs matched the one that was producing the error with the certificate...
then:
eval $(docker-machine env)
make the configuraiton in place for me...

[PatMyron]

minikube stop && minikube start fixed this for me

[Kihagi]

minikube stop && minikube start fixed this for me

@PatMyron Surprisingly, this worked for me too!

[solotvun]

I had the same exact issue just now and apparently it was fixed by resetting docker to factory settings and enabling the Kubernetes cluster again.

EDIT: I managed to reproduce the fix on a second machine. The exact steps in my case were:

start Docker Desktop
update to 3.2.1 -> immediately after this another updated was available to 3.2.2
update to 3.2.2
enable Kubernetes cluster -> wait until the error appears
right click on Docker in the System Tray -> choose Troubleshoot
click Reset to factory defaults -> wait until the reset is finished
right click on Docker in the System Tray -> choose Quit Docker Desktop
open Docker Desktop again
select only Enable Kubernetes

[tommasosansone91]

I had the same problem and I solved it by doing this:

In your command prompt run:

docker-machine ls

it will return

NAME ACTIVE DRIVER STATE URL SWARM DOCKER ERRORS
<name> <status> <driver> <state> <url> <swarm> <docker> <errors>

get the of the machine you want to fix

then run:

docker-machine kill <name>
docker-machine create <name>
docker-machine env <name>

The last command will output more lines, copy the last one, in my case it was like this:

# eval $("C:\my\path\to\docker-machine.exe" env <name>)

then paste it in the command prompt without the # and run it, the problem should be fixed now.

https://stackoverflow.com/a/67028893/7658051