minikube not starting on Windows 10 with Hyper-V

[ericmaia]

Is this a BUG REPORT or FEATURE REQUEST? (choose one): Bug Report

Please provide the following details:

Environment:

Minikube version
v0.23.0

• OS
  Windows 10 Enterprise version 1703 build 15063.483
• VM Driver
  Hyper-V
• ISO version

What happened:
minikube will not start. Sometimes it hangs at "Starting VM..." and sometimes it gives an error:
PS C:\WINDOWS\system32> minikube start --vm-driver=hyperv --hyperv-virtual-switch=ExternalSwitch
Starting local Kubernetes v1.8.0 cluster...
Starting VM...
E1109 20:44:16.687707 10552 start.go:150] Error starting host: Error starting stopped host: exit status 1.

Retrying.
E1109 20:44:16.725211 10552 start.go:156] Error starting host: Error starting stopped host: exit status 1

What you expected to happen:
minikube starts and the local cluster is available to deploy pods and services.

How to reproduce it (as minimally and precisely as possible):
Open powershell as administrator
Enter minikube start --vm-driver=hyperv --hyperv-virtual-switch=ExternalSwitch

Output of minikube logs (if applicable):
F1109 21:07:39.528007 15212 logs.go:50] Error getting cluster bootstrapper: getting localkube bootstrapper: getting ss
h client: Error creating new ssh host from driver: Error getting ssh host name for driver: Host is not running

Anything else do we need to know:
I did get minikube to start once, and was able to run a ReplicationController with a set of pods, and then it suddenly quit running for no reason. I have restarted multiple times since then, but have not been able to successfully start minikube. Hyper-V is running and I can successfully run VMs. Docker is running and I was able to create a container and use the ExternalSwitch.

[dsebastien]

Same behavior here.
Have you tried following that doc? https://github.com/kubernetes/minikube/blob/master/docs/drivers.md#hyperv-driver (I haven't yet)

[dsebastien]

Here's a more detailed error log I got after deleting everything and trying again with minikube start --vm-driver=hyperv -v 9999:

$ minikube start --vm-driver=hyperv -v 9999
Aliases:
Aliases:
map[string]string{}
Override:
map[string]interface {}{"v":"9999"}
PFlags:
map[string]viper.FlagValue{"registry-mirror":viper.pflagValue{flag:(*pflag.Flag)(0xc04251f2c0)}, "iso-url":viper.pflagValue{flag:(*pflag.Flag)(0xc04251e8c0)}, "keep-context":viper.pflagValue{flag:(*pflag.Flag)(0xc04251e640)}, "vm-driver":viper.pflagValue{flag:(*pflag.Flag)(0xc04251e960)}, "xhyve-disk-driver":viper.pflagValue{flag:(*pflag.Flag)(0xc04251ee60)}, "docker-opt":viper.pflagValue{flag:(*pflag.Flag)(0xc04251efa0)}, "feature-gates":viper.pflagValue{flag:(*pflag.Flag)(0xc04251f540)}, "profile":viper.pflagValue{flag:(*pflag.Flag)(0xc0421bf7c0)}, "docker-env":viper.pflagValue{flag:(*pflag.Flag)(0xc04251ef00)}, "extra-config":viper.pflagValue{flag:(*pflag.Flag)(0xc04251f680)}, "kvm-network":viper.pflagValue{flag:(*pflag.Flag)(0xc04251edc0)}, "memory":viper.pflagValue{flag:(*pflag.Flag)(0xc04251eaa0)}, "network-plugin":viper.pflagValue{flag:(*pflag.Flag)(0xc04251f4a0)}, "container-runtime":viper.pflagValue{flag:(*pflag.Flag)(0xc04251f400)}, "disk-size":viper.pflagValue{flag:(*pflag.Flag)(0xc04251ebe0)}, "hyperv-virtual-switch":viper.pflagValue{flag:(*pflag.Flag)(0xc04251ed20)}, "insecure-registry":viper.pflagValue{flag:(*pflag.Flag)(0xc04251f220)}, "kubernetes-version":viper.pflagValue{flag:(*pflag.Flag)(0xc04251f360)}, "bootstrapper":viper.pflagValue{flag:(*pflag.Flag)(0xc0421bf860)}, "mount-string":viper.pflagValue{flag:(*pflag.Flag)(0xc04251e780)}, "apiserver-name":viper.pflagValue{flag:(*pflag.Flag)(0xc04251f0e0)}, "cpus":viper.pflagValue{flag:(*pflag.Flag)(0xc04251eb40)}, "disable-driver-mounts":viper.pflagValue{flag:(*pflag.Flag)(0xc04251e820)}, "dns-domain":viper.pflagValue{flag:(*pflag.Flag)(0xc04251f180)}, "host-only-cidr":viper.pflagValue{flag:(*pflag.Flag)(0xc04251ec80)}, "mount":viper.pflagValue{flag:(*pflag.Flag)(0xc04251e6e0)}, "cache-images":viper.pflagValue{flag:(*pflag.Flag)(0xc04251f5e0)}}
Env:
map[string]string{}
Key/Value Store:
map[string]interface {}{}
Config:
map[string]interface {}{}
Defaults:
map[string]interface {}{"reminderwaitperiodinhours":24, "wantreporterror":false, "wantreporterrorprompt":true, "wantkubectldownloadmsg":true, "v":"0", "alsologtostderr":"false", "log_dir":"", "wantupdatenotification":true}
Starting local Kubernetes v1.8.0 cluster...
Starting VM...
Downloading Minikube ISO
 140.01 MB / 140.01 MB [============================================] 100.00% 0s
Creating CA: C:\CloudStation\Configuration\SebHome\.minikube\certs\ca.pem
Creating client certificate: C:\CloudStation\Configuration\SebHome\.minikube\certs\cert.pem
[executing ==>] : C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive @(Get-Command Get-VM).ModuleName
[stdout =====>] : Hyper-V

[stderr =====>] :
[executing ==>] : C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive @([Security.Principal.WindowsPrincipal][Security.Principal.WindowsIdentity]::GetCurrent()).IsInRole("Hyper-V Administrators")
[stdout =====>] : True

[stderr =====>] :
[executing ==>] : C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive (Get-VMSwitch).Name
[stdout =====>] : DockerNAT

[stderr =====>] :
Downloading C:\CloudStation\Configuration\SebHome\.minikube\cache\boot2docker.iso from file://C:/CloudStation/Configuration/SebHome/.minikube/cache/iso/minikube-v0.23.6.iso...
Creating SSH key...
Creating VM...
[executing ==>] : C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive (Get-VMSwitch).Name
[stdout =====>] : DockerNAT

[stderr =====>] :
Using switch "DockerNAT"
[executing ==>] : C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive @([Security.Principal.WindowsPrincipal][Security.Principal.WindowsIdentity]::GetCurrent()).IsInRole([Security.Principal.WindowsBuiltInRole] "Administrator")
[stdout =====>] : False

[stderr =====>] :
Creating VHD
[executing ==>] : C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive New-VHD -Path 'C:\CloudStation\Configuration\SebHome\.minikube\machines\minikube\fixed.vhd' -SizeBytes 20000MB -Fixed
[stdout =====>] :
[stderr =====>] : New-VHD : Failed to create the virtual hard disk.
The system failed to create 'C:\CloudStation\Configuration\SebHome\.minikube\machines\minikube\fixed.vhd'.
At line:1 char:1
+ New-VHD -Path 'C:\CloudStation\Configuration\SebHome\.minikube\machin ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [New-VHD], VirtualizationException
    + FullyQualifiedErrorId : OperationFailed,Microsoft.Vhd.PowerShell.Cmdlets.NewVhd


E1114 07:30:00.042555   11168 start.go:150] Error starting host: Error creating host: Error executing step: Creating VM.
: exit status 1.

 Retrying.
E1114 07:30:00.043541   11168 start.go:156] Error starting host:  Error creating host: Error executing step: Creating VM.
: exit status 1
================================================================================
An error has occurred. Would you like to opt in to sending anonymized crash
information to minikube to help prevent future errors?
To opt out of these messages, run the command:
        minikube config set WantReportErrorPrompt false
================================================================================
Please enter your response [Y/n]:

Actually it might just be because I don't have enough disk space available! ;-)

[drivesmoothly]

I had the same behavior, but due to 2 reasons:

1. First, I did not have enough disk space on drive C: for the HyperV machine (<1.75GB left). Even if my HyperV is set to store the VMs on D:, the minikube scripts are still using drive C: regardless of that setting. Didn't dig too much into this, just freed some space.
2. The HyperV virtual switch was set to Internal. After I set it to external, everything worked. So, please create a virtual switch in HyperV, set it to External, name it "minikube-switch" or whatever else, and use "minikube start --vm-driver=hyperv --hyperv-virtual-switch=minikube-switch"

Now I am experiencing some CPU overusage by minikube, but that's a different topic :) So it's still fun :)
I hope this helps.

[ericmaia]

I have 700 GB free space, and I am using an external switch I set up in HyperV for Docker.
It looks like my "minikube" VM is gone. How does that get created, and is there any way to recreate it if it is missing?

Starting VM...
[executing ==>] : C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive ( Get-VM minikube ).state
[stdout =====>] :
[stderr =====>] : Get-VM : Hyper-V was unable to find a virtual machine with name "minikube".
At line:1 char:3

• ( Get-VM minikube ).state

• + CategoryInfo          : InvalidArgument: (minikube:String) [Get-VM], VirtualizationException
  + FullyQualifiedErrorId : InvalidParameter,Microsoft.HyperV.PowerShell.Commands.GetVM
  

[dsebastien]

I've tried again with more disk space available and also after creating a dedicated (external) vSwitch, but I still get errors:

$ minikube start --vm-driver=hyperv hyperv-virtual-switch=Minikube --v=7 --alsologtostderr
W1115 18:41:58.464004   10748 root.go:148] Error reading config file at C:\CloudStation\Configuration\SebHome\.minikube\config\config.json: open C:\CloudStation\Configuration\SebHome\.minikube\config\config.json: The system cannot find the file specified.
I1115 18:41:58.465005   10748 notify.go:109] Checking for updates...
I1115 18:41:58.594064   10748 cache_images.go:226] Attempting to cache image: gcr.io/google_containers/pause-amd64:3.0 at C:\CloudStation\Configuration\SebHome\.minikube\cache\images\gcr.io\google_containers\pause-amd64_3.0
Starting local Kubernetes v1.8.0 cluster...
I1115 18:41:58.594064   10748 cache_images.go:226] Attempting to cache image: gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.5 at C:\CloudStation\Configuration\SebHome\.minikube\cache\images\gcr.io\google_containers\k8s-dns-dnsmasq-nanny-amd64_1.14.5
Starting VM...
I1115 18:41:58.594064   10748 cache_images.go:226] Attempting to cache image: gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.5 at C:\CloudStation\Configuration\SebHome\.minikube\cache\images\gcr.io\google_containers\k8s-dns-kube-dns-amd64_1.14.5
I1115 18:41:58.594064   10748 cache_images.go:226] Attempting to cache image: gcr.io/google-containers/kube-addon-manager:v6.4-beta.2 at C:\CloudStation\Configuration\SebHome\.minikube\cache\images\gcr.io\google-containers\kube-addon-manager_v6.4-beta.2
I1115 18:41:58.594064   10748 cache_images.go:226] Attempting to cache image: gcr.io/google_containers/k8s-dns-sidecar-amd64:1.14.5 at C:\CloudStation\Configuration\SebHome\.minikube\cache\images\gcr.io\google_containers\k8s-dns-sidecar-amd64_1.14.5
I1115 18:41:58.594064   10748 cache_images.go:226] Attempting to cache image: gcr.io/google_containers/kubernetes-dashboard-amd64:v1.6.3 at C:\CloudStation\Configuration\SebHome\.minikube\cache\images\gcr.io\google_containers\kubernetes-dashboard-amd64_v1.6.3
I1115 18:41:58.595050   10748 cluster.go:64] Machine does not exist... provisioning new machine
I1115 18:41:58.597051   10748 cluster.go:65] Provisioning machine with config: {MinikubeISO:https://storage.googleapis.com/minikube/iso/minikube-v0.23.6.iso Memory:2048 CPUs:2 DiskSize:20000 VMDriver:hyperv XhyveDiskDriver:ahci-hd DockerEnv:[] InsecureRegistry:[10.0.0.0/24] RegistryMirror:[] HostOnlyCIDR:192.168.99.1/24 HypervVirtualSwitch: KvmNetwork:default Downloader:{} DockerOpt:[] DisableDriverMounts:false}
Downloading Minikube ISO
 82.33 MB / 140.01 MB [=========================>------------------]  58.80% 12sI1115 18:42:16.450544   10748 cache_images.go:77] Successfully cached all images.
 140.01 MB / 140.01 MB [============================================] 100.00% 0s
Creating CA: C:\CloudStation\Configuration\SebHome\.minikube\certs\ca.pem
Creating client certificate: C:\CloudStation\Configuration\SebHome\.minikube\certs\cert.pem
[executing ==>] : C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive @(Get-Command Get-VM).ModuleName
[stdout =====>] : Hyper-V

[stderr =====>] :
[executing ==>] : C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive @([Security.Principal.WindowsPrincipal][Security.Principal.WindowsIdentity]::GetCurrent()).IsInRole("Hyper-V Administrators")
[stdout =====>] : True

[stderr =====>] :
[executing ==>] : C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive (Get-VMSwitch).Name
[stdout =====>] : DockerNAT
Minikube

[stderr =====>] :
Downloading C:\CloudStation\Configuration\SebHome\.minikube\cache\boot2docker.iso from file://C:/CloudStation/Configuration/SebHome/.minikube/cache/iso/minikube-v0.23.6.iso...
Creating SSH key...
Creating VM...
[executing ==>] : C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive (Get-VMSwitch).Name
[stdout =====>] : DockerNAT
Minikube

[stderr =====>] :
Using switch "DockerNAT"
[executing ==>] : C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive @([Security.Principal.WindowsPrincipal][Security.Principal.WindowsIdentity]::GetCurrent()).IsInRole([Security.Principal.WindowsBuiltInRole] "Administrator")
[stdout =====>] : False

[stderr =====>] :
Creating VHD
[executing ==>] : C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive New-VHD -Path 'C:\CloudStation\Configuration\SebHome\.minikube\machines\minikube\fixed.vhd' -SizeBytes 20000MB -Fixed
[stdout =====>] :

ComputerName            : TRANKILLUX
Path                    : C:\CloudStation\Configuration\SebHome\.minikube\machines\minikube\fixed.vhd
VhdFormat               : VHD
VhdType                 : Fixed
FileSize                : 20971520512
Size                    : 20971520000
MinimumSize             :
LogicalSectorSize       : 512
PhysicalSectorSize      : 512
BlockSize               : 0
ParentPath              :
DiskIdentifier          : C292D1AA-E207-45F9-A211-06AB90013066
FragmentationPercentage : 0
Alignment               : 1
Attached                : False
DiskNumber              :
Number                  :




[stderr =====>] :
Writing magic tar header
Writing SSH key tar header
[executing ==>] : C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive Convert-VHD -Path 'C:\CloudStation\Configuration\SebHome\.minikube\machines\minikube\fixed.vhd' -DestinationPath 'C:\CloudStation\Configuration\SebHome\.minikube\machines\minikube\disk.vhd' -VHDType Dynamic -DeleteSource
[stdout =====>] :
[stderr =====>] :
[executing ==>] : C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive New-VM minikube -Path 'C:\CloudStation\Configuration\SebHome\.minikube\machines\minikube' -SwitchName 'DockerNAT' -MemoryStartupBytes 2048MB
[stdout =====>] :
Name     State CPUUsage(%) MemoryAssigned(M) Uptime   Status             Version
----     ----- ----------- ----------------- ------   ------             -------
minikube Off   0           0                 00:00:00 Operating normally 8.0



[stderr =====>] :
[executing ==>] : C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive Set-VMProcessor minikube -Count 2
[stdout =====>] :
[stderr =====>] :
[executing ==>] : C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive Set-VMDvdDrive -VMName minikube -Path 'C:\CloudStation\Configuration\SebHome\.minikube\machines\minikube\boot2docker.iso'
[stdout =====>] :
[stderr =====>] :
[executing ==>] : C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive Add-VMHardDiskDrive -VMName minikube -Path 'C:\CloudStation\Configuration\SebHome\.minikube\machines\minikube\disk.vhd'
[stdout =====>] :
[stderr =====>] :
Starting VM...
[executing ==>] : C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive Start-VM minikube
[stdout =====>] :
[stderr =====>] :
Waiting for host to start...
[executing ==>] : C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive ( Get-VM minikube ).state
[stdout =====>] : Running

[stderr =====>] :
[executing ==>] : C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive (( Get-VM minikube ).networkadapters[0]).ipaddresses[0]
[stdout =====>] :
[stderr =====>] :
[executing ==>] : C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive ( Get-VM minikube ).state
[stdout =====>] : Running

[stderr =====>] :
[executing ==>] : C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive (( Get-VM minikube ).networkadapters[0]).ipaddresses[0]
[stdout =====>] :
[stderr =====>] :
[executing ==>] : C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive ( Get-VM minikube ).state
[stdout =====>] : Running

[stderr =====>] :
[executing ==>] : C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive (( Get-VM minikube ).networkadapters[0]).ipaddresses[0]
[stdout =====>] :
[stderr =====>] :
[executing ==>] : C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive ( Get-VM minikube ).state
[stdout =====>] : Running

[stderr =====>] :
[executing ==>] : C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive (( Get-VM minikube ).networkadapters[0]).ipaddresses[0]
[stdout =====>] :
[stderr =====>] :
[executing ==>] : C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive ( Get-VM minikube ).state
[stdout =====>] : Running

[stderr =====>] :
[executing ==>] : C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive (( Get-VM minikube ).networkadapters[0]).ipaddresses[0]
[stdout =====>] :
[stderr =====>] :
[executing ==>] : C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive ( Get-VM minikube ).state
[stdout =====>] : Running

[stderr =====>] :
[executing ==>] : C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive (( Get-VM minikube ).networkadapters[0]).ipaddresses[0]
[stdout =====>] :
[stderr =====>] :
[executing ==>] : C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive ( Get-VM minikube ).state
[stdout =====>] : Running

[stderr =====>] :
[executing ==>] : C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive (( Get-VM minikube ).networkadapters[0]).ipaddresses[0]
[stdout =====>] :
[stderr =====>] :
[executing ==>] : C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive ( Get-VM minikube ).state
[stdout =====>] : Running

[stderr =====>] :
[executing ==>] : C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive (( Get-VM minikube ).networkadapters[0]).ipaddresses[0]
[stdout =====>] :
[stderr =====>] :
[executing ==>] : C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive ( Get-VM minikube ).state
[stdout =====>] : Running

[stderr =====>] :
[executing ==>] : C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive (( Get-VM minikube ).networkadapters[0]).ipaddresses[0]
[stdout =====>] :
[stderr =====>] :
[executing ==>] : C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive ( Get-VM minikube ).state
[stdout =====>] : Running

[stderr =====>] :
[executing ==>] : C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive (( Get-VM minikube ).networkadapters[0]).ipaddresses[0]
[stdout =====>] :
[stderr =====>] :
[executing ==>] : C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive ( Get-VM minikube ).state
[stdout =====>] : Running

[stderr =====>] :
[executing ==>] : C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive (( Get-VM minikube ).networkadapters[0]).ipaddresses[0]
[stdout =====>] :
[stderr =====>] :
[executing ==>] : C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive ( Get-VM minikube ).state
[stdout =====>] : Running

[stderr =====>] :
[executing ==>] : C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive (( Get-VM minikube ).networkadapters[0]).ipaddresses[0]
[stdout =====>] :
[stderr =====>] :
[executing ==>] : C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive ( Get-VM minikube ).state
[stdout =====>] : Running

[stderr =====>] :
[executing ==>] : C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive (( Get-VM minikube ).networkadapters[0]).ipaddresses[0]
[stdout =====>] : fe80::215:5dff:fe00:a66f

[stderr =====>] :
[executing ==>] : C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive ( Get-VM minikube ).state
[stdout =====>] : Running

[stderr =====>] :
setting hostname "minikube"
[executing ==>] : C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive ( Get-VM minikube ).state
[stdout =====>] : Running

[stderr =====>] :
[executing ==>] : C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive (( Get-VM minikube ).networkadapters[0]).ipaddresses[0]
[stdout =====>] : fe80::215:5dff:fe00:a66f

[stderr =====>] :
Using SSH client type: native
&{{{<nil> 0 [] [] []} docker [0x8474d0] 0x847480  [] 0s} fe80::215:5dff:fe00:a66f 22 <nil> <nil>}
About to run SSH command:
sudo hostname minikube && echo "minikube" | sudo tee /etc/hostname
Error dialing TCP: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
SSH cmd err, output: <nil>: minikube

[executing ==>] : C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive ( Get-VM minikube ).state
[stdout =====>] : Running

[stderr =====>] :
[executing ==>] : C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive (( Get-VM minikube ).networkadapters[0]).ipaddresses[0]
[stdout =====>] : fe80::215:5dff:fe00:a66f

[stderr =====>] :
Using SSH client type: native
&{{{<nil> 0 [] [] []} docker [0x8474d0] 0x847480  [] 0s} fe80::215:5dff:fe00:a66f 22 <nil> <nil>}
About to run SSH command:

                if ! grep -xq '.*\sminikube' /etc/hosts; then
                        if grep -xq '127.0.1.1\s.*' /etc/hosts; then
                                sudo sed -i 's/^127.0.1.1\s.*/127.0.1.1 minikube/g' /etc/hosts;
                        else
                                echo '127.0.1.1 minikube' | sudo tee -a /etc/hosts;
                        fi
                fi
SSH cmd err, output: <nil>:
set auth options {CertDir:C:\CloudStation\Configuration\SebHome\.minikube CaCertPath:C:\CloudStation\Configuration\SebHome\.minikube\certs\ca.pem CaPrivateKeyPath:C:\CloudStation\Configuration\SebHome\.minikube\certs\ca-key.pem CaCertRemotePath:/etc/docker/ca.pem ServerCertPath:C:\CloudStation\Configuration\SebHome\.minikube\machines\server.pem ServerKeyPath:C:\CloudStation\Configuration\SebHome\.minikube\machines\server-key.pem ClientKeyPath:C:\CloudStation\Configuration\SebHome\.minikube\certs\key.pem ServerCertRemotePath:/etc/docker/server.pem ServerKeyRemotePath:/etc/docker/server-key.pem ClientCertPath:C:\CloudStation\Configuration\SebHome\.minikube\certs\cert.pem ServerCertSANs:[] StorePath:C:\CloudStation\Configuration\SebHome\.minikube}
setting up certificates
[executing ==>] : C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive ( Get-VM minikube ).state
[stdout =====>] : Running

[stderr =====>] :
[executing ==>] : C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive (( Get-VM minikube ).networkadapters[0]).ipaddresses[0]
[stdout =====>] : fe80::215:5dff:fe00:a66f

[stderr =====>] :
generating server cert: C:\CloudStation\Configuration\SebHome\.minikube\machines\server.pem ca-key=C:\CloudStation\Configuration\SebHome\.minikube\certs\ca.pem private-key=C:\CloudStation\Configuration\SebHome\.minikube\certs\ca-key.pem org=trankill.minikube san=[fe80::215:5dff:fe00:a66f localhost]
[executing ==>] : C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive ( Get-VM minikube ).state
[stdout =====>] : Running

[stderr =====>] :
[executing ==>] : C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive (( Get-VM minikube ).networkadapters[0]).ipaddresses[0]
[stdout =====>] : fe80::215:5dff:fe00:a66f

[stderr =====>] :
I1115 18:44:48.327376   10748 ssh_runner.go:57] Run: sudo rm -f /etc/docker/ca.pem
I1115 18:44:48.370390   10748 ssh_runner.go:57] Run: sudo mkdir -p /etc/docker
I1115 18:44:48.376376   10748 ssh_runner.go:57] Run: sudo rm -f /etc/docker/server.pem
I1115 18:44:48.379376   10748 ssh_runner.go:57] Run: sudo mkdir -p /etc/docker
I1115 18:44:48.386378   10748 ssh_runner.go:57] Run: sudo rm -f /etc/docker/server-key.pem
I1115 18:44:48.389376   10748 ssh_runner.go:57] Run: sudo mkdir -p /etc/docker
Setting Docker configuration on the remote daemon...
[executing ==>] : C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive ( Get-VM minikube ).state
[stdout =====>] : Running

[stderr =====>] :
[executing ==>] : C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive (( Get-VM minikube ).networkadapters[0]).ipaddresses[0]
[stdout =====>] : fe80::215:5dff:fe00:a66f

[stderr =====>] :
Using SSH client type: native
&{{{<nil> 0 [] [] []} docker [0x8474d0] 0x847480  [] 0s} fe80::215:5dff:fe00:a66f 22 <nil> <nil>}
About to run SSH command:
sudo mkdir -p /lib/systemd/system && printf %s "[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network.target docker.socket
Requires=docker.socket

[Service]
Type=notify

# DOCKER_RAMDISK disables pivot_root in Docker, using MS_MOVE instead.
Environment=DOCKER_RAMDISK=yes


# This file is a systemd drop-in unit that inherits from the base dockerd configuration.
# The base configuration already specifies an 'ExecStart=...' command. The first directive
# here is to clear out that command inherited from the base configuration. Without this,
# the command from the base configuration and the command specified here are treated as
# a sequence of commands, which is not the desired behavior, nor is it valid -- systemd
# will catch this invalid input and refuse to start the service with an error like:
#  Service has more than one ExecStart= setting, which is only allowed for Type=oneshot services.
ExecStart=
ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --tlsverify --tlscacert /etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/docker/server-key.pem --label provider=hyperv --insecure-registry 10.0.0.0/24
ExecReload=/bin/kill -s HUP $MAINPID

# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity

# Uncomment TasksMax if your systemd version supports it.
# Only systemd 226 and above support this version.
TasksMax=infinity
TimeoutStartSec=0

# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes

# kill only the docker process, not all processes in the cgroup
KillMode=process

[Install]
WantedBy=multi-user.target
" | sudo tee /lib/systemd/system/docker.service
SSH cmd err, output: <nil>: [Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network.target docker.socket
Requires=docker.socket

[Service]
Type=notify

# DOCKER_RAMDISK disables pivot_root in Docker, using MS_MOVE instead.
Environment=DOCKER_RAMDISK=yes


# This file is a systemd drop-in unit that inherits from the base dockerd configuration.
# The base configuration already specifies an 'ExecStart=...' command. The first directive
# here is to clear out that command inherited from the base configuration. Without this,
# the command from the base configuration and the command specified here are treated as
# a sequence of commands, which is not the desired behavior, nor is it valid -- systemd
# will catch this invalid input and refuse to start the service with an error like:
#  Service has more than one ExecStart= setting, which is only allowed for Type=oneshot services.
ExecStart=
ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --tlsverify --tlscacert /etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/docker/server-key.pem --label provider=hyperv --insecure-registry 10.0.0.0/24
ExecReload=/bin/kill -s HUP

# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity

# Uncomment TasksMax if your systemd version supports it.
# Only systemd 226 and above support this version.
TasksMax=infinity
TimeoutStartSec=0

# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes

# kill only the docker process, not all processes in the cgroup
KillMode=process

[Install]
WantedBy=multi-user.target

[executing ==>] : C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive ( Get-VM minikube ).state
[stdout =====>] : Running

[stderr =====>] :
[executing ==>] : C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive (( Get-VM minikube ).networkadapters[0]).ipaddresses[0]
[stdout =====>] : fe80::215:5dff:fe00:a66f

[stderr =====>] :
Using SSH client type: native
&{{{<nil> 0 [] [] []} docker [0x8474d0] 0x847480  [] 0s} fe80::215:5dff:fe00:a66f 22 <nil> <nil>}
About to run SSH command:
sudo systemctl daemon-reload
SSH cmd err, output: <nil>:
[executing ==>] : C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive ( Get-VM minikube ).state
[stdout =====>] : Running

[stderr =====>] :
[executing ==>] : C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive (( Get-VM minikube ).networkadapters[0]).ipaddresses[0]
[stdout =====>] : fe80::215:5dff:fe00:a66f

[stderr =====>] :
Using SSH client type: native
&{{{<nil> 0 [] [] []} docker [0x8474d0] 0x847480  [] 0s} fe80::215:5dff:fe00:a66f 22 <nil> <nil>}
About to run SSH command:
sudo systemctl -f restart docker
SSH cmd err, output: <nil>:
setting minikube options for container-runtime
[executing ==>] : C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive ( Get-VM minikube ).state
[stdout =====>] : Running

[stderr =====>] :
[executing ==>] : C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive (( Get-VM minikube ).networkadapters[0]).ipaddresses[0]
[stdout =====>] : fe80::215:5dff:fe00:a66f

[stderr =====>] :
Using SSH client type: native
&{{{<nil> 0 [] [] []} docker [0x8474d0] 0x847480  [] 0s} fe80::215:5dff:fe00:a66f 22 <nil> <nil>}
About to run SSH command:
sudo mkdir -p /etc/sysconfig && printf %s "
CRIO_MINIKUBE_OPTIONS='--insecure-registry 10.0.0.0/24 '
" | sudo tee /etc/sysconfig/crio.minikube
SSH cmd err, output: <nil>:
CRIO_MINIKUBE_OPTIONS='--insecure-registry 10.0.0.0/24 '

[executing ==>] : C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive ( Get-VM minikube ).state
[stdout =====>] : Running

[stderr =====>] :
[executing ==>] : C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive (( Get-VM minikube ).networkadapters[0]).ipaddresses[0]
[stdout =====>] : fe80::215:5dff:fe00:a66f

[stderr =====>] :
Using SSH client type: native
&{{{<nil> 0 [] [] []} docker [0x8474d0] 0x847480  [] 0s} fe80::215:5dff:fe00:a66f 22 <nil> <nil>}
About to run SSH command:
sudo systemctl daemon-reload
SSH cmd err, output: <nil>:
[executing ==>] : C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive ( Get-VM minikube ).state
[stdout =====>] : Running

[stderr =====>] :
[executing ==>] : C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive (( Get-VM minikube ).networkadapters[0]).ipaddresses[0]
[stdout =====>] : fe80::215:5dff:fe00:a66f

[stderr =====>] :
Using SSH client type: native
&{{{<nil> 0 [] [] []} docker [0x8474d0] 0x847480  [] 0s} fe80::215:5dff:fe00:a66f 22 <nil> <nil>}
About to run SSH command:
sudo systemctl -f restart crio
SSH cmd err, output: Process exited with status 1: Job for crio.service failed because the control process exited with error code.
See "systemctl status crio.service" and "journalctl -xe" for details.

Error setting container-runtime options during provisioning ssh command error:
command : sudo systemctl -f restart crio
err     : Process exited with status 1
output  : Job for crio.service failed because the control process exited with error code.
See "systemctl status crio.service" and "journalctl -xe" for details.

E1115 18:45:02.959478   10748 start.go:150] Error starting host: Error creating host: Error executing step: Provisioning VM.
: ssh command error:
command : sudo systemctl -f restart crio
err     : Process exited with status 1
output  : Job for crio.service failed because the control process exited with error code.
See "systemctl status crio.service" and "journalctl -xe" for details.
.

 Retrying.
E1115 18:45:02.962482   10748 start.go:156] Error starting host:  Error creating host: Error executing step: Provisioning VM.
: ssh command error:
command : sudo systemctl -f restart crio
err     : Process exited with status 1
output  : Job for crio.service failed because the control process exited with error code.
See "systemctl status crio.service" and "journalctl -xe" for details.

[dsebastien]

Digging into the logs using journalctl told me a bit more. crio.service seems to fail with:

msg="Unable to select an IP"
...
Failed to start Open Container Initiative Daemon.

[dsebastien]

Ok so it's really /usr/bin/crio that's not happy when starting up.
I guess it's either the default config used (/etc/cni/net.d/k8s.conf) that's not liked or I need to adapt something to the vSwitch I've created (?)

Some refs:
https://github.com/containernetworking/cni
https://github.com/containernetworking/cni/blob/master/SPEC.md

Sorry if my comments seem stupid but I'm discovering rkt, cni, etc just at the moment ;-)

[dsebastien]

Okay, I've finally got it working.
I was wrong all along; my external vSwitch wasn't associated with the correct nic.
Kubernetes is now running ;-)

[dsebastien]

Quick summary for those seeing similar issues:

First create a vSwitch in Hyper-V

• type: external
• name: Minikube (or anything you like)
• make sure to associate the vSwitch with the correct NIC
• reboot after creating the vSwitch just in case (routing tables, ...)
• go take a look at the network devices on your Windows host to make sure that you have a Virtual Ethernet Adapter configured and actually connected to your network
  • Control Panel\Network and Internet\Network Connections. In my case i have: "vEthernet (Minikube)"

Once done make sure to start clean, remove ~/.minikube and remove any VM you tried to create before.

Once you're ready, make it happen:

minikube start --vm-driver "hyperv" --hyperv-virtual-switch "Minikube" --disk-size 10g --memory 4096 --v 9999 --alsologtostderr

[ccarns]

@dsebastien Thanks the detailed fix -I was able to get mine running after reading the post above. Clearing out the ./minikube was the kicker for me.

[kambbado]

I have tried @dsebastien suggestions but did not work on my case. I'm still running into issues with "crio".

Using SSH client type: native
&{{{ 0 [] [] []} docker [0x83feb0] 0x83fe60 [] 0s} fe80::215:5dff:fec8:6412 22 }
About to run SSH command:
sudo systemctl -f restart crio
SSH cmd err, output: Process exited with status 1: Job for crio.service failed because the control process exited with e
rror code.
See "systemctl status crio.service" and "journalctl -xe" for details.

Error setting container-runtime options during provisioning ssh command error:
command : sudo systemctl -f restart crio
err : Process exited with status 1
output : Job for crio.service failed because the control process exited with error code.
See "systemctl status crio.service" and "journalctl -xe" for details.

E1211 15:33:45.206964 6120 start.go:150] Error starting host: Error creating host: Error executing step: Provisioning
VM.
: ssh command error:
command : sudo systemctl -f restart crio
err : Process exited with status 1
output : Job for crio.service failed because the control process exited with error code.
See "systemctl status crio.service" and "journalctl -xe" for details.
.

Retrying.
E1211 15:33:45.227177 6120 start.go:156] Error starting host: Error creating host: Error executing step: Provisionin
g VM.
: ssh command error:
command : sudo systemctl -f restart crio
err : Process exited with status 1
output : Job for crio.service failed because the control process exited with error code.
See "systemctl status crio.service" and "journalctl -xe" for details.

minikube version: v0.24.1

Any other suggestions? I have tried almost everything I could find on the net.

Thank you

[jelleverheyen]

What worked for me was first following dsebastien's instructions in his last comment, however it still didn't work after that so I tried launching the minikube VM through the Hyper-V Manager and it showed me an error saying I don't have enough memory, so after putting in some more RAM and following dsebastien's instructions again, I got it working.

[kambbado]

What is interesting is that I can get the Minikube to start once, it works as expected, however, once I reboot the server, issuing a start of Minikube does not work anymore.

PS C:> minikube start --vm-driver=hyperv --hyperv-virtual-switch=Minikube --bootstrapper kubeadm
Starting local Kubernetes v1.8.0 cluster...
Starting VM...
Downloading Minikube ISO
140.01 MB / 140.01 MB [============================================] 100.00% 0s
E1215 09:30:46.923908 3824 start.go:150] Error starting host: Error creating host: Error executing step: Provisioning
VM.
: ssh command error:
command : sudo systemctl -f restart crio
err : Process exited with status 1
output : Job for crio.service failed because the control process exited with error code.
See "systemctl status crio.service" and "journalctl -xe" for details.

I find this very puzzling. If I roll back hte VM checkpoint I can create the Minikube again. I tried to clean up everything related to Minikube and it still errors.

[old-coder-404]

Creating an external virtual switch cuts my connection to our network (and as such, the internet). Is there a way to run minikube w/o loosing internet connection?

[Moncefmd]

Had the same issue, fixed by selecting the correct NIC, purging all caches (basically rm ~/.minikube/) and then running minikube start again, with the correct flags. Thank you @dsebastien !

[jan-hudec]

Had the same issue, fixed by not setting the virtual switch.

Here, the cluster will not get external IP, because the DHCP server only provides addresses to registered devices. So the external switch is actually guaranteed to break everything.

[jevy-wangfei]

Please have a try of a solution posted here: #2131

[thangchung]

I have found the solution for this on #2255

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[juant72]

Quick summary for those seeing similar issues:

First create a vSwitch in Hyper-V

• type: external

• name: Minikube (or anything you like)

• make sure to associate the vSwitch with the correct NIC

• reboot after creating the vSwitch just in case (routing tables, ...)

• go take a look at the network devices on your Windows host to make sure that you have a Virtual Ethernet Adapter configured and actually connected to your network

  • Control Panel\Network and Internet\Network Connections. In my case i have: "vEthernet (Minikube)"

Once done make sure to start clean, remove ~/.minikube and remove any VM you tried to create before.

Once you're ready, make it happen:

minikube start --vm-driver "hyperv" --hyperv-virtual-switch "Minikube" --disk-size 10g --memory 4096 --v 9999 --alsologtostderr

Remember use "minikube delete" for clean configurations.
This works for me.
thankyou

[cda963]

Quick summary for those seeing similar issues:

First create a vSwitch in Hyper-V

• type: external

• name: Minikube (or anything you like)

• make sure to associate the vSwitch with the correct NIC

• reboot after creating the vSwitch just in case (routing tables, ...)

• go take a look at the network devices on your Windows host to make sure that you have a Virtual Ethernet Adapter configured and actually connected to your network

  • Control Panel\Network and Internet\Network Connections. In my case i have: "vEthernet (Minikube)"

Once done make sure to start clean, remove ~/.minikube and remove any VM you tried to create before.

Once you're ready, make it happen:

minikube start --vm-driver "hyperv" --hyperv-virtual-switch "Minikube" --disk-size 10g --memory 4096 --v 9999 --alsologtostderr

Even though the above solution works just fine, I was also able to keep the virtual switch internal, by going to Control Panel\Network and Internet\Network and Sharing Center\Network Adapter (wifi or ethernet)"Properties" button"Sharing" tab, then ticked the "Allow other network users to connect through this computer's Internet connection" and picked from the dropdown list the "minikube" virtual switch.