Update vagrant to include kubeworkers and refator edge, worker loop #1365

andreijs · 2016-04-19T23:43:03Z

Hey guys,

Here is the updated vagrant file for kubeworker.

[done ] Installs cleanly on a fresh build of most recent master branch
[ done] Upgrades cleanly from the most recent release
[ done] Updates documentation relevant to the changes

langston-barrett · 2016-04-20T04:57:38Z

Vagrantfile

@@ -26,6 +30,36 @@ else
 config_hash = config_hash.merge(YAML.load(File.read(config_path)))
 end

+def spin_up(config_hash:, config:, server_array:, hostvars:, hosts:, server_type:)


This function could use an explaining comment

langston-barrett · 2016-04-20T04:58:35Z

This looks great, thanks @andreijs! I left just a few comments

langston-barrett · 2016-04-20T05:00:33Z

Vagrantfile

+          "ansible_ssh_host" => ip,
+          "private_ipv4" => ip,
+          "public_ipv4" => ip,
+          "role" => server_type


server_type could just be renamed to "role" for consistency

langston-barrett · 2016-04-20T05:05:14Z

We should also probably add a line for kubeworkers here:
https://github.com/CiscoCloud/mantl/pull/1365/files#diff-23b6f443c01ea2efcb4f36eedfea9089L141

andreimc · 2016-04-20T09:24:14Z

@siddharthist thanks for your feedback I had made the changes pointed out 👍

langston-barrett · 2016-04-20T17:13:59Z

roles/consul/defaults/main.yml

-consul_package: consul-0.6.3
-consul_ui_package: consul-ui-0.6.3
+consul_package: consul-0.6.4
+consul_ui_package: consul-ui-0.6.4


Can we put this in a separate PR? Doesn't seem directly related.

Sorry I didn't realize I commited this

langston-barrett · 2016-04-20T19:03:22Z

I got this error on vagrant up:

TASK: [kubernetes-addons | create/update skydns replication controller] ******* 
failed: [control-01] => {"failed": true}
msg: error running kubectl (/bin/kubectl --server=http://localhost:8085/ --namespace=kube-system create --filename=/etc/kubernetes/manifests/skydns-rc.yaml) command (rc=1): Error from server: error when creating "/etc/kubernetes/manifests/skydns-rc.yaml": namespaces "kube-system" not found


FATAL: all hosts have already failed -- aborting

andreimc · 2016-04-20T23:36:41Z

I get this one some times haven't seen the one you got @siddharthist

TASK: [kubernetes-addons | create/update elasticsearch service] ***************
failed: [control-01] => {"failed": true}
msg: error running kubectl (/bin/kubectl --server=http://localhost:8085/ --namespace=kube-system create --filename=/etc/kubernetes/manifests/es-svc.yaml) command (rc=1): Error from server: error when creating "/etc/kubernetes/manifests/es-svc.yaml": Internal error occurred: failed to allocate a serviceIP: cannot allocate resources of type serviceipallocations at this time

andreimc · 2016-04-21T01:56:39Z

Everything works for me apart from Kube UI and nginx-consul starting on kubeworker ref #1346

PLAY RECAP ********************************************************************
mesos | wait for zookeeper service to be registered -------------------- 38.28s
common | install system utilities -------------------------------------- 37.09s
consul | wait for leader ----------------------------------------------- 30.54s
kubernetes | pull hyperkube docker image ------------------------------- 24.81s
kubernetes-master | wait for apiserver to come up ---------------------- 16.20s
etcd | restart skydns -------------------------------------------------- 14.56s
kubernetes-master | download kubernetes binaries ----------------------- 14.36s
mantlui | ensure nginx-mantlui docker image is present ----------------- 12.17s
kubernetes-node | download kubernetes binaries ------------------------- 12.07s
zookeeper | install zookeepercli package ------------------------------- 10.47s
control-01                 : ok=271  changed=206  unreachable=0    failed=0
edge-001                   : ok=122  changed=91   unreachable=0    failed=0
kubeworker-001             : ok=152  changed=106  unreachable=0    failed=0
localhost                  : ok=0    changed=0    unreachable=0    failed=0
worker-001                 : ok=123  changed=88   unreachable=0    failed=0

➜  mantl (master) ✔ vagrant ssh kubeworker-001
No vagrant-config.yml found, using defaults
Last login: Wed Apr 20 23:54:41 2016 from control-01
[vagrant@kubeworker-001 ~]$

langston-barrett · 2016-04-21T18:03:25Z

I got another error this time:

TASK: [kubernetes-addons | create/update grafana service] ********************* 
failed: [control-01] => {"failed": true}
msg: error running kubectl (/bin/kubectl --server=http://localhost:8085/ --namespace=kube-system create --filename=/etc/kubernetes/manifests/grafana-service.yaml) command (rc=1): Error from server: error when creating "/etc/kubernetes/manifests/grafana-service.yaml": Internal error occurred: failed to allocate a serviceIP: cannot allocate resources of type serviceipallocations at this time


FATAL: all hosts have already failed -- aborting

andreimc · 2016-04-22T00:39:00Z

@siddharthist can you give me your machine specs OS, ansible version etc ?

langston-barrett · 2016-04-22T05:29:18Z

@andreimc I have Vagrant 1.8.1 and Oracle VM VirtualBox Manager 5.0.16_OSE. My host's version of ansible doesn't/shouldn't affect anything, the VMs are provisioned from the control node.

andreimc · 2016-04-22T11:07:48Z

@siddharthist I really don't know why it fails, me and a co-worker both tried to spin it up and it worked fine, Vagrant file updates should not really cause ansible to fail ... maybe get someone else to try it.

ryane · 2016-04-26T13:28:03Z

I'm also having a lot of trouble getting kubernetes to run on Vagrant. Provisioning fails intermittently with various errors. Here are a couple I have seen repeatedly:

TASK: [kubernetes-addons | create or update dashboard] ************************
failed: [control-01] => {"failed": true}
msg: error running kubectl (/bin/kubectl --server=http://localhost:8085/ --namespace=kube-system create --filename=/etc/kubernetes/manifests/kubernetes-dashboard.yaml) command (rc=1): You have exposed your service on an external port on all nodes in your
cluster.  If you want to expose this service to the external internet, you may
need to set up firewall rules for the service port(s) (tcp:30000) to serve traffic.

See http://releases.k8s.io/release-1.2/docs/user-guide/services-firewalls.md for more details.
service "kubernetes-dashboard" created

TASK: [kubernetes-addons | create or update dashboard] ************************
failed: [control-01] => {"failed": true}
msg: error running kubectl (/bin/kubectl --server=http://localhost:8085/ --namespace=kube-system create --filename=/etc/kubernetes/manifests/kubernetes-dashboard.yaml) command (rc=1): replicationcontroller "kubernetes-dashboard" created

Repeated provisioning attempts might ultimately complete but still seeing various problems with Kubernetes:

UI not accessible
No nodes registered
```
$ kubectl get nodes

# no results
```

Errors running kubectl

$ kubectl get po
Error from server: an error on the server has prevented the request from succeeding

@BrianHicks @Zogg any ideas on this?

SillyMoo · 2016-04-26T16:26:43Z

I get the same issue, but if I re-run with a 'vagrant provision' it all springs to life. Looks like a timing issue to me (I know that the ansible scripts wait for hyperkube to be pulled, but do they start for it to actually be up and listening?).

SillyMoo · 2016-04-26T17:05:46Z

Ok, I tell a bit of a lie. Ansible finishes ok, hyperkube is running and I see a node in kubectl. However I can't actually get kubernetes to pull any images (the pod just sits there, no image pull events, and no sign on the kubeworker that any images are being pulled).

andreimc · 2016-04-28T12:53:00Z

With latest master merged in it fails to restart skydns, not sure what would be causing it It just hangs for a while then I get the following error message:

NOTIFIED: [dnsmasq | restart dnsmasq] *****************************************
changed: [control-01]
changed: [kubeworker-001]
changed: [edge-001]

PLAY [role=worker] ************************************************************

TASK: [mesos | install mesos packages] ****************************************
FATAL: no hosts matched or all hosts have already failed -- aborting

Not sure why.

stevendborrelli · 2016-05-03T02:28:41Z

Docker fails on this

TASK: [docker | enable docker] ************************************************ 
failed: [control-01] => {"failed": true}
msg: Job for docker.service failed because a configured resource limit was exceeded. See "systemctl status docker.service" and "journalctl -xe" for details.

But this is due to the new docker implementation not creating a /etc/sysconfig/mantl-storage file on non-lvm based systems and not a problem with this PR.

andreimc · 2016-05-04T00:33:00Z

The problems with this will be fixed after #1409 and #1410 are merged in.

langston-barrett · 2016-05-04T01:36:33Z

@andreijs @andreimc Can you rebase this? Both those PRs have been merged.

andreimc · 2016-05-04T09:59:14Z

@siddharthist up to date.

ryane · 2016-05-04T13:04:21Z

Had a successful build but I am back to

Internal Server Error (500)

Get https://10.254.0.1:443/api/v1/replicationcontrollers: dial tcp 10.254.0.1:443: getsockopt: connection refused

when trying to access the Kubernetes UI. 10.254.0.1 is the cluster ip for the kubernetes service.

kubectl get svc --namespace=default
NAME         CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   10.254.0.1   <none>        443/TCP   1h

Logs from the kubernetes-dashboard pod's container:

2016/05/04 12:50:26 Incoming HTTP/1.0 GET /api/v1/replicationcontrollers request from 10.10.99.1:33787
2016/05/04 12:50:26 Getting list of all replication controllers in the cluster
2016/05/04 12:50:26 Get https://10.254.0.1:443/api/v1/replicationcontrollers: dial tcp 10.254.0.1:443: getsockopt: connection refused
2016/05/04 12:50:26 Outcoming response to 10.10.99.1:33787 with 500 status code

andreimc · 2016-05-07T09:08:31Z

hey guys, I had some time span this up in vagrant, I still get 502 for kube ui. :(, asnible ran ok tho.

andreimc · 2016-05-07T11:17:28Z

I get the following on control-01 when i try to list pods:

[vagrant@control-01 ~]$ kubectl --namespace kube-system get pods
Error from server: an error on the server has prevented the request from succeeding

manishrajkarnikar · 2016-05-14T11:36:13Z

@andreimc @siddharthist curious how is file groups_var/all/kubernetes_vars.yml read in vagrant run? or is it required at all.

Zogg · 2016-05-16T09:00:11Z

@manishrajkarnikar groups_var/all/kubernetes_vars.yml usage in local ansible run shouldn't be different from the remote case.
As far as I remember yes, the kubernetes_vars.yml was mandatory to have the kubernetes roles play nice.

manishrajkarnikar · 2016-05-16T13:55:17Z

@Zogg I don't see it being mentioned in the vagrant file. I added that in my vagrant file as raw parameter and I was able to get K8s multi node cluster up and running. I couldn't get single master and slave node going though probably because of bug reported in vagrant file.

…1.1.0

[skip ci]

andreijs · 2016-06-12T03:28:00Z

Opening new PR of a branch closing this

andreijs mentioned this pull request Apr 19, 2016

Mantl Kubernetes - kube workers - help is appreciated #1362

Closed

langston-barrett reviewed Apr 20, 2016
View reviewed changes

langston-barrett added the provider/vagrant label Apr 20, 2016

langston-barrett reviewed Apr 20, 2016
View reviewed changes

ryane modified the milestone: 1.1 Apr 22, 2016

BrianHicks added the core/kubernetes label Apr 26, 2016

langston-barrett mentioned this pull request May 3, 2016

Docker failures in Vagrant due to missing /etc/sysconfig/mantl-storage #1403

Closed

ryane mentioned this pull request May 3, 2016

Kube UI not working #1367

Closed

ryane modified the milestones: 1.2, 1.1 May 12, 2016

langston-barrett mentioned this pull request May 25, 2016

Install with vagrant blocked at waiting for apiserver #1482

Closed

ryane and others added 3 commits May 31, 2016 14:58

Merge commit '9b7277ac7e0ed96ccff6d66f0f56cfa804b06980' into release/…

01b54e3

…1.1.0

docs: 1.1 changelog update

171502b

[skip ci]

Merge remote-tracking branch 'upstream/master'

f877500

andreijs closed this Jun 12, 2016

andreimc mentioned this pull request Jun 12, 2016

Vagrant update to include Kube worker #1542

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update vagrant to include kubeworkers and refator edge, worker loop #1365

Update vagrant to include kubeworkers and refator edge, worker loop #1365

andreijs commented Apr 19, 2016

langston-barrett Apr 20, 2016

langston-barrett commented Apr 20, 2016 •

edited

Loading

langston-barrett Apr 20, 2016

langston-barrett commented Apr 20, 2016

andreimc commented Apr 20, 2016

langston-barrett Apr 20, 2016

andreimc Apr 20, 2016

langston-barrett commented Apr 20, 2016

andreimc commented Apr 20, 2016

andreimc commented Apr 21, 2016

langston-barrett commented Apr 21, 2016

andreimc commented Apr 22, 2016

langston-barrett commented Apr 22, 2016

andreimc commented Apr 22, 2016

ryane commented Apr 26, 2016

SillyMoo commented Apr 26, 2016

SillyMoo commented Apr 26, 2016

andreimc commented Apr 28, 2016

stevendborrelli commented May 3, 2016

andreimc commented May 4, 2016

langston-barrett commented May 4, 2016

andreimc commented May 4, 2016

ryane commented May 4, 2016

andreimc commented May 7, 2016 •

edited

Loading

andreimc commented May 7, 2016

manishrajkarnikar commented May 14, 2016

Zogg commented May 16, 2016

manishrajkarnikar commented May 16, 2016

andreijs commented Jun 12, 2016

Update vagrant to include kubeworkers and refator edge, worker loop #1365

Update vagrant to include kubeworkers and refator edge, worker loop #1365

Conversation

andreijs commented Apr 19, 2016

langston-barrett Apr 20, 2016

Choose a reason for hiding this comment

langston-barrett commented Apr 20, 2016 • edited Loading

langston-barrett Apr 20, 2016

Choose a reason for hiding this comment

langston-barrett commented Apr 20, 2016

andreimc commented Apr 20, 2016

langston-barrett Apr 20, 2016

Choose a reason for hiding this comment

andreimc Apr 20, 2016

Choose a reason for hiding this comment

langston-barrett commented Apr 20, 2016

andreimc commented Apr 20, 2016

andreimc commented Apr 21, 2016

langston-barrett commented Apr 21, 2016

andreimc commented Apr 22, 2016

langston-barrett commented Apr 22, 2016

andreimc commented Apr 22, 2016

ryane commented Apr 26, 2016

SillyMoo commented Apr 26, 2016

SillyMoo commented Apr 26, 2016

andreimc commented Apr 28, 2016

stevendborrelli commented May 3, 2016

andreimc commented May 4, 2016

langston-barrett commented May 4, 2016

andreimc commented May 4, 2016

ryane commented May 4, 2016

andreimc commented May 7, 2016 • edited Loading

andreimc commented May 7, 2016

manishrajkarnikar commented May 14, 2016

Zogg commented May 16, 2016

manishrajkarnikar commented May 16, 2016

andreijs commented Jun 12, 2016

langston-barrett commented Apr 20, 2016 •

edited

Loading

andreimc commented May 7, 2016 •

edited

Loading