cluster-init: [ERROR]: probe I/O capabilities: can't allocate write buffer

[caleno]

Version:

Describe the bug
Starting a new cluster with embedded db fails with error message
02:00:30.442 [ERROR]: probe I/O capabilities: can't allocate write buffer
FATA[2019-11-25T14:36:08.565595250Z] starting kubernetes: failed to create task object

To Reproduce
Run: 'K3S_TOKEN=hilfe k3s server --cluster-init' (on a Centos 7 server (?))

Expected behavior
It should k3s cluster with dqlite as backend which I can join more servers nodes to.

Actual behavior
K3S_TOKEN=hilfe k3s server --cluster-init
INFO[2019-11-25T15:50:05.381319277+01:00] Starting k3s v1.0.0 (18bd921)
03:23:31.137 [DEBUG]: data dir: /var/lib/rancher/k3s/server/db/state.dqlite
03:23:31.137 [DEBUG]: metadata1: version 191, term 0, voted for 0
03:23:31.137 [DEBUG]: metadata2: version 192, term 0, voted for 0
03:23:31.137 [DEBUG]: metadata: version 194, term 0, voted for 0
03:23:31.137 [ERROR]: probe I/O capabilities: can't allocate write buffer
FATA[2019-11-25T15:50:05.399366856+01:00] starting kubernetes: failed to create task object

Additional context
Ran same command on two different servers, same problem.
Server1: CentOS Linux release 7.4.1708 (Core)
Server2: CentOS Linux release 7.7.1908 (Core)

[bigl0af]

I'm having the exact same problem on Centos7 as well...

[f-bn]

Same here, same CentOS 7 version

[gitrootid]

### Same here, inite Ubuntu 18.04.3 LTS and try to join with Ubuntu 16.04.6 LTS ,and
docker version
Client:
Version: 18.06.3-ce
API version: 1.38
Go version: go1.10.3
Git commit: d7080c1
Built: Wed Feb 20 02:27:18 2019
OS/Arch: linux/amd64
Experimental: false

Server:
Engine:
Version: 18.06.3-ce
API version: 1.38 (minimum version 1.12)
Go version: go1.10.3
Git commit: d7080c1
Built: Wed Feb 20 02:26:20 2019
OS/Arch: linux/amd64
Experimental: false

### log:

#$ K3S_TOKEN=SECRET k3s server --docker --server https://10.0.1.68:6443

INFO[0000] Preparing data dir /var/lib/rancher/k3s/data/2492a28b74abfbdbd3f943c8109a93e979669647bbbe95eadae43b60c667df69
INFO[2019-11-28T14:47:56.282408598+08:00] Starting k3s v1.0.0 (18bd921)
00:13:44.749 [DEBUG]: data dir: /var/lib/rancher/k3s/server/db/state.dqlite
00:13:44.749 [DEBUG]: metadata1: version 0, term 0, voted for 0
00:13:44.749 [DEBUG]: metadata2: version 0, term 0, voted for 0
00:13:44.749 [DEBUG]: metadata: version 2, term 0, voted for 0
00:13:44.749 [ERROR]: probe I/O capabilities: can't allocate write buffer
FATA[2019-11-28T14:47:56.392257744+08:00] starting kubernetes: failed to create task object

[gitrootid]

the inite-cluster server log:

W1128 06:45:54.041551 10741 pod_container_deletor.go:75] Container "9a0d7df36e50318c52ceaf25089927468cd94d7df82e6a4bf627f789a291a6b8" not found in pod's containers
I1128 06:46:49.844151 10741 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I1128 06:47:49.850532 10741 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I1128 06:47:56.319868 10741 log.go:172] http: TLS handshake error from 10.0.1.99:43590: remote error: tls: bad certificate
I1128 06:47:56.341660 10741 log.go:172] http: TLS handshake error from 10.0.1.99:43598: remote error: tls: bad certificate
I1128 06:48:49.859101 10741 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io

[caleno]

I managed to get it working on Ubuntu 18.04.3 LTS

[geertvanheusden]

@caleno Could you please explain us how you fixed it? ;-)

[davideshay]

Seeing the same issue on Ubuntu 18.0.4.3 LTS and Debian 10 (odroid N2 image) both on arm64.
@caleno, any more details on resolution?

[bushchannelstepper]

Also seeing this on Slackware 14.2 with OpenRC

[bushchannelstepper]

FWIW Alpine 3.11.2 extended (also OpenRC) is not displaying this issue

[ethan-reynolds]

Same thing here. I haven't been able to track down the source for the error message, but it looks like something in the libuv I/O library or the raft library?
I wasn't aware that k3s now defaults to dqlite over sqlite in non-ha clusters, but that's the assumption I reach based on the location of the raft related code in k3s.

running under Hyper-V, 2048MB RAM, static IP assigned.
initial cluster bootstrap works fine, attempting to use --cluster-init seems to corrupt/ modify something before the process exits.

CentOS Linux release 7.7.1908 (Core)
Linux centos2 3.10.0-1062.9.1.el7.x86_64 #1 SMP Fri Dec 6 15:49:49 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
SELinux status: disabled
firewalld disabled

$ curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--node-external-ip 141.73.90.64" sh -s -
successful

$ systemctl stop k3s
successful

$ systemctl start k3s
successful

$ systemctl stop k3s
successful

$ k3s server --cluster-init --node-external-ip 141.73.90.64
INFO[2020-01-03T09:19:16.864609700-05:00] Starting k3s v1.0.1 (e94a3c6)
01:10:03.126 [DEBUG]: data dir: /var/lib/rancher/k3s/server/db/state.dqlite
01:10:03.126 [DEBUG]: metadata1: version 7, term 0, voted for 0
01:10:03.126 [DEBUG]: metadata2: version 8, term 0, voted for 0
01:10:03.126 [DEBUG]: metadata: version 10, term 0, voted for 0
01:10:03.126 [ERROR]: probe I/O capabilities: can't allocate write buffer
FATA[2020-01-03T09:19:16.896075600-05:00] starting kubernetes: failed to create task object

$ systemctl start k3s
Job for k3s.service failed because the control process exited with error code. See "systemctl status k3s.service" and "journalctl -xe" for details.

$ journalctl -xe
...
-- Unit k3s.service has begun starting up.
Jan 03 09:21:42 centos2 k3s[8887]: time="2020-01-03T09:21:42.578847900-05:00" level=info msg="Starting k3s v1.0.1 (e94a3c6)"
Jan 03 09:21:42 centos2 k3s[8887]: 01:12:28.844 [DEBUG]: data dir: /var/lib/rancher/k3s/server/db/state.dqlite
Jan 03 09:21:42 centos2 k3s[8887]: time="2020-01-03T09:21:42.614008500-05:00" level=fatal msg="starting kubernetes: failed to create task
Jan 03 09:21:42 centos2 systemd[1]: k3s.service: main process exited, code=exited, status=1/FAILURE
Jan 03 09:21:42 centos2 systemd[1]: Failed to start Lightweight Kubernetes.
...

[BruceWayneVsSuperMan]

@ethan-reynolds Same with you, on CentOS 7 version

[stale]

This repository uses a bot to automatically label issues which have not had any activity (commit/comment/label) for 180 days. This helps us manage the community issues better. If the issue is still relevant, please add a comment to the issue so the bot can remove the label and we know it is still valid. If it is no longer relevant (or possibly fixed in the latest release), the bot will automatically close the issue in 14 days. Thank you for your contributions.