ERROR: Cannot connect to the Docker daemon. Docker:dind

[mayurkumar09]

Server:
ERROR: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

Docker daemon isn't getting up in docker:dind or other images that has docker. Tried via brig project create, as well as helm chart for brigade project.
no problem in the gke cluster too, while i'm trying from another machine.

Please help.

Thanks :)

Output of brig version: v1.2.1

Output of kubectl version:

Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.0", GitCommit:"70132b0f130acc0bed193d9ba59dd186f0e634cf", GitTreeState:"clean", BuildDate:"2019-12-07T21:20:10Z", GoVersion:"go1.13.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"13+", GitVersion:"v1.13.11-gke.14", GitCommit:"56d89863d1033f9668ddd6e1c1aea81cd846ef88", GitTreeState:"clean", BuildDate:"2019-11-07T19:12:22Z", GoVersion:"go1.12.11b4", Compiler:"gc", Platform:"linux/amd64"

Cloud Provider/Platform (AKS, GKE, Minikube etc.): GKE

[krancour]

How are you planning to use Docker? Despite "dind" being in the image's name, there are two ways of using this, with one obviously being DinD (Docker in Docker) and the other being sometimes referred to as "DooD" (Docker outside of Docker).

What does your project configuration look like?

For DinD, your configuration needs to permit privileged containers and your job needs to launch its own Docker daemon (i.e. within the container) before proceeding with whatever else you intend to do with that job (e.g. build an image).

For DooD, your configuration needs to permit host mounts so that the Docker client within the container can connect to the Docker daemon running on whatever Kubernetes node the job gets assigned to.

Hope this helps. If neither of these do, then it could be that there is something wrong with your cluster that is not specifically Brigade-related.

[mayurkumar09]

@krancour -- Thanks for your reply.
When i created a project with brig cli the job was having privileged permission in brigade.js file but then also the docker daemon was not coming up.
However when i created the brigade project with helm chart, docker daemon was working perfectly fine with the same brigade.js file. Is it because project.yml(helm chart) file has privileged enabled by default???

[krancour]

When i created a project with brig cli the job was having privileged permission in brigade.js file...

I think I should clarify something here. In your brigade.js, if you configure a job to use privileged containers or host mounts, you're requesting those features. That doesn't mean they are permitted. Whether they are permitted or not is set in the project level configuration. i.e. To use a privileged container or host mount, those things need to be true at both the job level and project level.

If it's not clear why...

Imagine a hostile actor coming along and submitting a PR that modifies your brigade.js to elevate and leverage job permissions to some nefarious end. If these are disallowed at the project level, this stops them.

Is it because project.yml(helm chart) file has privileged enabled by default???

https://github.com/brigadecore/charts/blob/master/charts/brigade-project/values.yaml#L115-L122

In the chart, a project defaults to allowing privileged containers and whether host mounts are permitted is undefined (commented). The rendering logic in the chart's templates interprets undefined as false.

I'll circle back to mentioning that the right combination of these settings depends on whether you're trying to do "DinD" or "DooD" in your job.

[mayurkumar09]

@krancour -- thanks for the clarification, gained a lot of knowledge.

I was using docker:dind image.

[krancour]

@mayurkumar09 you may be using an image with "dind" in its name, but that image actually supports both the DinD and DooD approach. Can you clarify which you are attempting? DinD == start your own Docker daemon on tge container. DooD == use the host's Docker daemon.

[mayurkumar09]

@krancour -- i was using DinD -- which was starting the docker daemon on the container.
Below is my brigade.js file :

const { events, Job } = require("brigadier");
events.on("push", () => {

var job = new Job("dockerbuild", "docker:dind");
job.privileged = true;
job.env = {
DOCKER_DRIVER: "overlay"
}
job.tasks = [
    "dockerd-entrypoint.sh &",
    "sleep 10",
    "docker ps"
    ];

job.run();

});

[krancour]

Ok. That looks quite right. Are this working with that + privileged containers permitted at the project level?

[mayurkumar09]

yes this is working fine after creating brigade project with helm chart as by default there was allowPrivilegedJobs: "true" at project level.
But i was not able to do it via brig cli even when i have permitted jobs to have privileged permission at project level via advanced option.
how should i do it via brig cli ??

[radu-matei]

If you create the project via brig, if you choose Configure advanced options, there is an option to allow privileged jobs:

? Allow privileged jobs [? for help] (y/N)

[krancour]

@radu-matei it sounds like that is what he had done.

@mayurkumar09 I'd be curious... if you walk through the steps of creating the project (or a new one similar to it) via brig (and the advanced option) again, could you possibly extract the corresponding Kubernetes secret that gets created and show it to us?

[mayurkumar09]

@radu-matei -- ya i did same way previously. Thanks for the help!!

@krancour -- Now docker daemon is running perfectly fine. Might be i was using some wrong permutation. But now its working fine both ways via brig and helm too. Thanks a lot for ur support.

Below is the details of my brigade project :

$ brig project get mayurkumar09/hellonode
{
"metadata": {
"name": "brigade-68b44050a12acebea789025fecd5a569e7c90788a6f07ae67fe414",
"creationTimestamp": null,
"labels": {
"app": "brigade",
"component": "project",
"heritage": "brigade"
},
"annotations": {
"projectName": "mayurkumar09/hellonode"
}
},
"stringData": {
"allowHostMounts": "false",
"allowPrivilegedJobs": "true",
"brigadejsPath": "",
"buildStorageSize": "50Mi",
"cloneURL": "https://github.com/mayurkumar09/hellonode.git",
"defaultScript": "",
"defaultScriptName": "",
"genericGatewaySecret": "CSxW9",
"github.baseURL": "",
"github.token": "",
"github.uploadURL": "",
"imagePullSecrets": "",
"initGitSubmodules": "false",
"kubernetes.allowSecretKeyRef": "false",
"kubernetes.buildStorageClass": "nfs",
"kubernetes.cacheStorageClass": "nfs",
"namespace": "default",
"repository": "github.com/mayurkumar09/hellonode",
"secrets": "{}",
"serviceAccount": "",
"sharedSecret": "mysecrets",
"sshKey": "",
"vcsSidecar": "brigadecore/git-sidecar:latest",
"worker.name": "",
"worker.pullPolicy": "IfNotPresent",
"worker.registry": "",
"worker.tag": "",
"workerCommand": ""
},
"type": "brigade.sh/project"
}

[krancour]

@mayurkumar09 sounds like you're all set. Glad to hear it.