Set up keyfile for replica set in docker-compose

[rrriki]

Hi!

Sorry if this is the wrong place to ask this.

I'm trying to set up a docker-compose file to spin-up a mongo instance using replica sets (I just need to be able to use transactions), I got the error that

BadValue: security.keyFile is required when authorization is enabled with replica sets

So I started digging around and found that I needed to use a keyfile, so I generated the keyfile and updated my docker-compose a little bit:

version: '3.4'

services:
    mongo:
        hostname: 'mongodb'
        container_name: 'mongo'
        image: 'mongo:latest'
        ports:
            - 27017:27017
        environment:
            - MONGO_INITDB_DATABASE
            - MONGO_INITDB_ROOT_USERNAME
            - MONGO_INITDB_ROOT_PASSWORD
            - MONGO_USERNAME
            - MONGO_PASSWORD
            - MONGO_INITDB_DATABASE=${MONGO_INITDB_DATABASE}
            - MONGO_INITDB_ROOT_USERNAME=${MONGO_INITDB_ROOT_USERNAME}
            - MONGO_INITDB_ROOT_PASSWORD=${MONGO_INITDB_ROOT_PASSWORD}
        volumes:
            - mongodb_data_container:/data/db
            - $PWD/.docker/mongo/mongo-init.sh:/docker-entrypoint-initdb.d/mongo-init.sh
            - $PWD/.docker/mongo/replica.key:/data/replica.key
            
        command: ["--replSet", "replicaSet001", "--bind_ip_all",  "--keyFile", "/data/replica.key"]

But now I'm getting

Read security file failed","attr":{"error":{"code":30,"codeName":"InvalidPath","errmsg":"error opening file: /auth/file.key: bad file"}}}

I shell'ed into the container and can see that my file is being copied to the path i'm using, so I'm left to believe that I don't have the right permissions or that the mongod instance is not the owner of the file, but I'm not sure how can I change the ownership using the docker-compose.

I found this issue (#13) which is a little related, but the last question from @oxr463 wasn't anwered.

Can someone point me in the right direction to set this up?

Thanks.

[wglambert]

You can't do an image change like chmod from a docker-compose.yml, you'd have to make a Dockerfile like:

FROM mongo
COPY keyfile 
RUN chown 999:999 keyfile /data/replica.key

But since you're mounting the file into the container, then permissions set on it would be persisted when it's mounted into the image. So doing a chown 999:999 keyfile should work

[rrriki]

Thanks @wglambert

I think I've come up with something that works, not sure if I'm doing things right or not tho 😬

I didn't just want to do it on my machine and let the permissions carry on the copy to the container because I wanted it to work seamlessly for anyone cloning my repo and running the docker-compose up

Overwritting the entrypoint allows me to make the chown command needed to give access to the mongod instance

version: '3.4'

services:
        hostname: 'mongodb'
        container_name: 'mongo'
        image: 'mongo:latest'
        expose:
            - 27017
        ports:
            - 27017:27017
        environment:
            - MONGO_INITDB_DATABASE
            - MONGO_INITDB_ROOT_USERNAME
            - MONGO_INITDB_ROOT_PASSWORD
            - MONGO_USERNAME
            - MONGO_PASSWORD
            - MONGO_INITDB_DATABASE=${MONGO_INITDB_DATABASE}
            - MONGO_INITDB_ROOT_USERNAME=${MONGO_INITDB_ROOT_USERNAME}
            - MONGO_INITDB_ROOT_PASSWORD=${MONGO_INITDB_ROOT_PASSWORD}
        volumes:
            - mongodb_data_container:/data/db
            - $PWD/.docker/mongo/replica.key:/data/replica.key
            - $PWD/.docker/mongo/mongo-init.sh:/docker-entrypoint-initdb.d/mongo-init.sh

        entrypoint:
            - bash
            - -c
            - |
                chmod 400 /data/replica.key
                chown 999:999 /data/replica.key
                exec docker-entrypoint.sh $$@     
        command: "mongod --bind_ip_all --replSet replicaSet01 --keyFile /data/replica.key"

That solved the file access error, and now my issue was left on executing the rs.initiate() after the container started, because I started getting some “This node was not started with the replSet option” errors as described in this issue.

So I'm doing it in the mongo-init.sh with the hackish sleep referenced in that thread. Courtesy of @zhangyoufu 😅

[rrriki]

Closing since I found a work around.

[rrriki]

Hey @wglambert, sorry for bringing back and old issue. I just have one small problem that's been really annoying me and I'm not sure how to fix it, so I thought I'd ask.

I opted for the solution described above of changing the keyfile permissions on the entrypoint in the docker-file, by doing

      chmod 400 /data/replica.key
      chown 999:999 /data/replica.key

However, (I'm guessing because I set the file as a docker volume) the permission also get overwritten in my host machine, so I have to manually change them back when I'm making a commit, because otherwise I get a Permission Denied on the file by git.

Can you think of a workaround for this? 😅

[astrolemonade]

Alias it under a different name copy it and change the permissions on the copy

version: '3.4'

services:
        hostname: 'mongodb'
        container_name: 'mongo'
        image: 'mongo:latest'
        expose:
            - 27017
        ports:
            - 27017:27017
        environment:
            - MONGO_INITDB_DATABASE
            - MONGO_INITDB_ROOT_USERNAME
            - MONGO_INITDB_ROOT_PASSWORD
            - MONGO_USERNAME
            - MONGO_PASSWORD
            - MONGO_INITDB_DATABASE=${MONGO_INITDB_DATABASE}
            - MONGO_INITDB_ROOT_USERNAME=${MONGO_INITDB_ROOT_USERNAME}
            - MONGO_INITDB_ROOT_PASSWORD=${MONGO_INITDB_ROOT_PASSWORD}
        volumes:
            - mongodb_data_container:/data/db
            - $PWD/.docker/mongo/replica.key:/data/replica.key.devel
            - $PWD/.docker/mongo/mongo-init.sh:/docker-entrypoint-initdb.d/mongo-init.sh

        entrypoint:
            - bash
            - -c
            - |
                cp /data/replicaset.key.devel /data/replicaset.key
                chmod 400 /data/replica.key
                chown 999:999 /data/replica.key
                exec docker-entrypoint.sh $$@     
        command: "mongod --bind_ip_all --replSet replicaSet01 --keyFile /data/replica.key"

[wglambert]

Closing since this seems resolved, also in the future questions like these should over at the Docker Community Forums, Docker Community Slack, or Stack Overflow. Since these repos aren't really a user-help forum

[gustavz]

@rrriki what's inside your docker-entrypoint.sh?

[Camuvingian]

@rrriki what's inside your docker-entrypoint.sh?

It is defined in the mongo image see

https://github.com/docker-library/mongo/blob/master/4.2/Dockerfile
https://github.com/docker-library/mongo/blob/master/4.2/docker-entrypoint.sh#L303-L313

[SamSamhuns]

Thank you, guys, for all the information here. This had been driving me crazy for days. For future viewers, the following worked for me to set a three-container replica set with a separate initializing container and a mongo-express UI container.

# mongodb replicaset primary
  mongod1:
    container_name: mongod1
    image: mongo:6.0
    environment:
      MONGO_INITDB_ROOT_USERNAME: ${MONGO_INITDB_ROOT_USERNAME}
      MONGO_INITDB_ROOT_PASSWORD: ${MONGO_INITDB_ROOT_PASSWORD}
    ports:
      - ${MONGODB_PORT}:27017
    volumes:
      - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/mongodb/data1:/data/db
      - ${DOCKER_VOLUME_DIRECTORY:-.}/.docker/mongo/replica.key:/data/replica.key
    depends_on:
      - mongod2
      - mongod3
    restart: always
    command: "mongod --bind_ip_all --replSet dbrs --keyFile /data/replica.key"

  # mongodb replicaset secondary
  mongod2:
    container_name: mongod2
    image: mongo:6.0
    expose:
      - 27017
    volumes:
      - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/mongodb/data2:/data/db
      - ${DOCKER_VOLUME_DIRECTORY:-.}/.docker/mongo/replica.key:/data/replica.key
    restart: always
    command: "mongod --bind_ip_all --replSet dbrs --keyFile /data/replica.key"

  # mongodb replicaset arbiter
  mongod3:
    container_name: mongod3
    image: mongo:6.0
    expose:
      - 27017
    volumes:
      - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/mongodb/data3:/data/db
      - ${DOCKER_VOLUME_DIRECTORY:-.}/.docker/mongo/replica.key:/data/replica.key
    restart: always
    command: "mongod --bind_ip_all --replSet dbrs --keyFile /data/replica.key"

  mongo-setup:
    image: mongo:6.0
    container_name: mongo-setup
    environment:
      MONGO_INITDB_ROOT_USERNAME: ${MONGO_INITDB_ROOT_USERNAME}
      MONGO_INITDB_ROOT_PASSWORD: ${MONGO_INITDB_ROOT_PASSWORD}
    depends_on:
      - mongod1
      - mongod2
      - mongod3
    volumes:
      - ${DOCKER_VOLUME_DIRECTORY:-.}/scripts/mongodb_rs_init.sh:/scripts/mongodb_rs_init.sh
    restart: on-failure
    entrypoint: ["/bin/bash", "/scripts/mongodb_rs_init.sh"]

  mongo-express:
    container_name: mongo-express
    image: mongo-express
    restart: always
    ports:
      - ${MONGOEXP_PORT}:8081
    environment:
      ME_CONFIG_BASICAUTH_USERNAME: ${MONGOEXP_USERNAME}
      ME_CONFIG_BASICAUTH_PASSWORD: ${MONGOEXP_PASSWORD}
      ME_CONFIG_MONGODB_URL: "mongodb://${MONGO_INITDB_ROOT_USERNAME}:${MONGO_INITDB_ROOT_PASSWORD}@mongod1:27017/?replicaSet=dbrs"
    depends_on:
      - "mongod1"

The keyfile should be created with openssl based on the comments and links above.

The contents of mongodb_rs_init.sh which should be set to be executable mode chmod +x mongodb_rs_init.sh before running docker-compose:

#!/bin/bash

m1=mongod1
m2=mongod2
m3=mongod3
port=${PORT:-27017}

echo "###### Waiting for ${m1} instance startup.."
until mongosh --host ${m1}:${port} --eval 'quit(db.runCommand({ ping: 1 }).ok ? 0 : 2)' &>/dev/null; do
  printf '.'
  sleep 1
done
echo "###### Working ${m1} instance found, initiating user setup & initializing rs setup.."

# setup user + pass and initialize replica sets
mongosh --host ${m1}:${port} <<EOF
var rootUser = '$MONGO_INITDB_ROOT_USERNAME';
var rootPassword = '$MONGO_INITDB_ROOT_PASSWORD';
var admin = db.getSiblingDB('admin');
admin.auth(rootUser, rootPassword);

var config = {
    "_id": "dbrs",
    "version": 1,
    "members": [
        {
            "_id": 1,
            "host": "${m1}:${port}",
            "priority": 2
        },
        {
            "_id": 2,
            "host": "${m2}:${port}",
            "priority": 1
        },
        {
            "_id": 3,
            "host": "${m3}:${port}",
            "priority": 1,
            "arbiterOnly": true 
        }
    ]
};
rs.initiate(config, { force: true });
rs.status();
EOF

Hope this helps any frustrated soul in the future.

[n-0]

Tried to adapt and reuse astrolemonade solution for the docker-compose.yaml, but I only get validation errors, most of the time it's

services.entrypoint must be a mapping

Any idea how to fix this? Using docker compose version v2.19.1

[SamSamhuns]

I am using docker-compose version v2.17.2

[n-0]

Hm strange copy+pasting+editing was not okay but copying by hand gave no validation errors, though a

NotYetInitialized: Cannot use non-local read concern until replica set is finished initializing.

remained.
Playing around with @SamSamhuns solution did the trick, thank you so much (i still don't get why this goes wrong so easily).

I had one additional requirement, namely that I initialize another user for a different DB on the primary node (and also needed only one node in the replica set). For brevity posted the solution on stack overflow.

[apexbb]

follow @SamSamhuns @rrriki suggestion however, got

{"error":{"code":30,"codeName":"InvalidPath","errmsg":"permissions on /data/keyfile.pem are too open"}}}

already did the permission

RUN chmod 400 /data/keyfile.pem
RUN chown 999:999 /data/keyfile.pem

on Dockerfile, i was using mongo image version 6.0.3

[mit456]

I am using @SamSamhuns example and everything to working smooth. However, how do I connect from the host machine's Mongo Compass ?

I have been trying this connection string here mongodb://root:root%[email protected]:27017/?replicaSet=dbrs&authSource=admin but its not able to connect and throws getaddrinfo ENOTFOUND mongod2 or getaddrinfo ENOTFOUND mongod1

Please help.

[kasir-barati]

@mit456 I am connecting to a single node cluster like this, you can read my NodeJS code + .env. Read the README.md to learn how you should start it.

Consider giving it a star in case it was helpful

[clopezpro]

entrypoint:
            - bash
            - -c
            - |
                cp /data/replicaset.key.devel /data/replicaset.key
                chmod 400 /data/replica.key
                chown 999:999 /data/replica.key
                exec docker-entrypoint.sh $$@     

[clopezpro]

@mit456 I am connecting to a single node cluster like this, you can read my NodeJS code + .env. Read the README.md to learn how you should start it.

Consider giving it a star in case it was helpful
Thanks for sharing it, thanks to this thread I was able to understand in a practical way how the replication works in mongodb, I was stuck, I could already mount this on two VPS, now I have to investigate how to make the switch in the app if one fails

[lucifermorningstar1305]

Hey guys,

I followed @SamSamhuns solution but unfortunately I am still getting the following error:

{"error":{"code":30,"codeName":"InvalidPath","errmsg":"error opening file: /data/replica.key: bad file"}}

I generated the keyfile using the following command

openssl rand -base64 756 > replica.key
chmod 400 replica.key

I am currently running docker compose v2.27.0

[kasir-barati]

@lucifermorningstar1305 I am not sure if this still work or not but if you need a 3 nodes cluster. Or the other one for bitnami.

Lemme know if this worked for you. if this was helpful consider giving my repo a ⭐.