Support Mesos Framework Authorisation

[philwinder]

• Accept a principal as a command-line argument.
• Accept a path to a secret as a command-line argument.
• Pass principal and secret to the SchedulerDriver
• Set the principal value in the FrameworkInfo message during registration.
• Tested on Pure Mesos cluster. See below. (Support Mesos Framework Authorisation #218 (comment))

Still does not work on DCOS.

Notes: Secret should not be necessarily required, for maximum flexibility in auth modes.

Same as: d2iq-archive/marathon#638

Links: https://github.com/mesosphere/marathon/blob/c7624e0c3778da02919e743e8b9eda7248acd1a1/src/main/scala/mesosphere/marathon/MarathonSchedulerDriver.scala#L47
http://mesos.apache.org/documentation/latest/authorization/
http://mesos.apache.org/blog/framework-authentication-in-apache-mesos-0-15-0/

See DCOS PR d2iq-archive/universe#193

[philwinder]

Note from Mesosphere
alexander.mesosphere [3:25 PM]
@pnw.trifork: I’ve been investigating the issue and it is a bug, apparently the libsasl2 plugins are not being distributed with the dcos image, in particular libcrammd5.so. I will try to get the image updated
Notes: Secret should not be necessarily required, for maximum flexibility in auth modes.

[philwinder]

Note to self: When enabling auth on DCOS this is how to do it.

echo "user1 secret" | sudo tee -a /opt/mesosphere/etc/passwd
echo "MESOS_CREDENTIALS=/opt/mesosphere/etc/passwd" | sudo tee -a /opt/mesosphere/environment
echo "MESOS_AUTHENTICATE=true" | sudo tee -a /opt/mesosphere/environment
echo "MARATHON_MESOS_AUTHENTICATION_SECRET_FILE=/opt/mesosphere/etc/passwd" | sudo tee -a /opt/mesosphere/environment
echo "MARATHON_MESOS_AUTHENTICATION_PRINCIPAL=user1" | sudo tee -a /opt/mesosphere/environment

However, the MARATHON_ commands, even though they should. So we have to set this:

Append this to end of exec line in marathon.service:
--mesos_authentication_principal "user1" --mesos_authentication_secret_file "/opt/mesosphere/etc/passwd"

Then run the 218 branch with the following JSON:

{
  "id": "elasticsearch-mesos-scheduler",
  "container": {
    "docker": {
      "image": "pnwtrifork/scheduler-auth",
      "network": "HOST"
    }
  },
  "args": ["-n", "1", "-zk", "zk://52.18.157.54:2181/mesos", "-ram", "2048", "-principal", "user1", "-secret", "secret"],
  "cpus": 0.2,
  "mem": 512.0,
  "env": {
    "JAVA_OPTS": "-Xms128m -Xmx256m"
  },
  "instances": 1
}

Mesosphere bug still stands as of 27/08/15

[philwinder]

The Auth branch has been tested working on a pure mesos cluster. Instructions to replicate are below.

Setup

A single AWS micro instance was setup using the AMI: mesos-0.22.1-ubuntu-14.04_amd64_0.22.1.0 (ami-47b0c930)

This installs ubuntu, mess, zookeeper and marathon. Zookeeper starts as a service, but Mesos needs to be started from the command line.

Running Mesos

The following commands will write the username and passwords to two files. One for Mesos and one for the framework. Note how the echo commands use the -n parameter to prevent placing a newline at the end of the file. Auth will not work with a newline at the end. Also note that the framework secret file only contains the password, not the username.

echo -n "user1 secret" | sudo tee -a /etc/mesos/passwd && echo -n "secret" | sudo tee -a /etc/mesos/frameworkpasswd

Next we start a Mesos slave and Mesos master on the same machine. We also start marathon using the framework credentials. Zookeeper is required, since ES needs it to start up. Groups must be used, as this is not the default. Containerises must be used, as this is not the default.

Please change the ip address to match the local ip address of your test machine

nohup sudo mesos-master --ip=172.31.41.119 --work_dir=/var/lib/mesos --authenticate=true --credentials=/etc/mesos/passwd --zk=zk://127.0.0.1:2181/mesos --quorum=1 & nohup sudo mesos-sla ve --ip=172.31.41.119 --work_dir=/var/lib/mesos --master=zk://127.0.0.1:2181/mesos --containerizers=docker,mesos --isolation=cgroups/cpu,cgroups/mem & nohup sudo /opt/marathon/bin/start --master zk://127.0.0.1:21 81/mesos --mesos_authentication_principal "user1" --mesos_authentication_secret_file "/etc/mesos/frameworkpasswd" &

Marathon command

A volume is added so that the user can pass the secret file through to the container. The secret must be a file. In this case, we are using the frameworkpasswd file from before. The number of executors is zero, to reduce ram requirements. All we are testing here is that the scheduler is able to register the framework without crashing. Beware of using too little RAM for the scheduler. It may get OOM reaped by the kernel before it starts. It you see random "killed" messages in the docker logs, then check dmesg. It was probably OOM.

echo '{
  "id": "elasticsearch-auth",
  "cpus": 0.5,
  "mem": 256.0,
  "instances": 1,
  "args": ["--zookeeperMesosUrl", "zk://127.0.0.1:2181", "--frameworkPrincipal", "user1", "--frameworkSecretPath", "/etc/mesos/frameworkpasswd", "--elasticsearchNodes", "1"],
  "env": {
    "JAVA_OPTS": "-Xms128m -Xmx256m"
  },
  "container": {
    "type": "DOCKER",
    "docker": {
      "image": "mesos/elasticsearch-scheduler",
      "network": "HOST",
      "forcePullImage": true
    },
    "volumes": [
      {
        "containerPath": "/etc/mesos/frameworkpasswd",
        "hostPath": "/etc/mesos/frameworkpasswd",
        "mode": "RO"
      }
    ]
  },
  "ports": [31100],
  "requirePorts": true
}' > marathon-auth.json

curl -XPOST -H 'Content-Type:application/json' -d @marathon-auth.json http://52.19.121.213:8080/v2/apps

[philwinder]

Fixed in #220