Turn down log levels on most chatty services

[BrianHicks]

• Installs cleanly on a fresh build of most recent master branch
• Upgrades cleanly from the most recent release
• Updates documentation relevant to the changes
• Rebases cleanly onto the latest master

This PR turns down the log levels of quite a few components. It takes the total logs per hour down from 45,000-55,000 lines per hour to around 6,000 per hour on a cluster with minimal use.

Changes made:

• set Consul, consul-template, MantlUI, Mesos, ZooKeeper, Chronos, and Marathon's log levels to WARN
• install fail2ban to reduce log spam from sshd and improve cluster security when over IPv4

[BrianHicks]

Just to drive the point home, here's what this looks like in Kibana:

(two days over the weekend I left the cluster running, then fixed Marathon on Monday. That's why the long plateau.)

[langston-barrett]

Can we add the possible options to the README?

[BrianHicks]

yes, the options will all be documented once it looks good otherwise.

[ryane]

applied cleanly over a 1.0.3 cluster on GCE.

[ryane]

Installed cleanly on a fresh build of most recent master branch (GCE)

were you going to add more docs? otherwise, lgtm

[BrianHicks]

Yes, I'll add docs.

[ryane]

actually, do we want to change the log level for consul itself (in /etc/consul/consul.json)? afaict, this is not set and the default level is 'info'

[BrianHicks]

@ryane I'm not sure what you're asking actually. Are you saying that we should change it or should not change it?

[ryane]

The log level for consul appears to still be 'info'. I just wasn't sure if that was intentional or not. Otherwise, we should set it to 'warn', right?

[BrianHicks]

@ryane https://github.com/CiscoCloud/mantl/pull/1241/files#diff-2ecd0b0955c7961aca7aedfe5a118873R6

[ryane]

@BrianHicks I was referring to the consul service itself:

$ consul monitor
2016/03/09 22:36:37 [INFO] agent: Synced check 'distributive-zookeeper-checks'
2016/03/09 22:37:36 [INFO] agent: Synced check 'distributive-common-checks'
2016/03/09 22:37:59 [INFO] agent.rpc: Accepted client: 127.0.0.1:56836

log_level can be set in /etc/consul/consul.json (https://www.consul.io/docs/agent/options.html#log_level)

[ryane]

since this has already been tested as-is, I'm just going to merge it. I opened #1256 with the consul log level.