params.json

{
  "name": "Systemd-cloud-watch",
  "tagline": "systemd-cloud-watch",
  "body": "# systemd-cloud-watch\r\n\r\nThis is an alternative process to the AWS-provided logs agent.\r\nThe AWS logs agent copies data from on-disk text log files into [Cloudwatch](https://aws.amazon.com/cloudwatch/).\r\n\r\nThis utility reads from the systemd journal and sends the data in batches to Cloudwatch.\r\n\r\n\r\n## Derived\r\nThis is based on [advantageous journald-cloudwatch-logs](https://github.com/advantageous/journald-cloudwatch-logs)\r\nwhich was forked from [saymedia journald-cloudwatch-logs](https://github.com/saymedia/journald-cloudwatch-logs).\r\n\r\n## Status\r\nIt is close to being done. \r\n\r\n\r\nImprovements:\r\n\r\n* Added unit tests (there were none).\r\n* Added cross compile so I can develop/test on my laptop (MacOS).\r\n* Made logging stateless. No more need for a state file.\r\n* No more getting out of sync with CloudWatch.\r\n* Detects being out of sync and recovers.\r\n* Fixed error with log messages being too big.\r\n* Added ability to include or omit logging fields.\r\n* Created docker image and scripts to test on Linux (CentOS7).\r\n* Created EC2 image and scripts to test on Linux running in AWS EC2 (CentOS7).\r\n* Code organization (we use a package).\r\n* Added comprehensive logging which includes debug logging by config.\r\n* Uses actual timestamp from journal log record instead of just current time\r\n* Auto-creates CloudWatch log group if it does not exist\r\n\r\n\r\n## Log format\r\n\r\nThe journal event data is written to ***CloudWatch*** Logs in JSON format, making it amenable to filtering using the JSON filter syntax.\r\nLog records are translated to ***CloudWatch*** JSON events using a structure like the following:\r\n\r\n#### Sample log\r\n```javascript\r\n{\r\n    \"instanceId\": \"i-xxxxxxxx\",\r\n    \"pid\": 12354,\r\n    \"uid\": 0,\r\n    \"gid\": 0,\r\n    \"cmdName\": \"cron\",\r\n    \"exe\": \"/usr/sbin/cron\",\r\n    \"cmdLine\": \"/usr/sbin/CRON -f\",\r\n    \"systemdUnit\": \"cron.service\",\r\n    \"bootId\": \"fa58079c7a6d12345678b6ebf1234567\",\r\n    \"hostname\": \"ip-10-1-0-15\",\r\n    \"transport\": \"syslog\",\r\n    \"priority\": \"INFO\",\r\n    \"message\": \"pam_unix(cron:session): session opened for user root by (uid=0)\",\r\n    \"syslog\": {\r\n        \"facility\": 10,\r\n        \"ident\": \"CRON\",\r\n        \"pid\": 12354\r\n    },\r\n    \"kernel\": {}\r\n}\r\n```\r\n\r\nThe JSON-formatted log events could also be exported into an AWS ElasticSearch instance using the ***CloudWatch***\r\nsync mechanism. Once in ElasticSearch, you can use an ELK stack to obtain more elaborate filtering and query capabilities.\r\n\r\n\r\n## Installation\r\n\r\nIf you have a binary distribution, you just need to drop the executable file somewhere.\r\n\r\nThis tool assumes that it is running on an EC2 instance.\r\n\r\nThis tool uses `libsystemd` to access the journal. systemd-based distributions generally ship\r\nwith this already installed, but if yours doesn't you must manually install the library somehow before\r\nthis tool will work.\r\n\r\nThere are instructions on how to install the Linux requirements for development below see - \r\n[Setting up a Linux env for testing/developing (CentOS7)](#setting-up-a-linux-env-for-testingdeveloping-centos7).\r\n\r\nWe also have two excellent examples of setting up a dev environment using [packer](https://www.packer.io/) for both \r\n[AWS EC2](#building-the-ec2-image-with-packer-to-build-the-linux-instance-to-build-this-project) and \r\n[Docker](#building-the-docker-image-to-build-the-linux-instance-to-build-this-project). We setup CentoOS 7.\r\nThe EC2 instance packer build uses the ***aws command line*** to create and connect to a running image. \r\nThese should be instructive for how to setup this utility in your environment to run with ***systemd*** as we provide\r\nall of the systemd scripts in the packer provision scripts for EC2.  An example is good. A running example is better.\r\n\r\n## Configuration\r\n\r\nThis tool uses a small configuration file to set some values that are required for its operation.\r\nMost of the configuration values are optional and have default settings, but a couple are required.\r\n\r\nThe configuration file uses a syntax like this:\r\n\r\n```js\r\nlog_group = \"my-awesome-app\"\r\n\r\n```\r\n\r\nThe following configuration settings are supported:\r\n\r\n* `aws_region`: (Optional) The AWS region whose CloudWatch Logs API will be written to. If not provided,\r\n  this defaults to the region where the host EC2 instance is running.\r\n\r\n* `ec2_instance_id`: (Optional) The id of the EC2 instance on which the tool is running. There is very\r\n  little reason to set this, since it will be automatically set to the id of the host EC2 instance.\r\n\r\n* `journal_dir`: (Optional) Override the directory where the systemd journal can be found. This is\r\n  useful in conjunction with remote log aggregation, to work with journals synced from other systems.\r\n  The default is to use the local system's journal.\r\n\r\n* `log_group`: (Required) The name of the cloudwatch log group to write logs into. This log group must\r\n  be created before running the program.\r\n\r\n* `log_priority`: (Optional) The highest priority of the log messages to read (on a 0-7 scale). This defaults\r\n    to DEBUG (all messages). This has a behaviour similar to `journalctl -p <priority>`. At the moment, only\r\n    a single value can be specified, not a range. Possible values are: `0,1,2,3,4,5,6,7` or one of the corresponding\r\n    `\"emerg\", \"alert\", \"crit\", \"err\", \"warning\", \"notice\", \"info\", \"debug\"`.\r\n    When a single log level is specified, all messages with this log level or a lower (hence more important)\r\n    log level are read and pushed to CloudWatch. For more information about priority levels, look at\r\n    https://www.freedesktop.org/software/systemd/man/journalctl.html\r\n\r\n* `log_stream`: (Optional) The name of the cloudwatch log stream to write logs into. This defaults to\r\n  the EC2 instance id. Each running instance of this application (along with any other applications\r\n  writing logs into the same log group) must have a unique `log_stream` value. If the given log stream\r\n  doesn't exist then it will be created before writing the first set of journal events.\r\n\r\n* `buffer_size`: (Optional) The size of the local event buffer where journal events will be kept\r\n  in order to write batches of events to the CloudWatch Logs API. The default is 100. A batch of\r\n  new events will be written to CloudWatch Logs every second even if the buffer does not fill, but\r\n  this setting provides a maximum batch size to use when clearing a large backlog of events, e.g.\r\n  from system boot when the program starts for the first time.\r\n\r\n* `fields`: (Optional) Specifies which fields should be included in the JSON map that is sent to CloudWatch.\r\n\r\n* `omit_fields`: (Optional) Specifies which fields should NOT be included in the JSON map that is sent to CloudWatch.\r\n\r\n* `field_length`: (Optional) Specifies how long string fileds can be in the JSON  map that is sent to CloudWatch.\r\n   The default is 255 characters.\r\n\r\n* `debug`: (Optional) Turns on debug logging.\r\n\r\n* `local`: (Optional) Used for unit testing. Will not try to create an AWS meta-data client to read region and AWS credentials.\r\n\r\n\r\n\r\n### AWS API access\r\n\r\nThis program requires access to call some of the Cloudwatch API functions. The recommended way to\r\nachieve this is to create an\r\n[IAM Instance Profile](http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2_instance-profiles.html)\r\nthat grants your EC2 instance a role that has Cloudwatch API access. The program will automatically\r\ndiscover and make use of instance profile credentials.\r\n\r\nThe following IAM policy grants the required access across all log groups in all regions:\r\n\r\n```js\r\n{\r\n    \"Version\": \"2012-10-17\",\r\n    \"Statement\": [\r\n        {\r\n            \"Effect\": \"Allow\",\r\n            \"Action\": [\r\n                \"logs:CreateLogStream\",\r\n                \"logs:PutLogEvents\",\r\n                \"logs:DescribeLogStreams\"\r\n            ],\r\n            \"Resource\": [\r\n                \"arn:aws:logs:*:*:log-group:*\",\r\n                \"arn:aws:logs:*:*:log-group:*:log-stream:*\"\r\n            ]\r\n        }\r\n    ]\r\n}\r\n```\r\n\r\nIn more complex environments you may want to restrict further which regions, groups and streams\r\nthe instance can write to. You can do this by adjusting the two ARN strings in the `\"Resource\"` section:\r\n\r\n* The first `*` in each string can be replaced with an AWS region name like `us-east-1`\r\n  to grant access only within the given region.\r\n* The `*` after `log-group` in each string can be replaced with a Cloudwatch Logs log group name\r\n  to grant access only to the named group.\r\n* The `*` after `log-stream` in the second string can be replaced with a Cloudwatch Logs log stream\r\n  name to grant access only to the named stream.\r\n\r\nOther combinations are possible too. For more information, see\r\n[the reference on ARNs and namespaces](http://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html#arn-syntax-cloudwatch-logs).\r\n\r\n\r\n\r\n### Coexisting with the official Cloudwatch Logs agent\r\n\r\nThis application can run on the same host as the official Cloudwatch Logs agent but care must be taken\r\nto ensure that they each use a different log stream name. Only one process may write into each log\r\nstream.\r\n\r\n## Running on System Boot\r\n\r\nThis program is best used as a persistent service that starts on boot and keeps running until the\r\nsystem is shut down. If you're using `journald` then you're presumably using systemd; you can create\r\na systemd unit for this service. For example:\r\n\r\n```\r\n[Unit]\r\nDescription=journald-cloudwatch-logs\r\nWants=basic.target\r\nAfter=basic.target network.target\r\n\r\n[Service]\r\nUser=nobody\r\nGroup=nobody\r\nExecStart=/usr/local/bin/journald-cloudwatch-logs /usr/local/etc/journald-cloudwatch-logs.conf\r\nKillMode=process\r\nRestart=on-failure\r\nRestartSec=42s\r\n```\r\n\r\nThis program is designed under the assumption that it will run constantly from some point during\r\nsystem boot until the system shuts down.\r\n\r\nIf the service is stopped while the system is running and then later started again, it will\r\n\"lose\" any journal entries that were written while it wasn't running. However, on the initial\r\nrun after each boot it will clear the backlog of logs created during the boot process, so it\r\nis not necessary to run the program particularly early in the boot process unless you wish\r\nto *promptly* capture startup messages.\r\n\r\n## Building\r\n\r\n#### Test cloud-watch package\r\n```sh\r\ngo test -v  github.com/advantageous/systemd-cloud-watch/cloud-watch\r\n```\r\n\r\n\r\n#### Build and Test on Linux (Centos7)\r\n```sh\r\n ./run_build_linux.sh\r\n```\r\n\r\nThe above starts up a docker container, runs `go get`, `go build`, `go test` and then copies the binary to\r\n`systemd-cloud-watch_linux`.\r\n\r\n#### Debug process running Linux\r\n```sh\r\n ./run_test_container.sh\r\n```\r\n\r\n\r\nThe above starts up a docker container that you can develop with that has all the prerequisites needed to\r\ncompile and test this project.\r\n\r\n#### Sample debug session\r\n```sh\r\n$ ./run_test_container.sh\r\nlatest: Pulling from advantageous/golang-cloud-watch\r\nDigest: sha256:eaf5c0a387aee8cc2d690e1c5e18763e12beb7940ca0960ce1b9742229413e71\r\nStatus: Image is up to date for advantageous/golang-cloud-watch:latest\r\n[root@6e0d1f984c03 /]# cd gopath/src/github.com/advantageous/systemd-cloud-watch/\r\n.git/                      README.md                  cloud-watch/               packer/                    sample.conf                \r\n.gitignore                 build_linux.sh             main.go                    run_build_linux.sh         systemd-cloud-watch.iml    \r\n.idea/                     cgroup/                    output.json                run_test_container.sh      systemd-cloud-watch_linux  \r\n\r\n[root@6e0d1f984c03 /]# cd gopath/src/github.com/advantageous/systemd-cloud-watch/\r\n\r\n[root@6e0d1f984c03 systemd-cloud-watch]# ls\r\nREADME.md  build_linux.sh  cgroup  cloud-watch  main.go  output.json  packer  run_build_linux.sh  \r\nrun_test_container.sh  sample.conf  systemd-cloud-watch.iml  systemd-cloud-watch_linux\r\n\r\n[root@6e0d1f984c03 systemd-cloud-watch]# source ~/.bash_profile\r\n\r\n[root@6e0d1f984c03 systemd-cloud-watch]# export GOPATH=/gopath\r\n\r\n[root@6e0d1f984c03 systemd-cloud-watch]# /usr/lib/systemd/systemd-journald &\r\n[1] 24\r\n\r\n[root@6e0d1f984c03 systemd-cloud-watch]# systemd-cat echo \"RUNNING JAVA BATCH JOB - ADF BATCH from `pwd`\"\r\n\r\n[root@6e0d1f984c03 systemd-cloud-watch]# echo \"Running go clean\"\r\nRunning go clean\r\n\r\n[root@6e0d1f984c03 systemd-cloud-watch]# go clean\r\n\r\n[root@6e0d1f984c03 systemd-cloud-watch]# echo \"Running go get\"\r\nRunning go get\r\n\r\n[root@6e0d1f984c03 systemd-cloud-watch]# go get\r\n\r\n[root@6e0d1f984c03 systemd-cloud-watch]# echo \"Running go build\"\r\nRunning go build\r\n[root@6e0d1f984c03 systemd-cloud-watch]# go build\r\n\r\n[root@6e0d1f984c03 systemd-cloud-watch]# echo \"Running go test\"\r\nRunning go test\r\n\r\n[root@6e0d1f984c03 systemd-cloud-watch]# go test -v github.com/advantageous/systemd-cloud-watch/cloud-watch\r\n=== RUN   TestRepeater\r\nconfig DEBUG: 2016/11/30 08:53:34 config.go:66: Loading log...\r\naws INFO: 2016/11/30 08:53:34 aws.go:42: Config set to local\r\naws INFO: 2016/11/30 08:53:34 aws.go:72: Client missing credentials not looked up\r\naws INFO: 2016/11/30 08:53:34 aws.go:50: Client missing using config to set region\r\naws INFO: 2016/11/30 08:53:34 aws.go:52: AWSRegion missing using default region us-west-2\r\nrepeater ERROR: 2016/11/30 08:53:44 cloudwatch_journal_repeater.go:141: Error from putEvents NoCredentialProviders: no valid providers in chain. Deprecated.\r\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors\r\n--- SKIP: TestRepeater (10.01s)\r\n\tcloudwatch_journal_repeater_test.go:43: Skipping WriteBatch, you need to setup AWS credentials for this to work\r\n=== RUN   TestConfig\r\ntest DEBUG: 2016/11/30 08:53:44 config.go:66: Loading log...\r\ntest INFO: 2016/11/30 08:53:44 config_test.go:33: [Foo Bar]\r\n--- PASS: TestConfig (0.00s)\r\n=== RUN   TestLogOmitField\r\ntest DEBUG: 2016/11/30 08:53:44 config.go:66: Loading log...\r\n--- PASS: TestLogOmitField (0.00s)\r\n=== RUN   TestNewJournal\r\n--- PASS: TestNewJournal (0.00s)\r\n=== RUN   TestSdJournal_Operations\r\n--- PASS: TestSdJournal_Operations (0.00s)\r\n\tjournal_linux_test.go:41: Read value=Runtime journal is using 8.0M (max allowed 4.0G, trying to leave 4.0G free of 55.1G available → current limit 4.0G).\r\n=== RUN   TestNewRecord\r\ntest DEBUG: 2016/11/30 08:53:44 config.go:66: Loading log...\r\n--- PASS: TestNewRecord (0.00s)\r\n=== RUN   TestLimitFields\r\ntest DEBUG: 2016/11/30 08:53:44 config.go:66: Loading log...\r\n--- PASS: TestLimitFields (0.00s)\r\n=== RUN   TestOmitFields\r\ntest DEBUG: 2016/11/30 08:53:44 config.go:66: Loading log...\r\n--- PASS: TestOmitFields (0.00s)\r\nPASS\r\nok  \tgithub.com/advantageous/systemd-cloud-watch/cloud-watch\t10.017s\r\n```\r\n\r\n\r\n\r\n\r\n#### Building the docker image to build the linux instance to build this project\r\n\r\n```sh\r\n# from project root\r\ncd packer\r\npacker build packer_docker.json\r\n```\r\n\r\n\r\n#### To run docker dev image\r\n```sh\r\n# from project root\r\ncd packer\r\n./run.sh\r\n\r\n```\r\n\r\n#### Building the ec2 image with packer to build the linux instance to build this project\r\n\r\n```sh\r\n# from project root\r\ncd packer\r\npacker build packer_ec2.json\r\n```\r\n\r\nWe use the [docker](https://www.packer.io/docs/builders/docker.html) support for [packer](https://www.packer.io/).\r\n(\"Packer is a tool for creating machine and container images for multiple platforms from a single source configuration.\")\r\n\r\nUse `ec2_env.sh_example` to create a `ec2_env.sh` with the instance id that was just created. \r\n\r\n#### ec2_env.sh_example\r\n```\r\n#!/usr/bin/env bash\r\nexport ami=ami-YOURAMI\r\nexport subnet=subnet-YOURSUBNET\r\nexport security_group=sg-YOURSG\r\nexport iam_profile=YOUR_IAM_ROLE\r\nexport key_name=MY_PEM_FILE_KEY_NAME\r\n\r\n```\r\n\r\n##### Using EC2 image (assumes you have ~/.ssh config setup)\r\n```sh\r\n# from project root\r\ncd packer\r\n\r\n# Run and log into dev env running in EC2\r\n./runEc2Dev.sh\r\n\r\n# Log into running server\r\n./loginIntoEc2Dev.sh\r\n\r\n```\r\n\r\n\r\n\r\n\r\n\r\n## Setting up a Linux env for testing/developing (CentOS7).\r\n```sh\r\nyum -y install wget\r\nyum install -y git\r\nyum install -y gcc\r\nyum install -y systemd-devel\r\n\r\n\r\necho \"installing go\"\r\ncd /tmp\r\nwget https://storage.googleapis.com/golang/go1.7.3.linux-amd64.tar.gz\r\ntar -C /usr/local/ -xzf go1.7.3.linux-amd64.tar.gz\r\nrm go1.7.3.linux-amd64.tar.gz\r\necho 'export PATH=$PATH:/usr/local/go/bin' >> ~/.bash_profile\r\n```\r\n\r\n## Setting up Java to write to systemd journal\r\n\r\n#### gradle build\r\n```\r\ncompile 'org.gnieh:logback-journal:0.2.0'\r\n\r\n```\r\n\r\n#### logback.xml\r\n```xml\r\n<?xml version=\"1.0\" encoding=\"UTF-8\"?>\r\n<configuration>\r\n\r\n    <appender name=\"journal\" class=\"org.gnieh.logback.SystemdJournalAppender\" />\r\n\r\n    <root level=\"INFO\">\r\n        <appender-ref ref=\"journal\" />\r\n        <customFields>{\"serviceName\":\"adfCalcBatch\",\"serviceHost\":\"${HOST}\"}</customFields>\r\n    </root>\r\n\r\n\r\n    <logger name=\"com.mycompany\" level=\"INFO\"/>\r\n\r\n</configuration>\r\n```\r\n\r\n## Commands for controlling systemd service EC2 dev env\r\n\r\n```sh\r\n# Get status\r\nsudo systemctl status journald-cloudwatch\r\n# Stop Service\r\nsudo systemctl stop journald-cloudwatch\r\n# Find the service\r\nps -ef | grep cloud\r\n# Run service manually\r\n/usr/bin/systemd-cloud-watch_linux /etc/journald-cloudwatch.conf\r\n\r\n```\r\n\r\n## License\r\n\r\nCopyright (c) 2015 Say Media Inc\r\n\r\nPermission is hereby granted, free of charge, to any person obtaining a copy\r\nof this software and associated documentation files (the \"Software\"), to deal\r\nin the Software without restriction, including without limitation the rights\r\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\r\ncopies of the Software, and to permit persons to whom the Software is\r\nfurnished to do so, subject to the following conditions:\r\n\r\nThe above copyright notice and this permission notice shall be included in all\r\ncopies or substantial portions of the Software.\r\n\r\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\r\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\r\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\r\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\r\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\r\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\r\nSOFTWARE.\r\n\r\nAll additional work is covered under Apache 2.0 license.\r\nCopyright (c) 2016 Geoff Chandler, Rick Hightower\r\n",
  "note": "Don't delete this file! It's used internally to help with page regeneration."
}