Papertrail Logging Naming with AWS Autoscaling Groups

Off-box logging is vital to everyone's sanity and success when in AWS.

Using Papertrail has been painless and the features are great! There are a many different ways to integrate with them, and forwarding your syslog to them is a good first step.

The problem I ran into was that my logs were showing up by hostname.  In AWS, a hostname of "ip-nnn-nnn-nnn-nnn" is not very clear.  What application does that instance belong to?  Is that instance part of staging or production?  Add another complication - AutoScaling.  Now when an instance comes online and goes away after a few hours, it gets even more difficult to understand what is occurring when you look at the logs.

In my UserData, in the CloudFormation Template, I just added these simple bash lines to change the hostname to something meaningful and dynamic:

1 2 3 4 5 6 7

{ "Fn::Join" : ["", [ "hs=", { "Ref" : "AppName" }, "-", { "Ref" : "Environment" }, "/$(curl -s http://169.254.169.254/latest/meta-data/instance-id)\n" ] ] }, "ipAddr=$(ip addr | grep 'state UP' -A2 | tail -n1 | awk '{print $2}' | cut -f1  -d'/')\n", "echo '$ipAddr $hs' >> /etc/hosts\n", "echo '127.0.0.1 $hs' >> /etc/hosts\n", "echo $hs > /etc/hostname\n", "hostname -F /etc/hostname\n",

The CloudFormation Template has two input parameters (AppName and Environment) and uses a Join to put the hostname variable together.

So now the hostname will be <ApplicationName>-<Environment>/<InstanceID>.

Back in Papertrail, I can now group these systems using wildcards.