Stream Heroku Logs to AWS CloudWatch

by Mischa Spiegelmock

June 25, 2019

Heroku doesn't provide a native way of storing and retreiving old logs. Moreover, it doesn't provide any sort of monitoring or analyzing on the logs. This standalone app uses API Gateway and AWS Lambda to stream logs into CloudWatch in real time. After the logs are in CloudWatch, they can be further queried and analyzed by third party providers or even Cloudwatch Insights.

Quick Notes
Stream Heroku Logs to AWS CloudWatch

Solution Details

AWS API Gateway accepts the events from the Heroku Log Drain and parses it in a AWS Lambda. The logs can be further queried and analyzed by third party providers or even Cloudwatch Insights.

One advantage of this setup is that you can have your application emit logs with a minimum of blocking. At one point I had my application sending logs to Slack directly but this caused latency in the application any time I logged anything. By sending to Logplex on the other hand, I can process the application messages asynchronously without doing anything remotely fancy in my application. Another benefit is that you can handle your application, database, build, and deploy logs all the same unified fashion.

Setup

Prerequisites Setup AWS Credentials
Make sure you have AWS access key and secrete keys setup locally, following this video here

Download the code

git clone git@github.com:CodeRecipe-dev/Heroku-log-AWS-cloudwatch.git

Deploy to the cloud

cd Heroku-log-AWS-cloudwatch

npm install

serverless deploy --stage beta

Add endpoint as Log Drain on Heroku

heroku drains:add <api endpoint from sls deploy> -a <heroku app name>

After following the instructions above, any logs from heroku will also show up in CloudWatch for the Lambda function. Here's an example of what you will see in CloudWatch and its corresponding Heroku log:

Heroku Logs: Heroku Logs

CloudWatch Logs:

Starting from the second line, exact same logs show up in CloudWatch as well

CloudWatch Logs

source code
platforms

Heroku

AWS