Amazon generates a lot of logs via VPC Flow Logs, CloudTrail, S3 access logs, CloudWatch (See the end of the blog article for a full list.) Additionally, there are OS, Application, web server logs. That is a lot of data which provides valuable insight into your running AWS environment. What are you doing to manage this log files? What are you doing with those log files? What are you doing to analysis these log files?
There are a lot of logging solutions available that integrate with AWS. Honestly, I’m a big fan of Splunk and have set it up multiple times. However, I wanted to look at something else for this blog article. Something open source and relatively low cost. This blog is going to explain what I did to setup Graylog. Graylog has no charges for the software, but you’re going to get charged for the instance, Kinesis, SQS, and data storage. It actually a good exercise if to familiarize yourself with AWS services, especially for the Sysops exams.
Graylog provides great instructions. I followed the steps remember to use their image which is already self-built on Ubuntu. One difference with this setup, I didn’t use a 4GB memory systems. I picked a t2.small which proves 1vCPU and 2GB of memory. I didn’t notice performance issues. Remember to allow ports 443 and 9000 in security groups and the Networking ACLs. I prefer to run this over HTTPS. And it bugs me when you see NOT SECURE HTTP: I installed an SSL certificate, and this is how I did it.
- Create a DNS name
- Get a free certificate
- Install the Certificate as such
Now my instance is up, and I can log into the console. I want to get my AWS logs into Graylog. To do this is requires the logs sent to Kinesis or SQS. I am not going to explain the SQS setup as there plenty of resources for the specific AWS Service. Also, the Graylog Plugin describes how to do this. Graylog plugin for CloudTrail, CloudWatch and VPC Flow logs is available on Github at Graylog Plugin for AWS.
What about access_logs? Graylog has the Graylog Collector Sidecar. I’m not going to rehash the installation instructions here as there are great installation instructions. Graylog has a great documentation. Also if you are looking for something not covered here, it will be in the documentation or in their Github project.
What are you using as your log collection processing service on Amazon?
List of AWS Servers generating logs:
Amazon S3 Access logs Amazon CloudFront Access logs Elastic Load Balancer (ELB) logs Amazon Relational Database Service (RDS) logs Amazon Elastic MapReduce (EMR) logs Amazon Redshift logs AWS Elastic Beanstalk logs AWS OpsWorks logs (or this link) AWS Import/Export logs AWS Data Pipeline logs AWS CloudTrail logs
Amazon generates a lot of logs via VPC Flow Logs, CloudTrail, S3 access logs, CloudWatch (See the end of the blog article for a full list.) Additionally, there are OS, Application, web server logs. That is a lot of data which provides valuable insight into your running AWS environment. What are you doing to manage this log files? What are you doing with those log files? What are you doing to analysis these log files?
There are a lot of logging solutions available that integrate with AWS. Honestly, I’m a big fan of Splunk and have set it up multiple times. However, I wanted to look at something else for this blog article. Something open source and relatively low cost. This blog is going to explain what I did to setup Graylog. Graylog has no charges for the software, but you’re going to get charged for the instance, Kinesis, SQS, and data storage. It actually a good exercise if to familiarize yourself with AWS services, especially for the Sysops exams.
Graylog provides great instructions. I followed the steps remember to use their image which is already self-built on Ubuntu. One difference with this setup, I didn’t use a 4GB memory systems. I picked a t2.small which proves 1vCPU and 2GB of memory. I didn’t notice performance issues. Remember to allow ports 443 and 9000 in security groups and the Networking ACLs. I prefer to run this over HTTPS. And it bugs me when you see NOT SECURE HTTP: I installed an SSL certificate, and this is how I did it.
- Create a DNS name
- Get a free certificate
- Install the Certificate as such
Now my instance is up, and I can log into the console. I want to get my AWS logs into Graylog. To do this is requires the logs sent to Kinesis or SQS. I am not going to explain the SQS setup as there plenty of resources for the specific AWS Service. Also, the Graylog Plugin describes how to do this. Graylog plugin for CloudTrail, CloudWatch and VPC Flow logs is available on Github at Graylog Plugin for AWS.
What about access_logs? Graylog has the Graylog Collector Sidecar. I’m not going to rehash the installation instructions here as there are great installation instructions. Graylog has a great documentation. Also if you are looking for something not covered here, it will be in the documentation or in their Github project.
What are you using as your log collection processing service on Amazon?
List of AWS Servers generating logs:
Amazon S3 Access logs Amazon CloudFront Access logs Elastic Load Balancer (ELB) logs Amazon Relational Database Service (RDS) logs Amazon Elastic MapReduce (EMR) logs Amazon Redshift logs AWS Elastic Beanstalk logs AWS OpsWorks logs (or this link) AWS Import/Export logs AWS Data Pipeline logs AWS CloudTrail logs
