Python - Lambda-based application for generating a cost report with graphs, including month-on-month cost changes as an Excel sheet. This application generates month-on-month costs acquired by the resources in multiple AWS accounts held by the company and uploaded into the S3 bucket with the Static website.
Project Details
This project uses the AWS Cost Explorer API for data. This was done before AWS released the brand-new Cost Explorer which has more advanced features. The main reason behind this project was to create an aggregated report to analyse and find which resource was costing the company a lot. The project was based on the AWS Sample project but was tweaked to categorise the cost according to the business needs.
Project Requirements
- awscli
- Configure AWS credentials for the target account
- run
aws configure - Cost Explorer enabled
- Verfied Amazon SES Sender email
AWS Costs
- AWS Lambda Invocation
- Usually Free
- Amazon SES
- Usually Free
- Amazon S3
- Minimal usage
- AWS Cost Explorer API calls
- $0.01 per API call (about 25 calls per run)
Deployment
Since I was already using AWS CloudFormation as Infrastructure as Code to create the projects, this was my first attempt to use the AWS Serverless Application Model framework. The SAM framework creates Cloudformation and this project was right up my alley to make use of my knowledge of it. It was an easy deployment since the sample application was already configured to work with the us-east-1 region. Instead of using the easy_deploy.yaml and uploading it directly to the AWS Console, I created the SAM templates and checked in to the Github repository. The infrastructure was created using CI/CD pipeline using Jenkins so that anyone can update the infrastructure from anywhere thus avoiding the dependency of deploying locally. Once the infrastructure was created, I used bash scripts to create the lambda layers and then deploy them to the lambda function using the AWS CLI commands in the script.
The tweaks I made in the sample project are as follows.
- Connect the application with AWS CloudWatch Events to run it as a scheduled job
- The job runs once a week to send Engineering reports to the Engineering department
- The job runs once a month to send AWS Cost reports to the finance department & engineering department
- The email will also contain a graph with a spending pattern
- The email template was created with the help of the product team and engineering team
- Categorise the reports based on the Projects
- This was done using the tags in the resources
- Send a report of untagged resources once a week as a notice to the relevant departments
- CI/CD Pipeline to deploy the infrastructure and the lambda function using Jenkins
- Overall reports are uploaded to the S3 bucket as a static website
- Extended to use of multiple AWS accounts using AWS Organisations (After March 2017)
Project Update
AWS Organisations was generally available from Feb 2017 which made a lot of features in this project obsolete. AWS Cost Management had some features related to budget and cost categorisations. With the release of AWS Cost Explorer UI in August 2017, this entire project was made obsolete. We created a new AWS Account and put all of our accounts under AWS Organisation which made billings and cost management very easy to view and manage from a single AWS account.