MongoDB was outsourced from a third-party provider called Compose. This MongoDB mainly was used as archive storage and the applications very rarely connected with the DB to retrieve data. The monthly cost was more than $10,000 and the sole idea behind this project is to reduce cost.
Project Details
The project began with the idea of moving the MongoDB Cluster to an in-house AWS account. I started working on creating a MongoDB Cluster using CloudFormation as Infrastructure as Code. This was my introduction to MongoDB and NoSQL Databases. I went through deep learning about MongoDB with this project.
This was a three instances cluster design. One was the Primary which was used as READ and WRITE operations while the other two instances were Secondary clusters that have only READ operations. The cluster names were manually configured in the instance and the MongoDB was installed so that it was easy to identify the primary and secondary instances.
The instances are created from an AMI which was built using Packer so that it aligns with other EC2 Instances styles in the company. The operating system used was CentOS 7 same as the other instances in the estate. The instances were created using Vagrant. I also used Puppet to add users and other required configurations. The Puppet was made Masterless by using Github as the Puppet Master. This helped us to use only the version-controlled configuration changes when creating and configuring the instances. All the instances were EBS backed.
Once the instances were up and running, MongoDB was intelligent to identify the cluster members. MongoDB is versatile in this so it wasn’t difficult. The MongoDB configurations were predetermined using Puppet when the instances were deployed using Vagrant. The final piece was to retrieve data from Compose to our new cluster. Since Compose was a cloud hosting, our only choice was to retrieve the data from one DB and sync it with our in-house DB. This was a lift and shift operation that went on for 2 whole weeks.
We began with the archives and started dumping the data. We also tweaked the application that was putting data to Compose so that it can write to the new in-house operation. This made life easier as the New MongoDB was already collecting the new data when archives were being migrated. At last, we shut the write operation to Compose when all the data was completely replicated to the AWS MongoDB.
After a couple of months, the cost of running in-house MongoDB was about $4200 while the total cost of running just production Compose DB was slightly above $10,000. We saved a total of $9600 per month for all the environments (Prod and Test)
Project Requirements
- AWS Account
- EC2 Instances
- EBS Volumes
- S3 Bucket (to transfer MongoDB data from Compose)
- Packer
- Vagrant
- Puppet + Github
Project Update
This project is no longer required as AWS has a deployment guide for MongoDB in AWS. There are other elegant ways to deploy in-house MongoDB clusters but if you’re looking for fully managed cloud-hosted MongoDB alternatives, then DocumentDB by AWS will be the closest one to look at. However, from my research, I can see that it still has a long way to go to become a full replacement for MongoDB.