Achieving Faster, More Affordable Data Analysis with an AWS Glue Pipeline

A prominent ad tech company needed to build a sophisticated big data processing system, which would drastically improve the speed and reliability of their data processing.

The customer's original system was serverless and extremely expensive to operate, costing $5,000 per day. Furthermore, it took several days to reprocess data as new updates arrived, meaning the customer was often stuck waiting for new insights. The customer approached Big Cloud Country with a challenging requirement: they needed a reliable, cost-effective system that could reprocess large amounts of data quickly and accurately.

Big Cloud Country used our experience in building distributed systems to quickly architect a solution. We moved their system from serverless Lambda to Spark, which significantly reduced costs while improving reliability and scalability. This drastically reduced the cost of operation from $5,000 per day to $1,000 per day. We also improved reliability several times over and freed up the customer's team to focus on higher-value activities.

In addition to these improvements, Big Cloud Country also improved the time it took to reprocess data. Our team was able to dramatically reduce reprocessing time from several days to just 20 minutes. This allowed the customer to gain insights from their data much faster, enabling them to make better decisions in their ad tech business. Big Cloud Country also used cutting edge technology such as Docker containers to facilitate rapid development and deployments of new features. This allowed us to quickly react to changes as needed during the development process, while still maintaining stability and reliability of the system. This greatly increased the quality of our end product while keeping development costs low.

Overall, our work enabled the customer to reduce their costs while also improving reliability and scalability. We were able to cut reprocessing time from several days to 20 minutes and free up their team for higher-value activities. Additionally, our use of cutting edge technology allowed us to rapidly develop new features while still keeping costs down. We are proud of the result we achieved for this prominent ad tech company, and we look forward to helping more businesses in the future with our cloud computing solutions and big data processing services.