Explore our expert-made templates & start with the right one for you.
Gamoshi Saves 75% on Agile Pipelines with Upsolver and AWS
CUSTOMER STORY
INDUSTRY:
Online advertising
SOLUTION:
Advertising platform service
TECHNICAL USE CASE:
Exposing product data; productized analytics
75% reduction
in architecture spend
from weeks to hours
to modify data pipelines
Where it started
Gamoshi is a unique startup selling an advertising platform service designed to help companies create their own programmatic advertising solutions. It’s a one-stop-shop for brands, advertisers, media houses, and publishers who want to fast-track development without bearing the risk and costs of building from scratch. As a fast-growing startup building an innovative platform, Gamoshi must be able to provide data for advanced analytics to customers in real-time and keep it up to date as data requirements shift.
Building a better experience through scale
Moshe Moses, CEO of Gamoshi, is a veteran technology leader in building complex distributed systems. That experience enabled him and the Gamoshi team to build the many complex pieces required to run and win a real-time bidding system. When the work to win the bid is complete, they need pipelines, pulling from several applications within their ad processing platform, that report out the ad performance results to customers. Since customers’ needs are varied, and since their data source attributes as well as their needs can change, Gamoshi’s pipelines must be flexible, agile, and high-performing at scale.
”Every day someone can add another rule, property, or dimension that might need to be added to the data model,” Moshe says. “If we wanted to make changes we needed the right people to do it without making costly mistakes. Any changes we made would take weeks because of the amount of testing required before pushing to production. We knew there had to be a better way.”
At the time, Gamoshi was using Google Dataflow, a managed solution for building structured data pipelines. It proved to be a more rigid system than the team preferred. The main issue was that it required specialized skills. Adding a metric forced changes to the entire hard-coded pipeline plus a deep understanding of distributed systems. Moshe knew they needed to make a change so they could respond more quickly to customer demands. They started evaluating Upsolver on Amazon Web Services (AWS).
75% reduction
in architecture spend
from weeks to hours
to modify data pipelines
”Upsolver is elastic in the way it auto-scales compute, and in the way you make changes quickly using SQL, without tons of engineering overhead. You can teach any SQL-savvy non-developer to use Upsolver. It's very straightforward.” A better way to deliver data to customers
Moshe and the team knew if they were going to make a switch to something better, they needed several systemic improvements. Key criteria were system flexibility and agility – the ability to easily handle frequent changes.
The key architectural component was streaming data from their product. Upsolver connected directly to Amazon Kinesis, a managed streaming service, to continuously ingest data from their product. From there, the team leveraged Upsolver’s visual interface and SQL-based IDE to write, test, and run pipelines. They could also test the output of the data and add new dimensions without going through lengthy development cycles.
Once the data is processed, it is delivered into S3, Upsolver Lookup Tables, and Amazon Redshift outputs for consumption back into the Gamoshi product. “Upsolver is elastic in the way it auto-scales compute, and in the way you make changes quickly using SQL, without tons of engineering overhead,” says Moshe. “You can teach any SQL-savvy non-developer to use Upsolver. It’s very straightforward.”
“Upsolver provides us peace of mind, because now that we store everything in the data lake, I can reprocess the data in case we make a mistake or need to add new fields.” Implementation and future
The data that is ingested and processed by Upsolver enables Moshe and the team to move faster than ever to deliver immediate value to every one of their customers.
When the team started migrating from Google to Amazon the learning curve proved not to be steep. While the first few processes were moved and written by a consultant, the team quickly learned that they could make changes themselves, and update them directly in Upsolver in minutes. This led to numerous changes to pipelines that were accomplished in hours rather than weeks, since they can see the results of their changes immediately.
Additionally, the team found that since they now store all of their raw data in the data lake they can quickly fix any issues that arise. Upsolver allows them to “time travel” back to a previous state of the data, correct the pipeline logic, and replay the data up to the present. The team can respond quickly and fix bugs or troubleshoot an issue to determine whether an event occurred.
The team has seen such great success implementing Upsolver on AWS that Moshe is constantly on the lookout for new ways to implement it in their product. Their customers are asking for more data all the time, and Upsolver enables Gamoshi to constantly meet new demands as they arise. Says Moshe, “Upsolver provides us peace of mind, because now that we store all the data in the data lake I can reprocess the data in case that we make a mistake or need to add new fields.”
“After we changed and updated the processes with Upsolver, we saw a 75% savings over our previous architecture spend.”