Data Engineering Architecture: Optimizing For Cost Efficiency

Being cost effective is a must for companies to survive and be competitive in today’s markets. However, controlling costs is a constant struggle for many data engineering teams.

From reducing compute consumption by optimizing users’ modeling and transformation code to minimizing storage usage by constantly chasing folks to delete unused tables.
Not to mention designing and selecting tools to build a scalable and open architecture that allows you to avoid vendor lock-in and predatory licensing.

In this webinar Roy Hasson, VP Product at Upsolver, presents three modern design patterns to help you control and lower costs and future proof your data architecture to remain cost efficient as you scale.

Key design patterns presented:

• Open Table Formats: Discover how using an open format-based lakehouse can minimize duplication, increase accessibility and interoperability all while minimizing the cost of storage and eliminating redundant processing of data.
• Object Stores: Explore the considerations needed to determine when it’s ok to query data in object stores vs. when it should be loaded into optimized storage, often at a higher cost. Learn when to take advantage of the tiered storage model, becoming commonplace in various modern data solutions to reduce costs.
• Revitalizing ETL: See how modernizing Extract, Transform, Load (ETL) processes beyond traditional data warehouses can reduce compute costs and enhance efficiency.

About the presenter
Roy Hasson, in his role as VP of Product, contributes extensive knowledge from his previous position as a product manager for AWS Glue and AWS Lake Formation