Delivering near real-time price alerts with a scalable lakehouse architecture

Going, a cheap flight alert platform, helps customers find affordable flights through timely price alerts. With a quickly expanding user base and increasing data volumes, Going needed a modern data infrastructure to deliver actionable insights at scale. Their previous architecture lacked the flexibility and speed required to support near-real-time analytics, driving the need for modernization.

Going faced challenges in delivering airfare price intelligence to its customers, which led to a reassessment of its existing data infrastructure:

• Scaling beyond Snowflake: As data volumes grew to over 50 petabytes annually, Snowflake’s closed ecosystem and vendor lock-in limited Going’s ability to scale cost-effectively and adopt best-in-class tools. While it initially served their needs, its pricing structure and proprietary nature made integrating with their expanding architecture difficult.
• Latency requirements: Delivering airfare price alerts requires sub-minute data processing to ensure customers receive deals before prices change. Centralizing all data for processing created inefficiencies, making it challenging to meet these demands and deliver real-time insights.
• Complex data formats: Processing deeply nested airline data formats, such as JSON, required flexibility in data handling and transformation that Snowflake could not efficiently provide.
• Cost concerns: Snowflake’s “pay-to-go-fast” pricing model meant costs increased significantly with rising data volumes and computing needs. This prompted Going to seek a more cost-efficient, scalable solution to support their growth.

“Snowflake’s architecture made sense at first, but as we scaled, the costs and rigidity started to outweigh the benefits. We needed an open, future-proof solution that allowed us to move at the speed of our business,” said the VP of Engineering.

Going chose Starburst Galaxy as the core of its new lakehouse architecture, a decision driven by its ability to deliver real-time insights and support open data standards:

• Starburst Galaxy for open analytics: Going adopted Starburst Galaxy, powered by Trino, as their querying layer. Its open architecture allowed for the federation of queries across distributed data sources without requiring centralized ingestion, dramatically improving efficiency and reducing costs.
• Apache Iceberg for flexibility: Iceberg’s open table format provided schema evolution and transactional consistency, addressing the flexibility and performance limitations of Snowflake’s proprietary structure.
• Kafka for streaming ingestion: Confluent Kafka replaced batch-based data ingestion processes, enabling real-time updates that aligned perfectly with Going’s sub-minute latency requirements.
• Multi-tool ecosystem integration: Unlike Snowflake, Starburst Galaxy seamlessly integrates with Databricks for batch modeling and analytics, allowing users to use the best tools for their specific needs.

Moving to a lakehouse architecture powered by Starburst Galaxy and Iceberg delivered impactful outcomes for Going:

• Scalable performance: Seamlessly processed over 50 petabytes annually, supporting rapid data growth and new data sources.
• Real-time analytics: Reduced latency to minutes, enabling faster price alerts and enhancing the customer experience.
• Cost efficiency: Decoupled compute and storage, optimizing resource utilization while managing costs effectively.
• Self-service analytics: Empowered teams to independently access and analyze data, driving productivity and better decision-making.
• Future-ready architecture: Supported schema evolution and integration with emerging tools, ensuring long-term adaptability.

By transitioning to Starburst Galaxy, Going addressed its challenges with scalability, flexibility, and cost, unlocking the full potential of its data. This move reinforced the company’s commitment to innovation, enabling it to deliver real-time, actionable insights to customers while maintaining a scalable and cost-effective architecture.

More Resources: Keeping the Data Lake Full | Datanova 2024