BestSecret’s data journey: Moving beyond Snowflake

In June, I flew out to Munich to attend the DATA Festival. This festival brings data leaders together from across Europe – and in my case, abroad – to celebrate the constantly advancing world of data, analytics, and AI, and to showcase fascinating data journeys that can bring value to companies of all sizes. 

Of all the seventy-plus speakers at the festival, there was one presentation that I found to be particularly interesting – and not because the speaker also happens to be our customer. That presentation was from Lutz Künneke, Director of Engineering, and Isa Inalcik, Senior Data Engineer, at BestSecret, a leading European online destination for off-price fashion based near Munich, Germany. 

Moving off of Snowflake 

As Künneke got to the stage, the first words out of his mouth were: “We are moving off of Snowflake.” 

It immediately caught everyone’s attention because most of the industry is just starting to adopt Snowflake, and there stood one of the most innovative companies in Germany, moving away from what’s perceived as the ‘modern’ data warehouse. 

Until very recently, BestSecret’s data stack looked similar to what you’d find throughout thousands of organizations – utilizing the trio of dbt, Snowflake, and Airflow as their data warehousing solution. 

Künneke painted the picture of how the e-commerce company’s data landscape has evolved, and why that necessitated a rethink of their architecture. In particular, the reason for the change was the creation of “emerging projects” – basically, all the new ideas and new aspects of their business weren’t able to fit into their already existing data stack.

BestSecret’s evolving data landscape

BestSecret shared all the challenges of why emerging projects cannot be covered optimally with this model. First, they had to extract the data, load it, and transform it to a staging area. 

Next, they had to use dbt to transform that data. This created considerable effort in housekeeping for previous workflows. There’s also a central data engineering and analytics team that became a bottleneck because the company’s doing more research and innovating at a pace faster than the central data warehousing team can keep up with. 

All the changes in emerging projects caused fragmentation in the data warehouse and complicated things, so this was a model that became unsustainable as they continued to scale the business. 

That’s when they decided to introduce Starburst.

From Trino to a Proof of Concept with Starburst

BestSecret first started experimenting with Trino. As an open source project, Trino was readily available and easy to use. “I could just download it and start to experience it immediately and get a feel for what Trino does,” Inalcik said. The simplicity of being able to start to experience Trino is part of what drew them in.

Then they completed their own Proof of Concept with Starburst and started to get excited about it. Adopting the Trino on Ice design pattern, BestSecret decided that Iceberg would be their format of choice, and they’ve started to migrate their Snowflake workloads and move that data into Azure Blob Storage.

The open data stack: Move workloads from Snowflake to Trino and Starburst

BestSecret’s new data stack has a zero ELT approach. 

The sources are catalogs, and therefore everything is reachable at one connection with the same SQL syntax. It’s in open table formats, so there’s no vendor lock-in. They can create custom UDFs, custom functions, and even create custom connectors. 

The team learned some lessons during the implementation process, but the bottom line is that Trino has changed their architecture entirely, and this is their new modern and open data stack: open source, open formats, and open architecture.

This approach doesn’t mean that Snowflake is removed, but the important point is that BestSecret is no longer centralizing data and dependent on this one monolithic data warehouse.  They have introduced optionality into their architecture. By moving workloads off of Snowflake and onto Trino, they’re seeing greater efficiencies in doing so, including a notable 70% cost reduction.