Enhanced agility, materialized views, and efficient database analytics

About

A personal finance company, doxo helps more than 6 million people protect their financial health and accomplish their financial goals by breaking free of the bill-paying burden. The company facilitates millions of transactions daily, allowing people to manage their due dates and pay all their bills in one place.

The company maintains multiple internal databases with a wide variety of essential data, including payment flows, user activity, signups, telemetry data from doxo’s production website and mobile app, analytics from email marketing campaigns, and more. This data is stored in different formats on different platforms. Business intelligence (BI) and analytics users work with several tools to generate daily insights about the health of the business.

Challenge

The company executive leaders at doxo review critical data stored in different formats on different platforms on a daily basis. With data residing in various places, doxo faced several challenges: 

• Querying distributed data simultaneously was a chore. Users querying against a MySQL database, for example, couldn’t do anything with the data that sat in Redshift, Snowflake, Hive, and an Amazon S3 data lake.
• Wrangling data nightly meant taking snapshots of 10 different databases and creating copies. The DevOps team had to babysit the copying job each night and fix it when it broke down.
• Evaluating payments and transactions at the end of each month required a laborious process involving a great deal of manual work, including downloading spreadsheets from third-party providers.
• Certain teams and users had standing daily processes to query databases directly. Switching backend storage would have disrupted their queries. Yet the data infrastructure team wanted to avoid locking into any storage systems interminably.

Daniel Cook (Senior Product Director, Data Platforms and Analytics at doxo) and his team wanted a solution that would allow BI and analytics users to query multiple data sources simultaneously, without requiring complex ETL operations or incurring long delays.

Solution

Cook decided to deploy Starburst Enterprise on AWS, through Amazon’s Elastic Kubernetes Services (EKS), since doxo already hosted its corporate infrastructure and other systems on AWS. 

Today, Starburst Enterprise functions as an abstraction layer, allowing doxo analysts working in Tableau, Geckoboard, and other BI tools to quickly and easily query data in multiple database systems simultaneously.

Cook says his three favorite Starburst features are:

1. Materialized views allow users to see all data queries for all data in a single place.
2. Query federation of disparate data sources with a single source of secure access to all data.
3. Vibrancy of the community of Starburst developers and others in the Trino community, providing enhancements and augmentation.

“Starburst makes these different data storage platforms look like a single virtual warehouse,” says Cook. “We’ve taken to calling it our logical data warehouse, or LDW.”

Results

The core function of Starburst as a federated query engine has been the most impactful at doxo. In addition, doxo realized several improvements in efficiency and flexibility in querying its distributed data.

More efficient database analytics. Starburst lets users get all the information they need from any data source, without having to know where it’s stored. 

Minimizing data wrangling. The data copying process has been eliminated. Now Starburst allows the DevOps team to satisfy data customers with the least amount of effort. 

Accelerating core processes. The end-of-month evaluation and reconciliation process now takes less than six hours from start to finish. 

Enhancing agility and avoiding lock-in. Starburst gives doxo the flexibility to change backend storage systems without disrupting BI or other data consumers. 

“The fact that we can switch out the data and move it from one storage technology to another without our users knowing or even having to hear about it, that’s a very good business outcome,”  Cook says.

Starburst has become an integral part of doxo’s infrastructure and a key enabler of doxo’s virtual data warehouse. doxo is doing for its own data consumers and infrastructure teams what it does for its 6+ million customers — providing a simpler, faster, and more effective way to manage a previously complex mix of processes.

More resources: Dan Cook’s Voyager profile