Fast performance, timely access
and engineering support

About

Novant Health is a network of physician practices, hospitals, outpatient, and diagnostic imaging centers and other medical facilities. In addition to employing world-class physicians and caregivers, the organization relies on advanced technology solutions to optimize patient care and the overall patient experience. Novant Health ingests and stores a variety of data relating to patient health, as well as to clinical and financial information and more, in multiple systems. The organization needs to be able to extract insights from its distributed data to enhance the patient experience and drive better business outcomes.

Challenge

Today, Novant Health maintains an on-prem warehouse, with Databricks and Delta Lake layered on top, but the data team has moved most of its storage to an Azure-based data platform, with Microsoft ADLS serving as the landing zone. According to VP of Data Platform & Integration Justin Byrd, the goal is to centralize as much data as possible in its Azure data platform. At the same time, Byrd and his team know that establishing a single source of data could take years for a large organization. His customers – the analysts and data scientists who mine Novant’s distributed data for insights – cannot wait that long. 

He wanted to find a way to provide his customers with access to all of Novant’s data, whether it resides in that on-prem warehouse, the cloud data lake, an operational data store, or some mix of the three. His customers, who’d grown accustomed to the speed of querying a traditional warehouse, would expect fast performance, too. Byrd needed a solution that would check several boxes. 

Solution

Novant Health deployed the Starburst platform to function as an abstraction layer between its varied data sources and the analysts, data scientists and other end users. Data engineers who might have had to wait months to source and curate data to satisfy complex use cases now have immediate access through Starburst. 

“We needed to point our customers directly to data where it lives, whether that’s in Azure, on-prem, or in one of our clinical or financial operations systems, and give them the capability to query that data and ultimately deliver insights for the organization,” says Byrd. “Starburst hits the sweet spot for us because it delivers exactly what we need, and it’s built on proven technology that has been out there in the market and tested at large companies.”

Results

Starburst did not replace an existing solution for Novant Health. The capabilities are net new for the organization and its data team, and Byrd hones in on five primary benefits. 

• High-performance distributed queries
• Efficiency of data access
• Faster access to recent data
• Centralized management 
• Data platform initiative support

Starburst’s high-performance distributed query functionality makes it easier for end users to quickly access more datasets and generate better, faster insights. Novant’s users had grown accustomed to fast performance because of their on-prem warehouse, and after some early tuning and optimization, Starburst was able to meet that performance expectation and in many cases exceed it. 

Byrd notes that Starburst adds an element of timeliness by extending access to Novant Health’s operational databases and other systems that do not have streaming capabilities. Previously, Novant had to run a batch process to ingest and curate this data into its cloud platform, which could take weeks to months to complete. Starburst provides near real-time access (60% faster access to data on average). The speed of access, combined with the timeliness of the data, allows analysts to identify competitive opportunities for the business faster using Microsoft Power BI. Thanks to Starburst, they’re now working with much newer data that better reflects the current market. 

Centrally managing and monitoring data access via Starburst has proven to be another major advantage. This allows Byrd and his team to respond faster to alerts, study how their customers are utilizing Starburst, and ensure that everything is running optimally behind the scenes. 

Finally, Byrd and his team were pleased to discover that Starburst facilitates Novant Health’s gradual, long-term transition to its Azure-based data platform vision. As Novant Health runs data pipelines to gradually move more data into this data lake, Starburst maintains access throughout the process. Byrd and his team do all the work behind the scenes with no impact on the end user, who only deals with the Starburst catalog. These technical engineering benefits were a welcome surprise, but ultimately Starburst is helping Novant Health carry out a broader mission to transform the healthcare leader into a data-driven organization.  

“The goal is that every business department is making data-driven decisions. We want them to be making decisions based on data, not gut feel, and in order to do that you need to have data access query capabilities,” explains Byrd. “Having a distributed query engine like Starburst in a large organization with varied data assets like ours is essential.”