The more data sources grow, the more weight they hold. This phenomenon is known as data gravity. Specifically, data gravity describes the tendency of smaller, individual datasets to be pulled toward large datasets. As a dataset grows, it becomes increasingly difficult and expensive to move. Analogously, this is similar to the physical phenomenon where smaller physical objects are attracted to larger ones. In the data world, this mass attracts the services, applications, and compute power required to process it.
For data platform engineers, this presents a particular problem because data gravity comes with specific infrastructure scaling requirements. For this reason, understanding it is essential for designing architectures that remain functional as they scale. While the industry often focuses on consolidation as the primary solution for data management, the reality of the modern enterprise is one of permanent heterogeneity and consolidation around specific areas of data gravity.
The physics of data gravity in the enterprise
Data gravity may be an analogous term, but that doesn’t mean that it doesn’t have big impacts on data architectures. How does this work in practice? In a cloud environment, this is often seen within object storage services like Amazon S3. When you have petabytes of information stored in a specific region, the cost of moving that data to another cloud provider or even another region becomes a significant barrier. Egress fees and the time required for data transfer create a weight that anchors your architecture to a specific location.
This weight affects how you design your data pipelines. If you attempt to move all your data into a single, central repository, you quickly encounter the limits of bandwidth and budget. How enterprise data architecture is changing is a question that many architects ask as they realize that the traditional model of moving everything into one warehouse is no longer sustainable. Instead of fighting gravity by moving data to the compute, modern architectures are increasingly moving the compute to the data.
The latency involved in moving massive datasets also introduces a time penalty. When datasets are large, moving them presents logistical challenges around data ingestion. This leads to stale data in the destination system, which undermines the accuracy of analytics and decision making. As data lakes grow into petabyte scale, the gravity becomes so strong that the only logical choice is to perform processing locally or use a federated approach that minimizes data movement.
The limits of data consolidation
Many platforms promise a single source of truth by consolidating all enterprise data into a single data lakehouse or data warehouse. While these platforms offer significant strengths in terms of performance and integrated tooling for the data they contain, they cannot solve the problem of data that exists outside their boundaries. For a large enterprise, the idea of a single, all-encompassing data store is often more of a theoretical goal than a practical reality.
Consolidation works well for specific use cases, such as high-performance analytics on a curated subset of data. However, it fails to account for the last mile of data that remains in operational systems, legacy on-premise databases, or specialized cloud services.
The cost of consolidation is not just financial. It also involves the risk of vendor lock-in. Once a significant mass of data is moved into a proprietary format or a specific cloud provider, the gravity of that data makes it nearly impossible to leave. This creates a strategic vulnerability where the platform provider has more control over your data than your own engineering team.
Permanent heterogeneity and regulatory boundaries
The primary reason enterprise data estates remain heterogeneous is not technical immaturity, but legal and organizational necessity. Data residency laws require that certain types of information, such as personal health information (PHI) or personally identifiable information (PII), remain within specific geographic or jurisdictional boundaries. For example, you cannot move sovereign data from a European data center to a US-based lakehouse without violating strict regulations like GDPR.
Beyond regulation, organizational requirements also play a role. Different business units may have valid reasons for maintaining their own data stacks, whether for specialized performance needs or local autonomy. This creates a landscape where data is scattered across multiple cloud platforms, on-premises systems, and various database engines. The data landscape is permanently distributed, and your architecture must reflect that.
Data lakes and data warehouses will continue to coexist because they serve different purposes. A data lake might store raw telemetry data for long-term retention, while a data warehouse stores structured financial records. Attempting to force both into a single system often results in a compromise that serves neither use case well. Accepting heterogeneity allows teams to use the best tool for each specific job.
Why AI platforms are sensitive to data gravity
Artificial intelligence and machine learning workflows are only as effective as the data they can access. For generative AI and agentic workflows, the need for business context is paramount. If your AI agent only has access to the data consolidated in your lakehouse, it is missing the broader context stored in your operational systems, your on-premise archives, and other systems.
In this sense, AI workflows inherit the constraints of data gravity inherent in the data systems that feed them. If the data required to train a model or provide context for a RAG (Retrieval-Augmented Generation) system is too heavy to move, the AI application must be able to reach out to that data where it resides. This requires a foundation of AI and analytics solutions that can bridge the gap between different environments. Without this, your AI will provide incomplete or inaccurate answers because it lacks the full enterprise context.
Governing the distributed estate with Starburst
Starburst addresses the challenges of data gravity by providing a governed access layer that spans your entire estate. Whether you are using Starburst Galaxy for a managed cloud experience or Starburst Enterprise for self-managed environments, the goal is to provide a single point of access to data regardless of its location. This approach uses Trino, the high-performance distributed SQL query engine, to execute queries across multiple sources simultaneously.
A key component of this architecture is the ability to maintain high performance without moving data. In this context, query optimization becomes a critical topic. Starburst uses advanced query optimization techniques to push processing down to the source systems whenever possible, minimizing the amount of data that needs to be transferred over the network. This allows you to maintain the benefits of data gravity (keeping data where it is most efficient to store) while gaining the benefits of consolidation (a single interface for analysis).
For organizations moving toward an open data architecture, Starburst Icehouse architecture provides a managed lakehouse service built on Apache Iceberg. This allows you to build a high-performance lakehouse while still maintaining the ability to federate queries to other systems. This flexibility ensures that your architecture can adapt as new data sources are added.
Moving from raw data to data products
To effectively manage a distributed estate, you need to move beyond thinking about raw tables and start thinking about data products. A data product is a curated, governed, and documented dataset designed for a specific business purpose. By using data products, you can wrap your distributed data in a layer of business context that makes it usable for both humans and AI agents.
This approach allows you to define security policies, metadata, and business logic once and apply them across all your data sources. For example, Rippling uses a similar approach to provide a consistent view of data across disparate systems, ensuring that every department is working from the same definitions even if the underlying data is physically separated.
Data products help mitigate the negative effects of data gravity by making the data more discoverable and usable. When data is treated as a product, the focus shifts from the physical location of the bits to the value they provide to the consumer. This abstraction layer is what allows an organization to scale its data operations without being crushed by the weight of its own information.
Practical steps for managing data gravity
If you are a data platform practitioner, you should start by inventorying your data estate to identify where gravity is strongest. Determine which datasets are too large to move, which are restricted by regulation, and which are currently siloed in operational systems. Once you have this map, you can begin to implement a federated access strategy.
Instead of planning for a multi-year consolidation project that may never reach completion, focus on creating a governed access layer. This allows you to provide immediate value to your AI and analytics teams by giving them access to the data they need today. By acknowledging the reality of data gravity rather than fighting it, you can build a more resilient and flexible data architecture that supports the evolving needs of the enterprise.
The goal is not to eliminate data gravity, but to work within its constraints. By using tools like Starburst, you can ensure that your data remains accessible, governed, and useful, no matter where it lives. This is the only way to build a truly comprehensive platform for AI and analytics in a world where data will always be distributed. Success in the modern data landscape requires a strategy that balances the need for central governance with the reality of distributed data.
