For decades, the path from a business question to a data-driven answer followed a predictable sequence. A business leader asked a question, an analyst translated that question into SQL, a query engine executed the code, and the leader received a result. This process relied on one key factor. Before anything could happen, a human intermediary needed to perform an invisible but critical function. The analyst acted as a filter, making dozens of small decisions based on institutional knowledge. They knew which tables were reliable, which date ranges were appropriate for specific fiscal queries, and how to calculate metrics based on the specific department asking the question. Because it was used constantly on a daily basis, this knowledge was rarely encoded in a formal system; instead, it lived in spreadsheets, onboarding documents, or the analyst’s memory. This is context, and it is now the most valuable commodity in the world.
Context is critical for AI success, but accessing it in a way that works for AI is difficult. Part of this is due to the nature of context itself, which is diffuse and spread across the business. As organizations transition toward using autonomous agents and large language models to interface with data, that human filter is removed. The query path now often moves directly from a business user or an agent to the query engine. While this shift reduces the time spent waiting for manual report generation, it also exposes a significant gap in the metadata needed to build context appropriately. Without the analyst to provide context, agents are forced to reason against raw data structures. When an agent is asked for a revenue figure, it may find multiple conflicting definitions across various data silos. Without a layer of curated meaning, the agent will likely choose a definition at random, leading to answers that are technically successful but contextually incorrect.
The disappearance of the human filter
All of this signals a major shift around context. The primary challenge in modern data architecture is not necessarily a lack of data, but a lack of shared meaning. In a traditional human-centric workflow, context travels with the question. If a Finance executive asks for revenue, the analyst knows to provide recognized GAAP revenue. If a Sales manager asks the same question, the analyst provides booked Annual Recurring Revenue (ARR). These are different numbers, yet both are correct within their respective domains.
When you remove the analyst and replace them with an agent, the agent lacks the ability to distinguish between these definitions unless they are explicitly encoded. The model itself is often capable of reasoning, but it is reasoning against the wrong definitions. Even worse, more context is not always the key either. Providing an agent with every available table and piece of metadata in an Amazon S3 bucket or a data warehouse often makes the problem worse. Too much noise leads to distraction, where the agent might use deprecated tables or conflicting sources.
The enterprise context layer addresses this by providing curated, certified context. It ensures that the agent has access to the appropriate amount of information. This is an evolution of the work organizations have done for years to support human consumers, but the requirements for agents are more stringent. Humans can ask for clarification when a definition seems ambiguous; agents often guess silently.
Defining the enterprise context layer
An enterprise context layer is not a single product but a functional tier in the data stack. It serves as the organized, governed structure between your agents and your data. To be effective, this layer must provide three specific capabilities: structure, meaning, and governance.
Structure
Structure refers to the formal definitions, metrics, and business rules that reflect how your specific organization operates. These are not generic industry definitions. They are the specific logic used by your Finance, Sales, and Product teams. In this layer, metrics are not just column names; they are objects that include the underlying SQL logic, the required join keys, and the valid parameters for filtering.
Meaning
Meaning goes beyond a list of definitions to describe how different data entities connect. This is often represented as a semantic graph. For example, the layer should understand that a “Customer ID” in a marketing database on Amazon S3 is the same entity as a “Client Code” in a Starburst Enterprise cluster. This allows agents to reason across domains rather than just looking up individual facts in isolation. This connectivity is essential for AI and analytics solutions that require cross-functional data to provide comprehensive answers.
Governance
Governance in the context layer ensures that every answer provided by an agent is auditable and traceable. It applies access controls at the moment context is served. If an agent generates a response based on a specific metric, the context layer provides a trace back to the person who certified that definition and the source data used to calculate it. This level of transparency is necessary for maintaining trust in automated systems.
Distinguishing context from raw catalogs
It is a common misconception that a comprehensive data catalog is equivalent to a context layer. Most catalogs function as a huge repository of metadata, listing every table, view, and column available across the enterprise. While this is useful for a data engineer searching for a specific asset, it is insufficient for an AI agent.
Adding to the problem, a raw catalog dump often contains “dark data,” such as temporary tables, staging areas, and deprecated schemas. If an agent is given access to this entire list, the probability of it selecting an incorrect source increases. The enterprise context layer requires a process of harvesting, approving, and mapping.
Instead of showing everything, the context layer only exposes assets that have been certified by a data steward. This distinction is critical. In a context layer, metadata is harvested from sources like dbt projects, Tableau workbooks, and query histories, but it remains in a pending state until a human confirms its accuracy. This ensures that the agent is only reasoning against the “gold” standard of organizational data.
Data products as code and the role of DPaC
To build a durable context layer, organizations are increasingly moving toward treating data products as code (DPaC). This approach allows you to define data products using YAML files. These files contain not just the data location, but the associated metadata, access policies, and business logic.
Because these definitions are stored as code, they can be version-controlled and integrated into standard CI/CD workflows. This is a significant shift from traditional data management where metadata often sits in a separate wiki or a proprietary catalog tool that is disconnected from the actual data pipeline. When data products are defined as code, the context travels with the data. If a developer changes a column definition in the source, the DPaC framework ensures the context layer is updated accordingly.
This framework supports the heterogeneous nature of modern estates. Whether your data lives in a data lakehouse, a legacy data warehouse, or data lakes, the DPaC approach provides a consistent way to present that data to agents. This consistency is vital for understanding How is Enterprise Data Architecture Changing? as organizations move away from monolithic centralization toward distributed, governed domains.
Harvesting metadata from existing sources
The creation of an enterprise context layer begins with harvesting existing metadata. Most organizations already have the components of a context layer. They are simply scattered across different tools. Logic might exist in dbt, a Collibra catalog, or a user’s query history. In fact, many context insights are hidden within the underlying metadata that already exists in a Starburst environment.
The context layer pulls these disparate signals into a central environment. Here, data stewards can review the harvested metadata. For example, the system might identify a frequently used join between a “Sales” table and a “Region” table. The steward can approve this join as a “certified join,” which the agent can then use with confidence.
Integration with existing architectures
Importantly, the enterprise context layer is not entirely new. It builds on past foundations. As such, it does not require a complete replacement of existing infrastructure. Instead, it acts as a connective tissue, joining together what is already there in ways that work for the AI era.
For organizations using Starburst, the context layer leverages the existing connectivity to data lakes, data warehouses, Amazon S3, and more.
Choosing a context layer that moves at the speed of your business
A static context layer will eventually fail as the business evolves. The final stage of a mature enterprise context layer is the ability to learn from usage patterns. Every time an agent asks a question and a human provides feedback or a correction, that signal should flow back into the layer.
If an agent consistently struggles to find the correct definition for “churn,” and a steward manually points it to the correct dbt model, the context layer should record that association. Over time, the layer becomes a self-improving map of the organization’s knowledge. This reduces the burden on data stewards and ensures that the context remains fresh.
This iterative process transforms the context layer from a simple repository of definitions into a dynamic asset that grows more accurate with every query. It ensures that whether the consumer is a human analyst or an autonomous agent, they are operating from a single, certified version of the truth. By focusing on curated, certified context rather than raw metadata, practitioners can build AI systems that are not only capable of generating answers but are also trustworthy enough to support critical business decisions.
