That unwritten layer is business context, the organizational rules, fiscal calendars, exclusions, and conventions that dictate how raw data must be interpreted. While data catalogs document where tables live and semantic models define field names, business context supplies the tribal knowledge required to make sense of numbers in a real operational environment.

Without explicit business context, even the most capable AI models produce confidently wrong answers. Establishing this layer through governed data products and an enterprise context layer is what allows human teams and autonomous agents to reason from a single, auditable source of truth.

Key takeaways

• Business context is the layer of business rules, logic, definitions, exclusions, and conventions, such as your fiscal calendar or which customer table is authoritative, that tells people and AI systems how to correctly interpret raw data.
• Without business context, AI systems return confident, wrong answers because they lack the tribal knowledge an experienced analyst applies without thinking.
• Business context differs from technical context and semantic context. It governs how your organization interprets and constrains that meaning.
• Data products and an enterprise context layer are the practical containers that capture, govern, and serve business context to people and AI agents alike.
• Agentic AI in production depends on business context as much as it depends on model quality, since answers built on ungoverned context can’t be trusted or audited.

What is business context?

Business context is the set of business rules that turns raw numbers into a correct answer. It goes beyond the data itself and beyond a data catalog entry that tells you a column is named `revenue` and stored as a decimal. For example, it might mean the knowledge that this quarter’s revenue excludes intercompany transfers, that the fiscal year starts in February, and that the “customers” table in the older data warehouse was deprecated last year in favor of a new one.

Having data isn’t the same as understanding it. A table full of correct rows can still produce an incorrect report if the person or system reading it doesn’t know the exclusions, definitions, and conventions your business applies before anyone treats a number as final. Business context is what supplies that missing layer.

Why business context matters for AI and analytics

The garbage-in, garbage-out problem

Every analyst carries a mental list of caveats. They know how to filter out test accounts before running a revenue query. They know last quarter’s numbers in one dashboard don’t match another because the two reports use different margin definitions. This tribal knowledge rarely gets written down, because experienced humans apply it automatically.

AI systems don’t have that instinct. A large language model can write a syntactically perfect SQL query against your data warehouse and still return a wrong number, because it queried a deprecated table or skipped an exclusion a human analyst would have applied without thinking. Business metadata closes that gap by making the rule explicit instead of assumed.

Why bigger models don’t solve for business context

Model upgrades don’t fix this problem, because it isn’t a reasoning problem. A more capable model can still generate a fluent, confident answer from the wrong table or the wrong definition. The failure comes from the gap between the data and what the model knows about your business, not in how well it can write code or summarize text.

That’s why access to business context, not model quality alone, increasingly determines whether an AI system gives you a right answer or a wrong one delivered with total confidence.

Business context vs. technical and semantic context

It helps to place business context inside a broader framework. Many data teams describe four types of context that together let people and AI systems use data correctly: 

• Technical
• Semantic
• Business
• Operational

Technical context tells you where data lives, including which system, which table, which column. A data catalog usually documents this layer, listing tables and columns without explaining the business rules behind them. Semantic context is closer to what a semantic layer provides. It tells you what a field means: that `net_rev` refers to revenue after returns and discounts. Business context goes a step further. It governs how your organization interprets and constrains that meaning, including which fiscal calendar applies, which exclusions to apply, and which of two similarly named tables is the one to trust. Operational context adds freshness and reliability signals, such as when a table last updated or how often a pipeline fails.

Each layer answers a different question. Business context answers the one that decides whether a technically correct query produces a business-correct answer.

Examples of business context

Business context shows up in specific, recognizable forms across most organizations:

• Fiscal calendar. Your fiscal year might start in February, not January, which changes every quarter-over-quarter comparison.
• Revenue and margin definitions. Finance might define margin one way for board reporting and another way for a specific product line.
• Exclusions. Test accounts, internal usage, and intercompany transfers need to come out of a query before a revenue number means anything.
• Deprecated versus authoritative tables. Two tables can share a similar name, but only one reflects current, governed logic. The other is left over from a past migration.
• Entity definitions. What counts as an “active customer” or a “closed deal” often varies by team and needs a single, documented answer.

Any one of these, applied incorrectly, can turn a correct query into a confidently wrong report.

How organizations capture and govern business context

Business metadata and data products

Business context has to live somewhere both people and AI agents can find and trust it. Many organizations package it as business metadata attached to a data product. This provides a governed, documented unit that bundles data, business logic, and access rules together, rather than as a table left to speak for itself.

Data products work because they carry the rules alongside the data. A revenue data product doesn’t just expose numbers. It documents the fiscal calendar, states the exclusions already applied, and names the authoritative source table, so anyone who queries it inherits the correct context automatically.

The enterprise context layer

At a larger scale, organizations consolidate this metadata into an enterprise context layer, creating a governed layer that captures business context once and serves it consistently to every dashboard, report, and AI agent that needs it. Rather than re-explaining fiscal quarter definitions and exclusion rules to each new tool, teams build an enterprise context layer that every consumer can draw from.

Business context and agentic AI in production

The stakes rise sharply once AI agents start acting on data without a human double-checking every step. An agent asked to summarize quarterly performance needs to know which table is authoritative, which exclusions apply, and which fiscal calendar governs the quarter, or it will produce an answer that looks complete and is wrong.

Ungoverned business context breaks three things agents need to work in production: 

• Accuracy, since the agent may query the wrong table or skip a needed exclusion
• Consistency, since two agents given the same question can return two different numbers
• Auditability, since nobody can trace why an agent reached a given answer if the business rules it applied were never documented in the first place.

Getting agentic AI in production right depends on solving for business context as much as it depends on model choice, and getting it wrong can make or break enterprise AI initiatives that otherwise look ready to launch.

Getting started with business context

Start by finding the disagreements. Look for the metrics that produce different numbers in different dashboards, and ask why. Those mismatches between reports usually point to undocumented business context.

From there, document the rules you find, including fiscal calendars, margin definitions, exclusions, and which tables are authoritative. Package that documentation as business metadata attached to the data products people already query, rather than leaving it in a wiki page nobody checks before running a report.

As you scale AI use across the business, treat this documentation as a prerequisite, not an afterthought. The organizations getting reliable answers from AI agents aren’t necessarily using better models. They’re the ones that solved context engineering and gave both people and AI systems the same business context a trained analyst already carries around. Starburst for AI agents helps you package and serve that context consistently across every team and every agent that needs it.

FAQs

What is business context in AI and analytics?

Business context is the set of business rules, definitions, and exclusions that turns raw data into a correct answer for both people and AI systems. It tells you things such as which fiscal calendar applies, how your finance team defines margin, and which table is authoritative. Without it, an AI system can write a technically correct query and still return a wrong number, which is why more data teams treat business context as core to their AI data strategy rather than an afterthought.

How is business context different from a semantic layer?

A semantic layer defines what a field means, for example, that `net_rev` refers to revenue after returns and discounts. Business context goes further. It governs how your organization interprets and constrains that meaning, including which exclusions to apply and which of two similarly named tables to trust. The two work together, but only business context tells you the rules specific to your company.

What is an example of business context?

A fiscal calendar that starts in February instead of January is a common example, since it changes every quarter-over-quarter comparison a report runs. Other examples include which customer table is authoritative after a migration, and which exclusions, such as test accounts or intercompany transfers, come out of a revenue query before anyone treats the number as final. Any one of these, applied incorrectly, turns a correct query into a confidently wrong report.

Why do AI agents need business context to avoid hallucinating?

An AI agent that queries a deprecated table or skips an exclusion a human analyst would apply without thinking produces an answer that looks complete and is wrong. Ungoverned business context breaks three things agents need in production: accuracy, consistency, and auditability, since nobody can trace which business rules the agent applied. Governing that context before an agent acts on it is what makes the difference between a trustworthy answer and a confident, wrong one.

How do data products capture and govern business context?

A data product bundles data, business logic, and access rules together, rather than leaving a table to speak for itself. A revenue data product documents the fiscal calendar, states the exclusions already applied, and names the authoritative source table, so anyone who queries it inherits the correct context automatically. Organizations that consolidate this metadata at scale build it into an enterprise context layer that serves every dashboard, report, and AI agent consistently.

AI is a major reason why.

Enterprise AI systems often depend on data spread across countries, clouds, business units, operational systems, and governance domains. Some data can move between environments with few restrictions. Other data can move only under specific conditions. Some must remain within a particular country, region, or controlled environment.

These requirements affect where data is stored, how it is accessed, where processing occurs, and which controls have to follow it.

For architects, the challenge is increasingly familiar: how do you give applications and AI systems access to the data they need when that data cannot always be centralized?

Answering that question starts with some terminology.

Sovereignty, residency, localization, privacy, and governance are different requirements

These terms frequently appear together in discussions about enterprise data. They describe related concerns, but each has different implications for architecture.

Data sovereignty

Data sovereignty refers to the principle that data is subject to the laws and governance frameworks of the jurisdictions in which it is collected, stored, or otherwise controlled.

Physical location is part of the picture, but sovereignty also involves jurisdiction and control.

Architects may need to understand which laws apply to a dataset, who can access it, which organization or entity controls the environment, whether the data can cross a geographic boundary, and what restrictions apply to its use.

Those considerations can influence architecture long before an application reaches production.

Data residency

Data residency concerns where data is physically stored.

A customer, regulator, or internal policy might require a dataset to remain in Germany, Canada, Australia, or another location. An organization may address that requirement through a particular cloud region, local data center, or other approved environment.

Residency establishes location. It does not, by itself, define every condition governing access or use.

Data localization

Data localization typically imposes stronger geographic restrictions. Data may be required to remain within a specific jurisdiction and, in some cases, be processed there as well.

That can have immediate consequences for application design.

If processing must occur locally, architects need to consider where compute runs, which services can interact with the data, what can leave the environment, and how results are shared with other systems.

For some workloads, bringing compute closer to the data becomes a practical requirement.

Data privacy

Data privacy focuses on how organizations collect, use, share, and protect personal information.

Privacy requirements may govern consent, access, retention, purpose, sharing, and cross-border transfers. They often intersect with sovereignty requirements, but the two address different concerns.

A dataset can reside in an approved country and still be used in a way that violates privacy requirements. Similarly, an organization may have strong privacy controls in place while still facing restrictions on where certain data can be stored or processed.

For many organizations, GDPR is the reference point for these conversations, but it’s one privacy framework among a growing set of regional and sector-specific requirements shaping how data can be used.

AI governance

AI governance adds requirements around how AI systems are designed, deployed, monitored, and controlled. The specific obligations vary by jurisdiction, use case, and risk profile.

The EU AI Act, for example, ties obligations to a system’s risk classification, so the same AI application can face very different governance requirements depending on how it’s used and who it affects.

For enterprise architects, these requirements rarely appear in isolation.

A single AI application may need access to personal data governed by privacy rules, customer information covered by contractual commitments, operational records controlled by internal policy, and datasets stored across several jurisdictions.

Architecture has to account for all of those conditions at the same time.

Why the distinctions matter in practice

Consider a global company building an internal AI assistant.

The assistant needs access to support history, customer information, policy documents, and operational data across several regions. From a purely technical perspective, the problem looks manageable. Connect the sources, provide the right context, and give the application a way to retrieve the information it needs.

The design becomes more complicated once the control requirements are mapped.

Customer data in one jurisdiction may need to remain local. Another dataset may be transferred only for specific approved purposes. Employee information may carry additional privacy requirements. A business unit may restrict AI access to particular categories of operational data. Certain processing may need to take place within a regional environment.

Each requirement places a different condition on the system.

When teams group all of them under a broad heading such as “compliance,” they can miss the architectural differences between them. The result can be unnecessary restrictions, late redesign, or new data copies created simply to work around existing controls.

A better starting point is to identify which condition applies to which dataset, user, workload, and environment. Data control is shaped by a combination of legal, geographic, contractual, and operational requirements, each of which can influence how AI systems are designed and operated.

Sovereignty becomes an operating concern

Understanding the requirements is only the first step.

The harder problem is applying them consistently once applications are running.

A pilot can often rely on a small number of datasets, one-time approvals, and narrowly defined access. Production environments work differently. More users need access. More regions become involved. Data changes. Policies change. New AI use cases appear.

Governance that depends heavily on manual reviews and individual interpretation becomes difficult to scale.

This is where sovereignty becomes operational.

Architecture needs to support decisions such as:

• Who is requesting access?
• Where is the user or workload running?
• Which policies apply to the dataset?
• Is the requested use allowed?
• Can the data cross a particular boundary?
• Does processing need to remain local?
• What controls need to be applied?
• Can the organization audit the access afterward?

These questions put identity, metadata, policy enforcement, entitlements, observability, workload placement, and auditability directly into the architecture.

This is where governance as an operating capability becomes critical. Policies need to be clear enough to apply consistently, and access decisions cannot depend indefinitely on repeated manual interpretation.

That becomes especially important for AI. A governance process that works for one pilot can become a major source of delay when dozens of applications need access to distributed data.

How sovereignty changes architecture decisions

Once sovereignty requirements are considered early in the design process, several familiar architecture decisions take on new importance.

1. Access does not always require another copy of the data

Many enterprise data architectures were built around a common pattern: move the data to the platform where it will be used.

That approach remains useful in many cases. Problems arise when every new access requirement results in another copy.

A dataset is replicated into a warehouse. Another copy is created for an AI team. A regional implementation requires another pipeline. A new application creates another transformed version. An employee copies and pastes analytical data into an Excel spreadsheet: welcome to shadow IT.

Each copy creates additional work.

Teams have to secure it, refresh it, document it, govern it, monitor it, and determine whether it remains consistent with the source. Under sovereignty requirements, every movement may also introduce additional legal, contractual, security, or policy considerations.

For many workloads, the better architectural option is to give applications and AI systems governed access to data where it already resides.

That approach reduces dependence on replication as the default integration pattern and puts more emphasis on consistent access across systems and environments.

2. Consistent control becomes as important as consistent storage

Central data platforms still have an important role in enterprise architecture.

Warehouses, data lakes, lakehouses, and other shared environments can provide consistency, performance, and common services for many workloads.

Sovereignty introduces situations where some data will remain outside those environments for valid reasons.

In those cases, consistency also has to exist at the control layer.

Identity, metadata, access policies, entitlements, governance, and auditability can provide a common framework across data that remains physically distributed.

This gives enterprises a way to keep data in the environments where it needs to remain while applying a more consistent approach to access and governance across the broader data estate.

It also changes the role of the central platform. Rather than serving as the required destination for every useful dataset, it can be part of an architecture that provides coherence across distributed environments.

3. Hybrid and multi-cloud environments need to be treated as durable

Large enterprises often operate across several infrastructure models at once.

Data may sit in public clouds, private clouds, object stores, SaaS applications, operational systems, and on-premises environments. Acquisitions add more platforms. Regional requirements create additional boundaries. Some systems remain in place because moving them would introduce more cost or risk than value.

Sovereignty adds another reason to expect this diversity to persist.

An architecture built around eventual consolidation can struggle when important systems and datasets continue to remain distributed.

A more practical design accounts for that reality. Data will continue to live in different systems, regions, and control domains, and the architecture needs to make those environments easier to work across.

That means creating consistency in areas such as discovery, access, governance, identity, policy enforcement, and observability, even when the underlying infrastructure remains diverse.

4. Workload placement becomes part of data architecture

Sovereignty also affects where processing happens.

If data needs to remain in a particular environment, analytics, retrieval, inference, or other processing may need to run closer to that data.

This can be particularly relevant for AI systems that depend on large, current, or sensitive datasets.

In some cases, moving a limited result set is more practical than transferring an entire source dataset into another environment. In others, both the data and the processing may need to remain within a controlled region.

This makes bringing compute closer to the data an important architectural principle, particularly when moving governed or high-volume datasets introduces cost, latency, or additional control requirements.

Workload placement therefore becomes part of the governance discussion, alongside infrastructure, performance, and application requirements.

5. Governance needs to be enforced where access occurs

Distributed architectures make manual governance harder.

If policies are interpreted separately for every platform, region, and use case, teams accumulate approval processes, exceptions, and one-off controls.

AI increases the pressure on that model because a single system may interact with several types of data across multiple control boundaries.

Governance-aware architecture brings policy closer to the point of access.

Identity, role, location, data sensitivity, jurisdiction, and usage context can all help determine whether a user or workload should be allowed to interact with a dataset.

Metadata also becomes more important. Architects need information about where data came from, what it contains, who controls it, and which restrictions may apply.

When governance is built into how data is discovered, queried, shared, and used, organizations can extend governed access more consistently across applications and environments.

That becomes increasingly important as AI initiatives expand beyond isolated pilots and require access to a broader range of enterprise data.

Questions technology leaders should be asking

Technology leaders do not need to become experts in every sovereignty law or regulatory framework. They do need enough visibility to understand how those requirements affect architecture.

A useful assessment starts with a few practical questions.

About the data:

• Which datasets are most important to current and planned AI use cases?
• Where does that data reside today?
• Which legal, contractual, security, or internal policy requirements apply?
• Which data has a firm requirement to remain local?
• Where are teams creating copies primarily because access across systems is difficult?

About access:

• Which users, applications, and AI workloads need access to each dataset?
• Can they work with the data in its current location?
• Where does access currently depend on manual approvals or custom integrations?
• Which access patterns introduce unnecessary movement or replication?

About governance:

• Can policies be applied consistently across environments?
• Do access controls account for identity, role, location, sensitivity, and usage?
• Can teams explain which policy allowed a particular access decision?
• Can access be monitored and audited across the distributed estate?

About architecture:

• Where should processing occur relative to governed data?
• Can the architecture support data across multiple clouds, regions, and platforms?
• Which parts of the environment truly need consolidation?
• Which parts would benefit more from consistent, governed access?
• How many new pipelines, copies, and control surfaces will a proposed design create?

These questions also help distinguish fixed sovereignty requirements from architectural practices that may simply have become standard over time.

Some constraints are defined by regulation, contracts, customer commitments, or security requirements. Others may come from internal conventions, existing processes, or earlier architecture decisions. Understanding which constraints are fixed and which can be addressed through better controls can prevent unnecessary complexity from becoming embedded in the design..

Designing for the data environment you actually have

Most enterprises already have distributed data estates.

Data lives across regions, clouds, business units, operational systems, and legacy platforms. AI systems increasingly need to work across those boundaries, often under different rules for access, movement, and use.

Architecture needs to account for those conditions from the beginning.

For many organizations, that means becoming more deliberate about when data moves, creating greater consistency in how access is governed, and designing workloads to operate across environments that will remain distributed.

The objective is to give applications and AI systems reliable access to the data they need while maintaining control over where that data lives, how it is accessed, and how it can be used.

This post is the first in a series exploring how data sovereignty is reshaping enterprise data and AI strategy. In the posts ahead, we’ll look at why sovereignty has become a global concern, how data movement can create new control challenges, and what these realities mean for organizations building and calling AI across distributed environments. 

AI is being held back by data access. Solving this challenge requires shifting from a mindset focused on data centralization to a federated data model. Model choice still matters. It does not matter if the model cannot see the tables, files, and events your business already runs on. By placing a unified, governed access layer over your existing datasources, you give analysts and AI agents secure, real-time access to your entire data estate without forcing unnecessary data movement.

Key takeaways

• AI is held back by data access. That bottleneck is what now limits enterprise AI, and a federated access model is how you relieve it.
• Copy-first consolidation is often too slow, expensive, or blocked by residency for the data AI needs.
• A federated query layer gives analysts and agents one path across data lakes, data warehouses, and operational systems.
• Put identity, row and column controls, and audit on that path, or agents will be blocked or unsafe.
• Start with one use case, an inventory of hold-in-place sources, and data products that carry business definitions.

Why AI stalls on data access

Most teams can access some data because some data is easy to access. Summaries of tickets, wikis, and PDFs land fast because those files already sit in one datasource. The problem arises when you need access to the data that’s harder to get at, including balances, claims, device telemetry, or customer events. That data lives in data silos, and is often spread across multiple data sources that legal will not allow you to copy.

Data centralization has always treated that as a relocation job. You plan a new data warehouse, whether Amazon Redshift or Snowflake, wait on pipelines, and hope data sovereignty rules allow you to move the data in the first place. 

There is a better way. 

Starburst believes that enterprise AI success comes down to data access, and that AI lives and dies on the basis of its access to context. 

What data access for AI means

Data access is a core requirement for AI, as models and agents require the ability to discover, query, and use data sources from across your organization. For this reason, agents require one access layer that holds the business’ logic. 

That layer has to span data lakes, data warehouses, and systems that will never land in the lake. Seen this way, data federation is now the key to AI as it allows organizations to access the context they need by querying across sources. 

Importantly, centralization is an option when it makes sense. You can still copy data when a use case needs a local table for heavy reuse. You federate some data and copy other data when it helps. Federation creates the ability to choose when each one makes the most sense. 

Take stock of what you have, and what can and cannot move

How do you get started improving your data access? Start with a list. For each source record the use case needs, record owner, system type, freshness, sensitivity, and whether the data can leave its region or network.

Separate data that can be copied from data that should stay put. Payment rows, health attributes, and data held in certain geographies often cannot move. Those are the federation-first sources. 

Keep the inventory short. Two to four sources is enough for a first project, and you can build from there. 

Access all of your data to begin using it immediately

Federation makes all of your data accessible. Connectors reach the data lake, the data warehouse, and the operational database. Users write one query. The engine pushes work down where it can and joins the rest.

With that layer, you apply the same policies whether the reader is a person or an agent. Row filters, column masks, and audit logs sit on the query path. You do not reimplement them in every pipeline.

Wrap access with business context

But you shouldn’t stop there. Access without meaning still fails AI. A column named `rev_amt` is not a metric that agents can act upon. Agents need definitions, owners, and grain. That is what reusable data products with business metadata are designed to do, providing curated datasets plus the contract a consumer can trust.

Additionally, governed access for agents and teams is the same idea aimed at AI consumers. Package the product once. Serve people and agents from it.

Do not publish a data product you have not tested. Test data products before agents rely on them. Broken grain and empty values become confident wrong answers.

You may attach business context sitting with the data. AIDA-style assistants and other tools then share the same definitions. The product is the contract. The assistant is a consumer.

A 90-day getting-started path

As with anything, it’s best to approach the rollout of this approach in phases. Pick one AI use case with a named owner and a decision it should improve. Identify two to four sources. Mark which ones cannot move. Stand up federated query for those sources. Attach metadata and the minimum row and column rules. Publish one data product with definitions. Point the assistant or notebook at that product, not at raw catalogs.

Starburst Galaxy and Starburst Enterprise are both designed to help make federation and data access easy. If you primarily use the cloud, Galaxy is designed as an easy, managed service. For more complex, hybrid use cases, Enterprise is the self-managed solution capable of scaling for any enterprise. 

Next steps

Want to know more about governed AI data access? Check out this ebook: Sovereign AI ebook.

FAQs

Why is data access the bottleneck for enterprise AI?

Models need context from systems that already hold the business. If that context sits behind copies and tickets, the model sees a slice or it sees nothing. Access at query time is what turns a demo copilot into a system that can use operational data.

Do agents need a data lakehouse first?

No. A data lakehouse is useful when you want warehouse-like tables on the data lake. It is not a precondition for your first AI workload. Federate all sources to begin with and then copy frequently used data into the lakehouse when a workload needs a local table.

How does data federation differ from ETL into a data warehouse?

ETL copies data from multiple sources into a single source. Federation queries sources in place using connectors and a distributed SQL engine. You still use ETL when it makes sense, but you do not need to do it in every use case. 

How do access policies apply to AI agents?

Put identity, row filters, column masks, and audit on the query path both people and agents use. If an agent has a separate ungoverned connection, it will bypass the rules you wrote for people. One access layer is the point.

What is a reasonable first project for governed AI data access?

Start slowly. Take a few sources first, and explore them using data federation. As you add more, you can centralize data that makes sense to centralize, knowing that it is always your choice and a choice driven by your needs rather than the needs of your technology. 

Worst of all, blindly moving every table into one place mistakes location for unity. Relocating data to a central warehouse changes where files sit, but it does not reconcile the conflicting definitions, mismatched identifiers, and unassigned ownership that dictate whether two tables agree on what a customer, an order, or a deal actually means.

True integration is about reconciling meaning, not forcing physical co-location. Rather than treating centralization as an all-or-nothing requirement, modern data architectures require a balanced approach. By pairing a shared semantic layer with data federation, organizations can query, govern, and integrate data across systems on their own terms without burning millions of dollars on endless ETL migrations.

Key takeaways

• Centralizing data in one data warehouse relocates tables, but it does not reconcile the conflicting definitions and IDs that make data usable across teams.
• Mergers and acquisitions, SaaS sprawl, and undecided data ownership are the most common reasons a centralized data warehouse still behaves like a set of data silos.
• Real integration needs a shared identifier per entity, such as one customer ID, and a single owner for each entity’s definition.
• Data federation and a shared semantic layer let teams query and reconcile contextual data across sources without first copying everything into one data warehouse.
• Data governance, not additional storage consolidation, is what turns centralized tables into a trustworthy, integrated source of answers.

Centralizing versus integrating data: what centralization means, and where it stops

Centralizing data means moving tables from many systems into one data warehouse or data lakehouse. Every team can query the same storage layer instead of pulling exports from a dozen source systems. That solves a real access problem.

Integration needs to determine whether those tables mean the same thing. As SeattleDataGuy has written, many companies have centralized their data without integrating it. A data warehouse is not integrated just because every table sits in the same place. That is why a lot of consolidation projects end up with a data warehouse full of tables that never learned to talk to each other. Integration means every team can ask the same question of the data and get one answer. That happens because the underlying entities, such as customer, order, or account, share a single definition across every source system that feeds them.

Why moving tables into one data warehouse doesn’t create integration

Let’s unpack this with two examples, one where centralization stalls, and one where full integration works.

A Salesforce table and a product table that don’t connect

Picture a common setup. A Salesforce customer table serves the sales team, and a separate customer table inside the product database serves engineering. A consolidation project can copy both tables into one data warehouse without much trouble. But if the Salesforce record keys on an account ID and the product table keys on an email address, the two tables still don’t connect. An analyst can query both, yet still can’t join them into one row per customer. The data sits in one place, but it still describes two disconnected views of the same customer.

The single customer ID: what full integration looks like

Contrast that with a company where engineers build a single customer ID at the application layer. Every system, including Salesforce and the product database, writes and reads that same ID. When that groundwork exists, a data team can join customer records across systems without guessing which ID means what. Leadership can ask how many customers the company has, split by industry or buying pattern, and get a direct answer, because one shared identifier already ties every system together. That upfront reconciliation work, not the storage location, is what makes data integrated.

How data warehouses turn back into data silos

Even a data warehouse that starts integrated can drift back into a set of disconnected data silos. Three patterns show up most often.

Mergers and acquisitions that never finish integrating systems

Mergers and acquisitions usually come with a plan to centralize operations onto one CRM, one ERP, and one set of core business tools. That plan takes time, and companies often move on to the next deal before the last one finishes. The result is two or more customer tables, two order histories, and two definitions of revenue, all sitting in the same data warehouse without ever reconciling with each other.

SaaS sprawl across Salesforce, Gainsight, NetSuite, and similar tools

Every team adopts its own SaaS product with overlapping functionality. Sales runs on Salesforce, customer success prefers Gainsight, and finance needs NetSuite. Each tool defines customer, account, and user a little differently, and each team wants questions answered that pull data from every system at once. Copying all three into a data warehouse preserves those different definitions instead of resolving them.

When no one owns the decision, every team defines customer differently

Not deciding who owns the customer entity is itself a decision. Without a single owner, each team defines customer based on whichever source system it trusts, then builds its own reporting, processes, and metrics around that definition. Those choices show up in dashboards, dbt models, and spreadsheets long after anyone remembers why they differ. Eventually leadership asks a simple question, such as how many customers the company has, and there is no simple answer. Clear data governance, meaning decision rights over who defines and owns each entity, is what closes that opening before it forms.

Real integration requires more than a shared data warehouse

Real integration needs three things that copying tables into one data warehouse does not provide on its own. First, shared entity definitions: every team agrees on one meaning for customer, order, or account, instead of building its own version. Second, a single owner per entity, so disagreements get resolved by one accountable person instead of by whichever team ships a dashboard first. Third, reconciliation work done up front, mapping every source system’s ID and field to that shared definition before the data lands in the data warehouse, not after.

A data catalog helps here because it tracks where each version of an entity lives, how it’s structured, and who owns it. That gives teams a starting point for reconciling definitions instead of guessing. Reconciling meaning also needs a context layer that captures the definitions, calculations, and relationships an analyst used to carry around in their head. That way, the knowledge survives staff turnover and reorganizations instead of leaving with whoever built the original report.

Data federation and the semantic layer as a faster path to integration

Copying every table into one data warehouse isn’t the only route to integration, and for many companies it isn’t the fastest one either. With data federation, you can query data across Salesforce, the production database, and every other system as if they were a single database. You get that view without first copying it all into one place. Query federation works the same way at the level of a single question. Instead of loading every source into a data warehouse before analysis, one query reaches across systems and returns a combined answer directly.

Federation only closes the difference between centralizing and integrating data if the query layer also carries a shared definition of what customer or revenue means. That is the job of a context layer, which defines metrics, dimensions, and relationships consistently across every tool that queries the data. Pair federation with a context layer, and your team can reconcile meaning across systems without first moving every table into a single data warehouse.

Where to start if your data warehouse is centralized but not integrated

Start by picking one entity, such as a customer, and mapping every system that stores a version of it. Assign one owner for that definition, and begin the reconciliation work before any more tables move into the data warehouse, not after.

That reconciliation work is also what it takes to build an enterprise context layer, the shared definitions, calculations, and relationships that used to live only in an analyst’s head. Making that knowledge durable means any dashboard, model, or agent can rely on it. This work matters beyond traditional reporting, too. An AI data strategy built on centralized but unreconciled data hands models and agents the same disconnected definitions that already confused human analysts. Start the reconciliation work now. Every dashboard, model, and AI agent you build on the data warehouse afterward inherits one consistent answer, instead of a jumble of tables that happen to share an address.

Why the data lakehouse is built for this problem 

This shift toward semantic reconciliation finds its ideal foundation in an open data lakehouse architecture powered by Apache Iceberg. By decoupling compute from storage and using open table formats, an Iceberg-based lakehouse eliminates the vendor lock-in and rigid schemas of traditional proprietary warehouses. It gives you a performant, cost-effective home for your core datasets, allowing you to centralize key workloads on your terms without forcing a brute-force migration of every single table across your enterprise.

When you pair an Iceberg lakehouse with a federated context layer, you get the best of both worlds, including high-performance storage for reconciled data products and instant, governed access to remote sources that do not need to move. This architecture ensures that whether data lives natively in your lakehouse or remains in external systems, it adheres to the same unified definitions, access rules, and metric logic. Centering your data strategy on an Iceberg lakehouse turns your storage into a flexible, scalable foundation built for human analytics and autonomous AI agents alike.

Want to know more about data lakehouse architecture. Download the ebook today. 

FAQs

What’s the difference between data centralization and data integration?

Centralizing data means moving tables from many systems into one data warehouse or data lakehouse, so every team queries the same storage layer. Integration is a separate step: it means every table shares the same definition for entities such as customer, order, or account, so teams get one answer instead of several disconnected ones. A data warehouse can be centralized without being integrated.

Why isn’t one data warehouse enough to integrate data?

Copying tables into one data warehouse solves an access problem, not a meaning problem. If a Salesforce customer table keys on account ID and a product database customer table keys on email address, moving both into the same data warehouse does not make them join. The reconciliation work, mapping every system’s ID and field to a shared definition, still has to happen on its own.

What turns a data warehouse back into data silos?

Three patterns show up most often: mergers and acquisitions that stall before systems are ever unified, SaaS sprawl across tools such as Salesforce, Gainsight, and NetSuite that each define customer differently, and no single owner deciding which definition of an entity is correct. Any of these leaves conflicting definitions sitting inside the same data warehouse.

How does data federation help integrate data without copying it first?

With data federation, you can query data across Salesforce, a product database, and other systems as if they were a single database, without moving it all into one place first. Query federation applies the same idea to a single question, reaching across systems and returning a combined answer directly instead of loading everything into a data warehouse before analysis.

What role do a semantic layer and data governance play in integration?

A semantic layer defines metrics, dimensions, and relationships consistently across every tool that queries the data, so federation carries a shared definition of what customer or revenue means instead of just fast access. Data governance assigns decision rights over who defines and owns each entity, which is what stops teams from building their own version of customer or account in the first place.

Who should own shared data definitions across business units?

One accountable person or team should own each entity’s definition, such as customer or order, rather than leaving it to whichever team ships a dashboard first. Without that owner, every team defines the entity based on whichever source system it trusts, and those differences surface later as conflicting numbers in dashboards and reports.

Enterprise intelligence goes beyond data. It is the combination of business meaning, governance, relationships, policies, and institutional knowledge that allows AI to understand how a business actually operates. Without it, even the most capable AI models struggle to produce trusted, explainable, and actionable outcomes.

The challenge is not creating enterprise intelligence from scratch. It is connecting fragmented business knowledge into a governed layer, and continuously evolving it as the business changes. This is where many AI transformations stall. Consider a bank defining a “high-risk customer.” Compliance flags them for suspicious activity, Credit for likely default, Fraud for unusual transactions. Each team is right by its own rules, but the definitions don’t match, and that is the gap governance closes, by giving every system and AI agent one shared answer.

This article presents a blueprint for enterprise AI. Starburst demonstrates how organizations create trusted enterprise intelligence across distributed data and metadata. Artefact demonstrates the operating model that enables that intelligence to be continuously discovered, certified, reused, and improved. Together, they provide the technology foundation and organizational model required to operationalize AI at enterprise scale.

I. Why Conversational Data Agents Are Unable to Scale After Repeated Successful Pilots

Most organizations have proven that conversational AI can work. They build impressive pilots, demonstrate real business value, and generate excitement across the organization. However, few succeed in scaling those successes into an enterprise capability.

The challenge is enterprise readiness. Before conversational data agents can operate reliably at scale, organizations must answer three foundational questions:

• Who defines the business context the AI should operate within?
• How do we ensure AI recommendations are reliable and explainable tied to organizational accountability? 
• How does the system continuously learn and improve as the business evolves?

II.  The Enterprise Intelligence Layer: The Data Foundations You Already Have In Place Are Your Biggest AI Assets

Artificial intelligence can only produce trustworthy business outcomes when three things happen in sequence: enterprise data must be reachable, business meaning must be organized, and trusted intelligence must be activated wherever decisions are made.

The Enterprise Context Layer creates that foundation through a simple execution model:

Connect. Enterprise data and metadata already exist across operational systems, cloud platforms, data lakes, catalogs, semantic models, BI tools, and documentation.

Rather than requiring organizations to centralize those assets, Starburst federates across both data and metadata, allowing AI initiatives to begin immediately without waiting for migration or consolidation projects. Organizations start with the intelligence they already have.

Organize. Connected data becomes reusable enterprise intelligence.The Enterprise Context Layer organizes federated data into AI-ready data products aligned to the way the business operates—Customer, Product, Policy, Account, Risk, Supplier, and other core business entities.

Each data product combines four elements:

• Trusted data
• Business meaning
• Business logic
• Embedded governance

Together these create reusable units of enterprise intelligence that can be shared across analytics, applications, and AI. Rather than rebuilding business context for every dashboard or agent, organizations build it once and reuse it everywhere.

Activate. AI-ready data products become the common intelligence foundation for dashboards, applications, AIDA, and approved AI agents. Because every data product already contains trusted business meaning, relationships, governance, and reusable business logic, AI can reason from enterprise intelligence instead of raw data.

As AI models continue to improve, the value of the Enterprise Context Layer grows with them. Smarter models increase the value of trusted enterprise intelligence.

The multiplier is:

Model × Enterprise Intelligence

Connect. Organize. Activate. establishes the technology foundation for enterprise AI.

Operationalizing that foundation requires one additional capability: Continuous improvement.

In financial services, that foundation is closer than most organizations realize. While financial institutions initially invested in governance to satisfy regulatory mandates, this compulsory effort has yielded the core building blocks of a context layer:

• Data stewards who possess granular understanding of business definitions at the field level
• Data domains featuring established ownership structures and clear lineage
• Business glossaries developed under regulatory scrutiny, already encoding the logic an AI assistant needs to function with trust

Deploying Conversational Data Agents in financial services is not a net-new transformation. It is a natural evolution of the data governance programs already in place, this time to satisfy more than a regulatory need.

IV. Example — Who Is the Customer?

Consider a simple question in a large bank: Who is the customer? It sounds easy, but it represents one of the most complex data challenges in financial services.

The answer depends on who is asking. Retail banking defines a customer as an account holder. Commercial banking uses a legal entity or corporate group. Wealth management uses a household. AML and KYC teams define a customer as a risk-assessed individual with screening and watchlist status. Marketing views a customer as an addressable profile built from behavioral and transactional signals.

Now ask each team the same question: How many active customers do we have? Each returns a different number because each uses a different, yet valid, business definition. Without business context, those answers cannot be reconciled. A Conversational Data Agent that applies a single definition to every query will inevitably produce inconsistent responses and erode user trust.

The solution is not another AI model. It is governance.

Business functions own and validate their definitions, data stewards translate them into governed business rules, and the semantic layer preserves that context. The Conversational Data Agent then selects the appropriate definition based on the user’s role, business function, and intent. Moreover, it now knows who is asking the question, what is the context of the user and makes a determination on what data product is to be tapped into. 

V. The Enterprise Intelligence Operating Model

“Connect. Organize. Activate.” establishes the technology foundation for enterprise AI.

Operationalizing AI at enterprise scale requires one additional capability: Improve.

Enterprise intelligence is not a static asset.  Business definitions evolve. Products change. Regulations change. AI capabilities improve. Every interaction creates new opportunities to strengthen the enterprise’s intelligence foundation.

Continuous improvement happens through a simple operating cycle.

Discover

AI should be leveraged to continuously discover opportunities to improve enterprise intelligence.

By observing metadata, query patterns, business rule usage, relationships across data products, and consumer behavior, the platform identifies opportunities to strengthen business logic, recommend new AI-ready data products, improve metadata, and expand reusable intelligence across domains. 

Every interaction becomes new learning.

Certify

AI discovers. People establish trust.

Trusted AI requires clear ownership, not more governance. Business and technology teams each play a distinct role in ensuring AI operates with the right business context and continues to improve as the organization evolves.

• Context Domain Owners govern business meaning by defining how terms are used, by whom, and for which decisions.
• Context Enablement Teams (Data & AI Architects) translate business-defined context into governed, AI-ready data products.
• Context Controls & Orchestration Team version, govern, and audit the context layer to ensure every AI output is traceable to a documented business decision

Reuse

Certified intelligence immediately becomes reusable across analytics, applications, AIDA, and approved AI agents. Every new consumer generates feedback that improves the enterprise intelligence foundation.

Enterprise intelligence becomes more trusted, more complete, and more valuable with every iteration.

This operating model does not require a new organization. It expands the mission of existing roles. Together, technology and people create a continuous intelligence system.

Starburst provides the technology foundation. The operating model ensures enterprise intelligence continuously evolves as the business evolves.

VI. What Could Go Wrong without the Operating Model?

Without an intelligent operating model, AI lacks the trusted business context needed to deliver consistent, reliable answers. Imagine a bank deploying AI assistants before business definitions, policies, and rules have been governed and certified. When executives ask for the number of active customers, different AI agents and dashboards return different answers.

The issue extends far beyond inconsistent numbers. Without clear ownership of the business context itself, no one can explain which definition was applied, why it was chosen, or who signed off on it as policy. When something goes wrong, accountability dissolves into “the AI said so,” and that is not an answer executives, regulators, or customers will accept. As AI becomes embedded in everyday decision making, organizations cannot afford intelligence that no one is accountable for.

Leading organizations recognize that the challenge is not building more AI agents. It is establishing who owns the business context every agent draws from, so that trust and accountability scale with it.

VI. Conclusion 

AI leaders are not just building better agents. They are building an intelligent operating model now because they see it as both an immediate accelerator and a long term competitive advantage.

A governed intelligence layer, owned and maintained by clear stewards, accelerates every AI initiative at once. Governance is already built in, so teams do not have to bolt it on later. The layer works with any agent a team chooses to build, so every new initiative starts from the same trusted foundation instead of rebuilding it from scratch. And because business users can serve themselves from a source they can trust, they stop waiting for IT just to get an answer. Across the board, teams stop reinventing definitions, rules, and context for every new use case, because someone has already done that work and owns keeping it current.

Where should organizations begin? The path to trusted AI starts with focused use cases, not an enterprise-wide overhaul. Leading organizations identify where business context can create the greatest impact, assign clear owners for that context from the start, and use those early learnings to evolve their operating model over time.

As AI models evolve, organizations with trusted enterprise intelligence and clear accountability will scale faster and create a lasting competitive advantage over those constrained by fragmented data and unowned business context.

Getting intelligence ready today is not just preparation for the future of AI. It is a strategic advantage for tomorrow, and it starts with deciding who is accountable for getting it right.

Published in concert with our partners at Artefact.

If you’d like to see how we’re helping other clients navigate this, reach out to the Artefact and Starburst teams: 

Artefact: Akhilesh Kale (akhilesh.kale@artefact.com) | Nivetha Aravindan (nivetha.aravindan@artefact.com) 

Starburst: Adran Estala (adrian.estala@starburstdata.com) | Christian Velez (christian.velez@starburstdata.com) | Evan Smith (evan.smith@starburstdata.com)

The recognition comes as enterprises confront one of the hardest challenges in putting AI into production: giving models, applications, and teams secure access to trusted data spread across clouds, data lakes, warehouses, databases, and other systems.

Enterprise AI depends on access to distributed data

AI initiatives depend on relevant, accessible data. But for most enterprises, that data does not live in one place. Moving and centralizing everything before it can be used can add cost, complexity, governance challenges, and delays.

Starburst gives organizations a way to access and use distributed data without requiring them to first move everything into a single environment. By providing consistent access across heterogeneous data environments while maintaining governance and control, Starburst helps enterprises put more of their data to work for AI, analytics, and data-driven applications.

As AI moves from experimentation into production, access to governed data across the business becomes a critical part of the underlying AI infrastructure.

Recognition grounded in customer impact

The TechForward Awards recognize enterprise and emerging technologies driving meaningful business results. Starburst was selected from a competitive field of nominees by a panel of industry experts and technology leaders.

“Congratulations to this year’s finalists,” said Dave Vellante, co-founder and co-CEO of SiliconANGLE Media. “Every company on this list earned its place through customer results including renewal rates, expansion revenue and direct feedback from the individuals using these products daily. Our judges evaluate that evidence when deciding who deserves this recognition.”

That emphasis on customer results makes the recognition especially meaningful to us.

Across industries, organizations use Starburst to make distributed data more accessible, reduce unnecessary data movement, support governed self-service analytics, and build a stronger data foundation for AI.

Those outcomes can take different forms. Lockheed Martin is using Starburst Enterprise as part of its intelligent factory initiative, integrating machine data to help improve operational efficiency and accelerate responses to production issues. Talkdesk used Starburst Galaxy to modernize its Iceberg lakehouse, achieving 85% faster P99 query times and a 150x reduction in query failure rates.

As enterprises put more AI models and applications into production, the ability to connect them with governed data across the business becomes increasingly important. That requires a data architecture that can support access across systems without sacrificing security, control, or flexibility.

“Each year, the TechForward Awards recognize the technology decisions that pay off for the businesses making them,” said John Furrier, co-founder and co-CEO of SiliconANGLE Media. “This year’s finalists span AI, security, cloud and data platforms, and each one solved a true problem for the customers who adopted it.”

We’re honored to be recognized among this year’s finalists in the AI-Optimized Data Platforms category, and grateful to the customers, partners, and Starburst team members whose work made it possible.

Learn more about the SiliconANGLE TechForward Awards.

About SiliconANGLE Media | theCUBE

SiliconANGLE Media operates at the intersection of media, technology, and AI through SiliconANGLE, theCUBE Network, theCUBE Research, CUBE365, theCUBE AI, and theCUBE SuperStudios. Founded by John Furrier and Dave Vellante, theCUBE broadcasts from its studio on the NYSE trading floor and from Silicon Valley, delivering leading live coverage at major tech events worldwide. SiliconANGLE reaches more than 15 million technology professionals and has interviewed over 11,000 C-suite and technical leaders.

Snowflake provides an exceptional engine for storing and querying petabyte-scale data. However, connecting an AI agent directly to raw warehouse tables assumes the model can automatically decipher your internal metrics, regional filters, and security rules. Closing this gap requires shifting from basic prompt engineering to context engineering, establishing a governed semantic layer that provides models with true organizational alignment before a single query executes.

This guide explores the architectural limits of relying solely on warehouse tables for AI, what an enterprise context layer brings to your stack, and how to deploy this pattern alongside Snowflake without duplicating data or disrupting existing workflows.

Key takeaways

• While Snowflake manages storage and compute reliably, it does not natively govern the organizational meaning behind raw columns, leaving AI models to guess business logic.
• Implementing a context layer establishes a single source of truth for metrics, ensuring automated agents and human analysts calculate KPIs using identical business logic.
• Building a context layer does not require migrating off Snowflake. Using data federation, you can connect external datastores while keeping Snowflake as your primary analytical hub.
• By establishing access policies at the context layer, security rules travel with your data across multi-cloud environments rather than being locked inside a single proprietary tool.
• Protocols like MCP provide AI agents with a secure, managed gateway to access organizational definitions without relying on shadow integrations.

What context engineering means for Snowflake teams

Context engineering is the work of connecting AI agents and copilots to your Snowflake data through a governed layer of shared business definitions, metadata, and policy enforcement, ensuring that every answer traces back to a source both people and machines can trust. For most enterprise data platform leaders, that layer is missing today. 

Analysts typically know which table holds the real revenue number and which fiscal calendar finance uses, but an AI agent querying the same data warehouse doesn’t have that context. That difference matters because it makes the difference between success and failure of AI in production. When you connect a copilot straight to raw Snowflake tables, you’re betting the model can guess your business logic correctly every time. Context engineering removes that bet by giving agents the same governed definitions, metadata, and access rules your analysts already rely on.

Why context engineering matters now

As Snowflake teams put more agents into production, the limiting factor is changing. It’s no longer which model you choose but how well that model understands your business context, and that typically comes down to the question of whether it can access it. Seen this way, the problem of context engineering is really a data access problem.

The bottleneck has moved from models to context

Foundation models keep improving, but a better model doesn’t know your business definitions on its own. For this reason, context is the new AI bottleneck. Context, not model quality, is what determines whether an AI agent answers a business question correctly. Two teams can run the identical model on the identical Snowflake data warehouse and get different answers, because one gave the agent governed definitions and the other didn’t.

For Snowflake teams, that means the next advantage doesn’t come from a better model. It comes from richer, more governed context around the data you already have.

Where Snowflake alone falls short for AI agents

Snowflake stores and runs your data reliably, but it wasn’t built to hold your company’s shared business logic. A data warehouse can tell an agent what columns exist. It can’t tell the agent which of three “customer” tables is authoritative, what your company means by “active user,” or which rows a given requester is allowed to see.

Without a governed context layer, every agent ends up building its own private interpretation of the same Snowflake tables. One agent calculates churn one way, another calculates it a different way, and both sound equally sure. As models commoditize, shared business definitions and governed metadata, the parts Snowflake alone doesn’t provide, are what separate a right AI answer from a confidently wrong one.

What a context layer adds on top of Snowflake

An enterprise context layer doesn’t replace Snowflake. Instead, it sits alongside your data warehouse, your analytics engine, and your agentic tools, and it applies common definitions and policy across every data domain you have.

Business definitions and metadata

Every enterprise has a handful of terms that mean more than one thing depending on who you ask. A context layer stores a single, governed definition for each term and its supporting metadata, so an analyst in finance and an AI agent answering a board question pull from the same source.

Additionally, open standards are starting to make these definitions portable across vendors. In November 2025, Starburst joined the Open Semantic Interchange, an open-source initiative led by Snowflake and industry partners to standardize semantic metadata across vendors. That means a business definition you write once doesn’t lock you into a single tool.

Governance and policy enforcement

A context layer also enforces who can see what, at the row and column level, before a query ever reaches an agent or an analyst. Instead of every application rebuilding its own permission checks against Snowflake, the context layer applies one set of rules everywhere it’s used. That consistency matters more as agents start acting on behalf of many users, each with different access, at once.

Data products as the delivery mechanism

Governed definitions need a way to reach the people and agents who use them. In this model, data products operate as the delivery mechanism for this. Instead of every agent querying raw Snowflake tables directly, teams package curated views, metrics, and definitions into data products that carry their governance rules wherever they’re used.

How context engineering works with Snowflake 

None of this needs you to move off Snowflake. Instead, you should use Starburst to augment your access to your organization’s entire context and business logic, and apply that access to your Snowflake data. 

Here’s what that looks like in practice. 

Federating Snowflake with other systems without copying data

With federation, you can connect the other systems you already run, such as data lakes on Amazon S3 or tools built for other cloud platforms, while Snowflake stays in place as your data warehouse of record. Starburst can also query Snowflake-managed Iceberg tables directly through the Snowflake Catalog Metastore, so teams get federated access across a data warehouse and other systems while limiting the need to copy data between them.

That approach shows up in results. BestSecret, a European retailer, cut Snowflake-related costs by 70 percent after it began to scale beyond Snowflake with Starburst, by federating workloads across systems instead of running everything inside the data warehouse alone. Separately, GigaOm’s total cost of ownership research studied a similar Snowflake alternative approach. It found that a Starburst data lakehouse setup delivered about 3 times faster time to insight, with a lower total cost of ownership, than a Snowflake-only setup over a 3-year period.

Serving governed context to agents over MCP

Once governed definitions and data products are in place, you need a safe way to serve them to agents. The Model Context Protocol (MCP) is emerging as a common way to do that. Starburst Galaxy runs a hosted Model Context Protocol (MCP) server, so agents get governed, observable access to your data instead of connecting through unmanaged local servers no one on your team can see or check.

Steps to start context engineering on your Snowflake data

To get started, you don’t need to solve every part of this at once. A few sequential steps help you get moving.

1. List the definitions your agents will need first, such as your fiscal calendar, your churn definition, and which table is the source of truth for each domain. Skipping this step leaves agents guessing at exactly these questions.
2. Pick one pilot use case, put a governed context layer in front of it, and connect that layer to your Snowflake data warehouse and to any other systems the use case touches.
3. Enforce policy at the context layer instead of inside each application, so analysts and agents inherit the same access rules.
4. Package the pilot’s definitions and data as a data product, so the next team can reuse it instead of rebuilding it from scratch.
5. Add a governed protocol for agent access, such as MCP, before you let agents query production data directly.

Start small, prove the pattern on one use case, then extend the same governed context layer to the next team asking Snowflake questions with an AI agent.

Want to know more about optimizing your cost and performance on Snowflake? Download the Data Engineer’s Guide to Reducing Your Cloud Data Warehouse Spending.

FAQs

What is context engineering, and how is it different from prompt engineering?

Prompt engineering shapes how you word a single question to a model. Context engineering gives every agent and copilot governed business definitions, metadata, and policy enforcement, so it works from the same source of truth an analyst already trusts. Instead of crafting a clever prompt for each request, you build a context layer that any agent can draw on consistently.

Why isn’t Snowflake data enough for an AI agent to answer correctly?

Snowflake tells an agent what tables and columns exist, but it doesn’t tell the agent which of three “customer” tables is authoritative, what your company means by “active user,” or which rows a given requester can see. Without governed context on top of that data, each agent builds its own private interpretation of the same tables, and different agents can produce different, equally confident answers.

Do you need to move Snowflake data to build a context layer?

No. Context engineering works through federation, so you connect other systems you already run, such as data lakes on Amazon S3, while Snowflake stays in place as your data warehouse of record. Starburst can also query Snowflake-managed Iceberg tables directly, an approach covered in Snowflake Iceberg catalog support, so you get federated access without copying data between systems.

What’s the difference between a semantic layer and a context layer?

A semantic layer usually covers business definitions and metrics for one analytics tool or team. A context layer extends that idea across every data domain you have, adding governance and policy enforcement on top of the definitions, so the same rules apply whether an analyst or an AI agent is asking the question.

What governance should a context layer enforce before agents get access?

At minimum, a context layer should enforce row and column level access rules before a query reaches an agent or an analyst, so every application isn’t rebuilding its own permission checks against Snowflake. Package governed definitions and access rules into data products, an approach covered in context layer, so the rules travel with the data wherever it’s used.

How does the Model Context Protocol (MCP) relate to context engineering?

MCP gives agents a governed, observable way to reach the definitions and data products your context layer already provides, instead of connecting through unmanaged local servers no one on your team can see or check. Starburst Galaxy runs a hosted Model Context Protocol (MCP) server for this purpose, so agents get access to Snowflake data under the same governance rules your analysts use.

Amazon Athena is a good test case for this problem. Amazon Athena gives you serverless SQL access to data sitting in Amazon S3. That access has made it a default choice for teams running fast, on-demand queries without managing infrastructure. 

Connecting an agent directly to Amazon S3 through Athena provides performance for data from that source, but it requires data centralization for data held in other sources. When you point an AI agent at Athena, serverless query access alone doesn’t tell the agent what a customer, a region, or a churn event means across your business. Solving this gap requires shifting focus from prompt engineering to context engineering, creating a governed semantic layer that equips agents with true organizational meaning before a single query executes.

This article looks at why Athena’s query model wasn’t built for agentic AI or agent grounding, what a governed context layer adds on top of it, and how you can start building that layer without moving off Athena or replacing what already works.

Key takeaways

• Athena gives AI agents serverless SQL access to Amazon S3 data, but not the governed business context they need to answer correctly and safely.
• Context engineering is a process that creates a governed layer of business definitions, metadata, and access policy that provides AI agents with the raw Athena access they need.
• Most AI agent projects stall due to lack of data and context access rather than model quality, and recent estimates put agent-project production deployment at only 30 percent.
• A federated context layer lets agents query Athena alongside Snowflake, Redshift, and other sources while limiting the need to copy the underlying data.
• Governance, lineage, access policy, and shared definitions have to be engineered into the context layer up front, not added after agents are already querying Athena.

What context engineering means for an Athena environment

From prompt engineering to context engineering

For the past few years, most AI work focused on prompt engineering, wording a request so a model responds well. Context engineering asks instead what business definitions, metadata, and access rules does an agent need before it ever sees a prompt. With context engineering, you build a governed layer of shared definitions and metadata that sits between your data and any model you connect to it. As models commoditize, the context layer becomes what separates an agent that gives correct answers from one that guesses.

Why Athena’s serverless model isn’t built for agent grounding

Athena isn’t built for this. It runs SQL against Amazon S3 data on demand, and you pay per query scan. That model works well for ad hoc analysis, but it wasn’t designed to carry business meaning at the contextual level. Athena has no built-in concept of a shared metric definition, a data product boundary, or a policy that travels with a table wherever it’s queried. When an agent queries Athena directly, it sees columns and rows, not what those columns mean to your finance or product team. Agent grounding closes that distance by giving agents a consistent, governed reference for what the data represents before they answer a question.

The context gap Athena leaves for AI agents

SQL access isn’t the same as business meaning

Give an agent a valid Athena connection, and it can write and run correct SQL. That’s a query problem, and Athena solves it well. The harder problem is a semantic one, and it’s much harder to solve because it requires knowledge of context.

For example, does “active customer” mean the same thing in your Athena tables as it does in your CRM, and will the agent apply that definition consistently? Without a governed layer, two agents can query the same Athena table and land on two different answers to the same question, each technically correct and neither usable for a decision.

Cost, performance, and governance limits compound the gap

There are other considerations as well. Athena’s pay-per-scan pricing penalizes exploratory queries, which is exactly the pattern agents produce when they ask several follow-up questions to reason through a problem. Multiple agents scanning the same partitions can also create noisy-neighbor contention that slows every other user on the account.

None of this comes with a governance framework built in. There’s no lineage, no consistent access policy, and no audit trail that shows why an agent reached a conclusion. Each of those has to be engineered in, not assumed.

What an enterprise context layer adds on top of Athena

Federated access solves this problem. It provides a universal point of access, allowing for universal data access. This approach is perfect for AI agents that require access to a contextual layer because it allows access to that context, regardless of where it lives, without requiring data centralization to achieve it. 

Let’s look at how this works in practice. 

Federated access across Athena, Snowflake, Redshift, and other sources

In production, an enterprise context layer sits above Athena and your other systems, connecting to Snowflake, Redshift, and additional sources so an agent can query across all of them while limiting the need to copy data into a new store. The context layer combines that federated access using shared data products, so an agent asking about revenue by region gets one governed answer whether the underlying rows live in Athena or somewhere else.

Data products as agent-consumable context

A context layer works because it packages data into data products that an agent can trust. Each one carries a definition, machine-readable metadata, and version history instead of column names alone. This creates a consistent, versioned interface, so a model doesn’t need to guess what a raw Athena table means or infer a business rule from column names alone.

Governance, lineage, and policy enforcement

Access policy, lineage, and shared definitions need to be built into the context layer from the start, not added after agents are already running queries against Athena. That includes tracking agentic data, the reasoning logs and tool calls agents themselves generate, under the same governance rules as the data they queried to produce it. Additionally, every policy enforced at this layer applies once, centrally, instead of being rebuilt separately for Athena, Snowflake, and every other source an agent might touch.

How context engineering changes what agents can do with Athena data

From natural-language SQL to grounded, explainable answers

With a context layer in place, an agent can take a natural-language question, translate it into SQL using Athena as a data source, combine it with data from another source, and return an answer tied to a governed definition instead of a guess. You can trace that answer back to the data product, the definition, and the underlying rows that produced it, and that traceability is what makes the answer explainable rather than a black box.

Building a context engineering roadmap for an existing Athena data estate

How do you create this workflow in production? Starburst is designed to provide access to your context layer, and combine that data alongside data from your entire organization using data federation to provide universal data access. 

Here’s how that works in practice. 

Audit current Athena workloads and where context is missing

Start by listing which Athena tables and workloads any agent project would touch, then check each one against a simple question: 

• Does a shared, documented definition already exist for the fields an agent would need?

Most teams find the missing piece isn’t the SQL. It’s the definition layer. That finding lines up with the data. Recent estimates put agent-project production deployment at just over 30 percent, up from a reported 88 percent of AI projects that never reached production in 2025, largely because of enterprise AI data access problems rather than model quality.

Add a context layer without a rip-and-replace migration

None of this needs you to migrate off Athena or consolidate every source into one platform. A context layer connects to Athena where it already runs, adds governed data products and shared definitions on top, and extends the same policy to Snowflake, Redshift, and other sources as you add them. You can also expose that layer to agents through a governed, multi-tenant hosted MCP service, so tools and context reach agents through one controlled path instead of a separate integration for every source. That’s your AI data strategy built on what you already run, not a replacement for it.

Want to know more about accessing Athena alongside your business context using Starburst? Download the Data Engineer’s Guide to Reducing Your Cloud Data Warehouse Spending.

FAQs

What is context engineering, and how is it different from prompt engineering?

Prompt engineering focuses on wording a request so a model responds well. Context engineering asks what business definitions, metadata, and access rules an agent needs before it ever sees a prompt. You build a governed layer of shared definitions and metadata that sits between your data and any model you connect to it, so accuracy depends on that layer instead of on model choice alone.

What’s missing when agents query Amazon Athena directly?

Give an agent a valid connection to Amazon Athena and it can write and run correct SQL. That solves the query problem, not the semantic one. Athena has no built-in concept of a shared metric definition or a policy that travels with a table, so two agents can query the same table and land on two different, technically correct answers to the same question.

Do you need to migrate off Athena to add a context layer?

No. A context layer connects to Athena where it already runs and adds governed data products and shared definitions on top of it. You extend that same policy to Snowflake, Redshift, and other sources as you add them, building your AI data strategy on what you already run instead of a rip-and-replace migration.

How does an enterprise context layer relate to a semantic layer?

An enterprise context layer sits above Athena and your other systems, combining federated access with data products that carry a metric definition, machine-readable metadata, and version history. It extends what a semantic layer does by making those shared definitions consumable by agents as well as BI tools, so an agent asking about revenue by region gets one governed answer no matter which system holds the rows.

What governance controls does a context layer add?

A context layer adds lineage, a consistent access policy, and an audit trail that shows why an agent reached a conclusion, none of which Athena provides on its own. It also tracks agentic data, the reasoning logs and tool calls agents generate, under the same governance rules as the data they queried to produce it, so every policy applies once, centrally, instead of being rebuilt for each source.

How do you get to that point? This is a field guide to that second half, drawn from the patterns Starburst worked through while readying data products for our own internal analytics agent.

How data products can get detached from their source of truth 

At the heart of this problem is the question of a data product’s value, and the value of a data product is directly tied to the value of its use within an organization. That organization, apart from anything else, has to be understood fully. It needs to have the business logic and context of the actual organization encoded into the logic of the data product. If you do not start there, then you will have a mismatch between the logic of the data product and the logic of your business. 

We know this already from semantic layer considerations in other domains. A query that executes tells you that the SQL was valid, but it tells you nothing about whether the answer matched the question. This is exactly the same trap with data products. You can create a data product that is perfectly valid and returns results that are perfectly valid, but this does not mean that the results adhere to the logic of your business or generate returns that you need. Assessing this match is the first consideration when determining whether a data product works or doesn’t work for you in real world environments.  

How AI data products accelerate the process, but that can accelerate the problems as well

AI brings a special acceleration to the process. An agent will take whatever the product declares, run it at full speed, and return something that reads as authoritative whether or not it is correct. This is the problem of trust, but dialled up in speed. 

A person has defenses against this. They notice when a total looks off, they know roughly what last quarter’s number was, and they ask a colleague when something seems strange. An agent has none of that instinct. It accepts the product’s logic as ground truth and moves on. That is exactly why the testing burden shifts onto you, before the product is published, rather than onto the consumer at query time.

Start with your subject matter as the source of truth

The way around this is to start with the logic of your business first. You need to start there, because that’s the inherent business logic that your data product is going to mirror. If that’s off, then all bets are off too. In this sense, the single most effective test is also the least technical. 

SMEs know your business, and they should know your data products too

Sit down with the subject-matter expert who will rely on the product and ask it the real questions they would put to it in their actual work. Not synthetic test queries. The questions that matter to their job.

Why this approach works

Two things make this work. First, the SME can judge completeness, not just correctness. An answer can be accurate and still be missing rows, and only someone who knows the domain will notice the count is short. Second, it surfaces the questions the product was never designed to answer, which tells you where the next iteration needs to go.

Introducing a logic gate between the business and the data product

Treat this as a quality gate, not a formality. A product does not graduate to an agent until an expert has confirmed its answers against the questions they genuinely ask. Everything upstream of that gate is preparation. This is the gate itself.

How to test for failure when an answer looks right but isn’t correct

Here is the failure mode that makes the case for SME testing, drawn from a real session.

Let’s say that an agent was asked to list a sales leader’s open opportunities. It returned 13. The leader knew the real number was 17. The agent had not crashed, thrown an error, or done anything visibly wrong. It confidently returned the wrong number.

The investigation traced the gap to a record-type flag in the view logic. Renewals were being filtered out by a rule that no one had flagged as significant when the view was built. The model reasoned correctly. The SQL ran correctly, but underneath it all, the data product had quietly encoded an assumption that did not match the question being asked.

Now, how did you catch this? It wasn’t through testing, and it wasn’t the agent. A human caught it who knew the context and the business logic, and could see that the answer should have been 17. That is the entire argument for SME Q&A. You always need the experts in the loop, and their understanding of the business and the problem needs to constantly correct the logic of the data product. Without that person, the problem goes unnoticed, that wrong answer ships, and it keeps being wrong every time anyone asks a similar question.

When an answer is wrong, look in three places

What should you do when you discover a bad answer? First, resist the urge to treat each failure as unique. In practice, the wrong answer can usually be traced to a problem in one of the following layers. Checking them in order turns debugging from an open-ended hunt into a short checklist.

Check your product metadata 

Product metadata encodes the business rules and context attached to the product itself. Ask yourself the following questions. 

• Does it encode which categories to include or exclude? 
• Does it distinguish between two similar-looking measures that mean different things?

This is the most common failure point, and the easiest to overlook because the data looks fine.

Check your column descriptions 

The column descriptions govern the field-level definitions in the underlying view. Did the agent misread what a column represents because the description was thin, absent, or misleading? This is the second most common culprit.

Check your view logic 

The view logic is another key checkpoint, and involves the SQL itself, including the filters and joins. A rule that silently drops records, as in the example above, often results from errors here. This is the least frequent cause, but the highest impact when it happens, because the numbers look internally consistent and nothing seems off.

The ranking matters. Teams instinctively suspect the SQL first because it is the most technical layer and seems the most opaque, but the metadata and column descriptions fail far more often. Start where the failures actually cluster, and most issues resolve in the first two checks.

Version your metadata, or lose your ability to explain its logic

At some point a leader will ask why the agent gave a particular answer, and the honest response has to accord with the business logic of your business. This is impossible if the product’s definitions have changed since and no one recorded the change.

To fix this, treat metadata with the same discipline as application code. Keep definitions in version control, kept in sync with your query engine, so every rule the agent reasoned from can be traced to a specific point in time. When an answer from last week comes into question, you can reconstruct exactly what the agent knew when it answered. Debugging becomes a lookup instead of an archaeology project. For the broader case on why this layer matters as much as it does, business metadata is what earns trust in the answer.

Don’t skip governance because testing felt like the finish line

It’s worth noting that there is a specific trap once a data product passes SME Q&A. At that point, it feels finished, and it can be tempting to leave it at that. But this is the most expensive corner to cut, because a data product’s real value is found in its economies of scale, and as you roll out your production line using the data product, errors will propagate and worsen over time. One of these errors could involve governance, and that’s a key thing to consider as you roll a data product into a production environment. 

Why governance of AI data products matters so much

An agent inherits the reach of whoever queries it. If a broad analyst role already grants wide data access, layering an agent on top without tighter controls exposes sensitive data to a far larger audience than intended. 

The fix is a role structure defined per data product, designed with the data and platform teams rather than retrofitted after the agent is live. Governance added late means unwinding access patterns people already depend on, which is far harder than defining them correctly at the start. This is a core theme of AI data governance, and it belongs in the test plan, not after it.

The data product testing loop from start to finish

Where does all of this leave us? Pulling it together, a reliable validation pass looks like this:

1. Assemble the product and its metadata, and confirm every field the agent will touch has a real description.
2. Sit with the SME and ask the genuine questions they would ask in their work.
3. For every wrong or incomplete answer, check the three layers in order: product metadata, column descriptions, then view logic.
4. Fix and re-test the same questions. Iterate until the answers are accurate and complete.
5. Define access controls per product before the agent goes live, not after.
6. Keep the whole definition in version control, so any future answer can be reconstructed and explained.

None of this is glamorous, and that is the point. The slow, deliberate work of testing and tracing is what separates a data product that produces answers from one your leaders can actually act on. If you are putting an agent in front of the people who run your business, it is not a corner worth cutting.

Ready to build on a foundation designed for this? Check out my recent webinar, Building Data Products With AI.

Let’s start with the basics. A data warehouse is a specialized analytical database designed for large-scale, SQL-driven analysis of structured and semi-structured data.

Unlike operational databases that handle day-to-day transactions, warehouses use techniques like massively parallel processing (MPP) and columnar storage to crunch through enormous datasets efficiently.

Think of platforms like Google BigQuery with its serverless, decoupled storage and compute model, or Amazon Redshift‘s managed petabyte-scale MPP architecture. These aren’t just bigger databases – they’re engineered from the ground up for analytical workloads where you’re aggregating millions of rows rather than looking up individual records.

Today, data warehouses serve as the central analytical hub where data from various sources gets consolidated, cleaned, and transformed into formats that power business intelligence, reporting, and increasingly, machine learning and AI initiatives. In this space, BigQuery and Snowflake position themselves as core platforms for analytics-to-AI workflows, offering built-in governance, lineage tracking, and ML integration hooks. Meanwhile, Redshift integrates seamlessly with data lakes and supports querying lake data through Spectrum while enabling exports to Amazon S3 for downstream processing with tools like Athena, EMR, and SageMaker.

But that’s where it gets interesting for data engineers. While getting data into warehouses is well-understood, extracting data from warehouses efficiently and reliably presents unique challenges that many teams struggle with.

As organizations build more sophisticated data architectures spanning multiple cloud platforms and platforms, understanding how to move data out of warehouses becomes critical for feeding AI/ML pipelines, operational systems, and cross-platform analytics.

How data warehouses anchor modern analytics strategies

Data warehouses have become indispensable because they solve the fundamental problem of enabling fast, reliable analytics across massive datasets without disrupting operational systems? They achieve this through specialized architecture optimizations that traditional databases simply can’t match.

Designed to be the analytical powerhouse your organization needs

In enterprise environments, data warehouses typically serve three primary functions. First, they’re the backbone of business intelligence and reporting, handling everything from executive dashboards to detailed financial analysis. Second, they enable ad-hoc analytics where analysts can explore data without predefined queries or reports. Third, and increasingly important, they support AI and analytics solutions by providing clean, consistent training data and feature engineering capabilities.

Consider a retail company tracking customer behavior across web, mobile, and in-store touchpoints. All this data flows into their warehouse where it gets joined, aggregated, and transformed into actionable insights. Marketing teams query for customer segments, finance pulls revenue metrics, and data scientists extract features for recommendation engines. In this context, the warehouse becomes the single source of truth that different teams can trust. Organizations in the retail analytics solutions space often see significant value from this unified approach.

Heterogeneity drives real-world architectural patterns that drive value

Warehouses occupy a privileged space in modern data stacks. In fact, many modern data architectures position warehouses as the centerpiece of of larger ecosystems. But increasingly, there is another pattern, based around data lakes. In this model, you’ll often see patterns where warehouses integrate with data lakes, creating hybrid environments where raw data lives in cheap object storage while curated, frequently-accessed data resides in the warehouse for fast queries. Today, these open data lakehouse architectures are becoming increasingly popular as organizations adopt multi-cloud strategies where different business units use different warehouse platforms, requiring federation and cross-platform analytics capabilities.

Financial services data analytics companies, for example, might keep regulatory reporting data in one cloud’s warehouse while running customer analytics in another cloud’s platform.

Manufacturing companies often need to analyze data that’s distributed globally due to data sovereignty requirements, making cross-region warehouse coordination essential. Healthcare data analytics organizations face similar challenges with data residency requirements.

Moving data out of warehouses brings unexpected complexity

While ingesting data into warehouses is straightforward thanks to well-established ETL patterns and vendor-provided tools, extracting data presents a different set of challenges that catch many teams off guard. These aren’t just technical hurdles. They have real business impact in terms of costs, performance, and operational overhead.

The hidden costs of data egress

One of the first surprises teams encounter is egress pricing. Cross-region or cross-cloud data transfer is billable for platforms like Snowflake, while BigQuery offers free batch extracts for compute but charges for cross-region transfers. When you’re moving large datasets regularly, these costs add up quickly.

We’ve seen organizations discover they’re spending thousands monthly on data transfer fees they didn’t anticipate when they designed their initial architecture. The solution often requires rethinking data placement and compute location to minimize cross-region movement, sometimes necessitating data migration solutions.

Performance bottlenecks and quota limits

There are other considerations as well. BigQuery imposes quotas on interactive queries and API connections, with daily limits on exports and constraints like 1 GB maximum file sizes before automatic sharding kicks in. Snowflake requires running warehouses for data access, often necessitating multi-cluster scaling for concurrent workloads.

These limits aren’t arbitrary. They protect multi-tenant services and ensure fair resource usage. But they mean you can’t just throw hardware at the problem like you might with on-premises systems. You need to design around these constraints from the beginning.

Consistency and format complications

Extracting large result sets while maintaining data consistency requires careful planning. You need techniques like BigQuery’s FOR SYSTEM TIME AS OF or Snowflake’s Time Travel to create consistent snapshots without holding locks that could impact other workloads.

Format handling adds another layer of complexity. Different export formats have different limitations. For instance, BigQuery’s CSV export doesn’t support nested or repeated fields, forcing you to choose between data fidelity and format compatibility. Additionally, schema evolution becomes particularly tricky when downstream systems expect stable schemas but your warehouse data models are constantly evolving. Understanding comparing open table formats can help address some of these challenges.

Security and governance gaps

Perhaps the most concerning challenge is governance portability. Fine-grained policies like row access controls and column masking are enforced inside the warehouse. When you export data, these protections don’t automatically follow unless explicitly applied. While Snowflake applies masking to COPY INTO output for unauthorized roles, once files leave the warehouse environment, external systems must implement their own controls.

This creates potential security gaps where sensitive data that was properly protected in the warehouse becomes exposed in downstream systems that lack equivalent controls. Organizations in regulated industries like federal government solutions face particularly stringent requirements around data governance.

Building robust data extraction patterns that scale

Successfully extracting data from warehouses requires a strategic approach that considers performance, cost, governance, and operational sustainability. The key is matching your extraction pattern to your specific use case and constraints.

Choosing the right extraction approach

For one-off exports or archival purposes, native bulk exports work well – think Redshift UNLOAD to Amazon S3 or BigQuery EXPORT DATA operations. These are simple to implement and cost-effective for infrequent large transfers.

For high-throughput programmatic access, you’ll want to leverage APIs like BigQuery’s Storage Read API or consider federation approaches. This is where tools like the Starburst Galaxy become valuable, offering native connectors that use optimized read paths and support features like predicate pushdown to minimize data movement. Meanwhile, Starburst Enterprise provides additional capabilities for large-scale enterprise deployments.

For operational analytics where you’re syncing warehouse insights back to operational systems, reverse ELT tools or data federation through query engines often provide better governance and consistency than file-based approaches, especially when building data applications.

Optimizing for cost and performance

Geographic alignment is crucial for cost control. Running processing in the same cloud and region as your warehouse minimizes transfer fees and latency. This means if your BigQuery datasets are in US-Central, your extraction jobs should run there too.

Starburst’s connector architecture takes this further with features like dynamic filtering and cost-based join pushdown, ensuring work gets pushed down to the warehouse where it’s most efficient rather than pulling unnecessary data across the network. This is one of the key benefits of Starburst over building custom extraction solutions.

For reliability at scale, consider fault tolerance mechanisms. Starburst’s Fault-Tolerant Execution enables retrying failed tasks and spooling intermediate results, which becomes essential for long-running extraction jobs that can’t afford to restart from scratch on transient failures.

Implementing governance that travels with your data

Security should be built into your extraction pipeline, not bolted on afterward. Private networking through AWS PrivateLink or Azure Private Link keeps data movement off the public internet, while identity federation and credential pass-through maintain audit trails showing who accessed what data.

For organizations with complex governance requirements, Starburst’s integration with Apache Ranger enables global access control that can apply consistent policies across multiple data sources and destinations. Galaxy’s built-in RBAC and column masking capabilities let you implement data protection that follows your data rather than being tied to specific platforms.

Starting with intelligent caching and acceleration

Rather than immediately building complex extraction pipelines, consider whether you can optimize access patterns first. Starburst’s Table Scan Redirection and Cache Service can create governed cached copies of frequently accessed warehouse tables in your data lake, reducing load on the source system while maintaining transparent access for users.

This approach is particularly effective for dashboard and reporting workloads where the same aggregated data gets queried repeatedly, including instances where those workloads are being taken over by AI. Instead of hitting the warehouse every time, queries get transparently redirected to cached results, improving performance while reducing costs.

The path forward involves starting with clear use cases, implementing proper governance from day one, and choosing tools that can grow with your needs. Whether you’re feeding ML pipelines, enabling cross-platform analytics, or creating and managing data products, success comes from treating data extraction as a strategic capability rather than a tactical afterthought.

Want to know more about data warehouses? Check out the Starburst Academy course, Exploring data warehouses.