Most guidance around data products stops at the build. You land the data, stage it, curate it, write the metadata, and ship the data product. But there’s more to data products than that. The build is just the starting point. Using data products effectively involves more than that. The unseen maintenance work comes afterward when you prove that the data product returns answers that a domain expert would sign off on. In other words, the job is only done when the data product is proven to improve actual work and actual efficiency.
How do you get to that point? This is a field guide to that second half, drawn from the patterns Starburst worked through while readying data products for our own internal analytics agent.
How data products can get detached from their source of truth
At the heart of this problem is the question of a data product’s value, and the value of a data product is directly tied to the value of its use within an organization. That organization, apart from anything else, has to be understood fully. It needs to have the business logic and context of the actual organization encoded into the logic of the data product. If you do not start there, then you will have a mismatch between the logic of the data product and the logic of your business.
We know this already from semantic layer considerations in other domains. A query that executes tells you that the SQL was valid, but it tells you nothing about whether the answer matched the question. This is exactly the same trap with data products. You can create a data product that is perfectly valid and returns results that are perfectly valid, but this does not mean that the results adhere to the logic of your business or generate returns that you need. Assessing this match is the first consideration when determining whether a data product works or doesn’t work for you in real world environments.
How AI data products accelerate the process, but that can accelerate the problems as well
AI brings a special acceleration to the process. An agent will take whatever the product declares, run it at full speed, and return something that reads as authoritative whether or not it is correct. This is the problem of trust, but dialled up in speed.
A person has defenses against this. They notice when a total looks off, they know roughly what last quarter’s number was, and they ask a colleague when something seems strange. An agent has none of that instinct. It accepts the product’s logic as ground truth and moves on. That is exactly why the testing burden shifts onto you, before the product is published, rather than onto the consumer at query time.
Start with your subject matter as the source of truth
The way around this is to start with the logic of your business first. You need to start there, because that’s the inherent business logic that your data product is going to mirror. If that’s off, then all bets are off too. In this sense, the single most effective test is also the least technical.
SMEs know your business, and they should know your data products too
Sit down with the subject-matter expert who will rely on the product and ask it the real questions they would put to it in their actual work. Not synthetic test queries. The questions that matter to their job.
Why this approach works
Two things make this work. First, the SME can judge completeness, not just correctness. An answer can be accurate and still be missing rows, and only someone who knows the domain will notice the count is short. Second, it surfaces the questions the product was never designed to answer, which tells you where the next iteration needs to go.
Introducing a logic gate between the business and the data product
Treat this as a quality gate, not a formality. A product does not graduate to an agent until an expert has confirmed its answers against the questions they genuinely ask. Everything upstream of that gate is preparation. This is the gate itself.
How to test for failure when an answer looks right but isn’t correct
Here is the failure mode that makes the case for SME testing, drawn from a real session.
Let’s say that an agent was asked to list a sales leader’s open opportunities. It returned 13. The leader knew the real number was 17. The agent had not crashed, thrown an error, or done anything visibly wrong. It confidently returned the wrong number.
The investigation traced the gap to a record-type flag in the view logic. Renewals were being filtered out by a rule that no one had flagged as significant when the view was built. The model reasoned correctly. The SQL ran correctly, but underneath it all, the data product had quietly encoded an assumption that did not match the question being asked.
Now, how did you catch this? It wasn’t through testing, and it wasn’t the agent. A human caught it who knew the context and the business logic, and could see that the answer should have been 17. That is the entire argument for SME Q&A. You always need the experts in the loop, and their understanding of the business and the problem needs to constantly correct the logic of the data product. Without that person, the problem goes unnoticed, that wrong answer ships, and it keeps being wrong every time anyone asks a similar question.
When an answer is wrong, look in three places
What should you do when you discover a bad answer? First, resist the urge to treat each failure as unique. In practice, the wrong answer can usually be traced to a problem in one of the following layers. Checking them in order turns debugging from an open-ended hunt into a short checklist.
Check your product metadata
Product metadata encodes the business rules and context attached to the product itself. Ask yourself the following questions.
- Does it encode which categories to include or exclude?
- Does it distinguish between two similar-looking measures that mean different things?
This is the most common failure point, and the easiest to overlook because the data looks fine.
Check your column descriptions
The column descriptions govern the field-level definitions in the underlying view. Did the agent misread what a column represents because the description was thin, absent, or misleading? This is the second most common culprit.
Check your view logic
The view logic is another key checkpoint, and involves the SQL itself, including the filters and joins. A rule that silently drops records, as in the example above, often results from errors here. This is the least frequent cause, but the highest impact when it happens, because the numbers look internally consistent and nothing seems off.
The ranking matters. Teams instinctively suspect the SQL first because it is the most technical layer and seems the most opaque, but the metadata and column descriptions fail far more often. Start where the failures actually cluster, and most issues resolve in the first two checks.
Version your metadata, or lose your ability to explain its logic
At some point a leader will ask why the agent gave a particular answer, and the honest response has to accord with the business logic of your business. This is impossible if the product’s definitions have changed since and no one recorded the change.
To fix this, treat metadata with the same discipline as application code. Keep definitions in version control, kept in sync with your query engine, so every rule the agent reasoned from can be traced to a specific point in time. When an answer from last week comes into question, you can reconstruct exactly what the agent knew when it answered. Debugging becomes a lookup instead of an archaeology project. For the broader case on why this layer matters as much as it does, business metadata is what earns trust in the answer.
Don’t skip governance because testing felt like the finish line
It’s worth noting that there is a specific trap once a data product passes SME Q&A. At that point, it feels finished, and it can be tempting to leave it at that. But this is the most expensive corner to cut, because a data product’s real value is found in its economies of scale, and as you roll out your production line using the data product, errors will propagate and worsen over time. One of these errors could involve governance, and that’s a key thing to consider as you roll a data product into a production environment.
Why governance of AI data products matters so much
An agent inherits the reach of whoever queries it. If a broad analyst role already grants wide data access, layering an agent on top without tighter controls exposes sensitive data to a far larger audience than intended.
The fix is a role structure defined per data product, designed with the data and platform teams rather than retrofitted after the agent is live. Governance added late means unwinding access patterns people already depend on, which is far harder than defining them correctly at the start. This is a core theme of AI data governance, and it belongs in the test plan, not after it.
The data product testing loop from start to finish
Where does all of this leave us? Pulling it together, a reliable validation pass looks like this:
- Assemble the product and its metadata, and confirm every field the agent will touch has a real description.
- Sit with the SME and ask the genuine questions they would ask in their work.
- For every wrong or incomplete answer, check the three layers in order: product metadata, column descriptions, then view logic.
- Fix and re-test the same questions. Iterate until the answers are accurate and complete.
- Define access controls per product before the agent goes live, not after.
- Keep the whole definition in version control, so any future answer can be reconstructed and explained.
None of this is glamorous, and that is the point. The slow, deliberate work of testing and tracing is what separates a data product that produces answers from one your leaders can actually act on. If you are putting an agent in front of the people who run your business, it is not a corner worth cutting.
Ready to build on a foundation designed for this? Check out my recent webinar, Building Data Products With AI.
