Data products have quickly become the foundation for delivering curated, domain-specific information across the modern enterprise. However, as the demand for trusted datasets accelerates to support both analytics and artificial intelligence, the actual process of building these assets creates a severe operational bottleneck.
This whole issue is encapsulated in one fundamental question. How do you quickly create data products at scale? The answer is more complex than it might seem.
Traditional data product construction often presents a binary choice. On the one hand, users can adopt a user interface to manually curate datasets. On the other, they could write complex custom code to manage the lifecycle of data assets.
Both approaches create problems, just different types of problems.
While a UI is accessible, it is difficult to version. Meanwhile, custom code allows for automation but creates a bottleneck for non-technical stakeholders. As organizations scale, this tension becomes a primary obstacle to efficiency.
The solution is neither. The shift toward data product solutions requires a middle ground that supports production AND consumption from both human analysts and autonomous agents.
The transition from UI to data products as code
Let’s begin by looking at what data products as code is and how it functions.
Data products through the UI
For years, creating data products was limited to two primary methods. Users could use the Starburst Galaxy or Starburst Enterprise user interface to select tables and define metadata, or they could interact directly with the API. The UI is effective for initial exploration but does not naturally support the version control and peer review processes that define modern software engineering. Conversely, direct API interaction requires significant boilerplate code that is often inaccessible to business domains.
Data products as Code (DPaC)
The introduction of Data Products as Code (DPaC) changes this dynamic by providing a YAML representation of the data product. This YAML file includes the domain, description, owner, and business rules in a single, portable package. Because YAML is a structured but human-readable format, it serves as a bridge. Users can open, read, and edit the file in a standard text editor, yet it remains structured enough for automated systems to process. That’s what lets humans and agents work on the same artifact. You describe what you want in plain language, the AI drafts the YAML, and if it isn’t right you tweak it, in natural language or in the file directly. Same document, either direction, every change versioned.
The YAML structure typically includes several key sections. The metadata section defines the identity of the product, including its unique business rules and definitions. The dataset section lists the underlying tables, views, or materialized views, often residing in data lakes or lakehouses. The dataset section contains the SQL logic that transforms raw data into a consumable format. By tying these elements together, the metadata definitions of a data product are never separated from the data implementation logic.
Using natural language to create data products
Today, there is another option–using natural language to aid in data product creation and management. Recent features in the Starburst platform allow users to describe a desired data product in plain English. An AI agent interprets this description, identifies the relevant underlying data in data lakes, lakehouses, or warehouses, and generates the corresponding YAML definition.
How this approach differs
This process does not eliminate the need for a structured artifact. Instead, it automates the generation of that artifact. The user provides the intent, and the system provides the structure. If the generated YAML does not perfectly match requirements, the user can iterate. A user might ask the agent to modify a specific business rule or add a new column, or they might choose to edit the YAML file directly. This feedback loop ensures that the technical definition remains accurate while lowering the barrier to entry for those who do not write structured configurations from scratch.
This approach also addresses the “cold start” problem in data documentation. Instead of staring at a blank screen, a data steward starts with a generated draft that already includes schema information and suggested descriptions. The steward then acts as an editor, refining the AI-generated content to ensure it meets organizational standards. This collaboration significantly reduces the time required to move from a data request to a published data product.
Check out the video below for more information.
Building for human and agent consumers
This approach has a number of advantages. In previous architectural patterns, data products were designed almost exclusively for human consumption. An analyst would find a product in a catalog, read the description, and use SQL to query the data. However, as AI and analytics solutions become more prevalent, the consumer of a data product is increasingly likely to be an AI agent.
Agents require a different level of context than humans. A human might infer the meaning of a column named “adj_rev” based on experience, but an agent needs explicit definitions. To make data products AI-ready, Starburst introduced new metadata fields within the DPaC framework.
These include:
1. Business Rules
These are plain-language explanations of the logic encoded within the data. For example, a business rule might specify how “churn” is calculated or which regions are included in “North American Sales.”
2. Gold Standard Questions
These are the specific questions the data product is designed to answer. By including these in the metadata, the system provides a signal to the AI agent. When a user asks a question that matches a gold standard question, the agent knows it has found the correct, certified source of truth.
3. Automatic Metadata Enrichment
AI can profile the schema of tables in Amazon S3 or other storage layers and suggest descriptions for tables and columns. This ensures that documentation is comprehensive from the start.
By embedding this information directly into the YAML file, the context travels with the data. This is a critical component of How Starburst and Dell Created Exactly What Enterprise AI… needs, as it focuses on providing curated, certified context rather than raw metadata dumps.
Managing the lifecycle with CLI and CI/CD
The shift toward DPaC is supported by a command line interface (CLI) that exposes the full data product lifecycle. This CLI allows users to perform import and export operations, which are the primary mechanisms for moving data products between Starburst and a local environment.
When a user exports a data product, they receive the YAML representation. They can then check this file into a version control system like GitHub. This enables a GitOps workflow for data management. For instance, when a change is made to a business rule in the YAML file and pushed to the repository, a GitHub Action can trigger the CLI to import the updated definition into Starburst.
This level of automation is vital for organizations that must maintain strict audit trails. Teams can see exactly who changed a data product definition, what the change was, and when it was deployed. This approach also facilitates peer review. A senior data steward can review a pull request containing a new data product definition before it is made available to the wider organization.
Furthermore, the CLI supports validation. Before a YAML file is imported, the system can check for syntax errors, broken references to underlying tables in Amazon S3, or missing mandatory metadata fields. This prevents broken data products from reaching the production catalog, ensuring a higher level of reliability for end users.
The role of agents as producers
One of the more significant shifts in the data product lifecycle is the role of AI agents as producers. In the past, only humans created data products. Now, an agent can analyze usage patterns and suggest the creation of a new data product to fill a gap in the catalog.
An agent might notice that many users are joining the same three tables from different data silos and performing the same calculations. The agent can then generate a YAML definition for a new data product that joins these tables and applies the common logic. A human data steward can then review this suggested product, make any necessary adjustments, and verify it for general use.
This collaboration between humans and agents is made possible by the common language of YAML. The agent writes the initial draft, the human refines it, and the system executes it. This workflow allows for a much higher volume of data products to be created and maintained than would be possible through manual effort alone. It also ensures that the data catalog remains relevant to the actual needs of the business.
Agentic AI and data products, underpinned by data federation
Overall, writing data products with natural language does not mean ignoring the underlying architecture. Instead, the effectiveness of a data product is tied to the performance and accessibility of the data.
We also see Starburst’s unique approach to data access at work. Starburst leverages Trino to provide a high-performance query engine that can access data across various data silos, including data lakes and legacy warehouses. This approach allows data products to be created from multiple data sources using data federation. There is tremendous power in this strategy as it allows universal access without data centralization.
All of this comes into play with data products, and AI agents make full use of the data foundation accessible by Starburst. This approach is an improvement over previous approaches to data product creation based around the UI or Data Products as Code. Most importantly, it is another example of agentic AI in action.
Want to know more about data products? Check out my recent webinar.
