Managing multiple data sources is one of the foundational demands of data engineering. As organizations grow, the problem can get worse. Data teams often find themselves trapped in a cycle of maintaining identical catalog definitions just to point to different regional projects or database instances.
Dynamic connection passthrough is designed to help. Let’s take a closer look.
What is dynamic connection passthrough?
Dynamic connection passthrough is a feature that lets a single Starburst catalog accept runtime connection context at query time instead of baking that context into static catalog properties. In practice, the client sends extra credentials with the query, and the connector uses them to decide which remote account, project, JDBC URL, or cloud configuration to use for that request.
The image below shows how dynamic passthrough works at a high level.
Standardizing dynamic access across all data sources
This highlights an important point. Dynamic connection passthrough is a platform capability, not a connector-specific one-off. The catalog definition is mostly a template, while the runtime connection details are injected with extra credentials when the query is executed.
The capability is growing too. Today, we support this pattern in connectors such as Snowflake, PostgreSQL, BigQuery, and Iceberg. As we add support to more connectors, the feature story stays the same even if the connector-specific properties differ.
Why dynamic connection passthrough matters
Modern data platforms rarely serve a single team or environment. There are multiple reasons for this. For example, SaaS vendors embed analytics into their products, or enterprise platforms spin up isolated environments on demand, whether for testing, compliance, or experimentation. In these deployments, the number of distinct data environments a single Starburst cluster needs to reach can grow quickly and unpredictably.
Why creating a separate catalog for each environment isn’t the right choice
Often, the natural inclination is to create a separate catalog for each environment. That works fine at a small scale. But when a platform needs to support dozens or hundreds of environments, each with its own database cluster, credentials, and lifecycle, managing individual catalogs becomes a significant operational burden. Every new environment means adding a catalog, propagating that configuration across all active clusters, and tearing it down again when the environment is deprovisioned.
The overhead compounds with scale.
Dynamic connection passthrough makes catalog management easy
Dynamic connection passthrough was designed specifically to break that pattern. Instead of maintaining one catalog per environment, a single catalog acts as a reusable template. The caller provides the connection details at query time, implemented as session parameters, and the connector resolves the right remote environment on the fly.
This results in a situation with no catalog changes, no cluster restarts, no propagation steps. A new environment becomes available the moment its credentials are ready to pass.
Shifting complexity from the platform layer to the application layer
Overall, this shifts focus from the platform layer to the application layer. This is a meaningfully different model from dynamic catalogs, which still require explicit catalog registration and cluster-level propagation. Dynamic connection passthrough keeps that complexity entirely out of the platform layer and puts it where it belongs, in the application passing the query.
When is this useful in production? In practice, this is useful in many applications, including the following objectives.
- Support ephemeral or sandbox environments that are created and destroyed on demand.
- Reduce catalog sprawl in shared or embedded deployments.
- Enable multi-tenant analytics where each tenant connects to their own isolated data source.
- Keep credentials scoped to the application layer and passed only when needed.
Putting dynamic connection passthrough into practice
Let’s see how this works in practice with an example. Regardless of the individual connectors involved, the pattern is mostly the same. You turn a standard catalog into an open template by decoupling the connection logic from the platform configuration, which requires four basic setup steps.
- Configure a catalog in dynamic mode.
- Map catalog properties to extra credential names.
- Pass the actual values from the client with each query or session.
Set up all client connections that use this catalog to append credentials using the extraCredentials attribute.
Client examples
Let’s look at a few client examples.
Instead of hardcoding a target database into the platform, as you would have done in the past, the application passes the specific connection string and credentials directly with the query.
The code below demonstrates the universal syntax pattern used to pass these runtime parameters. To do this, you can use either the Trino CLI or the Trino JDBC driver to pass runtime values. We show both examples below.
Trino CLI
trino \ --server http://<coordinator>:8080 \ --extra-credential key1=<VALUE1> \ --extra-credential key2=<VALUE2>
Trino JDBC
jdbc:trino://<coordinator>:8443/<catalog>/<schema>?extraCredentials=key1:<VALUE1>;key2:<VALUE2>
What changes from connector to connector is the shape of the runtime identity.
For example:
- Snowflake uses account URL plus user credentials.
- PostgreSQL uses JDBC URL plus user credentials.
- BigQuery uses project and credential context, plus optional view materialization settings.
- Iceberg uses AWS account, region, and Glue or Amazon S3 access settings.
Simple examples
Let’s dig into it a bit further. The examples below are intentionally small. They show the feature shape, not every supported property.
PostgreSQL configuration
Before a user can pass runtime credentials, a platform administrator defines a template catalog. This configuration maps the standard connector properties to the custom credential names the application will look for:
For PostgreSQL, the catalog can be configured to read the JDBC URL, username, and password from extra credentials.
connector.name=postgresql postgresql.authentication.type=DYNAMIC_CONNECTION postgresql-jdbc-url-credential-name=pg_url user-credential-name=pg_user password-credential-name=pg_password connection-pool.enabled=false
Similarly, a CLI example would look like this.
trino \ --server http://<coordinator>:8080 \ --extra-credential pg_url=jdbc:postgresql://db.example.net:5432/appdb \ --extra-credential pg_user=app_user \ --extra-credential pg_password=<PASSWORD>
In both cases, this simple mapping is what tells Starburst to intercept the pg_url, pg_user, and pg_password values from incoming queries and inject them into the PostgreSQL connector at runtime.
Snowflake configuration
The beauty of this feature is that the workflow remains identical even when switching to an entirely different architecture, like Snowflake. For Snowflake, the runtime values are the account URL and user credentials. Meanwhile, the administrator creates a similar template configuration, as in the example below.
connector.name=snowflake_parallel connection-url=jdbc:snowflake://snowflake-dynamic-passthrough snowflake.impersonation-type=dynamic_connection snowflake-account-credential-name=connection_url user-credential-name=user private-key-credential-name=private_key connection-pool.enabled=false
Once that template is live, an application or user can instantly target their specific Snowflake account using the exact same CLI pattern.
trino \ --server http://<coordinator>:8080 \ --extra-credential connection_url=https://acme.snowflakecomputing.com \ --extra-credential user=alice \ --extra-credential private_key=<BASE64_PRIVATE_KEY>
BigQuery example
This pattern also scales effortlessly to cloud data warehouses like BigQuery, where the runtime values are project-oriented rather than JDBC-oriented. The connector reads the project ID, parent project ID, and credentials key dynamically, with optional view materialization settings.
connector.name=bigquery bigquery.authentication.type=DYNAMIC_CONNECTION bigquery.project-id.credential-name=project_id bigquery.parent-project-id.credential-name=parent_project_id bigquery.credentials-key.credential-name=credentials_key bigquery.view-materialization-project.credential-name=view_materialization_project bigquery.view-materialization-dataset.credential-name=view_materialization_dataset
With the configuration handling the abstraction, the client simply passes the specific target project details, service account keys, and optional materialization settings at execution time, as in the example below.
trino \ --server http://<coordinator>:8080 \ --extra-credential project_id=<PROJECT_ID> \ --extra-credential parent_project_id=<PARENT_PROJECT_ID> \ --extra-credential credentials_key=<BASE64_SERVICE_ACCOUNT_JSON> \ --extra-credential view_materialization_project=<VIEW_MATERIALIZATION_PROJECT> \ --extra-credential view_materialization_dataset=<VIEW_MATERIALIZATION_DATASET>
Iceberg example
Finally, this pattern extends naturally to open table formats like Iceberg. While the implementation here is named dynamic configuration passthrough, the runtime pattern remains exactly the same.
In its current support, this approach is designed for use on AWS Glue and Amazon S3-backed Iceberg catalogs.
connector.name=iceberg iceberg.catalog.type=glue iceberg.dynamic-configuration-passthrough.enabled=true dynamic.aws-access-key-id.credential-name=aws_access_key_id dynamic.aws-secret-access-key.credential-name=aws_secret_access_key dynamic.aws-region.credential-name=aws_region
Just like the previous examples, the end user or application interacts with the catalog identically, injecting their specific AWS session parameters directly into the query stream, as in the example below.
trino \ --server http://<coordinator>:8080 \ --extra-credential aws_access_key_id=<AWS_ACCESS_KEY_ID> \ --extra-credential aws_secret_access_key=<AWS_SECRET_ACCESS_KEY> --extra-credential aws_region=<AWS_REGION>
One feature, connector-specific details
The architectural beauty of this approach is that while data ecosystems are fragmented, your access strategy doesn’t have to be. The important thing to keep consistent in documentation is the feature story, specifically:
- The catalog is mostly a template
- Runtime values are injected through extra credentials
- The same catalog can safely fan out to different remote environments.
The connector-specific details still matter, of course, but they are simply variations of the same underlying theme. Snowflake and PostgreSQL are JDBC-style examples. BigQuery is project-and-service-account oriented. Iceberg currently exposes the same idea through dynamic configuration for Glue and Amazon S3 rather than a JDBC connection string.
By isolating these differences inside the connector properties, Starburst ensures that your applications enjoy a uniform, predictable interface no matter where the underlying data lives.
Current scope and best practices
To get the most out of this architectural shift, platform teams should design their workflows around a few core operational principles.
A few practical guidelines help keep dynamic connection predictable:
- Treat the catalog as a reusable template and keep the runtime override surface intentionally small.
- Make sure the keys used in
extraCredentialsline up exactly with the configured*.credential-namemappings. - Prefer short-lived, scoped credentials passed at query time from the application layer.
- Expect connector-specific constraints. Some connectors disable features such as connection pooling, caching, or other shared state when dynamic mode is enabled.
- Document the required and optional runtime values clearly for each connector so client integrations stay simple.
These details are connector-specific, but the operational pattern is the same, requiring you to resolve runtime values per query, build the remote connection context from those values, and avoid assuming shared state across independent identities. By adhering to these guardrails, you can safely scale out multi-tenant and ephemeral data environments without introducing administrative chaos.
Your key takehome
Dynamic connection passthrough marks a fundamental shift in how we think about data virtualization and platform scaling.
It should be seen as a reusable feature for shared and embedded deployments. The connectors supported today are examples of that capability, not the boundary of the concept. As support expands, the core value stays the same, focused on one catalog definition, runtime connection context, and a much cleaner operational model for runtime access.
By decoupling the connection details from the platform itself, you unlock an architecture that is built to grow as fast as your application ecosystem demands.
Want to know more about Starburst? Try a free trial.
