An on-premises data lakehouse brings lakehouse capabilities into data centers you own or lease. It replaces Hadoop design with S3-compatible object storage, Apache Iceberg tables, a Trino-based query engine, and federation and governance layers.
Regulated teams can get lakehouse performance and open interoperability without sending data to a public cloud, provided hardware, network, table maintenance, and workloads are sized and tuned for it. You will learn when an on-premises lakehouse fits and how each layer works, including storage, table format, query engine, federation, governance, and availability. You will also see how to migrate from Hadoop in stages instead of in one big bang.
Key takeaways
- An on-premises lakehouse is a deliberate choice for sovereignty, latency, and cost, and it no longer requires Hadoop.
- The core stack is S3-compatible storage, Apache Iceberg, and Trino, a pattern we call the Icehouse.
- Federated query lets the lakehouse work with cloud and SaaS sources without forced centralization, so start federation-first and materialize hot paths into Iceberg.
- Build governance, security, and data products in from day one, especially if you plan to run AI on-premises.
- Migrate from Hadoop in stages: swap the engine first, land new data in Iceberg, and move Hive tables selectively.
Understand when an on-premises lakehouse fits
A data lakehouse pairs object storage with ACID (atomicity, consistency, isolation, durability) guarantees, governance, and, with proper tuning, compaction, and hardware sizing, warehouse-class performance through open table formats. An on-premises lakehouse applies those lakehouse principles in data centers you own or lease.
Object storage, open table formats, and separated storage and compute stay the same. On-premises, you own durability, capacity planning, network, and catalog operations. Teams stay on-premises for concrete reasons, including data sovereignty, regulatory control, low-latency operations, and egress fees.
Teams in regulated industries often can’t send their data to a public cloud. Compare the alternatives. A cloud-native lakehouse assumes cloud storage and managed services. A legacy Hadoop lake typically ties storage to compute through HDFS co-location and depends on aging components. The on-premises lakehouse keeps data local and removes that coupling.
Start with S3-compatible object storage
Use an S3-compatible object store as the center of the lake. Run it on servers or storage you already own, sized to meet your durability and throughput needs, and keep data in open file formats such as Parquet or Optimized Row Columnar (ORC). This replaces the Hadoop Distributed File System (HDFS) as the foundation. Trino is designed for S3-compatible systems, but only Amazon S3 and MinIO are tested for compatibility in open-source Trino. Starburst Enterprise documents the S3-compatible on-premises stores it supports, so if you run it, check that list first. If you choose a store that is not on it, test it yourself against your workloads before you commit. Separating storage from compute lets each scale on its own. Add storage nodes when data grows, and add query workers when concurrency grows. Additionally, you stop buying both together. Plan three design choices early:
- Capacity: Size for raw data, table snapshots, and growth.
- Durability: Choose erasure coding or replication based on your cost and recovery targets.
- Network: Provision enough bandwidth between compute and storage, since uncached reads go over the network.
Those choices set how storage, compute, and network fit together.
Use Apache Iceberg as the table format
Files alone don’t give you transactions. Apache Iceberg adds ACID transactions, schema evolution, time travel, and reliable updates and deletes on top of Parquet or ORC files. Analysts can query consistent snapshots while pipelines write new data. Open formats also protect you from lock-in. Engines that support your tables’ Iceberg format version, features, and catalog can use them, which keeps the stack interoperable as tools change. Check catalog interoperability first, because the catalog is where lock-in happens. Further, the format works the same on-premises and in the cloud. Choose the catalog and metadata store first, because every engine depends on them. Pick one that your security team can operate and back up inside your own environment. If you relied on our packaged Hive Metastore image, note that it reached end of support in 480-e and that 482-e removes it, along with the starburst-hive Helm chart. Bring-your-own Hive Metastore (3.13 or 4.x) remains supported, and we recommend migrating to Starburst Metastore. Plan and test the catalog you will run on-premises.
Run Trino as the query engine
Trino is a distributed SQL engine that turns the lake into a governed, queryable platform for business intelligence (BI), analytics, and artificial intelligence (AI). With Starburst Enterprise, we deliver Trino in your own environment, including private cloud, hybrid, on-premises, and air-gapped deployments. At lake scale, scan-heavy queries need caching and pushdown. In Starburst Enterprise, we include Warp Speed (on Kubernetes, with NVMe storage on every node), result caching, predicate pushdown, and cost-based optimization. Vendor speedup figures depend on workload, version, and method, so benchmark against open source Trino on your own queries. Size your clusters from measured workloads. Start with a baseline for concurrent users and scan volume, then measure and adjust. Additionally, separate workloads by source application and client tags so a heavy batch job doesn’t slow a dashboard. Use Trino resource groups within a cluster and, for multiple clusters, Control Plane routing. Your platform determines which AI workloads you can run on-premises.
Extend the lakehouse with federated query
Your data also lives outside the lakehouse, so plan for that. It sits in on-premises databases, cloud platforms, and SaaS applications. Federated query reaches those systems without forcing centralization. We offer 50+ native connectors and one ANSI SQL interface across them. You query in place, so you avoid duplicate copies of data. Access policies also follow the query. Start federation-first. Connect your sources, learn which queries run often, and then materialize or cache those hot paths into Iceberg tables. This limits copying to the queries that run most often. Decide early what to keep, connect, or add. When you’re ready, try federated query on your own data.
Build governance and security in from day one
Design access control before you open the lake to users. We support fine-grained role-based access control (RBAC), including column privileges, column masks, and row filters. The built-in access control audit log records those access decisions. Integrate the policy tools you already run. We connect with Apache Ranger as an access control system, so your security team can keep one policy model instead of one per system. Package curated datasets as governed data products. They become the unit of reuse for analytics and for on-premises AI, with clear owners and consistent rules. Further, sovereign AI means bringing compute to the data. Models and agents avoid copying data outside your boundary only if inference also runs inside it, because hosted LLM APIs send context out.
Plan for availability and scale
Some production workloads need more than one cluster, for example to balance load or isolate workloads. The multi-cluster architecture introduced in Starburst Enterprise 480-e LTS adds coordinator high availability (active-standby) within a cluster, load balancing across homogeneous clusters, and workload isolation. All three were in public preview in the 480-e release notes. These features depend on the Control Plane, which is included with the Elite license. Clusters without an Elite license, or running Starburst Enterprise earlier than 480-e, show as UNSUPPORTED. Treat this as a procurement decision. A control plane routes queries to healthy clusters. If a coordinator fails, the control plane fails over to the standby, which reduces downtime. On the AWS CloudFormation deployment, queries that were running when the coordinator failed do not complete, and you restart them on the new coordinator. Routing rules also keep critical workloads away from experimental ones. Because these features were in public preview in the 480-e release notes, do not make them your only availability plan for production until you confirm they are generally available in your release. Design recovery and failure handling that you can test without these features, for example with Trino fault-tolerant execution and client-side retry design. Plan the ongoing operations too:
- Run Iceberg maintenance, including compaction and snapshot expiry, on a schedule.
- Monitor query performance, cluster health, and storage growth.
- Review failure scenarios before users depend on the platform.
Migrate from Hadoop without a big bang
Work in stages and keep consumers running throughout.
- Replace Hadoop or Impala with Trino, and leave the HDFS setup untouched.
- Add S3-compatible storage, and write new data as Iceberg tables.
- Migrate Hive tables to Iceberg selectively, because not every table benefits from the move.
- Retire legacy components once consumers use the new path.
Step 1 limits disruption because the data stays put, though moving from Hive or Impala to Trino typically means porting SQL dialects. Step 3 needs judgment. Rarely queried or soon-to-expire tables may not justify the effort.
Next steps
Start with engine replacement, then move Hive tables to Iceberg.
FAQs
What is an on-premises data lakehouse, and how is it different from an on-premises data lake?
An on-premises data lakehouse pairs object storage with ACID guarantees and governance in data centers you own or lease. With proper tuning and sizing, it delivers warehouse-class performance through open table formats. A legacy Hadoop lake typically ties storage to compute and relies on aging components. The lakehouse separates storage from compute and adds a table format such as Apache Iceberg.
Which storage should replace HDFS in an on-premises lakehouse?
Use an S3-compatible object store running on servers or storage you already own. Open-source Trino is tested for compatibility only with Amazon S3 and MinIO. If you run Starburst Enterprise, check its list of supported S3-compatible stores first. Test any store that is not on it yourself. Keep data in open file formats such as Parquet or ORC. Plan capacity, durability, and network bandwidth between compute and storage early, since uncached reads go over the network.
Why choose Apache Iceberg over Hive tables on-premises?
Iceberg adds ACID transactions, schema evolution, time travel, and reliable updates and deletes on top of files. It is also an open format, so engines that support your tables’ format version, features, and catalog can use them. That said, not every Hive table needs to move, so migrate selectively.
What does Trino do in an on-premises lakehouse?
Trino is the distributed SQL engine that turns the lake into a governed, queryable platform for BI, analytics, and AI. With Starburst Enterprise, we run Trino in your own environment, including on-premises and air-gapped deployments. Separate workloads by source application and client tags so heavy jobs don’t slow dashboards. Use Trino resource groups within a cluster and Control Plane routing across clusters.
How do you keep an on-premises Trino deployment highly available?
Use coordinator high availability within a cluster, and add clusters when you need load balancing or workload isolation. The multi-cluster capability introduced in Starburst Enterprise 480-e LTS provides these. All three features were in public preview in the 480-e release notes. They require the Elite license. A control plane routes queries to healthy clusters and fails over to the standby coordinator if one fails, which reduces downtime. On the AWS CloudFormation deployment, queries that were running when the coordinator failed do not complete. Because these features were in public preview in the 480-e release notes, also plan a tested fallback that does not depend on them, such as Trino fault-tolerant execution and client-side retry design.
How do you migrate from Hadoop to a lakehouse with low risk?
Migrate in stages. First, replace Hive or Impala with Trino and leave HDFS untouched. Then add S3-compatible storage and write new data as Iceberg, migrate Hive tables selectively, and retire legacy components once consumers use the new path.
