Starburst-Delta lake and Unity catalog connector vs Tableflow

Starburst Enterprise
1

Hi,

Does Starburst have similar capabilities in working with Databricks and Kafka?

Can a implementation with a Kafka cluster use Starburst for similar use cases as Tableflow by Confluent?

Thanks!

2

Starburst Galaxy has a feature called Kafka streaming ingest as documented at Starburst | Kafka streaming ingestion. A recent webinar demonstrated this as well as you can see a replay at https://www.youtube.com/watch?v=N09DmkAeP94.

3

Thanks! Can this be used for Delta lake tables as well?

4

No, it can only created Iceberg tables. If you wanted to, you could run a little follow-along script that uses the CDC function, Iceberg connector — Trino 471 Documentation, to grab changes since the last Iceberg snapshot ID you caught-up to to get the next batch of changes and then apply them to a Delta Lake table.

Fundamentally, I’d guess it isn’t a GIANT engineering effort to allow the ingestion framework to use a switch to determine which table format to use, but that would. be something that you’d have to request from Starburst since the ingestion framework is not open-sourced.

2 Likes
5

here is the ingestion framework to refer Understanding Data Ingestion Architecture

Why Managed Iceberg Pipelines are the Key to your AI Understanding Data Ingestion Architecture | Starburst

1 Like
6

Thanks for calling that article out! The author appreciates it :wink:

To answer the question, it’s exactly as Lester says. Starburst can be used for streaming ingestion of Kafka into Iceberg. Additionally, you can use file ingestion from S3. The two work together and are part of the Managed Iceberg Pipelines, which allows for certain data maintenance tasks to be done automatically, like snapshot expiration, etc. as well as a more customized approach using the “Jobs” feature.

In terms of Delta, yes, it’s Iceberg first. It’s worth noting that Starburst does support connecting to Delta sources (and many people use this). For those deep in the Databricks ecosystem, we also support interoperable compute in Unity. Let me know if anyone needs more info on any of that. :slight_smile:

1 Like