Build a File Ingest Pipeline with Starburst Galaxy
Batch data ingestion should be simple, not slow. In this step-by-step tutorial, you’ll learn how to build a file ingest pipeline that continuously converts raw NDJSON files from Amazon S3 into Iceberg tables inside Starburst Galaxy, creating a foundation for analytics-ready data.
To do this, you will copy sample NDJSON files into your S3 bucket, connect the source to Galaxy, and create live tables that automatically detect and ingest new files. You will then query, explore, and validate the data using SQL, all within a single platform built for performance and simplicity.
By the end, you’ll know how to build and manage a file ingest pipeline that continuously loads NDJSON data from S3 into Iceberg tables in Starburst Galaxy, giving you a simple, reliable way to keep batch data analytics-ready.
What you’ll learn
- Configure S3 file ingest in Starburst Galaxy.
- Continuously load and transform NDJSON files into Iceberg tables.
- Query structured and nested data with SQL.
Ready to get hands-on?
Register now to start the tutorial
Batch data ingestion should be simple, not slow. In this step-by-step tutorial, you’ll learn how to build a file ingest pipeline that continuously converts raw NDJSON files from Amazon S3 into Iceberg tables inside Starburst Galaxy, creating a foundation for analytics-ready data.
To do this, you will copy sample NDJSON files into your S3 bucket, connect the source to Galaxy, and create live tables that automatically detect and ingest new files. You will then query, explore, and validate the data using SQL, all within a single platform built for performance and simplicity.
By the end, you’ll know how to build and manage a file ingest pipeline that continuously loads NDJSON data from S3 into Iceberg tables in Starburst Galaxy, giving you a simple, reliable way to keep batch data analytics-ready.
What you’ll learn
- Configure S3 file ingest in Starburst Galaxy.
- Continuously load and transform NDJSON files into Iceberg tables.
- Query structured and nested data with SQL.
Ready to get hands-on?
Register now to start the tutorial
