Last Updated: 2024-03-28

Background

Many of the Starburst tutorials require access to a writable data source to successfully complete all of the exercises. This tutorial will walk you through the steps to set up an environment that can be used for those tutorials.

To build the environment, you will create the following:

Scope of tutorial

In this tutorial, you will learn how to configure a catalog in Starburst Galaxy that connects to Amazon S3 object storage.

An S3 bucket with read/write permissions has been set up for you to use. It contains one table made up of csv files with sample payment transaction data.

Learning objectives

Once you've completed this tutorial, you will be able to:

Prerequisites

About Starburst tutorials

Starburst tutorials are designed to get you up and running quickly by providing bite-sized, hands-on educational resources. Each tutorial explores a single feature or topic through a series of guided, step-by-step instructions.

As you navigate through the tutorial you should follow along using your own Starburst Galaxy account. This will help consolidate the learning process by mixing theory and practice.

Background

You're going to begin by signing in to Starburst Galaxy and setting your role to begin the process of connecting the Amazon S3 data source.

This is a quick step, but an important one.

Step 1: Sign into Starburst Galaxy

Sign into Starburst Galaxy in the usual way. If you have not already set up an account, you can do that here.

Step 2: Set your role

Your current role is listed in the top right-hand corner of the screen.

Background

Adding a new Amazon S3 catalog follows the same process as adding other data sources in Starburst Galaxy. This is one of the main ways that Starburst Galaxy is used to connect to data lakes.

The steps below will show you how to start the process of configuring a new catalog.

Step 1: Create a new catalog

Create a new catalog for the Amazon S3 data source.

Step 2: Select Amazon S3 data source

Starburst Galaxy allows the creation of catalogs for a number of different data sources. In this case, you are going to create a new catalog in the Amazon S3 category.

Step 3: Input name and description

The catalog needs both a name and description. This ensures that you can find it later.

Example: Delete this catalog after use (data removed daily, but metastore still thinks it exists)

Background

When you connect Starburst Galaxy to a new data source, it is necessary to undergo an authentication process. This helps ensure that you are connecting the right data source and that you have the appropriate permissions.

Step 1: Choosing an authentication method

Starburst Galaxy allows you to configure several different authentication methods when creating a new catalog. This lets you connect to data sources of different types. In this particular example, we will use the AWS access key option.

Background

Starburst Galaxy uses a metastore to keep track of the location of your data when it is added to the data lake, in this case to Amazon S3.

You have three options when choosing a metastore. For the purposes of this tutorial, we will be using the Starburst Galaxy metastore.

Step 1: Select the metastore

Starburst Galaxy includes its own metastore, which can be used to easily store metadata. Using this option is often the simplest metadata management solution.

The choice of metastore is completely decoupled from the choice of storage option, allowing you to mix and match.

This will allow you to create external tables outside of the default S3 bucket.

This will allow you to write data into external tables outside of the default S3 bucket.

Background

Table formats control the way that data is stored. These include popular modern, open table formats like Iceberg or Delta Lake, or older table formats like Hive.

Step 1: Select the default table format

We will use Hive as the default table format.

Background

Every new catalog connection includes a test before you connect it. This helps to ensure that you have input the correct credentials and allows you to quickly fix any problems before actually connecting.

Step 1: Test and Connect

You're almost there! Time to test the connection and then complete the process of creating your new Amazon S3 catalog.

Background

Starburst Galaxy allows you to configure your catalog in a number of ways regarding access controls. The most important of these involves granting write access or restricting the catalog to read-only access.

Step 1: Save access controls

We need write access for our catalog, so we will leave the access controls as they are.

Step 2: Add catalog to cluster

You are going to add the new catalog to the cluster you created in the Starburst Galaxy: Getting started tutorial.

Background

With Starburst Galaxy, it's easy to create a new schema directly from the query editor. The previous section in this tutorial brought you to the Starburst Galaxy query editor, where the aws-us-east-1 cluster and tmp_cat catalog should be selected for use.

Step 1: Create schema

When you create a schema, it will create a folder with the schema name inside the Amazon S3 bucket. It is important that the name of that folder is unique, which is why this step has a required naming convention for the schema.

CREATE SCHEMA tmp_first_last_postalcode;

Tutorial complete

Congratulations! You have reached the end of this tutorial, and the end of this stage of your journey.

You're all set! Now you can use your new catalog and schema in our other tutorials.

Continuous learning

At Starburst, we believe in continuous learning. This tutorial provides the foundation for further training available on this platform, and you can return to it as many times as you like. Future tutorials will make use of the concepts used here.

Next steps

Starburst has lots of other tutorials to help you get up and running quickly. Each one breaks down an individual problem and guides you to a solution using a step-by-step approach to learning.

Tutorials available

Visit the Tutorials section to view the full list of tutorials and keep moving forward on your journey!

Start Free with
Starburst Galaxy

Up to $500 in usage credits included

  • Query your data lake fast with Starburst's best-in-class MPP SQL query engine
  • Get up and running in less than 5 minutes
  • Easily deploy clusters in AWS, Azure and Google Cloud
For more deployment options:
Download Starburst Enterprise

Please fill in all required fields and ensure you are using a valid email address.