Starburst vs. Athena

Some additional exploration

What is Amazon Athena Serverless used for?

Amazon Athena is AWS’s analytics engine that allows you to execute Athena queries terabytes and petabytes of data in and around S3. You can use Athena to execute data warehouse-like SQL queries on data in your lake, access data from federated sources, prepare data for machine learning models, build distributed data reconciliation engines, and perform multi-cloud data analysis while only being able to run on Amazon Web Services.

Is Amazon Athena an ETL tool?

Amazon Athena is not an ETL tool in the traditional sense, but it can be used to simplify ETL data pipelines using its federated SQL queries and user-defined functions. However, it is not uncommon for long-running queries like ETL jobs to fail without warning.

Are Amazon Athena and Amazon S3 the same?

No, they are not the same. Amazon Simple Storage Service (S3) is a cloud storage service that allows you to store and retrieve data stored within it (cloud data lake). Amazon Athena, on the other hand, is what you would use to run queries against S3 data using standard SQL that supports ANSI SQL.

Is Amazon Athena a SQL Server?

No, Amazon Athena is not a SQL server. It is an interactive query service that makes it easy to analyze data in Amazon S3 using standard SQL. Athena is serverless, so there is no infrastructure to manage, and you pay only for the queries that you run. Athena is built on top of Presto, a distributed SQL query engine, and can process large amounts of data in parallel. It supports a wide range of data formats, including CSV, JSON, ORC, Avro, and Parquet.

Can I use Amazon Athena as a data warehouse?

Athena allows you to create a ‘data warehouse’ like experience on Amazon S3. By defining schemas and running queries, you can efficiently organize and get your data. Furthermore, using APIs from AWS, visualization of the query results in business intelligence tools becomes possible.

What other AWS services do I need to use in conjunction with Amazon Athena?

To effectively use and manage Amazon Athena, instead of having the native built-in functionality you would expect of an analytics platform, Amazon Athena requires you to use several other AWS services. Here are some of the services that are required to make Athena work effectively.

Amazon S3 – Amazon S3 (Simple Storage Service) is an object storage service that offers scalability, data availability, security, and performance. It allows you to store and retrieve any amount of data from anywhere on the web. You can accomplish these tasks using the simple and intuitive web interface of the AWS Management Console. Data is stored in S3 buckets, which are containers for objects that you store in Amazon S3. S3 is the primary data source for Amazon Athena.

AWS Glue – this is the primary data catalog for Amazon Athena. The Glue data catalog is a fully managed, serverless data integration service that makes it easy to prepare and load data for analytics. Starburst Galaxy provides options in which data catalogs you use, including Starburst Gravity and AWS Glue.

Amazon Lake Formation – is a fully managed service that makes it easy to build, secure, and manage data lakes. Unlike Starburst Galaxy, which has these security and governance capabilities built in, Amazon Athena requires you to use Amazon Lake Formation to define and enforce database, table, and column-level access policies when using Athena queries to read data stored in Amazon S3

Amazon Redshift – this is AWS’s data warehouse. Similar to Starburst Galaxy, Amazon Athena also allows you to access data within Amazon Redshift.

AWS Lambda – You can use Lambda to execute SQL queries on Amazon Athena. You can create a Lambda function that uses the AWS SDK for Python (Boto3) to execute SQL queries on Amazon Athena. The Lambda function can be triggered by an event, such as an API Gateway invocation, to execute the query and return the query results.

Amazon DynamoDB – this is a fully managed NoSQL database service. The Amazon Athena DynamoDB connector (also available in the Starburst self-managed software offering) enables Athena to communicate with DynamoDB so that you can execute SQL queries on your tables.

AWS IAM – this is the identity and assessment service from AWS. Unlike the built-in capabilities with Starburst Galaxy, Amazon Athena uses AWS IAM policies as the primary means to restrict access to Athena operations. Users can create policies that grant or deny access to specific resources and configure permissions based on user roles or groups.

AWS Command Line Interface (CLI) – You can use the AWS CLI to interact with Amazon Athena. For example, you can use the aws athena start-query-execution command to run a query. You will then need to poll with aws athena get-query-execution until the query is finished. When that is the case, the result of that call will also contain the location of the query result on S3, which you can then download with aws s3 cp.

You cannot save results from the AWS CLI directly, but you can specify a Query Result Location, and Amazon Athena will automatically save a copy of the query results in an Amazon S3 location that you specify. You could then use the AWS CLI to download that results file.

Amazon Quicksight – this is AWS’s business intelligence (BI) service. Similar to the Starburst Galaxy and Quicksight experience, Amazon QuickSight retrieves data from Athena to enable visualization of the query results from Amazon Athena SQL queries.

Amazon EMR, formerly known as Amazon Elastic MapReduce – is a managed cluster platform that simplifies running big data frameworks, such as Apache Hadoop and Apache Spark, on AWS to process and analyze vast amounts of data. You would use the two together when building an Apache Iceberg data lake. You can use Amazon EMR Spark to create an Iceberg table and load sample data, then use Athena to query the table, perform schema evolution, and more with the AWS Glue Data Catalog.

Can I create data products using Amazon Athena?

Unlike Starburst, Amazon Athena does not offer capabilities to build, manage, secure, and share curated data sets in the form of federated data products.

What is the difference between Serverless and Dedicated in Amazon Athena?

Amazon Athena is serverless, so there is no infrastructure to manage, and its pricing structure is you pay only for the queries you run.

Amazon Athena also recently introduced the ability to provision dedicated capacity for your Athena queries. With provisioned capacity, you can reserve a dedicated set of compute resources to run your queries. This puts the management responsibilities on customers creating high risks of wasted resources and rising costs with poor management.