AWS Glue 101: What Is It And When Should You Use It?

By the time AWS Glue was being introduced in 2017, big data had already been widely recognized as a critical resource to any organization that intends to outperform its competitors. 

If you check out NewVantage Partners’ Big Data Executive Survey of 2017, you’ll notice that enterprises have been leveraging big data to drive success in a number of ways. Over 40% of the surveyed companies were capitalizing on it to set up new avenues for disruption and innovation, while nearly 50% were effectively using it to minimize their operating expenses. 

Well, the trend never slowed down. Rather, it’s increasingly expanding, as more and more enterprises continue to adopt managed data integration services like AWS Glue. The uptick is so widespread that 65% of organizations now prefer to ride on data integration solutions from cloud platforms or hybrid cloud. 

And how exactly do they stand to benefit? 

This article explains the craze by covering AWS Glue in detail, which happens to be one of the most popular cloud data integration services today. 

We’ll cover what exactly AWS Glue is, how it runs, when you might want to use the Amazon service, as well as the benefits and limitations of using AWS Glue. Additionally, since there is some confusion surrounding AWS Glue and AWS EMR, we’ll also compare the two and explain the principal differences you should expect between them.

Table Of Contents

• What Is AWS Glue?
• When Should You Use AWS Glue?
• What Are The Benefits And Limitations?
• AWS Glue Vs. EMR
• Measuring And Monitoring AWS Glue Costs

What Is AWS Glue?

To adequately define what AWS Glue is, you’ll first need to understand how data integration works. 

In essence, you can think of data integration as the process of setting up and putting together data for analytics, application development, and machine learning. It’s made up of multiple procedures - such as identifying and generating data from different sources, which is then followed by enriching, cleaning, normalizing, and merging data. In the end, the data is loaded and organized in data warehouses, databases, and data lakes. 

AWS Glue fits into the mix by providing a serverless solution that simplifies the entire operation of discovering, preparing, and combining data for application development, machine learning, and analytics. It facilitates all the data integration procedures so you can quickly put your merged data to good use. That means you get to analyze and leverage the data in minutes, instead of waiting around forever. 

Because of these capabilities, AWS Glue is technically described as a fully-managed ETL (Extract, Transform, and Load) data integration solution. Amazon even goes on to explain that the whole system is designed to provide an easy and cheap way to not only categorize your data, but also clean, enrich, and transfer it efficiently between different data streams and data stores. 

It doesn’t run across one front, though. Rather, AWS Glue comes as a multi-faceted system that powers data integration through three core components. 

The three components of AWS Glue

Under the hood, you’ll find these three AWS Glue components: 

• AWS Glue Data Catalog - This is basically a central repository for your metadata, built to hold information in metadata tables — with each table pointing to a single data store. In other words, it acts as an index to your data schema, location, and runtime metrics, which are then used to identify the targets and sources of your ETL (Extract, Transform, Load) jobs. 

• Job Scheduling System - The job scheduling system, on the other hand, is intended to help you automate and chain your ETL pipelines. It comes in the form of a flexible scheduler that’s capable of setting up event-based triggers and job execution schedules. 

• ETL Engine - AWS Glue’s ETL engine is the one component that handles ETL code generation. It automatically provides this in Python or Scala, and then proceeds to even give you the option of customizing the code. 

When Would You Want To Use AWS Glue? 

While AWS Glue continues to serve different types of users, it’s particularly popular among organizations that are trying to put up an enterprise-class data warehouse. 

They benefit from the fact that AWS Glue seamlessly facilitates the movement of data from various sources into their data warehouse. 

The process itself is quite simple and straightforward — you use AWS Glue to validate, cleanse, organize, and format data, which is ultimately stored in a centrally accessible data warehouse. You’ll further notice that the platform allows you to load such data from both data streaming and static sources. 

Now, the point of this whole approach is to bring in critical data from various parts of your business, and then consolidate it all into a central data warehouse. As such, you should be able to conveniently access and compute all your business information from a common source, as well as use the system to carry out tasks like: 

• Automatically scaling resources to cover the current needs of your situation.
• Error handling and retrying so as to avoid stalling issues.
• Gathering KPIs (Key Performance Indicators), metrics, and logs of your ETL procedures for the sake of monitoring and reporting.
• Executing ETL jobs based on specific events, schedules, or triggers.
• Automatically recognizing database schema changes, and subsequently tweaking the service to respond accordingly.
• Generating ETL scripts with the aim of enriching, denormalizing, and transforming the data while it’s transferring from its source to the target.
• Identifying your data stores’ and databases’ metadata, and then proceeding to archive them in AWS Glue Data Catalog.

What Are The Benefits And Limitations Of Using AWS Glue?

As with everything else in the world of big data computing, AWS Glue has its strengths and weaknesses. Although Amazon has, admittedly, done a fairly good job on it, there are still a couple of things about it that you might find a bit limiting. 

Here’s a breakdown of both sides:

Pros of AWS Glue

• Serverless - As a serverless data integration service, AWS Glue saves you the trouble of building and maintaining infrastructure. It is Amazon that provides and manages the servers. 

• Automatic ETL code - AWS Glue is capable of automatically generating ETL pipeline code in Scala or Python — based on your data sources and destination. This not only streamlines the data integration operations but also gives you the privilege of parallelizing heavy workloads. 

• Increased data visibility - By acting as the metadata repository for information on your data sources and stores, the AWS Glue Data Catalog helps you keep tabs on all your data assets. 

• Developer endpoints - For users who prefer to manually create and test their own custom ETL scripts, AWS Glue facilitates the whole development process through what it calls “developer endpoints.”

• Job scheduling - AWS Glue provides easy-to-use tools for creating and following up job tasks based on schedule and event triggers, or perhaps on-demand. 

• Pay-as-you-go - The service doesn’t force you to commit to long-term subscription plans. Instead, you can minimize your usage costs by paying only when you need to use it. 

Cons of AWS Glue

• Requires technical knowledge - Some aspects of AWS Glue are not very friendly to non-technical beginners. For instance, since all the tasks run in Apache Spark, you need to be well-versed in Spark to tweak the generated ETL jobs. What’s more, the ETL code itself can only be worked on by developers who understand Python or Scala. 

• Only two languages - When it comes to customizing ETL codes, AWS Glue only supports two programming languages, Python and Scala. 

• Limited integrations - AWS Glue is only built to work with other AWS services. That means you won’t be able to integrate it with platforms outside the Amazon ecosystem. 

AWS Glue Vs. EMR: How Does It Compare?

As it turns out, AWS Glue is not the only managed Amazon service that’s capable of handling big data. Other solutions include AWS Data Exchange, AWS Kinesis, AWS EMR, AWS Redshift, and Amazon Athena. 

While they have varying functionalities, some of them can, understandably, seem a bit confusing when you try to compare their capabilities. You might, for instance, find yourself torn between AWS Glue and AWS EMR — as they share quite a number of similarities. 

It’s worth noting, they also happen to have a couple of stark differences in the way they operate. 

AWS Glue, for example, exists as a serverless ETL system that offers its services on a pay-as-you-go basis. In essence, you only need basic infrastructure, and voila! Even without a server, you can count on AWS Glue to automate the bulk of the tasks in monitoring, executing, and writing ETL jobs. 

The rules, however, change slightly on the side of AWS EMR. Amazon Elastic MapReduce, as it’s known in full, reduces the costs of analyzing and processing huge volumes of data through a managed big data platform. Instead of restricting your configuration options, it allows you to set up custom EC2 (Amazon Elastic Computing) instance clusters, as well as create Hadoop ecosystem elements. 

There’s just one caveat. Apparently, AWS EMR requires you to have your own extensive infrastructure if you intend to leverage it for big data operations. This, of course, makes getting started a costly affair. 

On a brighter note, though, once you set up the infrastructure, you’ll have an easy time deploying AWS EMR — plus capitalizing on its power and flexibility. Data analysts can, for example, use it to perform SQL queries on Presto, while data scientists might have a field day running machine learning tasks. 

As for AWS Glue, it turns out you don’t get as much power and flexibility — which translates into a rather interesting relationship between the two. While organizations can proceed to replace AWS Glue with AWS EMR, the alternative is not possible. 

Also, you should note that since AWS Glue is serverless, it tends to be a bit costlier than AWS EMR. If you compare similar cluster configurations across the two, you’ll notice that the former is more expensive.

But, not by a huge margin. While AWS Glue would charge you around $21 per DPU (Data Processing Unit) for an entire day, Amazon EMR will bill you about $14 -16 for a similar configuration. 

How Can You Measure And Monitor Your AWS Glue Costs?

While Amazon Glue’s pay-as-you-go rate of $0.44 per DPU might seem reasonable at first, organizations commonly find themselves with bloated bills after prolonged use — which often run into thousands of dollars per month in extra or unnecessary costs. 

Such cost overruns are mostly due to poor AWS cost management practices. Something as simple as keeping tabs on your AWS Glue spend can be a challenge — since Amazon doesn’t readily provide comprehensive insights (like what you’re spending and why or how specific services drive your product and feature costs). 

With CloudZero, however, you gain complete insight into your AWS cloud spend. CloudZero’s cloud cost intelligence maps costs to your products, features, dev teams, and more. The platform also automatically detects cost issues and alerts you before they run for days or weeks.

With cloud cost intelligence, you’ll be able to drill into cost data from a high level down to the individual components that drive your cloud spend — and see exactly how services drive your cloud costs and why. To see CloudZero in action, .