<img height="1" width="1" style="display:none;" alt="LinkedIn" src="https://px.ads.linkedin.com/collect/?pid=1310905&amp;fmt=gif">

AWS Redshift 101: What Is It and When Should You Use It?

|April 27, 2021|

Having trouble connecting engineering decisions to business results? CloudZero  makes it easy for engineering teams to understand their AWS cloud spend (such  as unit cost) to make informed decisions.Click here to learn more.  <https://www.cloudzero.com/>

AWS Redshift is a popular data warehousing solution that can handle data on an exabytes scale. You may be considering the service for a number of use cases such as processing real-time analytics, combining multiple data sources, log analysis, or more.

Made possible by its Massively Parallel Processing (MPP) technology, Redshift is able to execute operations on a humongous volume of data at lightning speed  — while typically costing only a fraction of what competitors like Oracle and Teradata charge for comparable products. 

To cover the AWS data warehouse in more detail and help you decide if Redshift is the right solution for you, this article will answer:

Table of Contents

What Is AWS Redshift?

Amazon Redshift

AWS Redshift is a data warehousing solution from Amazon Web Services. Redshift shines in its ability to handle huge volumes of data — capable of processing structured and unstructured data in the range of exabytes (1018 bytes). However, the service can also be used for large-scale data migrations. 

Similar to many other AWS services, it can be deployed with just a few clicks and provides a plethora of options to import data. Additionally, the data in Redshift is always encrypted for added security.

Redshift helps to gather valuable insights from a large amount of data. With the easy-to-use interface of AWS, you can start a new cluster in a couple of minutes, and you don’t have to worry about managing infrastructure.

What Is Unique About Redshift?

Redshift is an OLAP-style (Online Analytical Processing) column-oriented database. It is based on PostgreSQL version 8.0.2. This means regular SQL queries can be used with Redshift. But this is not what separates it from other services. The fast delivery to queries made on a large database with exabytes of data is what helps Redshift stand out.

Fast querying is made possible by Massively Parallel Processing design or MPP. The technology was developed by ParAccel. With MPP, a large number of computer processors work in parallel to deliver the required computations. Sometimes processors situated across multiple servers can be used to deliver a process. 

Unlike most MPP vendors, ParAccel does not sell MPP devices. Their software can be used on any hardware to harness the power of multiple processors. AWS Redshift uses the MPP technology of ParAccel. In fact, Redshift was started following capital investment by AWS in ParAccel and using MPP technology from ParAccel. Now the company is part of Actian.

When Would You Want To Use Amazon Redshift?

Amazon Redshift is used when the data to be analyzed is humongous. The data has to be at least of a petabyte-scale (1015 bytes) for Redshift to be a viable solution. The MPP technology used by Redshift can be leveraged only at that scale. Beyond the size of data, there are some specific use cases that warrant its use. 

Real-time analytics

Many companies need to make decisions based on real-time data and often need to implement solutions quickly too. Take Uber for example

Based on historical and current data, Uber has to make decisions quickly. It has to decide surge pricing, where to send drivers, what route to take, expected traffic, and a whole host of data. 

Thousands of such decisions have to be made every minute for a company like Uber with operations across the globe. The current stream of data and historical data has to be processed in order to make those decisions and ensure smooth operations. Such instances can use Redshift as the MPP technology to make accessing and processing data faster.

Combining multiple data sources

There are occasions where structured data, semi-structured data, and/or unstructured data have to be processed to gain insights. Traditional business intelligence tools lack the capability to handle the varied structures of data from different sources. Amazon Redshift is a potent tool in such use cases.

Business intelligence

The data of an organization needs to be handled by a lot of different people. All of them are not necessarily data scientists and will not be familiar with the programming tools used by engineers. 

They can rely on detailed reports and information dashboards that have an easy-to-use interface. Highly functional dashboards and automatic report creation can be built using Redshift. It can be used with tools like Amazon Quicksight and also third-party tools created by AWS partners.

Log analysis

Behavior analytics is a powerful source for useful insights. Behavior analytics provide information on how a user uses an application, how they interact with it, the duration of use, their clicks, sensor data, and a plethora of other data. 

The data can be collected from multiple sources — including a web application used on a desktop, mobile phone, or tablet — and can be aggregated and analyzed to gain insight into user behavior. This coalescing of complex datasets and computing data can be done using Redshift.

Redshift can also be used for traditional data warehousing. But solutions like the S3 data lake would likely be better suited for that. Redshift can be used to perform operations on data in S3, and save the output in S3 or Redshift.

The Benefits of Using AWS Redshift

The very distinctive advantage of using AWS Redshift is the cost-benefit to your organization. It costs only a fraction (roughly one-twentieth) of the cost of competitors like Teradata and Oracle. 

In addition to cost, there are a number of benefits to using Redshift. 

  • Speed. With the use of MPP technology, the speed of delivering output on large data sets is unparalleled. No other cloud service providers can match the speed at the cost AWS provides the service. 
  • Data Encryption. Amazon provides the facility for data encryption for any part of Redshift operation. You as the user can decide which operations need encryption and those that do not need encryption. Data encryption provides an added layer of security.
  • Use familiar tools. Redshift is based on PostgreSQL. All the SQL queries work with it. Additionally, you are free to choose any SQL, ETL (Extract, Transform, Load), and Business Intelligence (BI) tools you are familiar with. There is no requirement to use the tools provided by Amazon.
  • Intelligent Optimization. For a large data set, there would be a number of ways to query data with the same parameters. The different commands will have different levels of data utilization. AWS Redshift provides tools and information to improve queries. It will also provide tips to improve the database automatically. These can be utilized for an even faster operation that is less intensive on resources.
  • Automate repetitive tasks. Redshift has the provisions by which you can automate tasks that have to be done repeatedly. This could be administrative tasks like generating, daily, weekly, or monthly reports. It could be resource and cost auditing. It can also be regular maintenance tasks to clean up data. You can automate all these with the provisions offered by Redshift.
  • Concurrent Scaling. AWS Redshift will scale up automatically to support increasing concurrent workloads.

Workload Management Configuration

  • Query Volume. The MPP technology shines in this aspect. You can send thousands of queries to the dataset at any given time. Still, Redshift will not slow down in any shape or form. It will dynamically allocate processing and memory resources to handle higher demand.
  • AWS Integration. Redshift works well with the rest of the tools from AWS. You can set up the integrations between all the services according to your needs and optimal setup.

Amazon Redshift Diagram

  • Redshift API. Redshift has a robust API with extensive documentation. It can be used to send queries and bain results using API tools. The API can also be used within a Python program for easier coding.
  • Security. The security of the cloud is handled by Amazon and the security of the applications within the cloud has to be provided by users. Amazon provides provision for access control, data encryption, and virtual private cloud to provide an added level of security.
  • Machine Learning. Redshift uses machine learning to predict and analyze queries. This, in addition to MPP, makes the performance of Redshift faster than other solutions in the market.
  • Easy Deployment. A Redshift cluster can be deployed in any part of the world from anywhere in a matter of just minutes. You can have a high-performing data warehousing solution at the fraction of the price set by competitors in mere minutes.
  • Consistent Backup. Amazon automatically backs up data regularly. This can be used to restore in the event of any faults, failures, or corruption. The backups are spread across different locations. So this eliminates the risk of faults at a location as a whole.
  • AWS Analytics. AWS offers plenty of analytical tools. All of these can work well with Redshift. Amazon provides support to integrate other analytical tools with Redshift. Redshift has native integration capabilities with AWS analytics services.
  • Open Formats. Redshift supports and can provide outputs in many open formats for data. The most common formats supported are Apache Parquet and Optimized Row Columnar (ORC) file formats.
  • Partner Ecosystem. AWS is one of the oldest cloud service providers. A lot of customers depend on Amazon for their infrastructure. In addition to that AWS has a strong network of partners that builds third-party applications and offers implementation services. This partner ecosystem can also be tapped to see if you can find an implementation solution that is perfect for your organizations.

The data collected will grow every day. Redshift is a hedge against the growing data with increasing analytical complexity. It can be used to build infrastructure that lasts into the future. 

Additionally, Redshift offers best in class performance at a fraction of the cost of competitors. This makes it a value proposition for any organization that has to handle large volumes of data.

What Are the Limitations of AWS Redshift?

Redshift has some drawbacks that need to be considered before choosing it as your data warehousing solution.

  • Parallel Uploads. Redshift does not support all databases for parallel upload. Amazon S3, EMR, and DynamoDB are supported by Redshift for parallel uploads using ultra-fast MPP. For other sources, separate scripts have to be used to upload data. This can be a very slow process.
  • Uniqueness. One of the basic tenets of a database is to have unique data and avoid redundancies. AWS Redshift does not provide any tool or means to ensure the uniqueness of data. If you are migrating overlapping data from different sources to Redshift, there will be redundant data points. 
  • Indexing. This becomes a problem when Redshift is used for data warehousing needs. Redshift uses distribution and sorts keys to index and store data. You will need to know the concepts behind the keys to work on the database. AWS does not provide any system to change the keys or manage them with minimal knowledge.
  • OLAP Limitations. OLAP databases (which Redshift is) are optimized for analytical queries on a large volume of data. Compared to traditional OLTP (Online Transaction Processing) databases, OLAP lacks in performing basic database tasks. Insert/update/delete operations have performance limitations in OLAP databases. It is often easier to recreate a table with changes than to insert/update tables in Redshift. While OLAP works well with static data, OLTP databases perform better for data modification operations.
  • Migration cost. Redshift is used in cases where the data to be stored or worked with is humongous. It will at least be in the range of petabytes. At this level, bandwidth becomes a problem. You will need to transfer this data to AWS locations before you can begin the project. This could be a potential problem for businesses that have network caps for bandwidth. The additional cost will have to be borne by the user. AWS does provide the option to send the data using physical storage devices.

AWS Redshift Pricing Model

AWS offers a very flexible pricing scheme for Redshift. The price starts at $0.25 per hour for a terabyte of data and it can be scaled from there. First, you need to opt for the node type you want. AWS Redshift offers three types of nodes.

  • RA3 nodes with managed storage. Here, you will need to pick the level of performance you require, and the managed storage will be billed on a pay-as-you-go basis. The number of RA3 clusters you have to choose will depend on the amount of data processed on a daily basis. 
  • DC2 Nodes. These should be chosen when you need high performance. Local SSD (Solid State Drive) storage is included with the nodes. You will need to add more nodes when the size of the data grows. DC2 nodes are best suited when the data is relatively small in size and needs superlative performance.
  • DS2 nodes. It should be opted for when there is a large data set that needs to be stored. DS2 provides only HDD (Hard Disk Drives) and has a slower performance compared to other nodes. But it is also considerably cheaper.

dc2

Prices at US-North California center

Redshift also has a pay-as-you-go pricing model according to the requirements.

  • Amazon Redshift spectrum pricing. When you need to run SQL queries on a large dataset in an S3 data lake, you pay according to the usage. Even if the data in S3 is of exabyte range, you will only need to pay according to the amount of data scanned. The price was $5 per terabyte scanned for the North California location.
  • Concurrency scaling pricing. Concurrency scaling enables you to dynamically allocate resources according to the demand. AWS automatically provides additional resources even if the number of queries and users multiply. You will only need to pay according to the usage. In addition, each cluster has one scaling credit every day. This will be sufficient for 97% of the customers according to past AWS data.
  • Redshift managed storage pricing. This pricing model will divide the computing and storage costs of RA3 nodes. That way you need not add more nodes when data requirement increases. RA3 nodes are costlier for storage purposes compared to using separate managed storage. 
  • Redshift ML. You can use SQL queries to train ML models. You will be able to use the free credits of Amazon Sagemaker before you have to pay for creating ML models.

Detailed pricing details are provided on the AWS Redshift pricing page

Managing and Understanding Your AWS Redshift Costs

AWS Redshift can integrate with Amazon S3, AWS Glue, Amazon Kinesis Data Firehose, Amazon Quicksight, and over 170 other AWS services. Though each of these services has its advantages, using all of them simultaneously can drastically inflate your AWS bill. 

Many of the services you use might be redundant, and there will be better ways to reduce costs.

Additionally, mapping costs from specific services that make up your products and features can be a nearly impossible task when done manually or using a traditional cloud cost management tool.

Whether your goal is to reduce your AWS bill, understand your cloud costs, or map costs to the products or features your business cares about, CloudZero can help you achieve all three and gain true cloud cost visibility. Request a demo to see CloudZero in action and learn more about how it can help you optimize your Redshift and other AWS costs.

See CloudZero in Action

STAY IN THE LOOP


Join thousands of engineers who already receive the best AWS and cloud cost intelligence content.