PySpark and AWS: Master Big Data with PySpark and AWS

Table of Contents

Chapter 1 : Introduction

Why Big Data

Applications of PySpark

Introduction to Instructor

Introduction to Course

Projects Overview

Chapter 2 : Introduction to Hadoop, Spark Ecosystems and Architectures

Why Spark

Hadoop Ecosystem

Spark Architecture and Ecosystem

Databricks Sign Up

Create Databricks Notebook

Download Spark and Dependencies

Java Setup

Python Setup

Spark Setup

Hadoop Setup

Running Spark

Chapter 3 : Spark RDDs

Spark RDDs

Creating Spark RDD

Running Spark Code Locally

RDD Map (Lambda)

RDD Map (Simple Function)

Quiz (Map)

Solution 1 (Map)

Solution 2 (Map)

RDD FlatMap

RDD Filter

Quiz (Filter)

Solution (Filter)

RDD Distinct

RDD GroupByKey

RDD ReduceByKey

Quiz (Word Count)

Solution (Word Count)

RDD (Count and CountByValue)

RDD (saveAsTextFile)

RDD (Partition)

Finding Average-1

Finding Average-2

Quiz (Average)

Solution (Average)

Finding Min and Max

Quiz (Min and Max)

Solution (Min and Max)

Project Overview

Total Students

Total Marks by Male and Female Student

Total Passed and Failed Students

Total Enrollments per Course

Total Marks per Course

Average Marks per Course

Finding Minimum and Maximum Marks

Average Age of Male and Female Students

Chapter 4 : Spark DFs

Introduction to Spark DFs

Creating Spark DFs

Spark Infer Schema

Spark Provide Schema

Create DF from RDD

Rectifying the Error

Select DF Columns

Spark DF with Column

Spark DF with Column Renamed and Alias

Spark DF Filter Rows

Quiz (Select, Withcolumn, Filter)

Solution (Select, Withcolumn, Filter)

Spark DF (Count, Distinct, Duplicate)

Quiz (Distinct, Duplicate)

Solution (Distinct, Duplicate)

Spark DF (Sort, OrderBy)

Quiz (Sort, OrderBy)

Solution (Sort, OrderBy)

Spark DF (Group By)

Spark DF (Group By - Multiple Columns and Aggregations)

Spark DF (Group By -Visualization)

Spark DF (Group By - Filtering)

Quiz (Group By)

Solution (Group By)

Quiz (Word Count)

Solution (Word Count)

Spark DF (UDFs)

Quiz (UDFs)

Solution (UDFs)

Solution (Cache and Persist)

Spark DF (DF to RDD)

Spark DF (Spark SQL)

Spark DF (Write DF)

Project Overview

Project (Count and Select)

Project (Group By)

Project (Group By, Aggregations and Order By)

Project (Filtering)

Project (UDF and WithColumn)

Project (Write)

Chapter 5 : Collaborative Filtering

Collaborative Filtering

Utility Matrix

Explicit and Implicit Ratings

Expected Results

Dataset

Joining Dataframes

Train and Test Data

ALS Model

Hyperparameter Tuning and Cross Validation

Best Model and Evaluate Predictions

Recommendations

Chapter 6 : Spark Streaming

Introduction to Spark Streaming

Spark Streaming with RDD

Spark Streaming Context

Spark Streaming Reading Data

Spark Streaming Cluster Restart

Spark Streaming RDD Transformations

Spark Streaming DF

Spark Streaming Display

Spark Streaming DF Aggregations

Chapter 7 : ETL Pipeline

Introduction to ETL

ETL Pipeline Flow

Data Set

Extracting Data

Transforming Data

Loading Data (Creating RDS-I)

Load data (Creating RDS-II)

RDS Networking

Downloading Postgres

Installing Postgres

Connect to RDS Through PGAdmin

Loading Data

Chapter 8 : Project - Change Data Capture / Replication Ongoing

Introduction to Project

Project Architecture

Creating RDS MySQL Instance

Creating S3 Bucket

Creating DMS Source Endpoint

Creating DMS Destination Endpoint

Creating DMS Instance

MySQL WorkBench

Connecting with RDS and Dumping Data

Querying RDS

DMS Full Load

DMS Replication Ongoing

Stopping Instances

Glue Job (Full Load)

Glue Job (Change Capture)

Glue Job (CDC)

Creating Lambda Function and Adding Trigger

Checking Trigger

Getting S3 File Name in Lambda

Creating Glue Job

Adding Invoke for Glue Job

Testing Invoke

Writing Glue Shell Job

Full Load Pipeline

Change Data Capture Pipeline