Understanding Tool Integration for Big Data Architecture

Table of Contents

Introduction to Clickstream Case Study

Requirements

Data Modeling

Data Ingest

Data Processing Engines - Part 1

Data Processing Engines - Part 2

Data Processing Patterns

Orchestration

Putting It All Together

Demo

Q

Introduction

Introduction to Time Series Problems

Kafka

Kafka Architecture and Deployment

Kafka Usage

Spark

Introduction to Spark

Spark Architecture

Spark Streaming

Spark Streaming: Windows Slides

Spark Streaming: Ingestion Sources Using Kafka

Sparks Streaming: Operations on the Stream

Cassandra

Introduction to Cassandra

Cassandra Basic Architecture

Replication, High Availability and Multi Datacenter

Cassandra Weather Website Example

Cassandra Query Language (CQL)

Cassandra Partitions Clustering

Cassandra Read and Write Path

Working with Cassandra

Cassandra Drivers and Access Patterns

Spark and Cassandra

Spark and Cassandra Architecture

Analyzing Cassandra Data Spark SQL

Spark and Cassandra DataStax Enterprise

Real World Use Cases

Real World Use Cases: Streaming Problems

Real World Use Cases: In-place Analytic Problems

Introduction

Course Introduction

About The Author

What Is Big Data

Historical Approaches

Modern-Day Approach

What Is Hadoop

Hadoop Core Vs Ecosystem

Hadoopable Problems

Hadoop Basics

HDFS And Yarn

Hive And Pig Interface Introduction

Introduction To Spark

Hadoop In The Cloud (Amazon Web Services Intro)

Installing Hadoop Into EMR Part - 1

Installing Hadoop Into EMR Part - 2

Installing Cloudera Quickstart VM

Web GUIs

Hadoop Distributed Filesystem (HDFS)

HDFS Architecture

HDFS File Write Walkthrough

Secondary Name Node

Basic HDFS Commands

Using HDFS Commands Part - 1

Using HDFS Commands Part - 2

HA And Federation Basics

HDFS Access Controls (Or Lack Thereof)

Yarn

Yarn Purpose

Yarn Architecture

Yarn With Spark

MapReduce

MapReduce Explained

MapReduce Architecture

MapReduce Code Walkthrough

MapReduce Details Walkthrough

Running MapReduce Job

HDFS Data Import And Export

Import/Export Options

Flume Introduction

Using Flume

Sqoop Introduction

Using Sqoop

HDFS Interaction Tools

Oozie Introduction

Spark Basics

Spark Value Propositions

Spark Run Modes (Yarn, Standalone, Mesos)

RDDs And Dataframes

Hands On Spark Part - 1

Hands On Spark Part - 2

Running Spark Part - 1

Running Spark Part - 2

Optimizing And Debugging Spark

Spark Libraries Overview

Spark Built-In Libraries

Spark SQL

Spark SQL Usage

MLlib Basics

Common MLlib Usage Part - 1

Common MLlib Usage Part - 2

Spark Streaming

GraphX

Hive And Pig

Hive Vs Pig

Hive Basics

Analysis With Hive

Pig Basics

ETL And Analytics With Pig

Hadoop In The Cloud

Hadoop/Cloud Use Cases

Elastic MapReduce (EMR)

Ecosystem

HBase Basics

Enterprise Integration

Wrap Up

Wrap Up

Introduction to Sqoop

Introduction

About The Author

Use Case #1: ELT

Use Case #2: ETL From DWH

Use Case #3: Data Analysis

Use Case #4: Data Archival

Use Case #5: Move Reports To Hadoop

Use Case #6: Data Consolidation

Importing Data To Hadoop From A Relational Database

Command Line Basics: Importing Data Using Sqoop

Importing Data With Column Filters, Row Filters, And Free Text Queries

Parallel Imports

Import Data Directory To HIVE Tables

Incremental Data Import Overview

Incremental Data Import And Using Sqoop Stored Jobs

Sqoop Hands-On: Exporting Data From Hadoop To A Relational Database

Exporting Data Back To A Relational Database Using Sqoop

Exporting data from Hadoop back to RDBMS

Advanced topics

Introduction to Sqoop2 Server

Course summary

Wrap Up

Continuous curation of event data for a customer event hub - Arvind Prabhakar (StreamSets)

Big data governance - Steven Totman (Cloudera), Mark Donsky (Cloudera), Kristi Cunningham (Capital One), Ben Harden (CapTech Consulting)

Preventing a big data security breach - Sam Heywood (Cloudera), Nick Curcuru (MasterCard Advisors), Ritu Kama (Intel)

Hive + Amazon EMR + S3 = Elastic big data SQL analytics processing in the cloud, a real-world case study - Jaipaul Agonus (FINRA)