ProgramGuru

Apache Spark

Learn big data by building with Spark.

Understand distributed computation, practise PySpark and Spark SQL, and complete a real data pipeline.

Start learning →

No account or installation needed. Your progress stays on this device.

Srinivas Namala, author
About the author

Srinivas Namala

Software Engineer

Srinivas focuses on backend development, databases, systems, and the technologies used to build reliable server-side applications.

View LinkedIn profile ↗

Your learning path

All lessons

50 lessons168 guided examples50 quiz questions

Your progress is saved on this device.

Estimated time includes reading, exploring code and answering quizzes. Learn at your own pace.

Module 1: Introduction to Big Data

4 lessons · 24 min
  1. What is Big Data?1guided examples1quiz questions5minutes estimated learning time
  2. 3Vs of Big Data: Volume, Velocity, Variety2guided examples1quiz questions6minutes estimated learning time
  3. Limitations of Traditional Data Processing2guided examples1quiz questions6minutes estimated learning time
  4. Overview of Big Data Tools: Hadoop, Spark…2guided examples1quiz questions7minutes estimated learning time

Module 2: Getting Started with Apache Spark

6 lessons · 50 min
  1. What is Apache Spark?1guided examples1quiz questions5minutes estimated learning time
  2. Use Cases of Apache Spark in Industry1guided examples1quiz questions5minutes estimated learning time
  3. Installing Apache Spark on Windows (Step-by…3guided examples1quiz questions7minutes estimated learning time
  4. Installing Apache Spark on a Local Linux Sy…10guided examples1quiz questions13minutes estimated learning time
  5. Installing Apache Spark on MacOS10guided examples1quiz questions14minutes estimated learning time
  6. Running Your First Spark Application2guided examples1quiz questions6minutes estimated learning time

Module 3: Apache Spark Architecture

4 lessons · 25 min
  1. Spark Ecosystem Components: Core, SQL, Stre…4guided examples1quiz questions8minutes estimated learning time
  2. Driver, Executors, and Cluster Manager in A…1guided examples1quiz questions5minutes estimated learning time
  3. Job, Stage, and Task in Apache Spark2guided examples1quiz questions6minutes estimated learning time
  4. DAG and Lazy Evaluation in Apache Spark2guided examples1quiz questions6minutes estimated learning time

Module 4: RDDs - Resilient Distributed Datasets

5 lessons · 34 min
  1. What is an RDD in Apache Spark?3guided examples1quiz questions7minutes estimated learning time
  2. Creating RDDs from Collections and Files4guided examples1quiz questions7minutes estimated learning time
  3. Transformations vs Actions in Apache Spark3guided examples1quiz questions7minutes estimated learning time
  4. Persistence and Caching in Apache Spark2guided examples1quiz questions6minutes estimated learning time
  5. Limitations of RDDs3guided examples1quiz questions7minutes estimated learning time

Module 5: Introduction to DataFrames

4 lessons · 32 min
  1. Why DataFrames over RDDs?2guided examples1quiz questions6minutes estimated learning time
  2. Creating and Displaying DataFrames in PySpark6guided examples1quiz questions10minutes estimated learning time
  3. Reading CSV, JSON, and Parquet Files in PyS…4guided examples1quiz questions8minutes estimated learning time
  4. Selecting, Filtering, and Transforming Data…5guided examples1quiz questions8minutes estimated learning time

Module 6: PySpark Essentials

4 lessons · 33 min
  1. Setting up PySpark in Jupyter Notebook5guided examples1quiz questions9minutes estimated learning time
  2. Basic DataFrame Operations in PySpark7guided examples1quiz questions10minutes estimated learning time
  3. Columns, Expressions, and User-Defined Func…3guided examples1quiz questions6minutes estimated learning time
  4. Practical Examples with Real Datasets5guided examples1quiz questions8minutes estimated learning time

Module 7: Spark SQL

4 lessons · 30 min
  1. Creating Temporary Views and Global Views i…2guided examples1quiz questions6minutes estimated learning time
  2. Using SQL Queries on DataFrames in Spark5guided examples1quiz questions8minutes estimated learning time
  3. Common Aggregations and Joins in Spark SQL4guided examples1quiz questions8minutes estimated learning time
  4. Optimization using Catalyst in Apache Spark4guided examples1quiz questions8minutes estimated learning time

Module 8: Working with Complex Data

4 lessons · 30 min
  1. Dealing with Nested JSON in Apache Spark4guided examples1quiz questions7minutes estimated learning time
  2. Exploding Arrays and Structs in Apache Spark6guided examples1quiz questions10minutes estimated learning time
  3. Flattening Hierarchical Data4guided examples1quiz questions7minutes estimated learning time
  4. Schema Evolution and Inference in Apache Spark2guided examples1quiz questions6minutes estimated learning time

Module 9: Data Cleaning & Transformations

4 lessons · 32 min
  1. Dropping Nulls and Handling Missing Values…5guided examples1quiz questions9minutes estimated learning time
  2. Replacing, Filtering, and Grouping Data in…4guided examples1quiz questions7minutes estimated learning time
  3. Window Functions in Apache Spark4guided examples1quiz questions7minutes estimated learning time
  4. Data Aggregation and Pivots in PySpark5guided examples1quiz questions9minutes estimated learning time

Module 10: Spark Streaming

4 lessons · 22 min
  1. What is Spark Streaming?1guided examples1quiz questions5minutes estimated learning time
  2. Micro-batching and Continuous Data Processi…1guided examples1quiz questions5minutes estimated learning time
  3. Reading from Kafka, Socket, and File Stream…3guided examples1quiz questions7minutes estimated learning time
  4. Window Operations and Aggregations in Spark…1guided examples1quiz questions5minutes estimated learning time

Module 11: Introduction to Machine Learning with Spark MLlib

6 lessons · 33 min
  1. Overview of Spark MLlib1guided examples1quiz questions5minutes estimated learning time
  2. Feature Engineering with VectorAssembler an…3guided examples1quiz questions6minutes estimated learning time
  3. Building Machine Learning Pipelines in Apac…1guided examples1quiz questions5minutes estimated learning time
  4. Classification Using Logistic Regression in…3guided examples1quiz questions6minutes estimated learning time
  5. Linear Regression in Spark MLlib1guided examples1quiz questions4minutes estimated learning time
  6. Clustering with KMeans in Spark MLlib4guided examples1quiz questions7minutes estimated learning time

Module 12: Project – Real-World Data Pipeline

1 lessons · 8 min
  1. Movie Ratings Project with Apache Spark5guided examples1quiz questions8minutes estimated learning time