Spark, No Tears

Preface

This book teaches students how to program in Apache Spark. The notebooks run with a local Spark master, local[*]. No Hadoop, HDFS, or standalone Spark cluster is required for the examples.

To execute the notebooks end-to-end, build the local Docker check image from the repository root.

docker build -t book-spark-intro-check:local docker/spark-intro-check
docker/spark-intro-check/execute-notebooks.sh

The runner writes executed notebooks to sphinx/spark-intro/build/executed-notebooks.

The diagram below shows the major groups in the book. The first chapters build the API foundation. The middle chapters focus on data movement, data quality, and specialized workloads. The final chapters cover applied projects, local testing, packaging, table formats, and runtime choices.

That left-to-right path matches the mental model most readers need: first understand distributed execution, then learn where Spark moves data, then use higher-level workloads, and finally shape the work into local tests, submitted jobs, and table-format-aware applications.

Foundations

• 1. Resilient Distributed Datasets (RDD)
  • 1.1. Local Spark setup
  • 1.2. Acquiring a RDD
  • 1.3. Transformations
  • 1.4. Actions
  • 1.5. Chaining transformations and actions
  • 1.6. Broadcasting variables
  • 1.7. Accumulator
• 2. DataFrames
  • 2.1. Local Spark setup
  • 2.2. Acquiring a DataFrame
  • 2.3. Data Types and Schema
  • 2.4. DataFrame operations
• 3. SparkSQL
  • 3.1. Local Spark setup
  • 3.2. Data
  • 3.3. Select
  • 3.4. Count
  • 3.5. Nested query
  • 3.6. Nested fields
  • 3.7. Left join
  • 3.8. Right join
  • 3.9. Inner join
  • 3.10. User defined function
• 4. Execution Plans
  • 4.1. Lazy Evaluation
  • 4.2. explain()
  • 4.3. Finding Shuffles
  • 4.4. Shuffle Joins And Broadcast Joins
  • 4.5. Caching
  • 4.6. Partitions
  • 4.7. Spark UI
  • 4.8. What To Remember

Data Movement and Quality

• 1. Input/Output
  • 1.1. Local Spark setup
  • 1.2. Data
  • 1.3. Writing data
  • 1.4. Reading
• 2. Joins, Skew, And Data Movement
  • 2.1. A Small Customer And Order Dataset
  • 2.2. Join Types
  • 2.3. Duplicate Key Explosions
  • 2.4. Shuffle Join Versus Broadcast Join
  • 2.5. Skewed Keys
  • 2.6. Salting A Skewed Join
  • 2.7. What To Remember
• 3. Schemas, Bad Data, And Defensive Reads
  • 3.1. Permissive JSON Reads
  • 3.2. Separate Parsing From Validation
  • 3.3. Fail Fast When Bad Rows Should Stop The Job
  • 3.4. Defensive CSV Reads
  • 3.5. What To Remember
• 4. UDFs Versus Built-In Functions
  • 4.1. Built-In Functions
  • 4.2. Regular Python UDFs
  • 4.3. Pandas UDFs
  • 4.4. Keep Conditions Inside The Expression
  • 4.5. What To Remember
• 5. Broadcast Variables And Accumulators
  • 5.1. Broadcast Variables
  • 5.2. DataFrame Alternative
  • 5.3. Accumulators
  • 5.4. What To Remember

Specialized Workloads

• 1. Spark Stream, DStreams
  • 1.1. Local Spark setup
  • 1.2. Transformations
  • 1.3. Window operations
  • 1.4. Output operations
• 2. Structured Streaming
  • 2.1. The Streaming Model
  • 2.2. A Local Event Stream
  • 2.3. Reading A Stream
  • 2.4. Aggregating Into A Memory Sink
  • 2.5. Inspecting Progress
  • 2.6. Event-Time Windows
  • 2.7. Writing A File Sink
  • 2.8. What To Remember
• 3. Graphs
  • 3.1. Local Spark setup
  • 3.2. Data
  • 3.3. In-degrees
  • 3.4. Group by, vertices
  • 3.5. Filter, vertices
  • 3.6. Filter, edges
  • 3.7. Motif finding
  • 3.8. Subgraph
  • 3.9. Algorithms
  • 3.10. Input/Output (IO)
  • 3.11. Message passing
• 4. Machine Learning
  • 4.1. Local Spark setup
  • 4.2. Vectors
  • 4.3. Basic statistics
  • 4.4. Pipelines
  • 4.5. Feature extractors
  • 4.6. Feature transformers
  • 4.7. Regression
  • 4.8. Classificaton
  • 4.9. Clustering
  • 4.10. Frequent pattern mining
  • 4.11. Model selection
• 5. Pandas API On Spark
  • 5.1. A Pandas-Like DataFrame
  • 5.2. Moving Between APIs
  • 5.3. Plans Still Matter
  • 5.4. Collect Intentionally
  • 5.5. What To Remember

Projects and Production Shape

• 1. End-To-End Local Spark Project
  • 1.1. Create A Small Raw Dataset
  • 1.2. Read With Schemas
  • 1.3. Clean And Feature Engineer
  • 1.4. Query With SQL
  • 1.5. Write A Curated Parquet Dataset
  • 1.6. Train A Tiny ML Pipeline
  • 1.7. What To Remember
• 2. Testing Spark Code Locally
  • 2.1. A Transformation Function
  • 2.2. Tiny Test Data
  • 2.3. Assertions
  • 2.4. Test The Edge Case You Expect To Break
  • 2.5. How This Looks In A Test Suite
  • 2.6. What To Remember
• 3. Packaging Spark Jobs
  • 3.1. A Script Entry Point
  • 3.2. Local Input Files
  • 3.3. Run With spark-submit
  • 3.4. Inspect The Result
  • 3.5. What To Remember

Runtime and Ecosystem

• 1. Lakehouse Table Formats
  • 1.1. Parquet Baseline
  • 1.2. Delta Lake
  • 1.3. Apache Iceberg
  • 1.4. Choosing A Format
• 2. Spark Connect
  • 2.1. Server And Client Shape
  • 2.2. Write Code Against A Session
  • 2.3. Practical Differences
  • 2.4. What To Remember
• 3. Tips
  • 3.1. Fine-tuning
• 4. Date and Time
• 5. Local Spark setup
  • 5.1. Pandas datetime
  • 5.2. Python datetime
• 6. Local Runtime Notes
  • 6.1. Docker Runner
  • 6.2. Session Settings
  • 6.3. Local Output Directories
  • 6.4. Practical Local Knobs
  • 6.5. Common Local Problems
  • 6.6. What To Remember

About

One-Off Coder is an educational, service and product company. Please visit us online to discover how we may help you achieve life-long success in your personal coding career or with your company’s business goals and objectives.

• Website

• Facebook

• Twitter

• Instagram

• YouTube

• LinkedIn

Copyright

This work is licensed under a Creative Commons Attribution 4.0 International License by One-Off Coder.



The full source code is available on GitHub.

Cite this book as follows.:

@misc{oneoffcoder_spark_intro_2019,
title={Spark, No Tears},
url={https://learn-spark.oneoffcoder.com},
author={One-Off Coder},
year={2019},
month={Oct}}

Author

Jee Vang, Ph.D.

• Patreon