Course Outline
Databricks Platform and Lakehouse Fundamentals
- Databricks Lakehouse architecture and components
- Organizing workspaces and catalogs
Databricks Workspace and Notebooks
- Workspace navigation and notebook-based development
- Structuring code into reusable notebooks
Apache Spark Architecture and Execution
- Spark runtime architecture and execution model
- Lazy evaluation and the job DAG
PySpark DataFrames and the DataFrame API
- DataFrame abstractions and schemas
- Core DataFrame operations and column expressions
Translating SQL to PySpark DataFrames
- Translating core SQL clauses to DataFrame operations
- Window functions and aggregations in PySpark
Reading and Writing Data in Databricks
- Reading from common file and database sources
- Writing and partitioning data in the Lakehouse
Delta Lake and Table Management
- Delta tables and ACID transactions
- Time travel and schema evolution
Data Cleaning and Transformation Patterns
- Data cleaning and type conversion
- Building reusable transformation logic
User-Defined Functions and Modular Code
- Python UDFs and pandas UDFs
- Modularizing procedural logic into functions
Performance Tuning and Optimization
- Partitioning and caching strategies
- Diagnosing bottlenecks with the Spark UI
Structured Streaming Fundamentals
- Batch versus streaming processing models
- Streaming DataFrames and basic aggregations
Databricks Jobs and Workflow Orchestration
- Scheduling notebooks as jobs and tasks
- Building multi-step workflows with dependencies
Unity Catalog and Data Governance
- Unity Catalog architecture and namespaces
- Access control and data lineage
Testing, Debugging, and Production Practices
- Unit testing PySpark logic
- Debugging and code quality standards
End-to-End Financial Services Use Cases
- Building an end-to-end banking ETL pipeline
- Translating legacy SQL processes to PySpark
Migrating SQL Workloads to PySpark
- Migration strategy and planning patterns
- Incremental conversion of SQL workflows to PySpark
Requirements
- Experience with Python programming, including functions and data types
- An understanding of SQL, including joins, aggregations, and subqueries
- No prior experience with Databricks or PySpark is required
Audience
- Data engineers, data analysts, and data professionals
- Teams migrating existing SQL-based workflows to Databricks and PySpark
Testimonials (2)
All the topics covered, although many were very quick, give us an idea of what we will need to delve into further. Additionally, I liked that we got to do some hands-on practice, although I still believe the course deserves more.
Sandra Mariela Lopez Bernal - Kueski
Course - Databricks
Machine Translated
I liked that it was practical. Loved to apply the theoretical knowledge with practical examples.
