Data Engineering · Advanced

Databricks: Modern Data Engineering

modern data engineering with a comprehensive 3-month, hands-on Databricks program. Learn Apache Spark, PySpark, Delta Lake, Unity Catalog, Lakeflow and cloud lakehouse architecture. Master enterprise data modeling, including dimensional modeling and build production-oriented data pipelines.

The Modern Data Engineering with Databricks program is a comprehensive, hands-on training course for experienced IT professionals who want to develop skills in modern cloud data engineering, data modeling and enterprise data architecture.
course starts with the fundamentals of the Databricks Lakehouse platform, including its architecture, compute, storage, delta lake and unity catalog. || participants then progress to Apache Spark, PySpark, SQL, data ingestion, streaming and automated pipeline development.
A major focus is on applying enterprise data modeling and architecture principles to the Databricks Lakehouse.
participants will learn how to design bronze, silver and gold data layers, implement dimensional models, and build scalable data solutions for enterprise workloads.
program also covers production-oriented topics such as pipeline orchestration, data quality, governance, security, CI/CD, monitoring, performance optimization and cost management.
throughout the course, participants will work on practical exercises and an integrated enterprise project involving source data ingestion, transformation, modeling, orchestration and reporting.

Duration: 60 hours
Mode: Live online
Next batch: 3 Nov 2026
Schedule: Mon to Fri, 7pm - 8pm

Who it is for

Prerequisites

basic SQL and relational database knowledge
understanding of ETL and data warehousing
familiarity with tables, joins, transformations and data pipelines

What you will be able to do

explain databricks lakehouse architecture and its core components.
databricks workspaces, compute resources, notebooks and SQL warehouses.
develop PySpark programs for large-scale data processing.
create, modify and optimize delta tables.
batch and incremental data ingestion using Auto Loader and other techniques.
build bronze, silver and gold pipelines using databricks.
develop data pipelines using lakeflow declarative pipelines and lakeflow jobs.
implement data quality, governance and security using unity catalog.
apply spark and delta lake performance optimization techniques.
deploy and manage data pipelines using git and databricks asset bundles.
monitor pipeline execution, troubleshoot failures and control compute costs.
build a complete enterprise lakehouse project with architecture and data models.

Course contents · 21 modules · 60 hours

Lakehouse Architecture and Databricks Platform

OBJECTIVE:

Understand the lakehouse architecture and how Databricks brings together data engineering, analytics and AI.

TOPICS:

evolution of data warehouses, data lakes and lakehouses
limitations of traditional two-tier architecture
lakehouse architecture and its characteristics
databricks data intelligence platform
control plane versus data plane
classic versus serverless compute
storage and compute separation
databricks on azure, aws and gcp
databricks workspace and account console
databricks Free Edition and workspace setup.

PRACTICAL EXCERCISES:

explore a databricks workspace
create and execute a notebook
examine the lakehouse architecture using a sample dataset
create a basic delta table.

ASSESSMENT:

explain the architecture and identify the role of each platform component.

2 h

Workspace, Compute and Development Environment

OBJECTIVE:

work with databricks development tools and configure appropriate compute resources.

TOPICS:

workspace, notebooks and workspace files
python, SQL, scala and R notebook support
notebook magic commands and widgets
all-purpose versus job compute | serverless compute and SQL warehouses
databricks runtime and photon
cluster policies, autoscaling and auto-termination
access modes and libraries
databricks cli, rest api and sdk
databricks assistant and vs code extension.

PRACTICAL EXCERCISES:

create notebooks using python and SQL
configure compute and explore runtime settings
use widgets and notebook parameters
run a notebook using different compute options.

ASSESSMENT:

select compute configurations for different workloads.

2 h

Apache Spark Architecture and Fundamentals

OBJECTIVE:

Understand distributed data processing and Spark execution.

TOPICS:

spark architecture: driver, executors, tasks, stages and jobs
RDDs, data frames and data sets
transformations and actions
lazy evaluation
narrow and wide transformations
shuffle operations
spark SQL and catalyst optimizer
adaptive query execution
spark ui and execution plans
spark connect.

PRACTICAL EXERCICES:

create data frames from csv and json
apply transformations and actions
examine spark execution plans
analyze shuffles using spark UI.

ASSESSMENT:

explain the execution flow of a Spark application.

2 h

PySpark Programming

OBJECTIVE:

Develop PySpark applications for data engineering workloads.

TOPICS:

python programming for data engineering
data frame creation and schema definition
column expressions and built-in functions
filtering, sorting and aggregations
joins and window functions
null handling and data cleansing
user-defined functions and pandas UDFs
partitioning, repartition and coalesce
caching and persistence
error handling and reusable transformations.

PRACTICAL EXERCICES:

build a PySpark ETL pipeline
cleanse customer and transaction data
perform joins and aggregations
develop reusable transformation functions

ASSESSMENT:

develop a PySpark pipeline from a source dataset to a curated data frame.

4 h

Databricks SQL and Data Analysis

OBJECTIVE:

Use SQL to query, transform and analyze lakehouse data.

TOPICS:

databricks SQL environment
SQL editor and query history
table and view creation
join, cte and window functions
advanced SQL functions
query plans and query profiles
SQL warehouses
AI/BI dashboards and genie overview
power bi connectivity.

PRACTICAL EXERCICES:

develop analytical SQL queries
create views and reporting tables
build a simple dashboard
query delta tables using a SQL warehouse.

ASSESSMENT:

create a reporting dataset and dashboard from transactional data.

3 h

Data Modeling Fundamentals for Lakehouse

OBJECTIVE:

Apply enterprise data modeling principles to a modern lakehouse.

TOPICS:

conceptual, logical and physical data modeling
business processes and data requirements
entities, attributes, keys and relationships
normalization and denormalization
oltp versus analytical models
enterprise data domains
data modeling considerations for distributed storage
modeling standards and documentation
data architecture and modeling responsibilities.

PRACTICAL EXERCICES:

analyze a banking or retail business case
develop a conceptual and logical data model
identify source-to-target mappings
map the model to the lakehouse layers.

ASSESSMENT:

design a logical data model and explain its implementation in Databricks.

2 h

Delta Lake and Lakehouse Storage

OBJECTIVE:

Use Delta Lake to build reliable, versioned and performant data tables.

TOPICS:

delta Lake architecture and transaction log
ACID transactions and optimistic concurrency
delta table creation and management
schema enforcement and schema evolution
merge, update and delete
time travel and restore
change data feed
managed and external tables
shallow and deep clones
deletion vectors and delta uniform
optimize, vacuum and liquid clustering.

PRACTICAL EXERCICES:

create and modify Delta tables
implement incremental merge
recover historical data using time travel
examine delta transaction history and optimize a table.

ASSESSMENT:

Implement an incremental Delta Lake solution with historical recovery.

2 h

Medallion Architecture and Pipeline Design

OBJECTIVE:

design and implement layered lakehouse pipelines.

TOPICS:

bronze, silver and gold architecture
raw data retention and replayability
data cleansing and standardization
data quality rules and validation
conformed data and reusable transformations
business-ready aggregates and data marts
pipeline dependencies and data lineage
batch versus incremental processing
architecture and naming standards.

PRACTICAL EXERCICES:

build a bronze-to-silver-to-gold pipeline
apply data quality rules
implement duplicate detection
develop a business-ready gold dataset.

ASSESSMENT:

Design and implement a complete medallion pipeline.

2 h

Dimensional Modeling on Databricks

OBJECTIVE:

Design analytical data models and implement them in the lakehouse.

TOPICS:

dimensional modeling principles
star and snowflake schemas
fact table grain and measures
transaction, periodic snapshot and accumulating snapshot facts
dimension design and surrogate keys
slowly changing dimensions: types 1, 2 and 3
role-playing and junk dimensions
bridge tables and many-to-many relationships
fact and dimension loading patterns
semantic modeling for BI.

PRACTICAL EXERCICES:

design a banking or retail star schema
implement fact and dimension tables in Delta Lake
implement scd type 2 using SQL or PySpark
create a gold-layer dimensional model.

ASSESSMENT:

build a dimensional model and demonstrate historical reporting.

2 h

Data Ingestion and Incremental Processing

OBJECTIVE:

build reliable ingestion pipelines from files, databases and event sources.

TOPICS:

batch ingestion patterns
auto loader and cloudFiles
file notification and directory listing modes
schema inference and evolution
copy into
lakeflow connect overview
database ingestion and lakehouse federation
ingestion from kafka, azure event hubs and amazon kinesis
checkpointing and incremental processing
idempotency and recovery.

PRACTICAL EXERCICES:

ingest csv, json and parquet files
implement auto loader
build an incremental file ingestion pipeline
apply schema evolution and handle corrupt records.

ASSESSMENT:

Implement a restartable ingestion pipeline with audit information.

2 h

Structured Streaming and Real-Time Data Processing

OBJECTIVE:

Develop streaming pipelines for event-driven data processing.

TOPICS:

structured Streaming architecture
streaming data frames and streaming sources
processing-time and available now triggers
checkpoints and fault tolerance
watermarks and late-arriving data
stateful aggregations and streaming joins
foreachBatch
deduplication and event-time processing
state store concepts
streaming pipeline monitoring.

PRACTICAL EXERCICES:

process a simulated stream of financial transactions
implement event-time aggregations
handle late-arriving records
build a streaming-to-delta pipeline.

ASSESSMENT:

Develop a streaming pipeline with checkpointing and late-data handling.

2 h

Lakeflow Declarative Pipelines

OBJECTIVE:

develop managed data pipelines using declarative programming.

TOPICS:

lakeflow declarative pipelines architecture
streaming tables and materialized views
views and flows
expectations and data quality rules
auto cdc and apply changes
scd type 1 and type 2 implementation
triggered and continuous pipelines
development and production modes
pipeline event logs
incremental refresh and enzyme concepts.

PRACTICAL EXERCICES:

create a declarative bronze-to-silver pipeline
apply expectations to reject or quarantine invalid records
implement scd type 2 using auto cdc
inspect pipeline execution and event logs.

ASSESSMENT:

build a managed pipeline with data quality and cdc processing.

2 h

Pipeline Orchestration with Lakeflow Jobs

OBJECTIVE:

Orchestrate enterprise data pipelines with dependencies, scheduling and error handling.

TOPICS:

jobs, tasks and DAGs
notebook, python, SQL and pipeline tasks
task dependencies and execution order
parameters and task values
for-each tasks and conditional execution
scheduling and file-arrival triggers
table-update triggers
retries, timeouts and notifications
repair runs and troubleshooting
multi-stage pipeline orchestration.

PRACTICAL EXERCICES:

build a multi-task data engineering workflow
parameterize the pipeline
configure schedules
retries and failure notifications
implement a dependent bronze, silver and gold workflow.

ASSESSMENT:

orchestrate an end-to-end pipeline and demonstrate failure recovery.

2 h

Unity Catalog and Data Governance

OBJECTIVE:

Implement enterprise data governance, access control and lineage.

TOPICS:

unity catalog architecture
metastore, catalog, schema, table, view and volume
managed and external locations | storage credentials
privileges, ownership, grant and revoke
row filters and column masks
tags and attribute-based access control
data lineage and audit
system tables and catalog Explorer
workspace-catalog binding
data classification and governance policies.

PRACTICAL EXERCICES:

create catalogs, schemas and volumes
implement role-based access
apply row filters and column masks
explore lineage and audit information.

ASSESSMENT:

Implement a governed data access model for a business domain.

2 h

Data Sharing, SQL Analytics and BI Integration

OBJECTIVE:

deliver governed lakehouse data to analysts and external consumers.

TOPICS:

databricks SQL warehouses
SQL query history and query profiles
AI/BI dashboards | AI/BI genie
SQL alerts and materialized views
unity catalog metric views
Power BI and Tableau integration
delta sharing
databricks marketplace and clean rooms overview
secure data sharing patterns.

PRACTICAL EXERCICES:

create a Gold-layer analytics dataset
build an AI/BI dashboard
connect Power BI to a SQL warehouse
configure a demonstration of governed data sharing.

ASSESSMENT:

build and present a business dashboard using governed data.

2 h

Spark and Delta Lake Performance Optimization

OBJECTIVE:

Diagnose and improve performance of large-scale data pipelines.

TOPICS:

spark execution plans and query profiles
catalyst optimizer and AQE
partitioning versus liquid clustering
data skipping and statistics
broadcast joins and join strategies
skew handling and salting
repartition and coalesce
caching and persistence
small files and file sizing
predicate pushdown
spill, memory and shuffle optimization
optimize and predictive optimization.

PRACTICAL EXERCICES:

analyze a slow Spark job
identify shuffles and data skew
optimize joins and table layout
compare query execution before and after tuning.

ASSESSMENT:

Submit a performance tuning report with measured improvements.

2 h

Security, Administration and Cost Management

OBJECTIVE:

operate databricks workloads securely and efficiently.

TOPICS:

users, groups and service principals
identity federation and SCIM
OAuth and personal access tokens
secrets and secret scopes
network security and private connectivity
encryption and customer-managed keys
cluster policies and access controls
system billing tables and usage dashboards
budgets, tags and cost allocation
job and query observability
monitoring data quality and drift.

PRACTICAL EXERCICES:

configure secure access for a service principal
explore job and compute usage
build a simple cost monitoring report
investigate failed jobs and data quality alerts.

ASSESSMENT:

review a platform configuration for security and cost controls.

2 h

Databricks DevOps and CI/CD

OBJECTIVE:

implement repeatable development and deployment practices for databricks projects.

TOPICS:

git folders and version control
development, test and production environments
databricks asset bundles
bundle configuration and deployment targets
CI/CD with GitHub actions or azure DevOps
service principals for automated deployment
databricks cli
rest api and sdk
terraform provider overview
unit testing with pytest and chispa.

PRACTICAL EXERCICES:

create a git-based databricks project
package a job using asset bundles
configure deployment targets
implement basic automated tests and deployment.

ASSESSMENT:

deploy a sample pipeline through a controlled CI/CD workflow.

2 h

Production Operations and Troubleshooting

OBJECTIVE:

Maintain reliable and observable Databricks data pipelines.

TOPICS:

Job and pipeline monitoring.
Spark UI and execution diagnostics.
Pipeline event logs.
Data quality monitoring and anomaly detection.
Retry, repair and recovery strategies.
Idempotent processing.
Logging and alerting.
Incident analysis and root-cause investigation.
Deployment rollback and recovery.
Production readiness checklists.

PRACTICAL EXERCICES:

Troubleshoot failed pipeline runs.
Implement recovery and restart mechanisms.
Diagnose data quality failures.
Create an operational runbook.

ASSESSMENT:

Complete a production incident simulation.

2 h

Machine Learning and AI on Databricks — Introduction

OBJECTIVE:

understand how the lakehouse supports data science and AI workloads.

TOPICS:

MLflow experiment tracking
model registry in unity catalog
feature engineering and feature store
autoML overview
model serving
vector search
mosaic AI and agent development overview
data engineering responsibilities in ML pipelines
governance and monitoring of ML assets.

PRACTICAL EXERCICES:

track a simple ML experiment
register a demonstration model
prepare a feature dataset from curated data
explore model serving and vector search.

ASSESSMENT:

explain the integration of data engineering and a basic ML workflow.

2 h

End-to-End Enterprise Lakehouse Implementation

OBJECTIVE:

build an integrated enterprise data engineering solution using databricks.

CASE STUDY:

Global Banking and Financial Services Lakehouse.

BUSINESS SCENARIO:

A financial services organization receives customer, account, transaction and investment data from multiple source systems. It requires a modern lakehouse for operational reporting, historical analysis and advanced analytics.

PROJECT ACTIVITIES:

ingest source files
implement bronze, silver and gold layers
build incremental pipelines using PySpark and delta lake
apply data quality rules and handle cdc
orchestrate pipelines using lakeflow jobs
configure unity catalog
optimize delta tables and spark workloads
implement deployment and monitoring practices.

DELIVERABLES:

enterprise architecture diagram
conceptual and logical data models
PySpark notebooks and Delta tables
automated pipeline and orchestration
deployment and operational documentation.

3 h

Hands-on approach

30% concepts, 70% hands-on labs on realistic business scenarios. AI practice support between sessions; every project is reviewed by the trainer.

Who it is for

Prerequisites

What you will be able to do

Course contents · 21 modules · 60 hours

Hands-on approach

Related trainings

Business Requirement to Enterprise Data Architecture

AI for Data Teams: LLMs, RAG and Reviewing AI Output

Business Requirement to Enterprise Data Architecture