Business Requirement to Enterprise Data Architecture

OBJECTIVE:

• Understand the lakehouse architecture and how Databricks brings together data engineering, analytics and AI.

TOPICS:

• evolution of data warehouses, data lakes and lakehouses

• limitations of traditional two-tier architecture

• lakehouse architecture and its characteristics

• databricks data intelligence platform

• control plane versus data plane

• classic versus serverless compute

• storage and compute separation

• databricks on azure, aws and gcp

• databricks workspace and account console

• databricks Free Edition and workspace setup.

PRACTICAL EXCERCISES:

• explore a databricks workspace

• create and execute a notebook

• examine the lakehouse architecture using a sample dataset

• create a basic delta table.

ASSESSMENT:

• explain the architecture and identify the role of each platform component.

OBJECTIVE:

• work with databricks development tools and configure appropriate compute resources.

TOPICS:

• workspace, notebooks and workspace files

• python, SQL, scala and R notebook support

• notebook magic commands and widgets

• all-purpose versus job compute | serverless compute and SQL warehouses

• databricks runtime and photon

• cluster policies, autoscaling and auto-termination

• access modes and libraries

• databricks cli, rest api and sdk

• databricks assistant and vs code extension.

PRACTICAL EXCERCISES:

• create notebooks using python and SQL

• configure compute and explore runtime settings

• use widgets and notebook parameters

• run a notebook using different compute options.

ASSESSMENT:

• select compute configurations for different workloads.

OBJECTIVE:

• Understand distributed data processing and Spark execution.

TOPICS:

• spark architecture: driver, executors, tasks, stages and jobs

• RDDs, data frames and data sets

• transformations and actions

• lazy evaluation

• narrow and wide transformations

• shuffle operations

• spark SQL and catalyst optimizer

• adaptive query execution

• spark ui and execution plans

• spark connect.

PRACTICAL EXERCICES:

• create data frames from csv and json

• apply transformations and actions

• examine spark execution plans

• analyze shuffles using spark UI.

ASSESSMENT:

• explain the execution flow of a Spark application.

OBJECTIVE:

• Develop PySpark applications for data engineering workloads.

TOPICS:

• python programming for data engineering

• data frame creation and schema definition

• column expressions and built-in functions

• filtering, sorting and aggregations

• joins and window functions

• null handling and data cleansing

• user-defined functions and pandas UDFs

• partitioning, repartition and coalesce

• caching and persistence

• error handling and reusable transformations.

PRACTICAL EXERCICES:

• build a PySpark ETL pipeline

• cleanse customer and transaction data

• perform joins and aggregations

• develop reusable transformation functions

ASSESSMENT:

• develop a PySpark pipeline from a source dataset to a curated data frame.

OBJECTIVE:

• Use SQL to query, transform and analyze lakehouse data.

TOPICS:

• databricks SQL environment

• SQL editor and query history

• table and view creation

• join, cte and window functions

• advanced SQL functions

• query plans and query profiles

• SQL warehouses

• AI/BI dashboards and genie overview

• power bi connectivity.

PRACTICAL EXERCICES:

• develop analytical SQL queries

• create views and reporting tables

• build a simple dashboard

• query delta tables using a SQL warehouse.

ASSESSMENT:

• create a reporting dataset and dashboard from transactional data.

OBJECTIVE:

• Apply enterprise data modeling principles to a modern lakehouse.

TOPICS:

• conceptual, logical and physical data modeling

• business processes and data requirements

• entities, attributes, keys and relationships

• normalization and denormalization

• oltp versus analytical models

• enterprise data domains

• data modeling considerations for distributed storage

• modeling standards and documentation

• data architecture and modeling responsibilities.

PRACTICAL EXERCICES:

• analyze a banking or retail business case

• develop a conceptual and logical data model

• identify source-to-target mappings

• map the model to the lakehouse layers.

ASSESSMENT:

• design a logical data model and explain its implementation in Databricks.

OBJECTIVE:

• Use Delta Lake to build reliable, versioned and performant data tables.

TOPICS:

• delta Lake architecture and transaction log

• ACID transactions and optimistic concurrency

• delta table creation and management

• schema enforcement and schema evolution

• merge, update and delete

• time travel and restore

• change data feed

• managed and external tables

• shallow and deep clones

• deletion vectors and delta uniform

• optimize, vacuum and liquid clustering.

PRACTICAL EXERCICES:

• create and modify Delta tables

• implement incremental merge

• recover historical data using time travel

• examine delta transaction history and optimize a table.

ASSESSMENT:

• Implement an incremental Delta Lake solution with historical recovery.

OBJECTIVE:

• design and implement layered lakehouse pipelines.

TOPICS:

• bronze, silver and gold architecture

• raw data retention and replayability

• data cleansing and standardization

• data quality rules and validation

• conformed data and reusable transformations

• business-ready aggregates and data marts

• pipeline dependencies and data lineage

• batch versus incremental processing

• architecture and naming standards.

PRACTICAL EXERCICES:

• build a bronze-to-silver-to-gold pipeline

• apply data quality rules

• implement duplicate detection

• develop a business-ready gold dataset.

ASSESSMENT:

• Design and implement a complete medallion pipeline.

OBJECTIVE:

• Design analytical data models and implement them in the lakehouse.

TOPICS:

• dimensional modeling principles

• star and snowflake schemas

• fact table grain and measures

• transaction, periodic snapshot and accumulating snapshot facts

• dimension design and surrogate keys

• slowly changing dimensions: types 1, 2 and 3

• role-playing and junk dimensions

• bridge tables and many-to-many relationships

• fact and dimension loading patterns

• semantic modeling for BI.

PRACTICAL EXERCICES:

• design a banking or retail star schema

• implement fact and dimension tables in Delta Lake

• implement scd type 2 using SQL or PySpark

• create a gold-layer dimensional model.

ASSESSMENT:

• build a dimensional model and demonstrate historical reporting.

OBJECTIVE:

• build reliable ingestion pipelines from files, databases and event sources.

TOPICS:

• batch ingestion patterns

• auto loader and cloudFiles

• file notification and directory listing modes

• schema inference and evolution

• copy into

• lakeflow connect overview

• database ingestion and lakehouse federation

• ingestion from kafka, azure event hubs and amazon kinesis

• checkpointing and incremental processing

• idempotency and recovery.

PRACTICAL EXERCICES:

• ingest csv, json and parquet files

• implement auto loader

• build an incremental file ingestion pipeline

• apply schema evolution and handle corrupt records.

ASSESSMENT:

• Implement a restartable ingestion pipeline with audit information.

OBJECTIVE:

• Develop streaming pipelines for event-driven data processing.

TOPICS:

• structured Streaming architecture

• streaming data frames and streaming sources

• processing-time and available now triggers

• checkpoints and fault tolerance

• watermarks and late-arriving data

• stateful aggregations and streaming joins

• foreachBatch

• deduplication and event-time processing

• state store concepts

• streaming pipeline monitoring.

PRACTICAL EXERCICES:

• process a simulated stream of financial transactions

• implement event-time aggregations

• handle late-arriving records

• build a streaming-to-delta pipeline.

ASSESSMENT:

• Develop a streaming pipeline with checkpointing and late-data handling.

OBJECTIVE:

• develop managed data pipelines using declarative programming.

TOPICS:

• lakeflow declarative pipelines architecture

• streaming tables and materialized views

• views and flows

• expectations and data quality rules

• auto cdc and apply changes

• scd type 1 and type 2 implementation

• triggered and continuous pipelines

• development and production modes

• pipeline event logs

• incremental refresh and enzyme concepts.

PRACTICAL EXERCICES:

• create a declarative bronze-to-silver pipeline

• apply expectations to reject or quarantine invalid records

• implement scd type 2 using auto cdc

• inspect pipeline execution and event logs.

ASSESSMENT:

• build a managed pipeline with data quality and cdc processing.

OBJECTIVE:

• Orchestrate enterprise data pipelines with dependencies, scheduling and error handling.

TOPICS:

• jobs, tasks and DAGs

• notebook, python, SQL and pipeline tasks

• task dependencies and execution order

• parameters and task values

• for-each tasks and conditional execution

• scheduling and file-arrival triggers

• table-update triggers

• retries, timeouts and notifications

• repair runs and troubleshooting

• multi-stage pipeline orchestration.

PRACTICAL EXERCICES:

• build a multi-task data engineering workflow

• parameterize the pipeline

• configure schedules

• retries and failure notifications

• implement a dependent bronze, silver and gold workflow.

ASSESSMENT:

• orchestrate an end-to-end pipeline and demonstrate failure recovery.

OBJECTIVE:

• Implement enterprise data governance, access control and lineage.

TOPICS:

• unity catalog architecture

• metastore, catalog, schema, table, view and volume

• managed and external locations | storage credentials

• privileges, ownership, grant and revoke

• row filters and column masks

• tags and attribute-based access control

• data lineage and audit

• system tables and catalog Explorer

• workspace-catalog binding

• data classification and governance policies.

PRACTICAL EXERCICES:

• create catalogs, schemas and volumes

• implement role-based access

• apply row filters and column masks

• explore lineage and audit information.

ASSESSMENT:

• Implement a governed data access model for a business domain.

OBJECTIVE:

• deliver governed lakehouse data to analysts and external consumers.

TOPICS:

• databricks SQL warehouses

• SQL query history and query profiles

• AI/BI dashboards | AI/BI genie

• SQL alerts and materialized views

• unity catalog metric views

• Power BI and Tableau integration

• delta sharing

• databricks marketplace and clean rooms overview

• secure data sharing patterns.

PRACTICAL EXERCICES:

• create a Gold-layer analytics dataset

• build an AI/BI dashboard

• connect Power BI to a SQL warehouse

• configure a demonstration of governed data sharing.

ASSESSMENT:

• build and present a business dashboard using governed data.

OBJECTIVE:

• Diagnose and improve performance of large-scale data pipelines.

TOPICS:

• spark execution plans and query profiles

• catalyst optimizer and AQE

• partitioning versus liquid clustering

• data skipping and statistics

• broadcast joins and join strategies

• skew handling and salting

• repartition and coalesce

• caching and persistence

• small files and file sizing

• predicate pushdown

• spill, memory and shuffle optimization

• optimize and predictive optimization.

PRACTICAL EXERCICES:

• analyze a slow Spark job

• identify shuffles and data skew

• optimize joins and table layout

• compare query execution before and after tuning.

ASSESSMENT:

• Submit a performance tuning report with measured improvements.

OBJECTIVE:

• operate databricks workloads securely and efficiently.

TOPICS:

• users, groups and service principals

• identity federation and SCIM

• OAuth and personal access tokens

• secrets and secret scopes

• network security and private connectivity

• encryption and customer-managed keys

• cluster policies and access controls

• system billing tables and usage dashboards

• budgets, tags and cost allocation

• job and query observability

• monitoring data quality and drift.

PRACTICAL EXERCICES:

• configure secure access for a service principal

• explore job and compute usage

• build a simple cost monitoring report

• investigate failed jobs and data quality alerts.

ASSESSMENT:

• review a platform configuration for security and cost controls.

OBJECTIVE:

• implement repeatable development and deployment practices for databricks projects.

TOPICS:

• git folders and version control

• development, test and production environments

• databricks asset bundles

• bundle configuration and deployment targets

• CI/CD with GitHub actions or azure DevOps

• service principals for automated deployment

• databricks cli

• rest api and sdk

• terraform provider overview

• unit testing with pytest and chispa.

PRACTICAL EXERCICES:

• create a git-based databricks project

• package a job using asset bundles

• configure deployment targets

• implement basic automated tests and deployment.

ASSESSMENT:

• deploy a sample pipeline through a controlled CI/CD workflow.

OBJECTIVE:

• Maintain reliable and observable Databricks data pipelines.

TOPICS:

• Job and pipeline monitoring.

• Spark UI and execution diagnostics.

• Pipeline event logs.

• Data quality monitoring and anomaly detection.

• Retry, repair and recovery strategies.

• Idempotent processing.

• Logging and alerting.

• Incident analysis and root-cause investigation.

• Deployment rollback and recovery.

• Production readiness checklists.

PRACTICAL EXERCICES:

• Troubleshoot failed pipeline runs.

• Implement recovery and restart mechanisms.

• Diagnose data quality failures.

• Create an operational runbook.

ASSESSMENT:

• Complete a production incident simulation.

OBJECTIVE:

• understand how the lakehouse supports data science and AI workloads.

TOPICS:

• MLflow experiment tracking

• model registry in unity catalog

• feature engineering and feature store

• autoML overview

• model serving

• vector search

• mosaic AI and agent development overview

• data engineering responsibilities in ML pipelines

• governance and monitoring of ML assets.

PRACTICAL EXERCICES:

• track a simple ML experiment

• register a demonstration model

• prepare a feature dataset from curated data

• explore model serving and vector search.

ASSESSMENT:

• explain the integration of data engineering and a basic ML workflow.

OBJECTIVE:

• build an integrated enterprise data engineering solution using databricks.

CASE STUDY:

• Global Banking and Financial Services Lakehouse.

BUSINESS SCENARIO:

• A financial services organization receives customer, account, transaction and investment data from multiple source systems. It requires a modern lakehouse for operational reporting, historical analysis and advanced analytics.

PROJECT ACTIVITIES:

• ingest source files

• implement bronze, silver and gold layers

• build incremental pipelines using PySpark and delta lake

• apply data quality rules and handle cdc

• orchestrate pipelines using lakeflow jobs

• configure unity catalog

• optimize delta tables and spark workloads

• implement deployment and monitoring practices.

DELIVERABLES:

• enterprise architecture diagram

• conceptual and logical data models

• PySpark notebooks and Delta tables

• automated pipeline and orchestration

• deployment and operational documentation.