Microsoft Fabric Advanced Production Data Engineering

Module 1 — Production Notebook Development in Fabric

• Apply notebook engineering standards (structure, idempotency, parameters)
• Use debugging and diagnostics techniques for notebooks
• Implement run logging and rerun-safe patterns for production
• Use delta log history/time travel concepts for recovery and troubleshooting
  Lab: Lab – Production Notebook Template & Standards

Module 2 — Medallion Architecture for Lakehouse and Warehouse

• Organize Lakehouse using Bronze/Silver/Gold layers
• Define table contracts (schema, keys, quality expectations)
• Design Silver as a stable interface for Gold and Warehouse
• Establish Lakehouse vs Warehouse serving decision framework
  Lab: Lab – Medallion Contracts and Silver Design

Module 3A — Pipeline Architecture: Metadata and Control Tables

• Design metadata-driven ingestion patterns (entities, sources, destinations)
• Implement control tables for incremental file ingestion (watermarks)
• Define operational states (new, processing, completed, failed)
• Establish repeatable pipeline templates for multiple entities
  Lab: Lab – Metadata Model + Control Tables for Incremental Ingestion

Module 3B — Advanced Pipeline Controls: Schema Evolution and Operations

• Handle schema drift and controlled schema evolution
• Implement run logging, retries, and reprocessing patterns
• Apply validation steps and failure handling strategies
• Standardize operational outputs (metrics, logs, audit trails)
  Lab: Lab – Metadata-Driven Pipeline with Logging + Schema Controls

Module 4 — Incremental Processing Patterns for Silver

• Ingest new and changed data safely (deduplication and replay handling)
• Upsert into Silver tables using Spark SQL patterns
• Implement data quality policies (fail vs quarantine vs warn)
• Validate correctness with reconciliations and repeatable checks
  Lab: Lab – Incremental Silver Upsert with Deduplication