Course – Cloud & Data Engineering
Azure AI — Data Engineering Expert (AI L2)
56 Hours L2 (Intermediate) Classroom / Online (ILT)
Duration
56 Hours
Level
L2 (Intermediate)
Format
Classroom / Online (ILT)
Domain
Azure OpenAI & AI Studio
Training Methodology
Learning by Doing
Every session in this program is built around hands-on execution, not passive slides — you leave having built something, not just watched a demo.
01
Explore
Hands-on labs from Day 1 — real or sandboxed environments, not slide decks.
02
Experiment
Work against scenarios that mirror your actual stack.
03
Engage
Solve real business problems in every session, not toy examples.
04
Apply
Capstone exercises tied to your team’s actual workflows.
Who This Is For
What You’ll Be Able to Do
- Design and build end-to-end Azure data pipelines (ingestion → processing → governance)
- Integrate Azure OpenAI and AI Search into data engineering workflows
- Architect and deploy enterprise RAG systems on Azure
- Build and monitor AI pipelines using Azure AI Studio and Prompt Flow
- Contribute to internal AI accelerators and POCs using Azure-native AI services
Prerequisites
- 1–2 years of hands-on experience working on the Azure platform in a data engineering or analytics role
- Working knowledge of Azure Storage (Blob, ADLS Gen2) and at least one processing service (Databricks, Synapse, or ADF)
- Familiarity with Python or PySpark for data processing
- Basic SQL proficiency and understanding of ETL/ELT, batch vs streaming concepts
Curriculum
Module 1
Azure Storage & Enterprise Data Lake Foundations
- Azure Storage Account types (Blob, ADLS Gen2, File, Queue)
- Data lake architecture — raw/curated/consumption zones
- Azure Data Factory: pipelines, linked services, datasets
- Delta Lake on Azure — ACID transactions, time travel
- Storage security (RBAC, SAS tokens, private endpoints) and performance tuning
Labs: Provision ADLS Gen2 with hierarchical namespace; build an ADF pipeline ingesting CSV to Delta format; implement RBAC and private endpoint for secure access. Case study: design a 3-zone lake architecture modeling an on-prem to Azure migration.
Module 2
Data Ingestion & Orchestration
- ADF advanced — triggers, dependencies, parameterization
- Azure Event Hubs — partitions, consumer groups, streaming ingestion
- Azure IoT Hub integration and Apache Kafka on Azure
- Change Data Capture with Debezium
- Azure Logic Apps & Data Factory hybrid orchestration
Labs: Build an event-driven ADF pipeline using an Event Hub trigger; set up Kafka-to-ADLS streaming ingestion; implement CDC from Azure SQL to Delta Lake. Case study: design an ingestion layer for 1M clickstream events/hour.
Module 3
Data Processing & Optimization
- Azure Databricks — clusters, notebooks, Spark internals
- Azure Synapse Analytics — dedicated vs serverless SQL pools
- Spark optimization — partitioning, caching, broadcast joins
- Stream processing with Spark Structured Streaming and Azure Stream Analytics
- Data quality patterns
Labs: Build a Databricks notebook for batch transformation with Delta Lake; write a Synapse serverless SQL query over Parquet; implement real-time aggregation with Stream Analytics. Case study: near-real-time KPI dashboard combining batch and stream layers.
Module 4
Security, Governance & Compliance
- Microsoft Purview — cataloging, lineage, classification
- Azure Key Vault — secrets management, managed identities
- Data masking & row-level security in Synapse
- GDPR/compliance patterns and Azure Policy for governance
- Unity Catalog integration with Databricks
Labs: Register ADLS assets in Purview and classify; implement row-level security for a multi-tenant scenario; set up Key Vault reference in an ADF pipeline. Case study: governance framework for a BFSI regulatory audit.
Module 5
Azure OpenAI Service — Foundations for Data Engineers
- Azure OpenAI vs OpenAI API — architecture, compliance, data residency
- Model catalogue: GPT-4o, text-embedding-3, DALL-E
- API basics — completions, chat, function calling
- Token management, cost optimization, and responsible AI principles
Labs: Deploy GPT-4o in Azure OpenAI Studio and call via REST API; write a function-calling prompt to extract schema from unstructured text. Case study: auto-generate pipeline documentation from an ADF JSON definition.
Module 6
Embeddings Generation & Vector Stores on Azure
- Embeddings and semantic vs keyword search
- Azure OpenAI embedding models
- Azure AI Search — vector, hybrid search
- Storing embeddings (Azure AI Search vs pgvector); chunking strategies
- Similarity search — cosine similarity, HNSW indexing
Labs: Generate embeddings for a document corpus; index in Azure AI Search and run vector similarity queries; compare keyword vs vector search. Case study: semantic search engine over 10,000 support tickets.
Module 7
Building Enterprise RAG Systems on Azure
- RAG architecture — retrieval + generation layers
- Azure AI Search as retrieval backbone; integrating with Azure OpenAI
- LangChain/Semantic Kernel on Azure
- Chunking, overlap, metadata filtering
- Grounding, hallucination mitigation, and evaluating RAG (precision@k, faithfulness)
Labs: Build an end-to-end RAG pipeline (ADLS → AI Search → Azure OpenAI → response); implement metadata filtering; evaluate with the Azure AI Evaluation SDK. Case study: 500+ document Q&A bot over an internal wiki.
Module 8
Azure AI Studio for Orchestration & Deployment
- Azure AI Studio overview — projects, hubs, model catalogue
- Prompt Flow — build, test, evaluate LLM pipelines visually
- Connecting Prompt Flow to AI Search, OpenAI, custom tools
- Deploying pipelines as managed endpoints
- Monitoring with Azure Monitor/Application Insights and CI/CD for AI pipelines
Labs: Build a Prompt Flow pipeline connecting AI Search + GPT-4o; deploy as a REST endpoint; set up monitoring alerts. Case study: productionize the Module 7 RAG bot with evaluation flow and CI/CD.
Delivery Details
- Designed for Instructor-Led Training (ILT) — classroom or virtual/online
- Hands-on labs use flexible, domain-adaptable datasets; case studies adapt to BFSI, Retail, Healthcare, or Telecom
- Data Engineering and Gen AI tracks run in sequence and are equally mandatory; recommended batch size 15–20 learners