GCP Data Engineering Expert (AI L2) — BigQuery | Dataflow | Dataproc | Cloud Composer | Vertex AI | Gemini | Vertex AI RAG

Duration: 58 Hours
Target Level: L2 (Intermediate)
Delivery Mode: Classroom / Online (ILT)
AI Platform: Vertex AI + Gemini

Course Overview

This course equips Data Engineering & Analytics professionals with core GCP data engineering skills and the AI layer required for L2 proficiency. It is designed for practitioners who already work on GCP and want to master enterprise-grade data pipelines alongside modern Gen AI capabilities — specifically Gemini on Vertex AI, AI-enriched pipeline patterns, and Vertex AI RAG systems.

The course follows a learn-by-doing philosophy. Every module includes hands-on labs and flexible case study scenarios that trainers can adapt to their domain context.

L2 Outcome Statement

On completing this course, a learner will be able to:

• Design and build end-to-end GCP data pipelines (ingestion → processing → governance)
• Leverage BigQuery as a unified analytics engine with Gen AI inference built in
• Integrate Gemini on Vertex AI into both batch and streaming data engineering workflows
• Architect and deploy enterprise RAG systems using Vertex AI Vector Search and RAG Engine
• Contribute to internal AI accelerators and POCs using GCP-native AI services

Prerequisites

Learners enrolling in this course must have:

• 1–2 years of hands-on experience working on the GCP platform in a data engineering or analytics role
• Working knowledge of GCS and at least one processing service (BigQuery, Dataflow, or Dataproc)
• Familiarity with Python or PySpark for data processing
• Basic SQL proficiency (BigQuery SQL preferred)
• Understanding of data pipeline concepts (ETL/ELT, batch vs streaming)

Helpful but not mandatory:

• Exposure to any Gen AI or LLM concept (prompt engineering, Gemini or GPT APIs)
• Experience with Cloud Composer (Airflow), Terraform, or CI/CD on GCP

Course Structure at a Glance

Data Engineering modules build the GCP platform foundation. Gen AI modules apply Vertex AI and Gemini in a data-engineering context. Both tracks run in sequence and are equally mandatory.

Detailed Course Modules

Part A — GCP Data Engineering Foundation

Module 1: GCP Infrastructure & Storage Foundations | 8 hrs

Topics Covered: GCP global infrastructure (regions, zones, VPCs, IAM hierarchy) · Cloud Storage buckets, storage classes, object lifecycle · Data lake architecture on GCS (raw/curated/consumption zones) · Cloud Storage security (IAM, ACLs, CMEK, VPC Service Controls) · Performance tuning and file formats (Parquet, Avro, ORC) · Cloud Data Catalog · Dataplex for unified governance

Hands-On Labs: Provision GCS buckets with 3-zone folder structure and IAM bindings · Configure Cloud Data Catalog crawler to tag a Parquet dataset · Set up a Dataplex lake with raw and curated zones

Case Study: Design a 3-zone data lake architecture with governance rules using Dataplex, modeling a migration from an on-prem warehouse to GCP.

Module 2: BigQuery – Analytical Data Warehousing | 8 hrs

Topics Covered: BigQuery architecture (storage/compute separation, columnar storage, Dremel engine) · Datasets, tables, views, materialized views · Partitioning strategies · Clustering · Performance optimization (query planning, slot management, BI Engine) · BigQuery Omni · Cost management · Loading data (batch, streaming inserts, Storage Write API)

Hands-On Labs: Create partitioned and clustered tables, compare query performance · Load data via batch job and Storage Write API · Build a materialized view and measure cost improvement

Case Study: Design a schema with partitioning and clustering strategy for multi-TB event data in a scalable analytics warehouse.

Module 3: Data Processing & Serverless ETL | 8 hrs

Topics Covered: Cloud Dataflow (Apache Beam model, pipelines, PCollections, windows, triggers) · Dataflow templates · Cloud Dataproc (managed Spark/Hadoop, Dataproc Serverless) · Spark optimization · Cloud Data Fusion · Stream processing window patterns · Data quality with Dataplex

Hands-On Labs: Build a Dataflow pipeline from Pub/Sub to BigQuery · Run a Dataproc Serverless Spark job on GCS Parquet files · Build a visual ETL pipeline with Cloud Data Fusion Wrangler

Case Study: Build a near-real-time KPI pipeline combining Dataflow streaming and Dataproc batch layers feeding a BigQuery dashboard.

Module 4: Pipeline Orchestration & Metadata Management | 6 hrs

Topics Covered: Cloud Composer (managed Airflow) architecture and DAGs · Designing robust DAGs (dependencies, branching, retries, SLAs) · Cloud Workflows · Eventarc triggers · Cloud Scheduler · Data lineage with Dataplex and Data Catalog · Pipeline monitoring (Cloud Monitoring, Cloud Logging)

Hands-On Labs: Deploy Cloud Composer and build a DAG orchestrating Dataflow + BigQuery load · Set up an Eventarc trigger on GCS object finalization · Configure pipeline monitoring alerts

Case Study: Design a fault-tolerant, end-to-end orchestrated daily pipeline for a financial reporting process with retries, alerting, and lineage tracking.

Module 5: Governance, Security & Compliance | 6 hrs

Topics Covered: GCP IAM (roles, custom roles, service accounts, Workload Identity Federation) · VPC Service Controls · Cloud DLP (PII detection, de-identification) · BigQuery column-level security and policy tags · Cloud KMS (CMEK) · Audit logging · GDPR / HIPAA compliance patterns

Hands-On Labs: Configure a Cloud DLP inspection job to detect PII in GCS · Set up BigQuery column-level security with policy tag taxonomy · Create a VPC Service Control perimeter around BigQuery and GCS

Case Study: Implement a data governance framework for regulatory compliance, mapping data lineage from raw GCS ingestion through Dataflow to BigQuery reporting.

Part B — GCP Gen AI for Data Engineers

Module 6: Gen AI Foundations & Gemini on Vertex AI | 6 hrs

Topics Covered: GCP Gen AI landscape (Vertex AI, Gemini API, Model Garden, AI Studio) · Gemini model family and use cases · Vertex AI model deployment (endpoints, predictions, model registry) · Vertex AI SDK and REST API · Prompt design for data engineering · Token management and safety settings · Responsible AI on GCP

Hands-On Labs: Access Gemini via Vertex AI SDK with structured prompts · Use function calling to extract structured schema from pipeline logs · Test multimodal input with a pipeline diagram image

Case Study: Use Gemini on Vertex AI to auto-generate business-readable documentation from a Dataflow pipeline’s DAG definition.

Module 7: Gen AI in GCP Data Pipelines | 8 hrs

Topics Covered: Integrating Gemini into Dataflow pipelines for real-time enrichment · Calling Gemini from Dataproc Spark jobs for batch enrichment · BigQuery ML and ML.GENERATE_TEXT · BigQuery + Vertex AI remote models · Cloud Functions / Cloud Run as AI enrichment microservices · Anomaly detection with Gemini in Dataplex · Cost, latency, and quota management

Hands-On Labs: Build a Dataflow pipeline calling Gemini for real-time classification · Use BigQuery ML.GENERATE_TEXT for in-SQL inference · Deploy a Cloud Run service wrapping Vertex AI for pipeline enrichment

Case Study: Design an AI-enriched pipeline (GCS → Dataflow → Gemini → BigQuery) for auto-categorizing a product catalogue.

Module 8: Embeddings, Vector Search & Vertex AI RAG | 8 hrs

Topics Covered: Embeddings on GCP (text-embedding-004, multimodal embedding models) · Vector search options (Vertex AI Vector Search, AlloyDB pgvector, BigQuery vector search) · Chunking strategies · Vertex AI RAG Engine · Grounding Gemini with Vertex AI Search · Hybrid search (dense + BM25) · Evaluating RAG (faithfulness, relevance, precision) · Vertex AI Agent Builder

Hands-On Labs: Generate embeddings for a document corpus via Vertex AI SDK · Index embeddings into Vertex AI Vector Search and run nearest-neighbour queries · Build a Vertex AI RAG Engine pipeline backed by a GCS corpus

Case Study: Design a RAG pipeline (GCS corpus → Vertex AI Vector Search → Gemini → response) for an enterprise Q&A bot over internal documentation.

Delivery Guidelines

• Designed for Instructor-Led Training (ILT) — in-person classroom or virtual/online delivery
• Hands-on labs use flexible, domain-adaptable datasets
• Case studies are scenario-based and industry-agnostic, adaptable to BFSI, Retail, Healthcare, or Telecom
• Recommended batch size: 15–20 learners for effective labs
• Online delivery uses breakout rooms for labs and screen-share for demos

Tools & Environment

• GCP Console + gcloud CLI
• Cloud Storage + Dataplex
• BigQuery
• Cloud Dataflow
• Cloud Dataproc Serverless
• Cloud Composer
• Vertex AI + Gemini API
• Vertex AI Vector Search
• Cloud DLP + Cloud KMS
• VS Code + Python (Google Cloud SDK)