GCP
Cloud Platform
BigQuery
Data Warehouse
Cloud Run
ETL Execution
Terraform
IaC
Production GCP data engineering pipeline using BigQuery as the warehouse and Cloud Run for containerized ETL execution. Pulls data from TheCocktailDB API, transforms it through a layered BigQuery schema, and deploys via Cloud Run with Terraform infrastructure.
GCP production stack: Cloud Run + BigQuery + Terraform + CI/CD. Real infra, not a local notebook.
Tech Stack
PythonGCPBigQueryCloud RunTerraformDockerCloud Storagedbt
Features
Layered Schema
Raw → staging → marts BigQuery schema. dbt transformations with full lineage.
Cloud Run ETL
Containerized Python ETL jobs run on Cloud Run — serverless, scalable, zero idle cost.
BigQuery Warehouse
Partitioned and clustered BigQuery tables for cost-efficient analytics queries.
Terraform IaC
All GCP resources defined in Terraform. Reproducible infra from scratch in minutes.
How It Works
01
Extract
TheCocktailDB API → Cloud Run job → Cloud Storage landing zone
→
02
Load
GCS → BigQuery raw tables with schema enforcement
→
03
Transform
dbt models: staging → intermediate → marts
→
04
Deploy
Terraform provisions all GCP resources. Cloud Run handles execution.