Study ref. DP-01
Data Pipeline Migration
Medallion Architecture & Schema-Driven Validation at Scale
The Situation
A B2B analytics platform was ingesting data from over 40 upstream sources via a single Python ETL monolith that had grown organically over four years. Source schemas were embedded as string literals inside transformation functions. There were no data contracts, no intermediate validation layers, and no visibility into where a record failed and why.
The consequence was a 6–8% daily pipeline failure rate — each failure requiring manual triage to locate the offending source row. A schema change upstream (a column renamed, a nullable field going non-null) would silently corrupt downstream aggregations for hours before an alert fired on the BI layer. Consumer teams had learned not to trust the platform for anything time-sensitive.
Constraints
- Live production system — no maintenance windows available
- Upstream sources owned by separate teams; schema changes happen without notice
- Existing consumers depend on current output table structure (cannot rename columns)
- Small data engineering team: three engineers, none with prior lakehouse experience
System Evolution
Left: raw-to-gold ETL monolith with no intermediate validation — schema errors propagate silently to consumers. Right: Medallion target state with Bronze landing, Silver validation, and Gold aggregation as distinct, observable stages.
The Medallion boundary guarantee: Gold consumers never touch a row that has not passed the Silver schema contract. A Bronze-to-Silver failure quarantines the offending records and pages the on-call engineer with the exact row, source, and field that violated the contract.
Migration Strategy: Parallel Write
Rather than cutting over sources one-by-one and risking consumer disruption, the new Medallion pipeline was built alongside the existing ETL. For the first four weeks, both pipelines wrote to separate output tables. A data reconciliation job ran nightly, comparing row counts and aggregate sums across 14 key metrics.
This approach surfaced three categories of silent data discrepancy in the old pipeline — differences the consumers had accepted as "normal variance." Consumer teams were briefed on the corrected figures before cutover, avoiding the appearance of a regression when accuracy improved.
Records flow Source → Bronze → Schema Gate → Silver → Gold; a fraction of each batch fails the gate and deflects into quarantine with a live pass-rate readout — the schema contract enforced in motion.
Schema Contract Design
Each of the 40+ sources received a Pandera schema definition checked into version control. Schema files specify field types, nullable constraints, value ranges, and cross-field invariants. The Bronze-to-Silver transition validates every incoming batch against the registered schema version for that source.
Schema versioning follows a two-step promotion process: a PR to register the new schema version, then a feature flag to activate it on the next batch run. This means upstream schema changes are visible in git history and always intentional — not discovered via downstream failures.
Pure Transformation Functions
All Silver-layer transformations were rewritten as pure functions: deterministic, no side effects, no database reads mid-transform. Given the same input batch, the same output is produced. This made the transformation logic trivially unit-testable without fixtures, mocks, or running infrastructure.
The test suite for transformation logic runs in under 30 seconds and covers 600+ property-based test cases generated from schema definitions. Any change to a transformation function requires a corresponding test that exercises the affected schema constraints.
Observability First
Each layer emits structured log events with a batch ID, row counts, validation failure counts, and latency. A lightweight Grafana dashboard was built before cutover — not after — so the team had baseline metrics from week one of the migration. The 12-minute mean time to failure isolation metric was achievable because operators could pinpoint the exact Bronze→Silver transition for any suspect batch without querying the data itself.
Pipeline failure reduction
6–8% daily failure rate → under 0.5% within 6 weeks of cutover
Mean time to failure isolation
Down from ~3 hours of manual triage per incident
Silent corruption incidents
Schema gate prevents invalid rows from ever reaching Gold
Sources under contract
Every source has a versioned schema definition in git
What I Would Do Differently
The parallel-write migration approach was the right call for data correctness, but it ran for seven weeks — three weeks longer than planned — because three source teams delayed schema registration while their own systems were in flux. In hindsight, schema registration should have been a hard pre-requisite for source onboarding with a published cutover date, rather than a soft deadline that kept slipping.
The Grafana dashboard was built manually and has since accumulated metric debt — adding a new source requires a manual dashboard update. Replacing it with auto-generated dashboards from the schema registry (each registered source automatically gets a row-count and failure-rate panel) would have removed the operational overhead and been achievable within the original project scope.