Pattern ◆◆◆◆◆ DP.001
Medallion Architecture
Progressively refine raw ingested data through Bronze, Silver, and Gold layers, each adding quality and semantic richness.
Open detail chart → Domain chart · iii
Data Pipeline
the Data Confluence
Medallion architecture, schema-driven validation, pure functional pipelines, and reliable data transformation patterns.
Celestial position
Featured visualisation — medallion refinement flow
Bronze → Silver → Gold
Raw records enter as Bronze, then refine through Silver into Gold. Each stage tightens the lane and raises quality; records failing validation are rejected and fade away. Quality rises left to right.
bronze · raw ingest
silver · validated
gold · curated
× · rejected
Featured visualisation — batch vs streaming processing
Batch vs Streaming
The top lane accumulates records into a bounded buffer then flushes the entire window in one pass — high throughput, higher latency. The bottom lane emits each event to a processor the moment it arrives — low latency, continuous. Choose based on your tolerance for staleness vs. per-event cost.
batch · windowed accumulation
stream · per-event flow
Featured visualisation — schema-driven validation flow
Schema-Driven Validation
Each record passes through a schema validator before entering the curated layer. Conforming records advance to the output; non-conforming records are deflected and tagged with the violation — null reference, type mismatch, missing field. The validator is the contract boundary.
record · approaching validator
schema pulse · validation event
pass · conforms to schema
reject · constraint violation
Featured visualisation — pure function composition pipeline
Pure Function Pipeline
Each stage is a pure function: same input always yields the same output, no hidden state, no side effects. Records flow Parse → Validate → Normalize → Enrich → Aggregate. Non-conforming records are rejected at Validate and deflect to the error bin — all other stages are guaranteed safe.
raw record · pre-parse
parsed · structured tokens extracted
validated · schema conformant
normalized · canonical form
enriched · context appended
aggregated · curated output
rejected · constraint violation
Skill progression — Data Pipeline Architecture
Pipeline Proficiency
Pure functions and schema contracts underpin every higher-order pattern. Select any node to see how skills build on one another.
Skill progression4 / 6
Select a skill to inspect
Pattern catalogue — Data Pipeline
Pattern ◆◆◇◇◇ DP.002
Schema-Driven Validation
Define and enforce data contracts at pipeline boundaries to catch structural violations before they propagate downstream.
Open detail chart → Principle ◆◇◇◇◇ DP.003
Pure Functions
Build transformation logic from functions that depend only on their inputs, enabling deterministic testing and safe composition.
Open detail chart → Pattern ◆◆◆◇◇ DP.004
Batch vs Streaming
Choose between bounded batch processing and unbounded stream processing based on latency, throughput, and consistency requirements.
Open detail chart →