"Modern analytics is not one tool — it is a pipeline of decisions. Every enterprise data platform, from the simplest to the most complex, follows the same four-layer pattern. Learn the pattern and you can design any platform. Learn only the tools and you can only copy what already exists."
Parts 1-3 covered the foundational concepts: cloud computing, storage/compute separation, and the three storage architectures. Now we put it all together into the complete end-to-end architecture that enterprises actually build.
The Four-Layer Pattern
Every modern analytics platform — regardless of vendor or scale — follows four layers:
MODERN ANALYTICS ARCHITECTURE — FOUR LAYERS
┌─────────────────────────────────────────────────────────────────┐
│ LAYER 1: INGEST │
│ Move data from source systems into the platform │
│ Tools: Event Hubs (streaming), Data Factory, Dataflow Gen2 │
│ On-premises Gateway, Logic Apps, REST APIs │
└────────────────────────────┬────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────────────┐
│ LAYER 2: STORE │
│ Persist data in an organised, queryable form │
│ Tools: OneLake, ADLS Gen2, Delta Lake tables │
│ Medallion pattern: Bronze → Silver → Gold │
└────────────────────────────┬────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────────────┐
│ LAYER 3: TRANSFORM │
│ Clean, enrich, aggregate data into business-ready form │
│ Tools: Fabric Spark (notebooks), SQL queries, Dataflow Gen2 │
│ dbt, stored procedures in SQL Analytics Endpoint │
└────────────────────────────┬────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────────────┐
│ LAYER 4: SERVE │
│ Deliver data to end consumers in the right format │
│ Tools: Power BI Direct Lake, SQL Analytics Endpoint │
│ APIs, Excel, Fabric notebooks for data science │
└─────────────────────────────────────────────────────────────────┘
Layer 2 in Detail — The Medallion Architecture
The Store layer is not a flat bucket. Modern platforms organise data into three tiers — collectively called the Medallion architecture (Bronze, Silver, Gold).
MEDALLION ARCHITECTURE — INSIDE THE STORE LAYER
BRONZE (Raw Zone) SILVER (Cleansed Zone) GOLD (Business Zone)
───────────────────── ───────────────────────── ──────────────────────
Exact copy of source data Validated, deduplicated Aggregated, enriched
No transformation Standardised formats Joined across domains
Append-only log Referential integrity checked Business logic applied
Retained forever for audit Schema enforced Ready for Power BI
───────────────────── ───────────────────────── ──────────────────────
"What arrived" "What is correct" "What business needs"
Example — Retail Sales:
Bronze: Raw POS transaction Silver: Deduped, valid Gold: Daily_Sales_Summary
JSON from each store amounts, matched by Region, Category
including cancelled, to product catalogue, excl. returns,
test, and error records normalised regions with targets
The key benefit: if a transformation bug corrupts Silver, you re-process from Bronze — the raw data is always there. If a business rule changes, re-process Gold from Silver — no need to re-ingest from source.
Enterprise Scenario — Retail Chain with 200 Stores
A retail chain operates 200 stores across India. Each store has a point-of-sale system generating transactions. Head office needs daily sales dashboards, weekly inventory reports, and monthly P&L — all before business opens each morning.
RETAIL CHAIN — END-TO-END ARCHITECTURE
INGEST
├── 200 POS systems → Azure Event Hubs (streaming, near-real-time)
├── Inventory system REST API → Dataflow Gen2 (hourly batch pull)
├── ERP system (on-premises SAP) → On-premises Gateway + Data Pipeline (nightly)
└── HR system Excel exports → Manual upload to OneLake Files (monthly)
STORE (OneLake — three Medallion zones)
├── Bronze Lakehouse: raw POS events, raw inventory JSON, raw ERP tables, raw Excel
├── Silver Lakehouse: validated sales, matched product IDs, standardised store codes
└── Gold Lakehouse: Daily_Sales, Store_Inventory, Monthly_PnL, Inventory_Alerts
TRANSFORM (overnight Spark jobs, Dataflow Gen2 for simpler logic)
├── Bronze → Silver: Spark notebook — dedup, validate, standardise (runs 1am)
├── Silver → Gold: SQL stored procedure — aggregate by store/region/category (runs 3am)
└── Gold refresh: complete by 5am for 7am dashboard readiness
SERVE
├── Store managers: Power BI dashboard via Direct Lake (Gold layer)
├── Finance team: SQL Analytics Endpoint for ad-hoc P&L queries
├── Supply chain: Automated alert when inventory below threshold (Logic App reads Gold)
└── Exec team: Mobile Power BI app, auto-refreshed
Which Tools Go Where — Microsoft Fabric Mapped to the Layers
Common Misconceptions
- "We should transform data as soon as it arrives" — No. The Bronze layer exists precisely so you do NOT transform immediately. Raw data preserved in Bronze is your safety net. Transform failures are recoverable. If you transform-on-ingest and the logic is wrong, you have lost the original.
- "Gold layer should contain everything" — Gold should contain only what business users consume. Putting all Silver data into Gold defeats the purpose — Gold is curated, not comprehensive. Analysts who need more than Gold should use Silver directly via SQL Analytics Endpoint.
- "Dataflow Gen2 replaces Spark for all transformations" — Dataflow Gen2 is excellent for small-to-medium data with simple transformations. For datasets above hundreds of millions of rows, or complex multi-step transformations, Spark notebooks are more appropriate. Both tools have their place.
- "The architecture must be built all at once" — No. Start with one data source, one Bronze table, one Gold table, one Power BI report. Prove the pattern works. Then scale. Enterprises that try to build all 200 store connections simultaneously before proving the pipeline works reliably always struggle.
🎯 Quick Check
Q1: A data engineer discovers that a Spark transformation introduced a bug that corrupted the Silver layer sales data for the past 3 days. In a Medallion architecture, what is the recovery path?
Show Answer
Re-process Silver from Bronze. Bronze is an append-only raw copy of source data — it is never modified. The engineer fixes the Spark transformation logic, then reruns the Bronze-to-Silver job for the affected 3 days. Bronze data is untouched because Silver corruption never propagates back. This is the core value of the Bronze layer.
Q2: A retail analyst needs to investigate an anomaly in last month's sales data — specifically the raw transaction records before any business rules were applied. Which Medallion layer do they query?
Show Answer
Bronze layer. Bronze contains the exact copy of source data with no transformation — including records that were later filtered out (cancelled transactions, test records, error records). Silver and Gold have already applied business rules. For investigating anomalies at the raw level, Bronze is the only authoritative source.
Q3: Why does the retail chain architecture use Event Hubs for POS data but Dataflow Gen2 for the inventory API, even though both are "data sources"?
Show Answer
Because the data characteristics are different. POS transactions are continuous, high-volume, time-sensitive events that need streaming ingestion (Event Hubs handles thousands of events per second). Inventory data updates hourly via a REST API — batch polling every hour is appropriate and Dataflow Gen2's low-code REST connector handles it without Spark or custom code. Tool selection follows data characteristics, not tool preference.
Key Takeaways — Part 4
- Every modern analytics platform follows four layers: Ingest, Store, Transform, Serve — understanding the pattern matters more than knowing specific tools
- The Medallion architecture organises the Store layer into Bronze (raw), Silver (cleansed), and Gold (business-ready) — each serving a distinct purpose
- Bronze is never modified — it is the safety net that makes transformation failures recoverable
- Tool selection at each layer should match data characteristics (volume, velocity, skill) not technology preference
- Design backwards from the SLA: if dashboards must be ready by 7am, work out what must complete by 5am, 3am, 1am — the schedule drives the architecture