"Scalability is not a feature you add. It is a property of decisions you make at design time — about how you partition data, how you size compute, and how you separate concerns. Get those decisions right and the platform scales invisibly. Get them wrong and you rebuild."
Two Types of Scaling — Vertical and Horizontal
Before discussing Fabric capacity units or Delta partitioning, you need to understand the two fundamental approaches to scaling:
VERTICAL SCALING (Scale Up)
─────────────────────────────────────────────────────────────────
One machine. Make it bigger.
Small server → Large server
4 CPU, 16GB RAM 64 CPU, 512GB RAM
Advantages: Simple. No code changes. Works for most workloads.
Limits: Maximum machine size exists. Downtime to resize.
Cost grows non-linearly — biggest machines expensive.
One machine = single point of failure.
In Fabric: Increasing Fabric capacity (F2 → F64 → F256)
More CUs = more concurrent queries handled
HORIZONTAL SCALING (Scale Out)
─────────────────────────────────────────────────────────────────
One machine becomes many. Distribute the work.
1 server → 10 servers (cluster)
each handling 1/10 of the data
Advantages: No theoretical limit. Add more nodes as needed.
No single point of failure (one node down ≠ outage).
Cost grows linearly with scale.
Limits: Data must be splittable across nodes (partitioning).
Coordination overhead for joins, aggregations.
In Fabric: Spark clusters (notebooks run distributed across nodes)
Delta table partitioning enables parallel reads
Fabric Capacity Units (CUs) — What You Are Actually Buying
When you provision Microsoft Fabric capacity, you are buying CUs (Capacity Units). Understanding what CUs represent prevents both under-provisioning (platform too slow) and over-provisioning (budget wasted).
FABRIC CAPACITY SIZES AND WHAT THEY MEAN
SKU CUs Rough guidance (not official limits)
──── ──── ─────────────────────────────────────────────────────
F2 2 Development, small pilots — limited concurrent users
F4 4 Small team BI (5-10 active users, light ETL)
F8 8 Medium team BI (10-25 users, daily pipelines)
F16 16 Departmental BI (25-50 users, multiple pipelines)
F32 32 Enterprise BI (50-100 users, complex transforms)
F64 64 Large enterprise (100+ users, heavy Spark workloads)
F128 128 Very large enterprise, real-time streaming + BI
F256 256 Largest enterprises, petabyte-scale workloads
CUs are SHARED across all workloads in the capacity:
- Power BI report rendering uses CUs
- Dataflow Gen2 transformations use CUs
- Spark jobs use CUs
- Pipelines use CUs
If all workloads run simultaneously → CUs are shared between them
If only reports run → all CUs available for report queries
Enterprise Scenario — Healthcare Analytics at 50 Million Records
A hospital network has 50 million patient encounter records spanning 10 years. The analytics team runs daily reports on recent encounters (last 90 days), monthly reports on quarterly trends, and annual compliance reports on full history.
Without partitioning, every query scans all 50 million records even when it only needs the last 90 days. With correct partitioning:
DELTA TABLE PARTITIONING — HEALTHCARE SCENARIO
WITHOUT PARTITIONING:
Query: "Show all encounters in Q3 2025"
Delta table scans: ALL 50M records → filters to Q3 2025
Files read: All Parquet files in the table
Query time: Long. Cost: High.
WITH PARTITIONING BY YEAR AND MONTH:
Delta table physical layout:
/encounters/year=2024/month=01/*.parquet (Jan 2024 data)
/encounters/year=2024/month=02/*.parquet (Feb 2024 data)
...
/encounters/year=2025/month=07/*.parquet (Jul 2025 data)
/encounters/year=2025/month=08/*.parquet (Aug 2025 data)
/encounters/year=2025/month=09/*.parquet (Sep 2025 data)
Query: "Show all encounters in Q3 2025"
Delta table reads: ONLY year=2025/month=07, 08, 09 folders
Files read: 3 months of data out of 120 months total
Query time: ~40x faster. Cost: ~40x less compute.
This is partition pruning — Spark/SQL skips partitions that
cannot contain the query result. Zero data scanned unnecessarily.
The key decision: partition by columns that appear in WHERE clauses most often. For time-series data, date-based partitioning (year, month) is almost always correct. For multi-region data, partitioning by region is common. Never partition by high-cardinality columns (patient ID has 50M unique values — partitioning by it creates 50M folders, which is worse than no partitioning).
Scaling Patterns — When to Use Which
Common Misconceptions
- "More CUs always makes everything faster" — CUs help with concurrency and parallel workloads. A poorly written query that scans 50 million rows unnecessarily will still scan 50 million rows on F256. The fix is query optimisation and partitioning, not more CUs.
- "Partition by every column you filter on" — Over-partitioning is a real problem. If a Delta table has 1 billion rows partitioned by year, month, day, region, and product category — you may have millions of tiny files. Spark performs better with fewer, larger files than millions of tiny ones. Partition by 1-2 high-impact columns only.
- "Scaling is an infrastructure problem, not a data problem" — Most analytics performance problems are data design problems — bad partitioning, missing indexes, poor schema design, unoptimised queries. Infrastructure scale is expensive. Data design fixes are free.
- "We need to provision for peak load permanently" — Fabric capacity can be scaled up and down. Provision for normal operations. Scale up temporarily for known peak periods (month-end, quarter-close). Scale back down. This is the cost advantage of cloud.
🎯 Quick Check
Q1: A hospital analytics team notices their Spark job processing 50 million records takes 3 hours every morning, even though only 200,000 new records were added the previous day. What is the most likely cause and fix?
Show Answer
The likely cause is full table scan with no incremental processing. The Spark job reads all 50 million records every run instead of only the new ones. Two fixes needed: (1) Partition the Delta table by date so only recent partitions are scanned, (2) Use incremental processing — filter WHERE encounter_date = yesterday rather than reading the entire table. After these changes, the job should process only 200,000 records, reducing runtime from 3 hours to potentially minutes.
Q2: A team wants to partition their patient encounters Delta table to speed up queries. They are considering partitioning by: (A) year+month, (B) patient_id, or (C) diagnosis_code. Which is correct and why?
Show Answer
Option A — year+month. Most analytics queries filter by time range. Year+month partitioning means time-range queries only read relevant partitions. Patient_id is high-cardinality (50M unique values) — partitioning by it creates 50M tiny folders, severely degrading performance. Diagnosis_code has thousands of values and queries rarely filter by a single diagnosis alone — poor partition key. Always partition by columns that appear in the most frequent WHERE clauses and have reasonable cardinality (not too high, not too low).
Q3: A Fabric workspace is provisioned at F32. During a quarter-end period, Power BI reports become slow and Spark jobs fail due to capacity throttling. What is the recommended approach?
Show Answer
Temporarily scale up the Fabric capacity to F64 or F128 for the quarter-end period, then scale back to F32 afterwards. Fabric capacity can be changed without downtime. Permanently running at F64 would cost roughly double the F32 cost for a 3-day spike that happens four times a year. The elastic scaling model exists precisely for this scenario — pay for peak capacity only during peak periods.
Key Takeaways — Part 6
- Vertical scaling (more CUs) addresses concurrency — more users, more simultaneous queries. Horizontal scaling (Spark clusters, partitioning) addresses data volume — faster processing of large datasets
- Fabric Capacity Units are shared across all workloads in the capacity — right-size from the Capacity Metrics app, not guesswork
- Delta table partitioning is the single highest-impact performance optimisation for large analytical datasets — partition by columns in the most frequent WHERE clauses
- Over-partitioning (too many small files) is as harmful as no partitioning — aim for 1-2 high-impact partition columns
- Scale Fabric capacity temporarily for known peak periods rather than permanently provisioning for peak load