Partitioning & Sharding
Intermediate · 17 min read · High-Level Design, Storing Data
Splitting data across machines: range vs hash partitioning, hot spots and rebalancing.
Replication copies the same data to many machines. Partitioning — called sharding in most databases — splits different data across machines so each holds only a subset. It's the tool for scaling beyond what one machine can store or write.
You need partitioning when:
Vertical vs horizontal partitioning
Choosing a partitioning strategy
The partition key (shard key) decides which shard a row lives on. The goal: spread data and load evenly, and keep common queries on a single shard.
Choosing a good shard key
A good shard key has:
Hot spots and skew
Even with hashing, a single hot key can overwhelm one shard: a celebrity's profile, a viral post, a flash-sale product.
Rebalancing
As data grows or nodes are added, partitions must move. Goals: keep load fair, move as little data as possible, and keep serving traffic throughout.
Request routing
How does a request find the right shard?
The costs of sharding
Sharding is powerful, but it removes conveniences you had on a single database:
Interview questions
- You shard an orders table by order_id. What problem will you hit?
- How would you handle a hot partition caused by a celebrity account?
- Why create more partitions than nodes?
- How do you enforce a unique email address across a database sharded by user_id?
Practice
Write your answer before opening the suggested one. Anything you mark "Partly" or "Missed it" comes back in your review deck.
Further reading
Key takeaways
- Sharding splits rows across machines to scale storage and writes; combine it with replication for availability.
- Range partitioning favours range queries but risks hot spots; hash partitioning distributes evenly but scatters ranges.
- Pick a shard key with high cardinality, even load, and alignment with your main queries.
- Avoid hash mod N; use fixed logical partitions, dynamic splitting, or consistent hashing for rebalancing.
- Sharding costs cross-shard queries, joins, transactions and global constraints — defer it until required.