Sharding Strategies

What Is Sharding?

Sharding splits a single database into multiple smaller databases (shards), each containing a subset of the data. Each shard handles a portion of the total load. Sharding is the nuclear option of database scaling — it is powerful but introduces significant complexity.

Hash-Based Sharding

Hash sharding uses a hash function on the shard key to determine which shard stores each row. This provides even distribution but makes range queries expensive because they must span all shards.

def get_shard(user_id: int, num_shards: int) -> int:
    return hash(user_id) % num_shards

# user_id=1001, num_shards=4
# hash(1001) % 4 = 2 → Shard 2

# user_id=1002, num_shards=4  
# hash(1002) % 4 = 0 → Shard 0

# Even distribution, but:
# SELECT * WHERE user_id BETWEEN 100 AND 200
# must query ALL shards

Range-Based Sharding

Range sharding assigns contiguous ranges of the shard key to each shard. User IDs 1-1M go to Shard 0, 1M-2M to Shard 1, etc. This preserves range query performance but can create hot shards if access patterns are skewed.

Shard 0: user_id 1 - 1,000,000
Shard 1: user_id 1,000,001 - 2,000,000
Shard 2: user_id 2,000,001 - 3,000,000
Shard 3: user_id 3,000,001 - 4,000,000

Range query (100-200): Hits only Shard 0 ✓
New users (4M+): All go to Shard 3 → Hot shard ✗

Directory-Based Sharding

Directory sharding uses a lookup table to map each key to a shard. This provides maximum flexibility — you can move individual keys between shards — but the directory itself becomes a single point of failure and bottleneck.

Resharding Without Downtime

Resharding is the process of changing the number of shards while the system is running. This is one of the hardest operations in distributed systems. The safest approach is consistent hashing, which minimizes data movement when shards are added or removed.

1. Add new empty shards alongside existing ones.
2. Begin dual-writing — write to both old and new shards.
3. Backfill existing data to new shards.
4. Switch reads to the new shard topology.
5. Stop writing to old shard positions.
6. Remove old shards after verification.

Common Pitfalls

• Cross-shard joins — Queries that join data across shards are expensive and complex.
• Distributed transactions — Two-phase commit across shards adds latency and complexity.
• Hot shards — Uneven data distribution causes some shards to handle more load.
• Shard key selection — Choosing the wrong shard key causes irreparable problems.
• Global uniqueness — Auto-increment IDs across shards require coordination.