🚀 Why Do We Even Need Hashing?

In distributed systems, we need to:

• Store data across multiple servers

• Retrieve it efficiently

A simple approach is:

server = hash(userId) % number_of_servers

✅ Benefits:

• Same input → same server (deterministic)

• Easy read/write mapping

---

⚠️ Problem with Simple Hashing

Let’s say:

userId % 3 → Server A

Now we add one more server:

userId % 4 → Server B

👉 Same user now maps to a different server

❌ Issues:

• Almost all keys get reassigned

• Massive data redistribution

• Not scalable

---

💡 Enter Consistent Hashing

Instead of % N, we:

• Map both servers and keys onto a ring

• Hash space typically ranges from 0 → 2^32

Think of it like a clock 🕒

---

🔄 How Consistent Hashing Works

1. Hash each server → place it on the ring

2. Hash each key → place it on the ring

3. Assign key to:

👉 First server in clockwise direction

---

🧠 Example

• Key → 250

• Servers:

  • S1 → 100

  • S2 → 200

  • S3 → 300

👉 Key (250) will be assigned to S3

---

➕ Adding a New Server

Suppose we add S3 between S2 and S4

Before:

• Keys in range (S2 → S4] → stored in S4

After:

• Keys in range (S2 → S3] → move to S3

👉 Only a small subset of keys move

---

➖ Removing a Server

If a server is removed:

• Its keys move to the next clockwise server

👉 Again, minimal data movement

---

🔥 Why Consistent Hashing is Powerful

Feature	Normal Hashing	Consistent Hashing
Add Server	Rehash almost all data ❌	Minimal movement ✅
Remove Server	Rehash almost all data ❌	Minimal movement ✅
Scalability	Poor ❌	Excellent ✅

---

⚖️ Virtual Nodes (VNodes) — Solving Uneven Load

🚨 Problem Without Virtual Nodes

If each server is placed only once:

S1 ---- S2 - S3 - S4 ---------------------- S1

👉 Large gaps can form on the ring

❌ Result:

• Some servers handle huge portions of data

• Load becomes uneven

---

💡 Solution: Virtual Nodes

Instead of placing each server once:

👉 Each server is placed multiple times on the ring

Example:

S1 → hash("S1-1"), hash("S1-2"), hash("S1-3")
S2 → hash("S2-1"), hash("S2-2"), hash("S2-3")

---

🔑 How Virtual Nodes Are Placed

Virtual nodes are created using:

• Same hash function

• Different identifiers

Example:

hash("S1-1")
hash("S1-2")
hash("S1-3")

✔ Deterministic

✔ Evenly spread

✔ Looks random but predictable

---

🔄 How Distribution Improves

Instead of one large chunk per server:

S1   S2   S3   S1   S4   S2   S3   S1   S4 ...

👉 Each server now owns multiple small ranges

✅ Result:

• No large gaps

• Better load balancing

---

🔥 Handling Hotspots

What is a Hotspot?

When:

• A particular key gets huge traffic

• Example: celebrity user, viral content

---

❌ Without Virtual Nodes:

• All traffic → one server
  👉 Server overload

---

✅ With Virtual Nodes:

• Data is distributed across multiple positions

• Requests are more evenly spread

👉 Load gets distributed across cluster

---

⚠️ Important Clarification

Virtual nodes:

Do NOT completely eliminate hotspots

They:

• Reduce uneven distribution

• Improve probability of balanced load

For real-world systems, we also use:

• Replication

• Caching (Redis, CDN)

• Load balancing

---

🌍 Real-World Systems Using This

• Cassandra

• DynamoDB

• Redis Cluster

• CDN systems

• Load balancers

---

🧩 Final Summary

Consistent hashing:

• Distributes data across servers efficiently

• Minimizes data movement during scaling

• Handles dynamic server addition/removal

• Works even better with virtual nodes

---

✍️ My Take

Initially, hashing felt like:

userId % N

But that approach breaks at scale.

Consistent hashing solves this elegantly by:

• Using a ring-based model

• Minimizing data reshuffling

• Enabling scalable distributed systems

---

🎯 When Should You Use This?

• Distributed caches

• Database sharding

• Load balancing

• CDN routing

---

🏁 Closing Thought

Consistent hashing is not just an optimization —

it’s what makes large-scale distributed systems practical.