System Design

Generating Unique IDs in Distributed Systems

A database auto-increment column gives you perfect unique IDs - right up until you have more than one database. Then it quietly becomes the thing that can't scale.

By Admin2026-06-1810 min readunique-idsdistributed-systemssnowflakeuuidscalabilitysystem-design

At Razorpay, every transaction needs an ID. Billions of them, never colliding, generated across many servers in many places at once. We started the way almost everyone starts: a database AUTO_INCREMENT column. The database hands out 1, 2, 3, 4… guaranteed unique, perfectly ordered, zero effort.

It worked beautifully until it didn't. As traffic grew and we needed more than one database - replicas, then shards across regions - that single auto-increment counter became a problem. Every ID had to come from one place, so that one place became a bottleneck and a single point of failure for creating anything at all. The simplest possible solution had quietly become the thing standing between us and scale.

Generating unique IDs in a distributed system is one of those problems that sounds trivial and isn't. Here's the full picture, from the obvious answer to the one most large systems actually use.

Why the obvious answer breaks

A single auto-incrementing counter has three properties we love: unique, sortable (ID order = creation order), and compact (a small integer). The trouble is the word single.

The moment your data lives on more than one machine, that counter has to be coordinated. Either every server asks one central authority for the next number - which serializes all your writes through a single chokepoint - or each server keeps its own counter, in which case two servers will both hand out "1,247" and you have a collision. At scale, you can't have one global counter without it becoming a bottleneck, and you can't have many independent counters without collisions. That contradiction is the whole problem.

So we need IDs that any server can generate on its own, with no coordination, that are still guaranteed unique across the entire fleet.

Option 1 - UUIDs

The first real escape is the UUID: a 128-bit random number. A version-4 UUID is essentially 122 random bits, and the space is so vast that the odds of two servers ever generating the same one are negligible. No coordination, no central authority - every server just rolls the dice.

Pros:
- Fully decentralized - any server generates one instantly, alone
- Effectively zero collision probability
- Dead simple, available in every language's standard library

Cons:
- Large: 128 bits (16 bytes), vs 8 bytes for a bigint - bloats every row and index
- NOT sortable - random IDs destroy database index locality
- Random inserts fragment B-tree indexes, hurting write performance

That "not sortable" point is subtler than it looks and it bit us. When IDs are random, new rows insert at random positions in the database's index, scattering writes across the whole structure instead of appending neatly at the end. On a high-write system, that fragmentation is a real, measurable performance cost. UUIDs solve uniqueness but sacrifice the ordering and compactness we had with auto-increment.

Option 2 - Snowflake IDs (the one big systems reach for)

This is the design that originated at Twitter and now shows up everywhere, because it threads the needle: decentralized like a UUID, but compact and time-sortable like an auto-increment. The trick is to construct a 64-bit ID out of meaningful parts instead of rolling it randomly.

The insight is elegant. The timestamp makes IDs roughly sortable by time (great for database locality - newer IDs are larger, so they append cleanly). The machine ID guarantees two different servers never collide, because their IDs differ in those bits no matter what. The sequence number lets a single machine generate thousands of IDs within the same millisecond without colliding with itself. No coordination required - each server only needs to know its own machine ID, assigned once at startup.

import time

class Snowflake:
    def __init__(self, machine_id):
        self.machine_id = machine_id & 0x3FF      # 10 bits
        self.seq = 0
        self.last_ms = -1
        self.epoch = 1609459200000                # custom epoch (2021-01-01)

    def next_id(self):
        ms = int(time.time() * 1000)
        if ms == self.last_ms:                    # same millisecond
            self.seq = (self.seq + 1) & 0xFFF     # 12-bit sequence, wraps at 4096
            if self.seq == 0:                     # ran out this ms: wait for next
                while ms <= self.last_ms:
                    ms = int(time.time() * 1000)
        else:
            self.seq = 0
        self.last_ms = ms
        return ((ms - self.epoch) << 22) | (self.machine_id << 12) | self.seq

gen = Snowflake(machine_id=7)
ids = [gen.next_id() for _ in range(5)]
print("Generated:", ids)
print("All unique? ", len(set(ids)) == len(ids))
print("Time-sorted?", ids == sorted(ids))

Run it: five IDs, all unique, all in ascending order - and the machine generating them never talked to any other server. That's the property that made our scaling problem disappear.

The catch: Snowflake depends on the clock

Nothing is free. Snowflake's superpower - the timestamp - is also its weakness. The IDs are only correct if the machine's clock moves forward. If a server's clock drifts and gets corrected backwards (NTP adjustments do this), it could generate a timestamp it has already used and risk a collision. Real implementations guard against this by detecting backwards clock movement and either waiting or refusing to issue IDs until time catches up.

Warning: "Clock moving backwards" sounds like a non-problem until it takes down ID generation in production. Any time-based ID scheme inherits the messy reality of clock synchronization across machines. If you propose Snowflake in an interview, mentioning the clock-skew failure mode unprompted is a strong signal of depth.

Choosing in an interview

Interview: The trap here is reaching for the fanciest option by reflex. If the interviewer's system has one database and modest traffic, "I'd just use the database's auto-increment" is the correct, senior answer - and proposing Snowflake would be over-engineering. The skill is matching the ID strategy to the actual scale, then naming the trade-off. Start simple; reach for Snowflake only when the scale demands it.

We reached for Snowflake when the auto-increment counter genuinely became our bottleneck - not before. That's the real lesson hiding inside a "generate unique IDs" question: it's never about the cleverest scheme, it's about reading the scale correctly and paying only the complexity you need. We work through exactly this kind of "start simple, escalate deliberately" reasoning across System Design Fundamentals.

Go deeper

• Universally Unique Identifier (UUID) - the formats, versions, and trade-offs.
• Snowflake ID - the bit-layout design and its variants across companies.
• Twitter's original Snowflake - the archived source that started the pattern.
• Designing Data-Intensive Applications - deeper treatment of ordering, time, and identity in distributed systems.

More articles