How Discord Stores Billions of Messages Without Going Broke?

May 15, 2026

Discord Engineering

engineering discord messaging scalability

A Database on Fire

It’s November 2015. Discord has just crossed 100 million stored messages. The team is celebrating growth, but their on-call engineer is watching something disturbing: MongoDB response times are creeping up. Not crashing — just creeping. A slow, grinding degradation that gets worse every week.

By 2017, with millions of users and thousands of active servers, reads that should take milliseconds are taking seconds. Hot servers — think a 200,000-member gaming community where everyone’s spamming during a game launch — are causing the database to thrash. The engineering team isn’t fighting a bug. They’re fighting the fundamental shape of their data.

This is the story of how they fixed it, and why the fix is more interesting than you’d expect.

Part 1: Why MongoDB Made Perfect Sense (Until It Didn’t)

Before judging the choice, let’s understand the context.

In 2015, Discord was a small startup trying to find product-market fit. MongoDB is a reasonable default for this phase: flexible schema means you don’t need to know exactly what a “message” looks like yet. You might add reactions, embeds, attachments, slash commands — the schema will shift. MongoDB handles that without migrations. You can move fast.

A message object looks reasonable in MongoDB’s document model:

{
  "_id": "snowflake-id-12345",
  "content": "gg no re",
  "author_id": 9876,
  "channel_id": 555,
  "guild_id": 111,
  "timestamp": "2015-11-01T12:34:56Z",
  "attachments": [],
  "reactions": {}
}

Each message is a document. You index on channel_id + created_at so you can query “all messages in this channel, newest first.” The classic pattern.

The problem is that this works fine at tens of millions of documents. At billions, the wheels come off.

Part 2: The Three Ways MongoDB Broke

2.1 Random I/O Is a Killer

MongoDB’s storage engine (WiredTiger) uses a B-tree index structure. Think of it as a sorted tree where every node has a pointer to where data lives on disk. To read messages for a channel, MongoDB walks this tree, finds matching document IDs, then fetches each document from its physical location.

Here’s the problem: messages for a given channel are not stored near each other on disk. As documents were inserted over time, they landed wherever there was space. Reading “the last 50 messages in channel X” might require jumping to 50 different physical locations on disk.

At small scale, this is fine — it’s all cached in RAM. But at Discord’s scale, the working set (data accessed frequently) stopped fitting in RAM. Every cache miss now means a disk seek. And with millions of users across thousands of channels all active simultaneously, cache misses pile up fast.

flowchart TD
    Q["Query: Get messages in channel #general"] --> IDX["B-Tree Index Walk"]
    IDX --> D1["Disk seek → Document at block 1,024"]
    IDX --> D2["Disk seek → Document at block 45,891"]
    IDX --> D3["Disk seek → Document at block 102,334"]
    IDX --> D4["Disk seek → Document at block 8,776"]
    IDX --> D5["50 more seeks..."]

    style D1 fill:#ef4444,color:#fff
    style D2 fill:#ef4444,color:#fff
    style D3 fill:#ef4444,color:#fff
    style D4 fill:#ef4444,color:#fff
    style D5 fill:#ef4444,color:#fff
    style Q fill:#3b82f6,color:#fff
    style IDX fill:#f59e0b,color:#fff

2.2 The Long Tail Kills You

Chat data has a nasty distribution: a small number of massive, highly active servers generate most of the traffic. But there are also millions of tiny servers — a friend group of 5 people, a school project, a hobbyist community — that send messages infrequently and then come back months later.

When someone opens a channel they haven’t visited in 8 months, that data is almost certainly cold — nowhere near the RAM cache. MongoDB has to go to disk. Multiply this across millions of users doing exactly this at any given moment, and you have constant, unpredictable disk pressure from what appears to be “quiet” data.

2.3 Writes Aren’t Free Either

Discord is fundamentally a write-heavy system. Every keystroke (in some UX states), every message sent, every reaction, every edit — all writes. MongoDB updates require finding the document in the B-tree, potentially moving it if it grew in size, and updating multiple index entries. Under sustained write load from millions of concurrent users, this becomes expensive.

The MongoDB setup was showing latency spikes during peak hours that no amount of tuning could fully resolve. The root cause wasn’t configuration — it was the fundamental mismatch between MongoDB’s data layout and Discord’s access patterns.

Part 3: The Right Tool for Chat

When Discord’s team sat down to evaluate alternatives, they needed to answer a few questions clearly:

What does “reading messages” actually look like? Answer: always by channel, always in time order.
What does the write pattern look like? Answer: append-only, high frequency, rarely updated.
What happens to old data? Answer: it still gets read — someone scrolls back, or a bot searches history.

This pattern has a name: time-series with grouped access. And the data structure that handles it well is a wide-column store, with Cassandra (and its faster C++ rewrite, ScyllaDB) being the most battle-tested examples.

graph TB
    subgraph MongoDB["MongoDB Architecture (Before)"]
        direction TB
        APP1["App Layer"] --> MONGO["Single MongoDB\nReplica Set"]
        MONGO --> SHARD1["Shard 1\nRandom docs"]
        MONGO --> SHARD2["Shard 2\nRandom docs"]
        MONGO --> SHARD3["Shard 3\nRandom docs"]
        note1["❌ Random I/O on reads\n❌ Hot shards on popular guilds\n❌ Working set > RAM = thrash"]
    end

    subgraph Cassandra["Cassandra / ScyllaDB Architecture (After)"]
        direction TB
        APP2["App Layer"] --> COORD["Coordinator Node\n(routes by partition key)"]
        COORD --> N1["Node 1\nchannel_id: A–F"]
        COORD --> N2["Node 2\nchannel_id: G–M"]
        COORD --> N3["Node 3\nchannel_id: N–Z"]
        note2["✅ Channel data co-located\n✅ Sequential reads per partition\n✅ Linear horizontal scale"]
    end

    style note1 fill:#fef2f2,stroke:#ef4444
    style note2 fill:#f0fdf4,stroke:#22c55e
    style MongoDB fill:#fff7ed,stroke:#f97316
    style Cassandra fill:#eff6ff,stroke:#3b82f6

Part 4: How Cassandra Actually Organizes Data

Understanding why Cassandra solves Discord’s problem requires understanding how it stores data differently.

The Wide-Column Model

A traditional relational table is a grid: rows and columns, every row has the same columns. A wide-column store is more like a sorted map of maps. Each partition holds a set of rows that are physically stored together and sorted by a clustering key. The magic is in that word physical — rows in the same partition live next to each other on disk.

Discord’s message table looks roughly like this:

CREATE TABLE messages (
    channel_id   bigint,
    bucket       int,         -- time bucket (groups ~10 days of messages)
    message_id   bigint,      -- Snowflake ID (encodes timestamp)
    author_id    bigint,
    content      text,
    attachments  list<text>,
    PRIMARY KEY ((channel_id, bucket), message_id)
) WITH CLUSTERING ORDER BY (message_id DESC);

Two things to notice:

(channel_id, bucket) is the partition key. This means all messages from the same channel, within the same time window, land on the same node and are stored sequentially on disk together.
message_id is the clustering key, ordered descending. Message IDs are Discord Snowflakes — 64-bit integers with the timestamp embedded. Sorting by message_id descending means the newest messages are physically first in the partition, which is exactly how you read them.

graph TB
    subgraph PartitionA["Partition: (channel_id=555, bucket=2024_01)"]
        direction TB
        R1["message_id: 9999 | content: 'gg no re'"]
        R2["message_id: 9998 | content: 'let's go!'"]
        R3["message_id: 9997 | content: 'ready?'"]
        R1 --> R2 --> R3
    end

    subgraph PartitionB["Partition: (channel_id=555, bucket=2023_12)"]
        direction TB
        R4["message_id: 5000 | content: 'happy new year'"]
        R5["message_id: 4999 | content: '3..2..1..'"]
        R4 --> R5
    end

    subgraph PartitionC["Partition: (channel_id=999, bucket=2024_01)"]
        direction TB
        R6["message_id: 8800 | content: 'different channel'"]
    end

    QUERY["Query: Latest messages in channel 555"]
    QUERY -->|"Read partition (555, 2024_01)"| PartitionA
    QUERY -..->|"If user scrolls back far\nread (555, 2023_12)"| PartitionB

    style PartitionA fill:#eff6ff,stroke:#3b82f6
    style PartitionB fill:#eff6ff,stroke:#3b82f6
    style PartitionC fill:#f5f5f5,stroke:#9ca3af
    style QUERY fill:#7c3aed,color:#fff

Now compare this to MongoDB’s approach. To read the last 50 messages in a channel from MongoDB, the engine walks a B-tree and fetches documents from 50 potentially scattered locations. With Cassandra, it reads one contiguous range from one partition — a single sequential scan. For spinning disks, this is the difference between 50ms and 2 seconds. Even on SSDs, it matters at scale because of I/O parallelism and page cache efficiency.

The Bucket: Keeping Partitions Healthy

You might wonder: why not just use channel_id alone as the partition key? Why the bucket?

Without bucketing, a channel that has been active for 5 years accumulates all its messages in one ever-growing partition. In Cassandra, unbounded partition growth is a well-known problem — it causes memory pressure, slower reads, and uneven distribution across nodes. A 3TB partition is nobody’s idea of fun.

The bucket solves this by capping partition size. Discord derives the bucket from the message’s timestamp, so all messages within roughly the same 10-day window land in the same partition. Older messages live in older buckets. The application layer knows to query bucket N, then bucket N-1, etc., when a user scrolls back in history.

Part 5: Why Writes Are Cheap in Cassandra

Cassandra’s write path is fundamentally different from MongoDB’s, and it’s why it handles Discord’s write volume well.

Under the hood, Cassandra uses a Log-Structured Merge Tree (LSM Tree) rather than a B-tree.

flowchart LR
    W["New Write\n(message)"] --> MEM["MemTable\n(in memory, sorted)"]
    MEM -->|"When full\nflush to disk"| SS1["SSTable 1\n(immutable)"]
    MEM -->|"Next flush"| SS2["SSTable 2\n(immutable)"]
    MEM -->|"Next flush"| SS3["SSTable 3\n(immutable)"]

    SS1 --> COMPACT["Compaction\n(merge + sort)"]
    SS2 --> COMPACT
    SS3 --> COMPACT
    COMPACT --> FINAL["Merged SSTable\n(optimized for reads)"]

    READ["Read Request"] --> BF["Bloom Filter\n(is this key here?)"]
    BF -->|"Probably yes"| SS1
    BF -->|"Probably yes"| SS2
    BF -->|"Probably yes"| SS3

    style W fill:#7c3aed,color:#fff
    style MEM fill:#3b82f6,color:#fff
    style COMPACT fill:#f59e0b,color:#fff
    style FINAL fill:#22c55e,color:#fff
    style READ fill:#ef4444,color:#fff

Here’s the core insight: writes never go directly to disk in a random location. They land in memory first (the MemTable), sorted by partition and clustering key. When the MemTable fills up, it’s flushed to disk as an SSTable — a sorted, immutable file. SSTable writes are purely sequential: you’re just appending a sorted stream to disk, which is the fastest possible disk operation.

MongoDB’s B-tree, by contrast, must find the right place in the tree to insert the new document, potentially splitting nodes and updating multiple index pointers — some of which are scattered across disk.

For Discord, where millions of messages arrive per second, this write path difference is significant.

Part 6: The Tombstone Problem — How Deletes Slow You Down

Here’s the surprising part of this story: deleting messages made the database slower.

Not temporarily. Structurally slower, in a way that required specific engineering to solve.

Why Deletes Don’t Delete

Cassandra is a distributed system. Data is replicated across multiple nodes. If you delete a row by simply removing it from one node, that deletion information is lost — when the replica syncs, it’ll just put the data back, because it doesn’t know about the delete.

Cassandra’s solution is the tombstone: instead of removing data, a delete operation writes a special marker with a timestamp. Every node that receives the tombstone knows: “this data is deleted as of this timestamp.” When reads happen, live data and tombstones are merged to produce the correct result.

sequenceDiagram
    participant App
    participant Cassandra as Cassandra Node
    participant SST1 as SSTable 1 (old)
    participant SST2 as SSTable 2 (tombstones)

    App->>Cassandra: DELETE message_id=9998
    Cassandra->>SST2: Write tombstone {message_id: 9998, deleted_at: T2}
    Note over SST1,SST2: Original data still exists in SSTable 1

    App->>Cassandra: SELECT * from channel 555 LIMIT 50
    Cassandra->>SST1: Scan partition (555, 2024_01)
    Cassandra->>SST2: Scan for tombstones
    Cassandra-->>App: Merge live rows + tombstones, return survivors
    Note over Cassandra: Every read must check ALL tombstones<br/>in the partition during compaction window

Tombstones stick around for gc_grace_seconds (10 days by default) to give all replicas time to learn about the deletion before the data is physically removed during compaction. During those 10 days, every read of that partition must process the tombstone.

When This Becomes a Real Problem

Imagine a Discord server with 500,000 members that runs a bot which posts auto-generated messages every minute, then deletes them after they expire. Over weeks, that channel accumulates millions of tombstones. A user opening the channel triggers a read that must scan through all of them to find the 50 most recent live messages. Response times crater.

Or consider server nukes — bulk deletions of every message in a channel. Each deleted message is a tombstone. Reading that channel afterward means wading through the entire graveyard.

Discord has two layers of defense against this:

Defense 1 — Bucketed partitions enable surgical cleanup. Because messages are partitioned by (channel_id, bucket), when a channel is deleted entirely, Discord can drop entire partitions rather than writing individual tombstones. Dropping a partition is a table-level operation — much faster, and it leaves no tombstone residue. The data is just gone.

Defense 2 — Separate delete tracking. For individual message deletes, Discord tracks deletion in a separate, purpose-built lookup table rather than writing tombstones directly into the messages partition. The main partition stays clean. Reads check the deletion table separately to filter out deleted messages.

This is a good example of the general principle: in distributed storage, you often can’t fix a problem by just configuring the database better. You have to change the shape of your data model to avoid the pathological case.

Part 7: Why ScyllaDB Over Cassandra

Discord eventually migrated from Cassandra to ScyllaDB. ScyllaDB is a drop-in Cassandra replacement written in C++ (Cassandra is Java) using the Seastar framework for lock-free, per-core I/O.

The reasons are operational rather than architectural:

No JVM GC pauses. Cassandra’s garbage collector occasionally causes latency spikes. In Java, this is hard to avoid at high heap sizes. ScyllaDB manages its own memory.
Per-core scheduling. ScyllaDB assigns each CPU core its own data shards and I/O queue, eliminating cross-core contention. Under heavy concurrent load, this matters.
Better tail latency. At Discord’s scale, even the 99.9th percentile latency matters — enough users hit it simultaneously to make it visible.

The data model and the query patterns remained identical. The migration was an infrastructure swap, not a re-architecture.

Putting It Together: The Full System View

flowchart TB
    subgraph Client["Client Layer"]
        DC["Discord Client\n(Web / Mobile / Desktop)"]
    end

    subgraph Gateway["Gateway Layer"]
        GW["WebSocket Gateway\n(handles real-time events)"]
    end

    subgraph API["API Layer"]
        MS["Message Service\n(reads, writes, validation)"]
    end

    subgraph Storage["Storage Layer"]
        COORD["ScyllaDB\nCoordinator Node"]
        N1["ScyllaDB Node 1\n(vnodes: token range A)"]
        N2["ScyllaDB Node 2\n(vnodes: token range B)"]
        N3["ScyllaDB Node 3\n(vnodes: token range C)"]
    end

    subgraph Cache["Cache Layer"]
        REDIS["Redis / In-memory\n(hot channel buffers)"]
    end

    DC <-->|"WebSocket"| GW
    DC <-->|"HTTPS REST"| MS
    GW --> MS
    MS <-->|"Cache-aside\nrecent messages"| REDIS
    MS -->|"Write: replicated\nto 3 nodes (quorum)"| COORD
    MS <-->|"Read: (channel_id, bucket)\n→ partition lookup"| COORD
    COORD --> N1
    COORD --> N2
    COORD --> N3

    style Client fill:#7c3aed,color:#fff
    style Gateway fill:#1d4ed8,color:#fff
    style API fill:#0369a1,color:#fff
    style Storage fill:#065f46,color:#fff
    style Cache fill:#92400e,color:#fff

What This Story Actually Teaches

The conventional takeaway from this story is “MongoDB bad, Cassandra good” — which misses the point. MongoDB is not bad. It was the right choice when Discord started. The lesson is more precise:

Every storage system makes assumptions about your access patterns. MongoDB assumes you read data by document, in unpredictable ways, and benefits from flexible indexing. Cassandra assumes you read data by partition, in predictable ways, and optimizes aggressively for that.

Chat data has a very specific shape: write-heavy, append-only, always-time-ordered, always-scoped-to-a-channel. Once Discord’s data volume made that shape impossible to ignore, the right tool was the one built for exactly that shape.

The tombstone issue is a subtler lesson: distributed systems externalize costs that centralized systems hide. A traditional database can delete data with a single write. A distributed, replicated store has to propagate that information safely across replicas — and the mechanism it uses has real performance consequences at scale. Understanding those consequences isn’t optional; it’s the job.

Discord eventually went further — they wrote about storing trillions of messages in a 2023 follow-up, introducing additional layers like message archival and tiered storage. But the foundation laid during the Cassandra migration remains intact. The core insight — organize your data around how you read it, not how it arrives — is still doing the heavy lifting.

Quick Reference: The Key Ideas

Concept	What It Means	Why It Mattered to Discord
B-tree index	Data sorted in a tree structure; random seeks for scattered documents	MongoDB used this; caused cache thrashing at scale
Wide-column store	Data physically co-located by partition key	Channel messages are always read together
LSM tree	Writes go to memory first, flushed as sorted sequential files	Makes high-throughput writes cheap
Partition key	Determines which node + which disk location	`(channel_id, bucket)` = all related messages are near each other
Clustering key	Sort order within a partition	`message_id DESC` = newest messages first, no extra sort step
Tombstone	A delete marker written instead of physically removing data	Accumulated tombstones slowed reads on channels with bulk deletes
Compaction	Background process merging SSTables, garbage-collecting tombstones	The mechanism that eventually removes deleted data
Bucket	Time-based grouping within a partition	Prevents unbounded partition growth; enables clean partition drops

This post is based on Discord’s publicly available engineering blog series: “How Discord Stores Billions of Messages” (2017) and “How Discord Stores Trillions of Messages” (2023). The architecture details are drawn from their public writeups and have been interpreted and explained here for educational purposes.

Enjoyed this deep dive? Staying updated on architectural shifts shouldn’t be a full-time job. At Knowledge Cafe, we act as your personal research team, filtering the fluff from the world’s top engineering blogs to bring you the insights needed to build the next generation of resilient systems.

Subscribe to Knowledge Cafe here