How Discord Stores 11 Billion Messages (Without Melting)
Every day, hundreds of millions of gamers, developers, and online communities send billions of chat messages on Discord. Whether it’s a simple "gg," a massive meme flood during an esports tournament, or a code snippet shared in a programming community, every single keystroke must be stored permanently, indexed instantly, and delivered across the world in milliseconds.
Back in 2017, Discord’s engineering team faced a terrifying milestone: their primary database was hitting a hard wall. They were storing over 11 billion messages, and the underlying database—Apache Cassandra—was starting to choke.
Read latencies were spiking, garbage collection freezes were causing random site-wide lags, and adding new server nodes felt like trying to put out a fire with gasoline.
This is the story of how Discord’s engineering team pulled off one of the most audacious database migrations in modern tech history: moving billions of live messages to a brand-new storage engine with zero downtime and zero dropped chats.
The Architectural Summary
If you are looking for the quick system design takeaway, here is how Discord solved their massive database bottleneck:
The Original Architecture (MongoDB to Cassandra): Discord started on MongoDB, quickly outgrew it, and migrated to Apache Cassandra for its wide-column, distributed key-value model.
The Cassandra Bottleneck: Java’s Garbage Collection (GC) pauses and unpredictable compaction cycles led to massive read-latency spikes ($100\text{ms}$ to several seconds).
The Solution (ScyllaDB): Discord migrated from Java-based Cassandra to ScyllaDB, a C++ rewrite of Cassandra that eliminates JVM garbage collection entirely and leverages thread-per-core hardware architecture.
The Custom Data Layer (Data Services): Discord built a intermediate service layer written in Rust to coalesce duplicate database requests, shielding the database from traffic surges.
1. The Early Days: MongoDB to Apache Cassandra
When Discord launched in 2015, the entire application ran on a single MongoDB replica set. MongoDB was fast to build on, but as the platform exploded in popularity, the message table outgrew the server's available RAM.
To survive, Discord needed a database that was:
Linearly Scalable: Easy to expand simply by adding more hardware nodes.
High Write Throughput: Capable of handling thousands of concurrent message writes per second.
Fault-Tolerant: Free of single points of failure.
In 2016, Discord migrated to Apache Cassandra. Cassandra is a distributed, wide-column NoSQL database designed at Facebook to handle massive write volumes across multiple server nodes.
How Discord Partitioned Data in Cassandra
To store billions of messages efficiently, Discord needed a clever database schema. They created a primary partition key based on the channel_id. Because messages in a channel are always retrieved together in chronological order, grouping messages by channel made fetching chat history incredibly fast.
+-------------------------------------------------------------------------+
| DISCORD CASSANDRA SCHEMA |
+-------------------------------------------------------------------------+
| Partition Key : channel_id (e.g., Channel #general) |
| Clustering Key: message_id (Snowflake ID: timestamp + worker + seq) |
+-------------------------------------------------------------------------+
| Row 1 | Message ID: 1001 | User: Alice | Text: "Hello!" |
| Row 2 | Message ID: 1002 | User: Bob | Text: "Hey there!" |
| Row 3 | Message ID: 1003 | User: Charlie| Text: "Ready to game?"|
+-------------------------------------------------------------------------+
To generate unique, chronologically sortable message IDs across distributed servers, Discord utilized Snowflake IDs (a 64-bit integer encoding a millisecond timestamp, worker process ID, and sequence number).
For years, Cassandra handled the load brilliantly. But by 2022, as the total message count skyrocketed past 100 billion, the system began to unravel.
2. The Nightmare: Java GC Pauses & Hot Partitions
Why did Apache Cassandra start struggling? The short answer comes down to Java and Disk I/O.
Problem A: The Java Virtual Machine (JVM) Garbage Collector
Apache Cassandra is written in Java. Java relies on an automatic Garbage Collector (GC) to clear unused memory. As Discord's dataset ballooned into hundreds of terabytes, Cassandra’s JVM heaps were constantly filling up with temporary query objects.
When Java’s Garbage Collector ran its deep cleaning cycle (Stop-the-World GC pauses), entire database nodes would freeze for several seconds. To the rest of the cluster, a frozen node looked like a crashed node, causing unnecessary cluster-wide panic and traffic rerouting.
Normal Operation: [ Query ] ──► [ DB Node Response (5ms) ]
During JVM GC Pause: [ Query ] ──► [ DB Node Frozen... ] ──► [ Timeout Spike (2000ms+) ]
Problem B: Tombstones and Compaction
In Cassandra, when a user deletes a message, the database doesn't instantly delete it from the disk. Instead, it writes a marker called a Tombstone.
When a user opened a channel with many deleted messages, Cassandra had to read through thousands of tombstones just to find the surviving messages. This caused massive read-latency spikes, pushing response times from a crisp $5\text{ms}$ up to several agonizing seconds.
3. The New Engine: Enter ScyllaDB
Faced with unacceptable latency spikes, Discord’s engineering team realized that tuning the Java Virtual Machine was no longer enough. They needed to replace the database engine itself.
They chose ScyllaDB.
ScyllaDB is a ground-up rewrite of Apache Cassandra written in C++. It is $100\%$ protocol-compatible with Cassandra (meaning Discord could use the exact same client drivers and query language), but its underlying performance architecture is fundamentally different.
+------------------------------------+------------------------------------+
| APACHE CASSANDRA (JAVA) | SCYLLADB (C++ REWRITE) |
+------------------------------------+------------------------------------+
| - Managed Memory (JVM GC Pauses) | - Manual C++ Memory Management |
| - Thread Pool Model | - Thread-Per-Core Architecture |
| - High CPU Overhead | - Direct Asynchronous I/O (io_uring)|
| - Susceptible to Latency Spikes | - Ultra-consistent Low Latency |
+------------------------------------+------------------------------------+
The Power of Thread-per-Core
In traditional Java applications, operating system threads are constantly context-switching between different CPU cores, causing lock contention and performance bottlenecks.
ScyllaDB uses a Thread-per-Core architecture. It pins a single execution thread to each individual CPU core. Memory and network queues are partitioned so that each core operates independently without waiting for locks from other cores.
By eliminating the JVM Garbage Collector and embracing bare-metal C++ efficiency, ScyllaDB delivered predictable, sub-millisecond latencies even under brutal traffic loads.
4. The Shield: Building "Data Services" in Rust
Replacing the database engine was only half the battle. Discord’s engineers knew that even the fastest database on Earth can be knocked offline if thousands of clients query the exact same data at the same instant (a phenomenon known as the Thundering Herd Problem).
To protect ScyllaDB, Discord built an intermediate caching and request coalescing layer written in Rust, called Data Services.
[ User Requests (10,000 Concurrent Users) ]
│
▼
[ RUST DATA SERVICES LAYER ]
(Coalesces 10,000 identical queries)
│
▼ (Only 1 query sent to DB!)
[ SCYLLADB CLUSTER ]
How Request Coalescing Works:
Imagine a popular streamer posts a message in a Discord channel with 100,000 active viewers. All 100,000 client apps will instantly send a read request for the exact same message ID at the exact same millisecond.
Instead of sending 100,000 separate queries to ScyllaDB:
The incoming requests hit the Rust Data Services layer.
The service detects that 10,000 users are asking for the exact same
message_id.It sends only ONE single query to ScyllaDB.
When ScyllaDB responds, the Data Services layer duplicates the response in memory and returns it to all 10,000 waiting users simultaneously.
This caching mechanism reduced database read traffic by orders of magnitude, smoothing out traffic spikes during viral events.
5. Live Migration: Moving Billions of Rows Zero Downtime
Migrating hundreds of billions of records while millions of users are actively chatting is like replacing the engine of a commercial jet mid-flight. Discord could not afford a maintenance window.
To pull off a seamless zero-downtime migration, they executed a Dual-Writing Engineering Strategy:
[ Incoming User Writes/Edits ]
│
├───────────────┬───────────────┐
▼ ▼
[ CASSANDRA (Old DB) ] [ SCYLLADB (New DB) ]
│ ▲
│ │
└───► [ BACKGROUND MIGRATOR ] ──┘
(Copies Historical Data)
Dual Writing: Discord updated their application code to write all new messages, edits, and deletions to both Cassandra and ScyllaDB simultaneously.
Historical Backfill: They built background worker tasks to read historical messages from Cassandra and copy them over to ScyllaDB in reverse chronological order.
Consistency Verification: Automated validation scripts compared records between both databases to ensure $100\%$ data integrity.
The Cutover: Once the backfill was complete and data parity was confirmed, Discord flipped the read switch. All user traffic was instantly pointed to ScyllaDB, and the old Cassandra cluster was safely decommissioned.
Summary & Key Takeaways
Discord’s migration from Cassandra to ScyllaDB is a masterclass in modern infrastructure scaling:
How Discord Stores 11 Billion Messages (Without Melting)
Every day, hundreds of millions of gamers, developers, and online communities send billions of chat messages on Discord. Whether it’s a simple "gg," a massive meme flood during an esports tournament, or a code snippet shared in a programming community, every single keystroke must be stored permanently, indexed instantly, and delivered across the world in milliseconds.
Back in 2017, Discord’s engineering team faced a terrifying milestone: their primary database was hitting a hard wall. They were storing over 11 billion messages, and the underlying database—Apache Cassandra—was starting to choke.
Read latencies were spiking, garbage collection freezes were causing random site-wide lags, and adding new server nodes felt like trying to put out a fire with gasoline.
This is the story of how Discord’s engineering team pulled off one of the most audacious database migrations in modern tech history: moving billions of live messages to a brand-new storage engine with zero downtime and zero dropped chats.
The TL;DR Architectural Summary
If you are looking for the quick system design takeaway, here is how Discord solved their massive database bottleneck:
The Original Architecture (MongoDB to Cassandra): Discord started on MongoDB, quickly outgrew it, and migrated to Apache Cassandra for its wide-column, distributed key-value model.
The Cassandra Bottleneck: Java’s Garbage Collection (GC) pauses and unpredictable compaction cycles led to massive read-latency spikes ($100\text{ms}$ to several seconds).
The Solution (ScyllaDB): Discord migrated from Java-based Cassandra to ScyllaDB, a C++ rewrite of Cassandra that eliminates JVM garbage collection entirely and leverages thread-per-core hardware architecture.
The Custom Data Layer (Data Services): Discord built a intermediate service layer written in Rust to coalesce duplicate database requests, shielding the database from traffic surges.
1. The Early Days: MongoDB to Apache Cassandra
When Discord launched in 2015, the entire application ran on a single MongoDB replica set. MongoDB was fast to build on, but as the platform exploded in popularity, the message table outgrew the server's available RAM.
To survive, Discord needed a database that was:
Linearly Scalable: Easy to expand simply by adding more hardware nodes.
High Write Throughput: Capable of handling thousands of concurrent message writes per second.
Fault-Tolerant: Free of single points of failure.
In 2016, Discord migrated to Apache Cassandra. Cassandra is a distributed, wide-column NoSQL database designed at Facebook to handle massive write volumes across multiple server nodes.
How Discord Partitioned Data in Cassandra
To store billions of messages efficiently, Discord needed a clever database schema. They created a primary partition key based on the channel_id. Because messages in a channel are always retrieved together in chronological order, grouping messages by channel made fetching chat history incredibly fast.
+-------------------------------------------------------------------------+
| DISCORD CASSANDRA SCHEMA |
+-------------------------------------------------------------------------+
| Partition Key : channel_id (e.g., Channel #general) |
| Clustering Key: message_id (Snowflake ID: timestamp + worker + seq) |
+-------------------------------------------------------------------------+
| Row 1 | Message ID: 1001 | User: Alice | Text: "Hello!" |
| Row 2 | Message ID: 1002 | User: Bob | Text: "Hey there!" |
| Row 3 | Message ID: 1003 | User: Charlie| Text: "Ready to game?"|
+-------------------------------------------------------------------------+
To generate unique, chronologically sortable message IDs across distributed servers, Discord utilized Snowflake IDs (a 64-bit integer encoding a millisecond timestamp, worker process ID, and sequence number).
For years, Cassandra handled the load brilliantly. But by 2022, as the total message count skyrocketed past 100 billion, the system began to unravel.
2. The Nightmare: Java GC Pauses & Hot Partitions
Why did Apache Cassandra start struggling? The short answer comes down to Java and Disk I/O.
Problem A: The Java Virtual Machine (JVM) Garbage Collector
Apache Cassandra is written in Java. Java relies on an automatic Garbage Collector (GC) to clear unused memory. As Discord's dataset ballooned into hundreds of terabytes, Cassandra’s JVM heaps were constantly filling up with temporary query objects.
When Java’s Garbage Collector ran its deep cleaning cycle (Stop-the-World GC pauses), entire database nodes would freeze for several seconds. To the rest of the cluster, a frozen node looked like a crashed node, causing unnecessary cluster-wide panic and traffic rerouting.
Normal Operation: [ Query ] ──► [ DB Node Response (5ms) ]
During JVM GC Pause: [ Query ] ──► [ DB Node Frozen... ] ──► [ Timeout Spike (2000ms+) ]
Problem B: Tombstones and Compaction
In Cassandra, when a user deletes a message, the database doesn't instantly delete it from the disk. Instead, it writes a marker called a Tombstone.
When a user opened a channel with many deleted messages, Cassandra had to read through thousands of tombstones just to find the surviving messages. This caused massive read-latency spikes, pushing response times from a crisp $5\text{ms}$ up to several agonizing seconds.
3. The New Engine: Enter ScyllaDB
Faced with unacceptable latency spikes, Discord’s engineering team realized that tuning the Java Virtual Machine was no longer enough. They needed to replace the database engine itself.
They chose ScyllaDB.
ScyllaDB is a ground-up rewrite of Apache Cassandra written in C++. It is $100\%$ protocol-compatible with Cassandra (meaning Discord could use the exact same client drivers and query language), but its underlying performance architecture is fundamentally different.
+------------------------------------+------------------------------------+
| APACHE CASSANDRA (JAVA) | SCYLLADB (C++ REWRITE) |
+------------------------------------+------------------------------------+
| - Managed Memory (JVM GC Pauses) | - Manual C++ Memory Management |
| - Thread Pool Model | - Thread-Per-Core Architecture |
| - High CPU Overhead | - Direct Asynchronous I/O (io_uring)|
| - Susceptible to Latency Spikes | - Ultra-consistent Low Latency |
+------------------------------------+------------------------------------+
The Power of Thread-per-Core
In traditional Java applications, operating system threads are constantly context-switching between different CPU cores, causing lock contention and performance bottlenecks.
ScyllaDB uses a Thread-per-Core architecture. It pins a single execution thread to each individual CPU core. Memory and network queues are partitioned so that each core operates independently without waiting for locks from other cores.
By eliminating the JVM Garbage Collector and embracing bare-metal C++ efficiency, ScyllaDB delivered predictable, sub-millisecond latencies even under brutal traffic loads.
4. The Shield: Building "Data Services" in Rust
Replacing the database engine was only half the battle. Discord’s engineers knew that even the fastest database on Earth can be knocked offline if thousands of clients query the exact same data at the same instant (a phenomenon known as the Thundering Herd Problem).
To protect ScyllaDB, Discord built an intermediate caching and request coalescing layer written in Rust, called Data Services.
[ User Requests (10,000 Concurrent Users) ]
│
▼
[ RUST DATA SERVICES LAYER ]
(Coalesces 10,000 identical queries)
│
▼ (Only 1 query sent to DB!)
[ SCYLLADB CLUSTER ]
How Request Coalescing Works:
Imagine a popular streamer posts a message in a Discord channel with 100,000 active viewers. All 100,000 client apps will instantly send a read request for the exact same message ID at the exact same millisecond.
Instead of sending 100,000 separate queries to ScyllaDB:
The incoming requests hit the Rust Data Services layer.
The service detects that 10,000 users are asking for the exact same
message_id.It sends only ONE single query to ScyllaDB.
When ScyllaDB responds, the Data Services layer duplicates the response in memory and returns it to all 10,000 waiting users simultaneously.
This caching mechanism reduced database read traffic by orders of magnitude, smoothing out traffic spikes during viral events.
5. Live Migration: Moving Billions of Rows Zero Downtime
Migrating hundreds of billions of records while millions of users are actively chatting is like replacing the engine of a commercial jet mid-flight. Discord could not afford a maintenance window.
To pull off a seamless zero-downtime migration, they executed a Dual-Writing Engineering Strategy:
[ Incoming User Writes/Edits ]
│
├───────────────┬───────────────┐
▼ ▼
[ CASSANDRA (Old DB) ] [ SCYLLADB (New DB) ]
│ ▲
│ │
└───► [ BACKGROUND MIGRATOR ] ──┘
(Copies Historical Data)
Dual Writing: Discord updated their application code to write all new messages, edits, and deletions to both Cassandra and ScyllaDB simultaneously.
Historical Backfill: They built background worker tasks to read historical messages from Cassandra and copy them over to ScyllaDB in reverse chronological order.
Consistency Verification: Automated validation scripts compared records between both databases to ensure $100\%$ data integrity.
The Cutover: Once the backfill was complete and data parity was confirmed, Discord flipped the read switch. All user traffic was instantly pointed to ScyllaDB, and the old Cassandra cluster was safely decommissioned.
Summary & Key Takeaways
Discord’s migration from Cassandra to ScyllaDB is a masterclass in modern infrastructure scaling:
| Architecture Era | Database Used | Primary Bottleneck | Performance Result |
| 2015 (Launch) | MongoDB | Outgrew RAM limits on single replica set. | Unstable under rapid user growth. |
| 2016 - 2022 | Apache Cassandra | JVM Garbage Collection freezes & Tombstones. | Spikes up to $2000\text{ms}+$ read latency. |
| 2023 - Present | ScyllaDB + Rust Data Services | Managed via C++ bare-metal efficiency & request coalescing. | Sub-millisecond latencies ($<5\text{ms}$) at $100\text{B}+$ messages. |
By replacing Java GC pauses with C++ hardware efficiency and shielding their database with a smart Rust middleware layer, Discord transformed a lagging infrastructure into a lightning-fast messaging platform capable of scaling effortlessly for years to come.
