Stalwart v1.0: The Fastest Mail Server on the Planet

Five years ago, on 3 October 2021, the first commit landed in a small repository called mail-parser. It was an e-mail parsing library written in Rust, and nothing more. Over the following five years it grew into a complete mail and collaboration server with JMAP, IMAP, POP3, SMTP, CalDAV, CardDAV and WebDAV, a spam filter, clustering, and support for half a dozen storage backends. Today, after five years of continuous development, we are proud to announce Stalwart v1.0, the first stable release.

Version 1.0 was never meant to be another feature release. It had two goals. The first was performance: making every part of the server as fast as it can be, and proving it with numbers. The second was stability of the data itself: finalising the database schema so that, from now on, upgrading Stalwart means installing a new version and nothing more, without the disruptive migrations some earlier releases required. Along the way a few new features found their way in too, including full-text search inside PDF and office documents.

Let’s walk through it.

Months spent chasing every microsecond

Most of the work in this release happened under a profiler. For months we measured where Stalwart spends its CPU cycles, its memory and its database round trips, rewrote whatever stood out, and measured again. The release contains close to 200 commits touching more than 1,300 files, and the vast majority of them exist to make something faster or lighter.

Parsers rewritten with SIMD and SWAR

A mail server spends a surprising share of its time looking for specific bytes: the end of a header line, a MIME boundary, an equals sign in a quoted-printable body, the space that ends an IMAP tag. Done one byte at a time, that is a lot of loop iterations for a 10 MB message. Two techniques let a CPU do this work many bytes at a time:

• SIMD (Single Instruction, Multiple Data) uses the wide vector registers of modern processors (SSE2, SSSE3 and AVX2 on x86-64, NEON on ARM) to compare or transform 16 or 32 bytes with a single instruction.
• SWAR (SIMD Within A Register) applies the same idea with ordinary 64-bit integers: a few carefully chosen arithmetic and bitwise operations test all eight bytes of a word at once. It needs no special instructions, so it works on every CPU.

Stalwart now uses both, everywhere it matters. At the bottom of the stack sit two libraries that reach 1.0 together with the server:

• mail-parser 1.0, the very library where Stalwart began, has been rewritten from scratch. SIMD kernels locate MIME boundaries and header fields, SWAR handles header names, the parsed message lives in a handful of flat vectors, and bodies are only decoded when someone actually reads them. The CPU-specific kernel is picked at runtime, and every one is fuzzed against a plain scalar reference. Against other parsers it is 3 times faster than the popular Rust mailparse crate, 9 times faster than Dovecot’s own parser, and 15 to 31 times faster than libEtPan!, VMime and GMime on a full parse.
• encodify is a new encoding library with SIMD kernels for Base64, Base32 and quoted-printable, the encodings that carry nearly every attachment and non-ASCII header in e-mail. Decoding a MIME-wrapped Base64 attachment is 22 times faster than with the standard Rust base64 crate on an AMD EPYC server, and quoted-printable decoding is 9 times faster than with the quoted_printable crate. It now handles every Base64, quoted-printable and IMAP UTF-7 conversion in the server.

Above them, the protocol layer got the same treatment. The IMAP command parser is now table-driven and scans whole runs of bytes instead of stepping a state machine per character. The SMTP command parser, the JSON parser behind JMAP, the spam filter tokenizers, the HTML-to-text converter, the variable-length integer codec and the log serializers all use SWAR scanning. Even the expression language, which administrators use for routing and filtering decisions, now compiles to a compact bytecode with specialised instructions for common comparisons and evaluates them in a recycled memory arena.

Fewer, faster trips to the database

Fast parsing is wasted if every request then waits on the database, so we spent just as long on how Stalwart talks to its storage. Identifiers are now assigned inside the same write transaction that uses them, removing whole round trips from every message delivery. Read-modify-write updates, such as adding a keyword or appending to a search index, run as merge operations inside the backend. Stored objects are compressed with zstd using dictionaries trained specifically on e-mail, calendar and contact data.

Each backend was also tuned individually:

• RocksDB: every kind of data now lives in a column family configured for how it is accessed, with bloom filters for point lookups, large blocks for scans and compaction tuned for data that churns. Hot counters, such as queue quotas, are spread across up to 256 slots to avoid write conflicts.
• PostgreSQL and MySQL: pre-generated and cached prepared statements, chunked iteration with automatic retry, and fewer round trips per write.
• FoundationDB: reads that a transaction will need are prefetched in parallel windows, and large values are assembled in chunks.
• SQLite: page size, cache, memory mapping and WAL settings tuned for a mail workload.

On top of the storage layer sits a redesigned cache subsystem. Each account’s message index is held in memory in a compact packed form, and IMAP sorting, threading, flag fetches, searches on message properties and resynchronisation are now answered from that cache with few or no database reads. Very large accounts get their own cache tier with a memory budget, and cache snapshots can be swapped out to local disk, Redis or the blob store instead of being rebuilt.

A new search engine

The built-in full-text search engine was rewritten from the ground up. The previous design stored one bitmap per word and had no notion of word positions. The new engine stores postings in blocks of 65,536 documents and keeps word positions for every message. That makes real phrase searches possible, and lets a search inside one mailbox read only the blocks that mailbox touches. It indexes word stems and prefixes, compresses its data with zstd, and indexes new mail through a background queue with concurrency control, retries and rebuilds. The PostgreSQL and MySQL search backends were reworked as well. On a mailbox of 250,000 messages, a full-text search of every message body now returns in under 4 milliseconds.

Stress tested on hundreds of vCPUs

Optimising code without measuring it under realistic load is guesswork, so we built a dedicated benchmarking and stress-testing lab to go with this release. It runs on Microsoft Azure, across more than 30 regions, with up to 18 complete test labs running in parallel and several hundred vCPUs busy at once.

The test catalogue covers 35 kinds of test, spread over more than 550 individual test cells, and they cover:

• Every major protocol: IMAP, JMAP (including push over EventSource and WebSocket), SMTP inbound, submission and relay, CalDAV, CardDAV and WebDAV.
• PostgreSQL, FoundationDB and RocksDB as data stores, Redis as the in-memory store, S3 and the filesystem as blob stores, and a three-node cluster under mixed load.
• Large datasets: 1.6 million messages and 750,000 calendar and contact items across 500 accounts, individual mailboxes of 1, 2 and 5 million messages, 100,000 accounts on a single server, mail queues of 10 million messages, and DMARC and TLS report floods of a million reports.
• Storms and abuse: 20,000 new connections per second, mass logins, thousands of IDLE and push subscribers per account, hostile and malformed messages, slow-loris clients and oversized literals.
• Six-hour soak tests that look for memory growth, cache drift and slow leaks, including one that pushed over 10 million messages through the MTA.

Runs were profiled throughout. CPU time was sampled with perf and turned into flame graphs, off-CPU analysis showed where threads waited on locks and I/O, the heap was profiled with jemalloc and heaptrack, and a special instrumented build counted every storage round trip, per command, so we could see exactly which requests hit the database more than they should. Each fix was then measured against the previous build on matched machines before it was accepted.

The first full round of this cycle turned up 64 problems, which were fixed and retested. The second round re-ran everything and fixed 33 more. Some examples of what the lab caught:

• Reopening a mailbox with 5 million messages went from 61 ms to 0.21 ms.
• A JMAP query on an account with 5 million messages went from 3.3 seconds to 12 ms.
• Peak inbound mail throughput on RocksDB nearly doubled, from 9,690 to 18,394 messages per second.
• A memory issue where the process held up to 9.6 GB while the actual heap was under 0.4 GB was found and fixed. In six-hour soak tests memory now stays flat.

This lab is not a one-off effort for the release. It is now part of how Stalwart is tested internally: every future release goes through the same benchmark and stress-test rounds before it ships, so performance gains stay gained.

Faster than Postfix and Dovecot

Internal numbers only tell you whether Stalwart got faster than it was. We also wanted to know how it compares to what most of the world runs. So we put Stalwart v1.0 head to head with the most widely deployed open-source mail stack: Postfix 3.11 for SMTP and Dovecot 2.4 for IMAP.

Both ran on identical Azure servers (8 AMD EPYC Zen 5 vCPUs, 32 GiB of RAM, local NVMe storage), loaded with the same 1,000 mailboxes plus three very large ones of up to 250,000 messages, and received exactly the same work: receiving mail, delivering it and serving it to IMAP clients. Encryption, password hashing and full-text search were configured the same way on both, and the two systems were measured alternately on the same hardware.

Stalwart was faster in 35 of the 38 headline comparisons.

Some of the results:

• IMAP: Stalwart is faster than Dovecot in 14 of 15 everyday client operations, by 1.4 to 19 times. A login with a full TLS handshake takes 5.1 ms instead of 28 ms, fetching a 1 MB message is 6 times faster, threading the INBOX 5 times faster, and moving or deleting 100 messages 18 to 19 times faster.
• SMTP: Stalwart accepts a message 3.6 to 12 times faster than Postfix, and sustains 5,445 incoming messages per second against Postfix’s 723.
• Delivery: a message is visible to an IMAP client 16 times sooner, and a client waiting in IDLE is notified in 3.3 ms instead of half a second.
• Load: Stalwart serves 5.4 times more concurrent IMAP sessions per second and relays 3.8 times more mail before it starts to slow down.

The gap grows with the size of the mailbox. On a 250,000-message mailbox, a full-text search of every message body takes 3.75 ms in Stalwart and 136 ms in Dovecot, and moving or deleting 25,000 messages finishes in about a tenth of a second instead of 24 to 34 seconds: more than 200 times faster.

The three tests Dovecot won are all the same operation: marking every message in a mailbox as read at once.

Only Postfix and Dovecot were tested, but they are not just any two products. They are the engine under nearly every popular self-hosted mail server: every open-source dockerized mail server with more than 10,000 stars on GitHub, including docker-mailserver and mailcow, runs Postfix and Dovecot under the hood, as does Mail-in-a-Box. Beating that combination on almost every benchmark means beating the stack behind most of the self-hosted mail on the internet. We are now comfortable saying it plainly: Stalwart is the fastest mail server on the planet.

Don’t take our word for it. Every script, configuration, test scenario and raw result is in the benchmarks repository, together with a methodology and a frank list of caveats, such as the differences in disk-sync behaviour between the two systems. Anyone with an Azure account can provision the same servers and run the whole suite. Benchmarks of Stalwart’s different storage backends and clustered setups are being prepared for the same repository.

What else is new

Performance was the headline, but v1.0 also brings a set of new features:

• Search inside documents: Stalwart now extracts and indexes the text of PDF, Microsoft Office (docx, xlsx, pptx), OpenDocument (odt, ods, odp), EPUB and RTF documents, both in e-mail attachments and in file storage. Searching for a phrase now finds the contract attached to last year’s message, not just the message that mentions it. The extractor is written from scratch in Rust, with no external document libraries, and processes every document under strict size and complexity limits, so a malicious or malformed file cannot exhaust the server.
• Full-text search of file storage, plus an index queue with per-partition status, retries and rebuilds.
• Metadata: JMAP Object Metadata for e-mails, mailboxes, calendars, contacts, files and more, and IMAP METADATA (RFC 5464).
• Post-quantum DKIM2: composite ML-DSA-44 and Ed25519 signatures for DKIM2.
• OAuth and OpenID Connect: OAuth Client ID Metadata Documents, per-client scopes for mail, contacts, calendars and files, a list of connected applications that users can revoke, and audience matching, opaque token validation and unscoped token policies for external identity providers.
• Storage and operations: a built-in in-memory store for single-node deployments, more authentication methods for Azure Blob Storage, secondary LDAP servers for failover, a memory budget for buffered data, SSRF protection for every outbound HTTP request, and configurable client IP sources and trusted proxy hops.
• AI models and an AI spam classifier in the Enterprise edition, with support for OpenAI-compatible, Anthropic, Gemini and Ollama models.

The full list is in the changelog.

Upgrading to v1.0

Finalising the database schema has a price that is paid once: Stalwart v1.0 uses a new database layout, and existing data has to be migrated to it. A v1.0 server will not open a store created by an earlier version.

We know that some past upgrades were bumpy, and we took that seriously. This migration was designed and tested to be as smooth as we could make it, and it is the last of its kind: the schema is now stable, and future releases will upgrade in place. There are two ways to get to v1.0.

The migration tool

For deployments running Stalwart v0.16, stalwart-migrate is the easiest route. It requires v0.16.27, so earlier 0.16 releases should be updated to it first. The tool converts your settings to the v1.0 schema and copies all data into a new, empty v1.0 store. It only reads from the old store, so the original deployment stays intact and can be restarted at any point before the switch.

It also migrates incrementally. The first copy runs while v0.16 keeps serving users, however long it takes. Each later run copies only what changed since the previous one, and the only downtime is the short final run, after v0.16 is stopped. Account and message identifiers, IMAP UIDs and change history are preserved, so IMAP and JMAP clients carry on without downloading anything again. Users who sign in with OAuth will need to sign in again, and the full-text index is rebuilt in the background after the switch.

The migration proxy and Vandelay

For anyone running Stalwart v0.15 or earlier, or anyone who prefers to move users one at a time, the migration guide describes the second route. The migration proxy sits in front of both the old and the new server and sends each user to whichever one holds their account, and Vandelay copies accounts across one by one, incrementally. Each account is switched by flipping one entry in a mapping table, with no downtime for that user, and the old server keeps a full copy until you decide to retire it. The same procedure also works for moving from Dovecot and Postfix to Stalwart. The migration overview compares the two approaches.

What changed

The structure of your settings is largely the same. Around a hundred keys were renamed to follow consistent naming conventions, a few objects (OIDC directories, AI models, Azure storage, forwarded headers) were restructured, and stalwart-migrate converts all of it automatically. Two changes need attention before you upgrade:

• Clusters need Redis. A cluster coordinator now requires a Redis-based in-memory store (Redis, Redis Cluster or Redis Sentinel). Earlier versions could fall back to the data store for this, but that was slow under load and has been removed. Single-node deployments get a built-in local in-memory store instead.
• OAuth now follows the open public client profile. Stalwart now fully enforces the OAuth Profile for Open Public Clients, the emerging standard that lets any mail client sign in to any mail server with OAuth without being registered in advance. Stalwart’s own authorization server supports it out of the box. Many external identity providers do not yet support everything the profile requires, such as Dynamic Client Registration, and some clients do not yet request the required scopes.

If you use an external identity provider that is not conformant, Stalwart’s token checks can be relaxed through the compatibility settings on the OIDC directory, at your own risk, since each one weakens a security check. These settings mainly keep web clients such as the WebUI working. What they cannot fix is a provider that lacks the profile itself, and that is going to matter soon: popular e-mail clients, including Apple Mail, are adopting the profile, and once they do, their users will not be able to authenticate against a non-conformant provider. If you depend on an external identity provider, now is a good time to ask them to support it.

A related change affects the WebUI. Instead of a fixed client id, it now identifies itself to the identity provider with a Client ID Metadata Document, a URL that describes the client. Some identity providers do not support this yet. With those, administrators can register a client for the WebUI with the provider and set its id in oauthClientId, and the WebUI signs in as before. The details are in client id of the WebUI.

We are here to help

If you run into a problem or have a question during the migration, post it on support.stalw.art. We will be answering questions there daily throughout the migration period.

Thank you

Five years ago this was one Rust library for parsing e-mail. Today it is a complete mail and collaboration server, and with v1.0 it is faster than anything else we have measured it against. None of it would have happened without the community: everyone who filed an issue, sent a patch, answered a question on the forum, or trusted Stalwart with their mail. Thank you.

Download Stalwart v1.0 and let us know what you think.