Continuous maildir export / replication

[aksdb]

Since I am still struggling with good backup strategies for Stalwart, a maybe slightly weird idea came to my mind: wouldn't it be relatively easy to have a secondary write-only mail storage into (nested) maildir? Let me expand that a bit:

• Every create/update/delete of mails is done to the primary store (rocksdb/filesystem/whatever) and also into a maildir. This should be relatively easy as long as the filenames are deterministic. Since for every operation we know the before and after state, we can easily apply that to the maildir without having to iterate or parse anything.
• Every read is done from the primary store only, so all optimizations towards file/database structure stay in place.
• The output directory of this secondary storage is treated as a virtual mail (vmail) directory; so each account has a directory under which the actual maildirs are found.

This would have the big advantage, that it's very easy to introspect (since a ton of tools have maildir support) and it would be super easy to backup, since all filesystem-based backup solutions (like restic, kopia, etc.) can derive incremental changes to the last time.

Now in addition to this (as a next step maybe), the root directory of the secondary storage could also contain a few metadata (simple account infos) while the account directories could contain account specific infos (like sieve scripts). That way, the Stalwart CLI could even offer a restore-from-(this-specific)-maildir. All indices and database structures can be derived again from the information in this vmail structure.

---

An alternative to this might also be, that this isn't a secondary storage, but to implement it as primary mail storage (like the existing filesystem storage, but with maildir structure). I assume though, the existing filesystem storage isn't structured the way it is for fun, but because there's a technical necessity to it, in which case having a completely different, non-block-based structure could be an issue.

[mdecimus]

Hi, thanks for the suggestion but Maildir is an ancient format that does not scale well. It is not going be supported by Stalwart.

[aksdb]

The scalability was the reason I contemplated if it might be worth it as a secondary write-only store (since that should not impact performance, unless I miss something). The big upside would be simplicity for backups.

[avsm]

I'm just using mbsync to maintain a backup Maildir via an external invocation over IMAP, while using pgsql as the main Stalwart store. Seems to be the best of both words: you get the simple rsync-based Maildir backup, and the primary Stalwart store uses a modern database.

[mdecimus]

There are also plans to expose the message store as Maildir via WebDAV, which will simplify import/export to the legacy Maildir format.
This feature is being tracked by another Github issue in case you want to subscribe to it.

[aksdb]

@avsm I do that for my own mailbox, but as I am hosting for friends and family, I can't do that on their behalf without letting them expose their credentials (and it doesn't scale nicely from an operational PoV, since it needs manual maintenance for each mailbox).

@mdecimus the WebDAV access to Maildir is a nice feature as well, but I believe it would fall short for my intended use case for the same reason as the imapsync/offlineimap/mbsync appraoch.

Btw. I am not (necessarily) requesting the feature to be implemented by you. If you are not completely opposed to it, I might put it on my backlog and try to tackle it myself. I would just like to avoid putting a lot of time into something that has no chance of being merged upstream.

[mdecimus]

@aksdb Yeah, sorry, anything beyond exporting/importing Maildir is out of scope for Stalwart. Making Stalwart write Maildirs to the filesystem would feel like adding an exhaust to an electric car ;-)