How to deduplicate emails after (multiple) imports?

[badboy]

Recently I migrated my old mailserver setup to stalwart.
One domain, several users and tens of thousands of emails.

As I set everything up I imported mails for users using the stalwart-cli import messages command from a maildir export from the old mailserver.
That looked good, so I proceeded with my migration.
Eventually I turned off the old mail server, took another maildir backup and ran the import again. That took a bit, but ultimately succeeded.

Unfortunately too late I realized it re-imported already existing emails and thus most mailboxes now contain nearly all mails twice.
I misinterpreted this line to assume it would dedupe upon import:

If the import process is interrupted, the CLI tool supports automatically resuming the task without duplicating messages or folders.

So my questions:

1. How to identify the duplicated mails? Quick check in my mail client looks like the Message-ID header is untouched, so that might be a solved thing.
2. How to actually best deduplicate those emails?

Any kind of solution to get me there appreciated. I don't mind writing a script to identify & delete from the database if that's needed

(This question is similar to #92, but that doesn't have a solution)

Edit: Related: #590 -- deduping on import

[HeWhoWatches]

I've used imapsync to do this recently. Take a look at this FAQ page, under "How can I remove duplicates in a lone account?". The FAQs are also available on the github page.

Obviously, backup everything before you give this a try, use some dummy accounts to make sure everything works as you expect, and make use of the --dry switch to dry-run before you pull the trigger.

A basic bash script might look a little like this:

user1passfile=credentials/username
user2passfile=credentials/username

host1=email.server.com
host2=email.server.com
user1=username
user2=username

includefolders='INBOX'

./imapsync \
--include "$includefolders" \
--host1 "$host1" --user1 "$user1" --passfile1 "$user1passfile" \
--host2 "$host2" --user2 "$user2" --passfile2 "$user2passfile" \
--automap --delete2duplicates --dry

Hope that helps!