Users are seeing duplicate emails after the migration — what causes this?

Solved 3 replies 21,050 views
Chris M.Original post

IT Support · June 1, 2026

We finished a migration last week and now about a dozen users report seeing every email twice in some folders. Not all folders, and not all users. What did we do wrong, and more importantly how do I clean it up without deleting real mail?

Marcus T.Staff

Migration Specialist · June 1, 2026

The pattern you describe — some folders, some users — points strongly at a migration pass that was re-run over mailboxes that had already partially transferred, using a tool that decides what to skip based on position rather than content.

Here is the mechanism. A naive migration tracks progress with a cursor: 'I copied the first 4,000 messages in this folder.' If the run is interrupted and restarted, and the folder's ordering has shifted — because new mail arrived, or because the server returns items in a different sequence — that cursor no longer points where it did. Everything after the drift gets copied a second time. Folders that finished cleanly the first time are unaffected, which is exactly why only some folders show duplicates.

The correct approach is content fingerprinting: before writing any message, compute a fingerprint from its stable properties — Message-ID, sender, timestamp, subject, size — and check whether that fingerprint already exists at the destination. If it does, skip it. This makes the migration idempotent. You can run it five times and the mailbox is identical to running it once, which is precisely the property you want when a run gets interrupted.

Chris M.

IT Support · June 2, 2026

That matches — we did restart two batches after a network problem. How do I clean up what is already there? I am nervous about anything that bulk-deletes mail.

Marcus T.Staff

Migration Specialist · June 2, 2026

You are right to be cautious. Do not let anyone run a bulk delete against production mailboxes without the following sequence:

  • Scope it first. Identify the affected mailboxes and folders precisely. You said roughly a dozen users — confirm the exact list before touching anything.
  • Run a dedup pass in report-only mode. It should produce a list of what it *would* remove without removing anything. Review that list. Look specifically for legitimate near-duplicates: a message the user genuinely received twice, or a thread where someone was CC'd and also BCC'd. Those are not duplicates and must survive.
  • Match on Message-ID, not subject and date. Subject-and-date matching will eat legitimate mail — recurring automated reports are the usual casualty. Message-ID is unique per message and is the only safe key.
  • Verify against one mailbox first. Run the cleanup on a single cooperative user, have them confirm their mailbox looks right, then proceed to the rest.

Do this and the cleanup is low-risk. Skip the report-only step and you will eventually delete something real. I have seen it happen and it is not a good week.

duplicatesdeduplicationtroubleshooting

This discussion is part of our curated migration knowledge base. Planning a move of your own? Our team answers within 48 hours.

Ask our migration team

Related discussions