Data Quality App duplicate rule settings in Dynamics 365

How to Clean Up Duplicate Records in Dynamics 365 After a Data Migration

A data migration is the moment duplicates multiply. Two source systems become one, a test load is followed by the final load, and integrations start writing to the new environment before cutover is finished. A few weeks after go-live, users find two Contoso accounts, each holding half of the history. This guide shows how to clean up duplicate records in Dynamics 365 after a migration, in an order that keeps your data and your users safe.

In short

  • Clean up post-migration duplicates in Dynamics 365 in six steps: measure, normalise matching values, detect with fuzzy rules, choose the surviving record, merge in order, block new duplicates.
  • Merge accounts first, then contacts and leads, and as a guide plan around two hours per 100,000 merged records when auto-merging strong matches as a background job.
  • Native merge handles two records at a time, too slow for migration backlogs; the Data Quality App merges up to 200 records manually and auto-merges in the background.

Why migrations create duplicates

  • Several sources, one customer. The old CRM, the ERP and a few spreadsheets all hold the same company, each spelled slightly differently.
  • Repeated loads. A pilot or test load that was not fully rolled back before the final load.
  • Integrations that started early. Web forms, marketing tools or ERP sync created new records with no link to the migrated ones.
  • Different formats. +45 31 31 38 38 versus 31313838, Ltd versus Limited. Exact matching does not see these as the same value.

Step 1: Measure before you merge

Count the suspected duplicates per table (accounts, contacts and leads first) and agree on what the same customer means for your business: the same registration number, the same email domain with a similar name, or the same phone number. Write these definitions down, because they become your duplicate rules. In the Data Quality App, Data Health Studio shows open duplicate pairs per record type and a data health score, so you have a baseline to report progress against.

Step 2: Normalise the values that matching depends on

Duplicates hide behind formatting. Phone numbers need one international format, email addresses should be compared in lowercase, and company names should be compared without legal forms such as ApS, Ltd or GmbH. The Data Quality App does this with match keys: it fills a normalised key column for phone, email, company name, web domain, registration number and a sounds-like (phonetic) key, without changing the original data. Read more about phonetic duplicate detection in Dynamics 365.

Step 3: Detect with fuzzy rules, not only exact ones

The native duplicate detection rules in Dynamics 365 compare exact values or the first characters of a field, so Jon Smith and John Smith slip through. After a migration you need rules with alternatives (for example the same email, or a similar name in the same city), similarity thresholds that tolerate typos, and a match score. Set two levels: one where a pair is worth reviewing and one where it is a strong match. See how this compares in Data Quality App vs native Dynamics 365 duplicate detection.

Step 4: Decide which record survives

Before anything is merged, agree on the master record. Common choices are the oldest record (often the one other systems reference), the newest, or the record with the most activities. Then decide per field: keep the master value, but take the value from the other record when the master field is empty. Related records such as activities, opportunities and cases move to the surviving record, so no history is lost.

Step 5: Merge in the right order

  1. Start with accounts, then contacts and leads, so contacts end up under the surviving account.
  2. Review a sample by hand. Open strong matches side by side and check that the rules pick the right master. The Data Quality App merges up to 200 records at once in a manual merge.
  3. Auto-merge the strong matches in the background. Run the merge as a scheduled job instead of keeping a browser window open. As a guide, plan around two hours per 100,000 merged records.
  4. Leave borderline pairs for review and mark real non-duplicates as not duplicates, so they do not come back in the next run.

The native merge in Dynamics 365 handles two records at a time, which is fine for a handful but not for a migration backlog. Our guide on how to merge duplicate records in Dynamics 365 explains the options in detail.

Step 6: Stop new duplicates at the door

Cleaning up once is not enough if integrations and users keep creating new duplicates. Turn on duplicate checks when records are created and updated, and choose per rule whether to warn the user or block the save. Integrations can be stopped completely, or specific service accounts can be allowed to create records that are merged automatically later, using the master record rules you defined in step 4. Add data quality rules for required fields and formats, and follow the data quality score so you see if the numbers start to slip.

Before your next migration

The cheapest duplicate is the one you never migrate. If another migration or a second wave is coming, read how to clean CRM data before a Dynamics 365 migration and work through our CRM data cleanup checklist.

Want to see how many duplicates your migration left behind? Start a free 15-day trial of the Data Quality App. The trial activates itself when you create your first rule.

Improve your data quality in Dynamics 365

The Data Quality App keeps records complete and correct with data quality rules at the point of entry, finds duplicates with fuzzy and phonetic matching, resolves them with bulk merge and measures your data health over 30, 60 and 90 days. Native in your own environment, free for 15 days.

Start free 15-day trial See the Data Quality App

Hasret Smajlović is the co-founder of Techdio, a Microsoft Certified Global ISV building data quality and migration solutions for Dynamics 365 and Power Apps. Working in the Microsoft business applications ecosystem since 2009, Hasret has delivered more than 100 end-to-end Dynamics 365 implementations, upgrades and migrations across cloud and on-premises environments. In 2024, Microsoft selected Hasret for its Partner Advisory Council as one of the world's top 25 Business Applications professionals among Microsoft partners. Hasret studied strategic management at Harvard Business School Online and holds a Mini MBA in organizational leadership.