Somehow Deduplication won’t Stick

psychographic MDM18 years ago I cruised into the data quality realm when making my first deduplication tool. Then it was an attempt to solve a business case of two companies who were considering merging and wanted to know the intersection of customers. So far, so good.

Since then I have worked intensively with deduplication and other data matching tools and approaches and also co-authored a leading eLearning course on the matter as seen here.

Deduplication capability is a core feature of many data quality tools and indeed the probably most mentioned data quality pain is lack of uniqueness not at least in party master data management.

However, most deduplication efforts don’t in my experience stick. Yes, we can process a file ready for direct marketing and purge the messages that might end up in the same offline or online inbox despite of spelling differences. But taking it from there and use the techniques in achieving a single customer view is another story. Some obstacles are:

In the comments to the latter 3 year old post the intersection (and non-intersection) of Entity Resolution and Master Data Management (MDM) was discussed.

During my latest work I have become more and more convinced that achieving a single view of something is a lot about entity resolution as expressed in the post The Good, Better and Best Way of Avoiding Duplicates.

Bookmark and Share

5 thoughts on “Somehow Deduplication won’t Stick

  1. garymdm 30th September 2013 / 06:10

    Henrik – I agree that trust is a critical element in getting matching to stick – as discussed in my comment here http://dataqualitymatters.wordpress.com/2013/09/12/deterministic-matching-versus-probabilistic-matching/

    Match approaches need to be easily understood by humans. We have successfully deployed matching by building test cases for each match rule that allowed the business to build trust in our results.

    A reference list is not really an option in South Africa – there is no definitive, quality source for many common objects such as companies, neither is there a definitive national address database.

  2. Henrik Liliendahl Sørensen 30th September 2013 / 06:54

    Thanks for commenting Gary.

    Lots of efforts have indeed been put into match implementation around and as mentioned I have also worked a lot with that. And surely some match tools work better than others.

    Using rich reference data is an easy way to take it into entity resolution and indeed the practical ways of doing that is for example very different between South Africa and Scandinavia.

    I remember once that I learned that D&B had identified their B2B match results to be best in the Nordics and France. As the supplier of the Nordic tool I was of course proud. But after working with the D&B Worldbase I realized that the main reason probably was that public registration of companies worked best in, among other places, the Nordics and France.

    Besides utilizing rich (or as I call it) big reference data tagging into many big data sources seems to be a future way applying entity resolution to achieving a single view of party master data.

  3. garymdm 30th September 2013 / 07:12

    Hi Henrik – there is definite potential in the big reference data approach.

    To me it requires a hybrid approach.

    Unless you are going to use a reference list as your single source of all truth then you will still need to do some kind of match between your data and the reference list. If you can trust this match then you can use the reference list to enrich or improve your existing data sets.

    And, as your Good, Better, Best post points out – internal content may already exist, so you don;t always need an external set. In South African banking, for example, we frequently see a customer with an existing account being asked to go through the entire KYC process (source documents etc) because the bank is product centric and does not recognise the customer that they have on record in another business unit.

  4. Gauthier Vasseur 30th September 2013 / 23:28

    Very good points Henrik. I would even add that “dedup” must always be part of a data governance process and engrained in MDM solutions.

    Stand-alone solutions are appealing but they will rarely support “do/undo/revert to previous versions” capabilities. Nor will they support workflows with different levels of validations to fix errors or supervise the quality of deduplication (the false positive you refer to, for instance).

    People and processes are definitely part of the equation and in my opinion, including de-duplication in an overall MDM/Governance process can make it stick.

  5. Henrik Liliendahl Sørensen 1st October 2013 / 06:52

    Thanks for commenting Gauthier. I do agree and have several times used deduplication as a successful part of making data ready for MDM. But then you could ask the question: Why don’t MDM Implementations Stick?.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s