Today I will like to summarize all the changes in the Gartner Magic Quadrant for Data Quality Tools that has happened between 2009 and 2010.
It goes like this:
- Netrics is gone
Data Quality Tools
What’s In a Given Name?
I use the term ”given name” here for the part of a person name that in most western cultures is called a ”first name”.
When working with automation of data quality, master data management and data matching you will encounter a lot of situations where you will like to mimic what we humans do, when we look at a given name. And when you have done this a few times you also learn the risks of doing so.
Here is some of the learning I have been through:
Gender
Most given names are either for males or for females. So most times you instinctively know if it is a male or a female when you look at a name. Probably you also know those given names in your culture that may be both. What often creates havoc is when you apply rules of one culture to data coming from a different culture. The subject was discussed on DataQualityPro here.
Salutation
In some cultures salutation is paramount – not at least in Germany. A correct salutation may depend on knowing the gender. The gender may be derived from the given name. But you should not use the given name itself in your greeting.
So writing to “Angela Merkel” will be “Sehr geehrte Frau Merkel” – translates to “Very honored Mrs. Merkel”.
If you have a small mistake as the name being “Angelo Merkel”, this will create a big mistake when writing “Sehr geehrter Herr Merkel” (Very honored Mr. Merkel) to her.
Age
In a recent post on the DataFlux Community of Experts Jim Harris wrote about how he received tons of direct mails assuming he was retired based on where he lives.
I have worked a bit with market segmentation and data (information) quality. I don’t know how it is with first names in the United States, but in Denmark you may have a good probability with estimating an age based on your given name. The statistical bureau provides statistics for each name and birth year. So combining that with the location based demographic you will get a better response rate in direct marketing.
Nicknames
Nicknames are used very different in various cultures. In Denmark we don’t use them that much and definitely very seldom in business transactions. If you meet a Dane called Jim his name is actually Jim. If you have a clever piece of software correcting/standardizing the name to be James, well, that’s not very clever.
Returns from Investing in a Data Quality Tool
The classic data quality business case is avoiding sending promotion letters and printed materials to duplicate prospects and customers.
Even as e-commerce moves forward and more complex data quality business cases as those related to multi-purpose master data management becomes more important I will like to take a look at the classic business case by examining some different kind of choices for a data quality tool.
As you may be used to all different kind of currencies as EUR, USD, AUD, GBP and so on I will use the fictitious currency SSB (Simple Stupid Bananas).
Let’s say we have a direct marketing campaign with these facts:
- 100,000 names and addresses, ½ of them also with phone number
- Cost per mail is 3 SSB
- Response is 4,500 orders with an average profit of 100 SSB
From investigating a sample we know that 10% of the names and addresses are duplicates with slightly different spellings.
So from these figures we know that the cost of a false negative (a not found actual duplicate) is 3 SSB. Savings of a true positive is then also 3 SSB.
The cost of a false positive (a found duplicate that actually isn’t a duplicate) is a possible missing order worth: 4,500 / (100,000 * 90 %) * 100 SSB = 5 SSB.
Now let’s examine 3 options for tools for finding duplicates:
A: We already have Excel
B: Buying the leader of the pack data quality tool
C: Buying an algorithm based dedupe tool
A: We already have Excel
You may first sort 100,000 rows by address and look for duplicates this way. Say you find 2,000 duplicates. Then sort 98,000 rows by surname and look for duplicates. Say you find 1,000 duplicates. Then sort 97,000 rows by given name. Say you find 1,000 duplicate. Finally sort 48,000 rows by phone number. Say you find 1,000 duplicates.
If a person can look for duplicates in 1,000 rows per hour (without making false positives) we will browse totally 343,000 sorted rows in 343 hours.
Say you hire a student for that and have the Subject Matter Expert explaining, controlling and verifying the process using 15 hours.
Costs are:
- 343 student hours each 15 SSB = 5.145 SSB
- 15 SME hours each 50 SSB = 750 SSB
Total costs are 5.895 SSB.
Total savings are 5,000 true positives each 3 SSB = 15.000 SSB, making a positive ROI = 9.105 SSB in each campaign.
Only thing is that it will take one student more than 2 months (without quitting) to do the job.
B: Buying the leader of the pack data quality tool
Such a tool may have all kind of data quality monitoring features, may be integrated smoothly with ETL functionality and so on. For data matching it may use so called match codes. Doing that we may expect that the tool will find 7,500 duplicates where 7,000 are true positives and 500 are false positives.
Costs may be:
- Tool license fee is 50.000 SSB
- Training fee is 7.000 SSB
- 80 hours external consultancy each 125 SSB = 10.000 SSB
- 60 IT hours for training and installation each 50 SSB = 3.000 SSB
- 100 SME hours for training and configuration each 50 SSB = 5.000 SSB
Total costs are 75.000 SSB
Savings per campaign are 7,000 * 3 SSB – 500* 5 SSB = 18.500 SSB.
A positive ROI will show up after the 5th campaign.
C: Buying an algorithm based dedupe tool
By using algorithm based data matching such a tool depending on the threshold setting may find 9,100 duplicates where 9,000 are true positives and 100 are false positives.
Costs may be:
- Tool license fee is 5.000 SSB
- 8 hours external consultancy for a workshop each 125 SSB = 1.000 SSB
- 15 SME hours for training, configuration and pushing the button each 50 SSB = 750 SSB
Total costs are 6.750 SSB
Savings per campaign are 9,000 * 3 SSB – 100* 5 SSB = 26.500 SSB
A remarkable ROI will show up in the 1st campaign.
The Slurry Project
When cleansing party master data it is often necessary to typify the records in order to settle if it is a business entity, a private consumer, a department (or project) in a business, an employee at a business, a household or some kind of dirt, test, comic name or other illegible name and address.
Once I made such a cleansing job for a client in the farming sector. When I browsed the result looking for false positives in the illegible group this name showed up:
- The Slurry Project (in Danish: Gylleprojektet)
So, normally it could be that someone called a really shitty project a bad name or provided dirty data for whatever reason. But in the context of the farming sector it makes a good name for a project dealing with better exploitation of slurry in growing crops.
A good example of the need for having the capability to adjust the bad word lists according to the context when cleansing data.
Eurovisions
Diversity in data quality is a recurring subject of mine. I think the issues with data quality and diversity resembles a recurring event in Europe being the yearly Eurovision Song Contest. This year the contest was held in Oslo the past week.
Every participating country brings a song. The text may be in any language which then mostly is either English or your different local language(s). Some songs have an international sound while other songs have a strong recognizable local sound. This year I noticed:
- The winning song from Germany was in the international category, performed in English.
- As UK songs usually have an international sound and are performed in English the British song handicapped itself with a +20 year old sound leading to a similar position in the finale.
- Netherlands had a winning strategy with a local sound performed in Dutch. Big hit in Holland I think, but didn’t make it to the finale.
The voting process was as usual criticized as there is a tendency that neighboring countries favors each other such as done by Balkan countries – and the Viking nations.
Post no. 100
This is post number 100 on this blog. Besides that this is a time for saying thank you to those who have read this blog, those who have re-tweeted the posts and not at least those who have commented on the posts on this blog, it is also time for a recapitulation on my opinions (based on my experiences and observations) about data quality.
Let me emphasize three points:
- Fit for purpose versus real world alignment
- Diversity in data quality
- The role of technology in data quality improvement
Fit for purpose versus real world alignment
According to Wikipedia data may be of high quality in two alternative ways:
- Either they are fit for their intended uses
- Or they correctly represent the real-world construct to which they refer
My thesis is that there is a breakeven point when including more and more purposes where it will be less cumbersome to reflect the real world object rather than trying to align all known purposes.
This theme is so far covered in 19 posts and pages including:
Diversity in data quality
International and multi-cultural aspects of data quality improvement have been a favorite topic of mine for a long time.
While working with data quality tools and services for many years I have found that many tools and services are very national. So you might discover that a tool or service will make wonders with data from one country, but be quite ordinary or in fact useless with data from another country.
I have made 15 posts on diversity in data quality so far including:
The role of technology in data quality improvement
Being a Data Quality professional may be achieved by coming from the business side or the technology side of practice. But more important in my eyes is the question whether you have made serious attempts and succeeded in understanding the side from where you didn’t start. I have always strived to be a mixed skilled person. As I have tried single handed to build a data quality tool – or to be more specific a data matching tool – I do of course write a lot about data quality technology.
This blog includes 37 posts on data quality technology so far including:
- Service Oriented Data Quality
- Upstream prevention by error tolerant search
- Data Quality Tools Revealed
Big Time ROI in Identity Resolution
Yesterday I had the chance to make a preliminary assessment of the data quality in one of the local databases holding information about entities involved in carbon trade activities. It is believed that up to 90 percent of the market activity may have been fraudulent with criminals pocketing 5 billion Euros. There is a description of the scam here from telegraph.co.uk.
Most of my work with data matching is aimed at finding duplicates. In doing this you must avoid finding so called false positives, so you don’t end up merging information about to different real world entities. But when doing identity resolution for several reasons including preventing fraud and scam you may be interested in finding connections between entities that are not supposed to be connected at all.
The result from making such connections in the carbon trade database was quite astonishing. Here is an example where I have changed the names, addresses, e-mails and phones, but such a pattern was found in several cases:
Here we have an example of a group of entities where the name, address, e-mail or phone is shared in a way that doesn’t seem natural.
My involvement in the carbon trade scam was initiated by a blog post yesterday by my colleague Jan Erik Ingvaldsen based on the story that journalists by merely gazing the database had found addresses that simply doesn’t exist.
So the question is if authorities may have avoided losing 5 billion taxpayer Euros if some identity resolution including automated fuzzy connection checks and real world checks was implemented. I know that you are so much more enlightened on what could have been done when the scam is discovered, but I actually think that there may be a lot of other billions of Euros (Pounds, Dollars, Rupees) to avoid losing out there by making some decent identity resolution.
A user experience
As a data quality professional it is a learning experience when you are the user.
During the last years I have worked for a data quality tool vendor with headquarter in Germany. As part of the role of serving partners, prospects and customers in Scandinavia I have been a CRM system user. As a tool vendor own medicine has been taken which includes intelligent real time duplicate check, postal address correction, fuzzy search and other goodies built into the CRM system.
Sounds perfect? Sure, if it wasn’t for a few diversity glitches.
The address doesn’t exist
Postal correction is only activated for Germany. This actually makes some sense since most activity is in Germany and postal correction is not that important in Scandinavia as company (and citizen) information is more available and then usually a better choice. Due to a less fortunate setup during the first years my routine when inserting a new account was to pick correct data from a business directory, paste into the CRM system and then angry override the warning that the address doesn’t exist (in Germany).
Dear worshipful Mr Doctor Oetker
In Germany salutation is paramount. In Scandinavia it is not common to use a prefixed salutation anymore – and if you do, you are regarded as very old fashioned. So having the salutation field for a contact as mandatory is an annoyance and setting up an automated salutation generation mechanism is a complete waste of time.
Data Quality and World Food
I have touched the analogy between food (quality) and data (quality) several times before for example in the posts “Bon Appétit” and “Under New Master Data Management”.
Why not continue down that road?
Let’s have a look at some local food that has become popular around the world.
寿司
Imagine you go to a restaurant where you order a fish dish. When starting to consume your dinner you realize that the fish hasn’t been boiled, fried or in any other way exposed to heat. Then I guess it is perfectly normal to shout out: THE FISH IS RAW – and demanding apologies from the chef, the head waiter, Gordon Ramsey or anyone else in charge. Unless of course if you are in a sushi restaurant where the famous Japanese dish that may include raw fish is prepared.
Köttbullar
Köttbullar is the Swedish word for meatballs. This had rightfully stayed as a fact only known to Swedes if it wasn’t for cheap furniture sold around the world by IKEA. By reasons still unclear to me IKEA has chosen to serve Köttbullar in the store cafeterias and even sell the stuff along with the particle board furniture on their e-commerce sites.
Pizza
Italian originated dish usually brought to you by someone on a bike or in extreme cases in a very old car.
McChicken™
Selling food of different kind in the form as a burger works in the United States – and by reasons that I can’t explain even in France.
Data Quality analogies
Well, let’s just say that data quality tools and services:
- May be regarded very different around the world,
- Usually are sold along with tools and services made for something completely different,
- Are brought to you in various ways by local vendors and
- By reasons I can’t explain often are made for use in the United States (no other pun intended but pure admiration of execution).
Bon appétit.
Merging Customer Master Data
One of the most frequent assignments I have had within data matching is merging customer databases after two companies have been merged.
This is one of the occasions where it doesn’t help saying the usual data quality mantras like:
- Prevention and root cause analysis is a better option
- Change management is a critical factor in ensuring long-term data quality success
- Tools are not important
It is often essential for the new merged company to have a 360 degree view of business partners as soon as possible in order to maximize synergies from the merger. If the volumes are above just a few thousand entities it is not possible to obtain that using human resources alone. Automated matching is the only realistic option.
The types of entities to be matched may be:
- Private customers – individuals and households (B2C)
- Business customers (B2B) on account level, enterprises, legal entities and branches
- Contacts for these accounts
I have developed a slightly extended version of this typification here.
One of the most common challenges in merging customer databases is that hierarchy management may have been done very different in the past within the merging bodies. When aligning different perceptions I have found that a real world approach often fulfils the different reasoning.
The fuzziness needed for the matching is basically dependent on the common unique keys available in the two databases. These are keys as citizen ID’s (whatever labeled around the world) and public company ID’s (the same applies). Matching both databases with an external source (per entity type) is an option. “Duns Numbering” is probably the most common known type of such an approach. Maintaining a solution for assigning Duns Numbers to customer files from the D&B WorldBase is by the way one of my other assignments as described here.
The automated matching process may be divided into these three steps:
During my many years of practice in doing this I have found that the result from the automated process may vary considerable in quality and speed depending on the tools used.
Liliendahl on Data Quality 