A while ago I wrote a short blog post about a tweet from the Gartner analyst Ted Friedman saying that clients are disappointed with the ability to support wide deployment of complex business rules in popular data quality tools.
Speaking about popular data quality tools; on the DataFlux Community of Experts blog Founder of DataQualityPro Dylan Jones posted a piece this Friday asking: Are Your Data Quality Rules Complex Enough?
Dylan says: “Many people I speak to still rely primarily on basic data profiling as the backbone of their data quality efforts”.
The classic answers to the challenge of complex business rules are:
- Relying on people to enforce complex business rules. Unfortunately people are not as consistent in enforcing complex rules as computer programs are.
- Making less complex business rules. Unfortunately the complexity may be your competitive advantage.
In my eyes there is no doubt about that data quality tool vendors has a great opportunity in research and development of tools that are better at deploying complex business rules. In my current involvement in doing so we work with features as:
- Deployment as Service Oriented Architecture components. More on this topic here.
- Integrating multiple external sources. Further explained here.
- Combining the best algorithms. Example here.
You raise a good point here Henrik, the reason most rules are not managed is because they are an absolute nightmare!
There is also an issue that business rules (if they exist in some electronic workflow system for example) are rarely reflected in the data quality rules that should align to them.
For example, a business rule may state that a customer who already has an existing account must be linked to a parent record, a business rule may be enforced to perform basic checks via workflow software but unless the workflow software has the power of modern matching technology it may still fail.
What we need is a unification of business and data quality rules so that there is no distinction.
Thanks for commenting Dylan. Indeed we need a unification of business and data quality rules.
In relation to the data quality tool market I think we have two trends for that:
• Mega vendors embracing all disciplines
• Specialized vendors making best-of-breed plug-ins