Metadata

The Intersection of Data Observability, MDM and Data Quality

Henrik Gabs Liliendahl2 Comments

Data observability is a new discipline on the rise within data management. As with many new disciplines everything is not new, though. There are several capabilities that come with a data observability solution that have been known for decades within Master Data Management (MDM) and not at least Data Quality Management (DQM).

The brief reason of being for data observability is to prevent data issues at scale. Compared to MDM and DQM you will usually utilize a data observability solution more upstream and have more data sources in scope. The emphasis of data observability is to early and continuously identify data issues. MDM and DQM is geared towards resolving the issues.

Below is a short walkthrough of the common capabilities you can deploy as part of the triangle of data observability, MDM and data quality.

Data Matching

Implementing a data observability solution will usually not extend to data matching capabilities. These capabilities will still reside in the intersection of MDM and data quality.

Data Discovery

Data discovery has been an adjacent part of many MDM solutions as touched on in the post How Data Discovery Makes a Data Hub More Valuable.

You will probably find a better home for data discovery in a data observability solution as this is better deployed for multiple upstream data flows.

Data Profiling

In Data Quality Management (DQM) solutions data profiling has often been seen as a one-off exercise that precedes data quality improvement and data matching, data migration and other data management initiatives.

With a data observability solution, you will be able to implement continuous data profiling and related monitoring.

Metadata Management

Metadata management is essential for data observability, MDM and data quality respectively and over essential for getting the full return of investment in a triangle of data observability, MDM and data quality solutions.

Three Augmented Data Management Flavors

Henrik Gabs LiliendahlLeave a comment

What is Augmented Data Management?

The term augmented data management has become a hyped topic in the data management world. “Augmented” is here used to describe an extension of the capabilities that is now available for doing data management with these characteristics:

  • Inclusion of Machine Learning (ML) and Artificial Intelligence (AI) methodology and technology to handle data management challenges that until now have been poorly solved using traditional methodology and technology
  • Encompassing graph approaches and technology to scale and widen data management coverage towards data that is lesser structured and have more variation than data that until now has been formally managed as an asset
  • Aiming at automating data management tasks that until now have been solved in manual ways or simply not been solved at all due to the size and complexity of the work involved.

Augmented data management can be applied to all the data management disciplines we know. In the following I will have a look at three data management disciplines where we today see solutions and implementations emerging. These are:

  • Augmented Metadata Management
  • Augmented Master Data Management
  • Augmented Data Quality Management

Augmented Metadata Management

The word metadata has been around for ages and the importance of metadata management as a prerequisite for proper data management is commonly agreed on among data management professionals. However, the concrete examples of successful enterprise-wide implementations are sparse. Even more, examples of solutions that are governed and maintained over time are rare.

Metadata management is a daunting task. Doing a snapshot of the metadata in play within an enterprise just now is hard enough. Maintaining this as new data types are utilized, applications are replaced, the organization changes, new standards are adopted, and more is even more daunting.

So, here augmented metadata management comes with a promise of automating this task by providing active metadata management, that is enabled by using machine learning and artificial intelligence components and relying on graph approaches that are able to picture complex relationships between metadata.

Augmented Master Data Management

Master Data Management (MDM) solutions are being implemented around the clock in large and midsize organizations. As these solutions become a part of business processes there are people responsible for controlling and maintaining master data. While some of this work can be automated through Robotic Automation Processes (RPA) there is still a substantial amount of work that relies on decision making not easily solved that way. Add to that, that more and more data will become part of MDM solutions.

So, here augmented master data management comes with a promise of automating these tasks by using machine learning and artificial intelligence components that where feasible can rely on graph approaches that are able to picture complex relationships between master data.

Augmented Data Quality

The promise of automating data quality tasks through machine learning and artificial intelligence is not new at all. For decades this approach has been tried out in areas such as data matching and product classification.

What we see now is that this approach has matured and is more widespread utilized, including going from being standalone specialty solutions to being components in broader data management solutions.  

One example of how data quality, master data management and metadata management is supported by augmented data management in a mature solution is showcased in a video embedded in the Semarchy blog post How Augmented Data Management Adds Value to Your Business.

PS: The term augmented stems from music – raising a chord to new height

Is it Called Vendor or Supplier?

Henrik Gabs Liliendahl7 Comments

One of the most addressed domains in Master Data Management (MDM) is the vendor domain – or is it called the supplier domain?

I have seen the terms vendor and supplier used synonymously many times at different MDM end user organizations, by MDM system integrators and by MDM platform vendors where I have been engaged. This ambiguity exists in English, the most used corporate language, whereas in other languages the distinction seems to be much clearer.

In a recent post by Jignesh Patel of Stibo Systems it is suggested that supplier and vendor are two opposite terms. The post is called What Vendor Data Is and Why It Matters to Manufacturers. I remember to have read the similar post before, probably in a previous version, and commented that this interpretation, in an MDM context, confuses me.

The linguistic issue is to me that a supplier is someone you buy from, and a vendor is someone who sells to you. But as interpreted in the above post, a vendor could also be someone who sells for you. In the latter case I however have mostly seen terms as dealer and reseller used.

Another possible distinction will be that a supplier is someone who deliver goods and services. A vendor will be someone who deliver the bill. So, in an MDM implementation supplier will be used if the MDM implementation is product oriented and vendor will be used if the MDM implementation is procurement and/or financial oriented or dominated.

This distinction reveals an interesting MDMish observation which is that usually the one who deliver the goods and services and the one who deliver the bill is the same entity – but not always.

What is your stance? Is vendor and supplier the same, a bit different or opposite?

2021 Data Management Mind Map

Henrik Gabs LiliendahlLeave a comment

Disciplines come and go in the data management world. Here is a mind map of the disciplines on top of my mind today. Some of the disciplines goes back to the emerge of IT in the previous millennium and some have risen during the latest years.

The Role of Data Discovery in Data Management

Henrik Gabs Liliendahl4 Comments

Data discovery is a term probably most mentioned in relation to business intelligence and data science. I this context data discovery can be seen as a more experimental and preliminary activity that can lead to a more continuous and integrated form of reporting and predictive analysis when hidden data sources, relationships and patterns are identified.

However, data discovery is useful in other data management disciplines as well.

Data Discovery

With the increasing awareness of data security, data protection and data privacy – and the regularity compliance enforced in this space – it is crucial for organisations to know what kind of data that flows and are stored within the organization. While you may argue that this should be available in already existing documentation, I have yet to meet an organization, where this is the case. And I come around a lot.

Data discovery is also a component of test data management and tool vendors package their offerings in this space with capabilities for data masking, data subsetting and data discovery in order to answer questions as:

  • Where are the data elements that should be masked when using production data in test scenarios without violating data privacy regulations?
  • How can you subset (minimize) test data sets derived from production (covering several databases) and still have proper relationships covered?

Within Data Quality Management, Data Governance and Master Data Management (MDM) data discovery also plays a role similar to the role in data reporting. We can use data discovery to map data lineage, find potential data relationships where data matching, data cleansing  and/or data stewardship might help with ensuring data quality and business process improvement and explore where the same data have different labels (metadata) attached or the same labels are used for different data types.

The Wisdom of Metadata

Henrik Gabs Liliendahl2 Comments

A previous post on this blog was called A Business Oriented Data Mind Map. In here, I took a snapshot of my brain when thinking through some different kinds of data used within doing business.

One omission of a kind of data was though metadata. So, let me just think metadata into those other kinds of data:

Metadata mind map

Metadata is data about data. Therefore, metadata applies to any other kind of data.

Metadata could be part of structured data but is unfortunately often only sparsely treated as data dictionaries and does only in a few cases reach the level of a business glossary not to say being continuously maintained in such a glossary. Thus, a lot of structured data is not used as the valuable information it could be.

Metadata should be applied to unstructured data. If not, unstructured data will just be data, and will not be transformed to information and used as knowledge.

Metadata must be used to link data on premise and data in the cloud. Metadata is the key to syndicate local data and global within an enterprise and across business ecosystems. Metadata must be used to combine internal and external data and make the transformation from data over information and knowledge to wisdom.

Data Matching, Machine Learning and Artificial Intelligence

Henrik Gabs Liliendahl8 Comments

Data matching is a sub discipline within data quality management. Data matching is about establishing a link between data elements and entities, that does not have the same value, but are referring to the same real-world construct. The most common example is establishing a link between two different data records probably describing the same person as for example:

  • Bob Smith at 1 Main Str in Anytown
  • Robert Smith at One Main Street in Any Town

Data matching can be applied to other master data entity types as companies, locations, products and more.

In the data matching world there has always been attempts to apply machine learning (or artificial intelligence if you like). This is because deterministic approaches usually result in too many false negatives being actual matching entities not found by the computer. Probabilistic / fuzzy logic approaches usually works better, but often not good enough.

One of my own attempts with machine learning was made within a solution at Dun & Bradstreet Nordic called GlobalMatchBox. One happy result of the machine learning capability was described in the post The Art in Data Matching.

In the recent years I have embraced product master data and product data quality within my business activities. The pain points in handling product information does in some cases include matching product entities but even more it is about matching the different taxonomies in use for product data, not at least between trading partners in business ecosystems.

So, machine learning leading to artificial intelligence is on my agenda again in a quest for matching metadata as told in the post It is time to apply AI to MDM and PIM.

How about you? Do you see a future with machine learning in data matching? Have you seen any happy results?

AI

Interenterprise Metadata Handling

Henrik Gabs LiliendahlLeave a comment

I am aware that the title of this blog post is a bit geeky.

However, the terms are important in data management and your organisations ability to prosper in a continuously data driven world.

The term interenterprise was part of the previous post on this blog. The post was called What is Interenterprise Data Sharing? In a comment on LinkedIn analyst Simon Walker of Gartner explained interenterprise data sharing this way: “Interenterprise data sharing = Organizations are increasingly required to provide data to, and receive data from, external trading partners (customers, suppliers, business partners and others).”

Metadata is data about data. Handling metadata is an important facet of data management including in data governance, data quality management and Master Data Management (MDM). When it comes to the new trends in data management as big data and handling data in data lakes, the importance of metadata management will in my eyes become even more obvious.

In a current venture (Product Data Lake) we are working on building in metadata management for business ecosystems, meaning that trading partners can share product information either using the same metadata or linking their different metadata.

Using international, national and industry standards for product information will be the perfect solution within business ecosystem sharing of metadata and indeed this is the preferred option we support. However, there are many competing standards for product information and they come in developing versions, so having everyone on the same page at the same time is quite utopic.

Add to that everyone do not speak English – and even not the same variant of English. Metadata originates and should exist in the languages that is used in trading partnerships.

In Product Data Lake we have started out with these principles:

  • Product attributes can be tagged with an attribute type telling about what standard (if any) in terms of product identification, product classification or product feature it adheres to. More about that in the post Connecting Product Information.
  • Attribute short and long descriptions can be represented in different languages.
  • Trading partners can link their product attributes and have visibility in the Product Data Lake of the standards and descriptions used in the different languages they exist.

pdl-how-2

 

Product Data Lake Version 1.3 is Live

Henrik Gabs LiliendahlLeave a comment

Product Data Lake is a cloud service for sharing product data in the business ecosystems of manufacturers, distributors, merchants, marketplaces and large end users of product information.

Yesterday some new capabilities have been included in the service.

As a manufacturer you can now utilize Product Data Lake even more as a cloud based lightweight Product Information Management (PIM) system. We have added better views of uploaded product information and better means of managing product data within the service. This will be of benefit for manufacturers who already handles product data in ERP and Product Lifecycle Management (PLM) solutions and needs a cost-effective solution to share these data with trading partners. Learn more about this option on our Product Data Push site.

Also, independent providers of hubs of product information within a given industry and/or geography can now self-register as a reservoir inside Product Data Lake and thus add a modern and generic way of collecting and distributing product information to existing specialized product data pools.

PDL sprints
We use Jira for collaboration between our Product Management in Copenhagen and development team in the Far East.

But we do not stop there. The next version 1.4 will be live just before our Far East development team takes some time off for the Lunar New Year. This version adds new possibilities for pushing product information through Product Data Lake. We already support file drops via FTP domains, traditional interactive upload from network drives and direct data entry. Next option is APIs.

Further versions during the coming months covers deeper integration of popular product information standards such as ETIM, eClass and UNSPSC. Learn more about these standards in the post Five Product Classification Standards.

If you want a presentation of current and future capabilities within Product Data Lake, please make contact below:

Go back

Your message has been sent

Better Product Specifications Makes you Sell More Online

Henrik Gabs LiliendahlLeave a comment

Yesterday Mark Leher of Wand, Inc. published a killer LinkedIn Pulse post with the title Five Reasons Detailed Product Specs are Critical for Killer eCommerce.

In here Mark says: “Digitizing and structuring data are critical to any business. Businesses that control and understand their data deliver superior results. Retail is no different. Product specifications are the DNA of your catalog and should be defined and collected so retailers can maximize sales and maximize intelligence.”

I agree. A survey made last year by the Danish E-Commerce Association (FDIH) revealed that 81% of visitors will leave a webshop with incomplete product information.

The key challenges in structuring and digitizing product information for merchants/retailers are:

  • “Customizing an ecommerce taxonomy hierarchy and specifications model and assigning specifications to the specific products” as mentioned as the service provided by Wand in Mark’s post.
  • Connecting to manufactures and pull detailed product information from them in one uniform way despite that manufacturers will provide data in each their ways as described on the page about Product Data Pull.

Sell More Wand