Data Pool vs Data Lake

Within Product Information Management (PIM) – or Product Master Data Management if you like – there is a concept of a data pool.

Recently Justine Rodian of Stibo Systems made a nice blog post with the title Master Data Management Definitions: The Complete A-Z of MDM. Herein Justine explains a lot of terms within Master Data Management (MDM). A data pool is described as this:

“A data pool is a centralized repository of data where trading partners (e.g., retailers, distributors or suppliers) can obtain, maintain and exchange information about products in a standard format. Suppliers can, for instance, upload data to a data pool that cooperating retailers can then receive through their data pool.”

Now, during the last couple of year I have been working on the concept of applying the data lake approach to product information exchange between trading partners. Justine describes a data lake this way:

“A data lake is a place to store your data, usually in its raw form without changing it. The idea of the data lake is to provide a place for the unaltered data in its native format until it’s needed…..” 

Product Data LakeFor a provider of product information, typically a manufacturer, the benefit of interacting via a data lake opposite to a data pool is that they do not have to go through standardization before uploading and thus have to shoehorn the data into a specific form and thereby almost certainly leave out important information and being depending on consensus between competing manufacturers.

For a receiver of information, typically a merchant as a retailer and B2B dealer, the benefit of interacting via a data lake opposite to a data pool is that they can request the data in the form they will use to be most competitive and thereby sell more and reduce costs in product information sharing. This will be further accelerated if the merchant uses several data pools.

In Product Data Lake we even combine the best of the two approaches by encompassing data pools in our reservoir concept – to stay in the water body lingo. Here data pools are refreshed with modern data management technology and less rigid incoming and outgoing streams as announced in the post Product Data Lake Version 1.3 is Live.

Seven Flavors of MDM

Master Data Management (MDM) can take many forms. An exciting side of being involved in MDM implementations is that every implementation is a little bit different which also makes room for a lot of different technology options. There is no best MDM solution out there. There are a lot of options where some will be the best fit for a given MDM implementation.

The available solutions also change over the years – typically by spreading to cover more land in the MDM space.

In the following I will shortly introduce seven flavours of MDM. A given MDM implementation will typically be focused on one of these flavours with some elements of the other flavors and a given piece of technology will have an origin in one of these flavours and in more or less degree encompass some more flavors.

7 flavours

The traditional MDM platform

A traditional MDM solution is a hub for master data aiming at delivering a single source of truth (or trust) for master data within a given organization either enterprise wide or within a portion of an enterprise. The first MDM solutions were aimed at Customer Data Integration (CDI), because having multiple and inconsistent data stores for customer data with varying data quality is a well-known pain point almost everywhere. Besides that, similar pain points exist around vendor data and other party roles, product data, assets, locations and other master data domains and dedicated solutions for that are available.

Product Information Management (PIM)

Special breed of solutions for Product Information Management aimed at having consistent product specifications across the enterprise to be published in multiple sales channels have been around for years and we have seen a continuously integration of the market for such solutions into the traditional MDM space as many of these solutions have morphed into being a kind of MDM solution.

Digital Asset Management (DAM)

Not at least in relation to PIM we have a distinct discipline around handling digital assets as text documents, audio files, video and other rich media data that are different from the structured and granular data we can manage in data models in common database technologies. A post on this blog examines How MDM, PIM and DAM Stick Together.

Big Data Integration

The rise of big data is having a considerable influence on how MDM solutions will look like in the future. You may handle big data directly inside MDM og link to big data outside MDM as told in the post about The Intersection of MDM and Big Data.

Application Data Management (ADM)

Another area where you have to decide where master data stops and handling other data starts is when it comes to transactional data and other forms data handled in dedicated applications as ERP, CRM, PLM (Product Lifecycle Management) and plenty of other industry specific applications. This conundrum was touched in a recent post called MDM vs ADM.

Multi-Domain MDM

Many MDM implementations focus on a single master data domain as customer, vendor or product or you see MDM programs that have a multi-domain vision, overall project management but quite separate tracks for each domain. We have though seen many technology vendors preparing for the multi-domain future either by:

  • Being born in the multi-domain age as for example Semarchy
  • Acquiring the stuff as for example Informatica and IBM
  • Extend from PIM as for example Riversand and Stibo Systems

MDM in the cloud

MDM follows the source applications up into the cloud. New MDM solutions naturally come as a cloud solution. The traditional vendors introduce cloud alternatives to or based on their proven on-promise solutions. There is only one direction here: More and more cloud MDM – also as customer as business partner engagement will take place in the cloud.

Ecosystem wide MDM

Doing MDM enterprise wide is hard enough. But it does not stop there. Increasingly every organization will be an integrated part of a business ecosystem where collaboration with business partners will be a part of digitalization and thus we will have a need for working on the same foundation around master data as reported in the post Ecosystem Wide MDM.

Ecosystem Wide MDM

Doing Master Data Management (MDM) enterprise wide is hard enough. The ability to control master data across your organization is essential to enable digitalization initiatives and ensure the competitiveness of your organization in the future.

But it does not stop there. Increasingly every organization will be an integrated part of a business ecosystem where collaboration with business partners will be a part of digitalization and thus we will have a need for working on the same foundation around master data.

The different master data domains will have different roles to play in such endeavors. Party master will be shared in some degree but there are both competitive factors, data protection and privacy factors to be observed as well.

MDM Ecosystem

Product master data – or product information if you like – is an obvious master data domain where you can gain business benefits from extending master data management to be ecosystem wide. This includes:

  • Working with the same product classifications or being able to continuously map between different classifications used by trading partners
  • Utilizing the same attribute definitions (metadata around products) or being able to continuously map between different attribute taxonomies in use by trading partners
  • Sharing data on product relationships (available accessories, relevant spare parts, updated succession for products, cross-sell information and up-sell opportunities)
  • Having access to latest versions of digital assets (text, audio, video) associated with products

The concept of ecosystem wide Multi-Domain MDM is explored further is the article about Master Data Share.

MDM vs ADM

The term Application Data Management (ADM) has recently been circulating in the Master Data Management (MDM) world as touched in The Disruptive MDM List blog post MDM Fact or Fiction: Who Knows?

Not at least Gartner, the analyst firm, has touted this as one of two Disruptive Forces in MDM Land. However, Gartner is not always your friend when it comes to short, crisp and easy digestible definitions and explanations of the terms they promote.

In my mind the two terms MDM and ADM relates as seen below:

ADM MDM.png

So, ADM takes care of a lot of data that we do not usually consider being master data within a given application while MDM takes care of master data across multiple applications.

The big question is how we handle the intersection (and sum of intersections in the IT landscape) when it comes to applying technology.

If you have an IT landscape with a dominant application like for example SAP ECC you are tempted to handle the master data within that application as your master data hub or using a vendor provided tightly integrated tool as for example SAP MDG. For specific master data domains, you might for example regard your CRM application as your customer master data hub. Here MDM and ADM melts into one process and technology platform.

If you have an IT landscape with multiple applications, you should consider implementing a specific MDM platform that receives master data from and provides master data to applications that takes care of all the other data used for specific business objectives. Here MDM and ADM will be in separated processes using best-of-breed technology.

Welcome Enterworks, Contentserv and SyncForce on The Disruptive MDM List

I am happy to welcome three new entries on The Disruptive Master Data Management Solutions List.

This site is meant to be a list of available:

  • Master Data Management (MDM) solutions
  • Customer Data Integration (CDI) solutions
  • Product Information Management (PIM) solutions
  • Digital Asset Management (DAM) solutions

Organizations on the look for a solution of the above kind can use this site as an alternative to the likes of Gartner, Forrester, MDM Institute and others, not at least because this site will include the market leaders as well as smaller and disruptive solutions with specific use case, geographical, industry or other best of breed capabilities.

The new entries are:

ew

  • EnterWorks who is among the market leaders in multi-domain master data solutions for acquiring, managing and transforming a company’s multi-domain master data into persuasive and personalized content for marketing, sales, digital commerce and new market opportunities.
  • Contentserv thumbCONTENTSERV who offers a real-time Product Experience Platform. This integrated and product centric solution seamlessly combines the functionalities of multi domain Master Data Management, Product Information Management & Marketing Content Management.
  • SyncForce-plus-iconSyncForce who makes your product portfolio digitally available with a click of a button, in every shape and form, both internal and external, so you can shift your attention from fire fighting to building successful business with your trading partners.

You can visit the list here.

New logos 20180313

 If you are a vendor, you can register your solution here.

Which MDM and/or PIM Solution to Choose?

More and more organizations are implementing Master Data Management (MDM) and Product Information Management (PIM) solutions.

When the implementation comes to the phase where you must choose one or more solutions and you go for the buy option (which is recommended), it can be hard to get a view on the available solutions. You can turn to the Gartner option, but their Quadrant only shows the more expensive options and Gartner is a bit old school as reported here.

An additional option will be to see how the vendors themselves present their solutions in a crisp way. This is what is going on at The Disruptive Master Data Management Solutions List.

mdmlist20180222

As a solution provider you can register your solution on this site in order to be a solution considered by organizations looking for a:

  • Master Data Management (MDM) solution
  • Customer Data Integration (CDI) solution
  • Product Information Management (PIM) solution
  • Digital Asset Management (DAM) solution

Registration takes place here.

Master Data or

How MDM Solutions are Changing

When Gartner, the analyst firm, today evaluates MDM solutions they measure their strengths within these use cases:

  • MDM of B2C Customer Data, which is about handling master data related to individuals within households acting as buyers (and users) of the products offered by an organisation
  • MDM of B2B Customer Data, which is about handling master data related to other organizations acting as buyers (and users) of the products offered by an organisation.
  • MDM of Buy-Side Product Data, which is about handling product master data as they are received from other organisations.
  • MDM of Sell-Side Product Data, which is about handling product master data as they are provided to other organisations and individuals.
  • Multidomain MDM, where all the above master data are handled in conjunction with other party roles than customer (eg supplier) and further core objects as locations, assets and more.
  • Multivector MDM, where Gartner adds industries, scenarios, structures and styles to the lingo.

QuadrantThe core party and product picture could look like examined in the post An Alternative Multi-Domain MDM Quadrant. Compared to the Gartner Magic Quadrant lingo (and the underlying critical capabilities) this picture is different because:

  • The distinction between B2B and B2C in customer MDM is diminishing and does not today make any significant differentiation between the solutions on the market.
  • Handling customer as one of several party roles will be the norm as told in the post Gravitational Waves in the MDM World.
  • We need (at least) one good MDMish solution to connect the buy-sides and the sell-sides in business ecosystems as pondered in the post Gravitational Collapse in the PIM Space.

How to Improve Completeness of Data

Completeness is one of the most frequently mentioned data quality dimensions. The different data quality dimensions (as completeness, timeliness, consistency, conformity, accuracy and uniqueness) sticks together, and not at least completeness is an aim in itself as well as something that helps improving the other data quality dimensions.

“You can’t control what you can’t measure” is a famous saying. That also applies to data quality dimensions. As pondered in the post Hierarchical Completeness, measuring completeness is usually not something you can apply on the data model level, but something you need to drill down in hierarchies and other segmentation of data.

Party Master Data

A common example is a form where you have to fill a name and address. You may have a field called state/province. The problem is that for some countries (like USA, Canada, Australia and India) this field should be mandatory (and conform to a value list), but for most other countries it does not make sense. If you keep the field mandatory for everyone, you will not get data quality but rubbish instead.

Multi-Domain MDM and Data Quality DimensionsCustomer and other party master data have plenty of other completeness challenges. In my experience the best approach to control completeness is involving third party reference data wherever possible and as early in the data capture as feasible. There is no reason to type something in probably in a wrong and incomplete way, if it is already digitally available in a righter and more complete way.

Product Master Data

With product master data the variations are even more challenging than with party master data. Which product information attributes that is needed for a product varies across different types of products.

There is some help available in some of the product information standards available as told in the post Five Product Classification Standards. A few of these standards actually sets requirements for which attributes (also called features and properties) that are needed for a product of certain classification within that standard. The problem is then that not everyone uses the same standard (to say in the same version) at the same time. But it is a good starting point.

Product data flows between trading partners. In my experience the key to getting more complete product data within the whole supply chain is to improve the flow of product data between trading partners supported by those who delivers solutions and services for Product Information Management (PIM).

Making that happen is the vision and mission for Product Data Lake.

6 Decades of the LEGO® Brick and the 2nd Decade of MDM

28th January 2018 marks the 60th anniversary of the iconic LEGO® brick.

As I was raised close to the LEGO headquarter in Billund, Denmark, I also remember having a considerable amount of LEGO® bricks to play with as a child back in the 60’s in the first decade of the current LEGO® brick design. At that time the brick was a brick, where you had to combine a few sizes and colours of bricks into resembling a usable thing from the real world. Since then the range of shapes and colours of the pieces from the Lego factory have grown considerably.

MDM BlocksMaster Data Management (MDM) went into the 2nd decade some years ago as reported in the post Happy 10 Years Birthday MDM Solutions. MDM has some basic building blocks, as proposed by former Gartner analyst John Radcliffe  back in 00’s and touched in the post The Need for a MDM Vision.

These blocks indeed look like the original LEGO® bricks.

Through the 2nd decade of MDM and in coming decades we will probably see a lot of specialised blocks in many shapes describing and covering the people, process and technology parts of MDM. Let us hope that they will all stick well together as the LEGO® bricks have done for the past 60 years.

PS: Some if the sticking together is described in the post How MDM, PIM and DAM Stick Together.

5 Product Data Levels to Consider

When talking about Product Master Data Management (Product MDM) Product Information Management (PIM) I like to divide the different kinds of product data into the schema below:

Five levels

Level 1, Basic Data

At the first level, we find the basic product data that typically is the minimum required for creating a product in any system of record.

Here we find the primary product identification number or code that is the internal key to all other product data structures and transactions related to the product within an organization.

Then there usually is a short product description. This description helps internal employees identifying a product and distinguishing that product from other products. Most often the product is named in the official language of the company.

If an upstream trading partner produces the product, we may find the identification of that supplier here too. If the product is part of internal production, we may have a material type telling about if it is a raw material, semi-finished product, finished good or packing material.

Level 2, Trading Data

The second level has product data related to trading the product. We may have a unique Global Trade Item Number (GTIN) that may be in the form of an International – former European – Article Number (EAN) or a Universal Product Code (UPC). Here we have commodity codes and a lot of other product data that supports buying, receiving, selling and delivering the product.

Level 3, Recognition Data

On the third level, we find the two basic pieces of product information that came to existence when we started producing product catalogues and had the first ecommerce solutions online.

The extended product description is needed because the usual short product description used internally have no meaning to an outsider as told in the post Customer Friendly Product Master Data. Some good best practices for governing the extended product description is to have a common structure of how the description is written, not to use abbreviations and to have a strict vocabulary as reported in the post Toilet Seats and Data Quality.

We often see that the extended product descriptions need to be present in the range of languages covering the locations where business is done either if the business is international or done in a country with multiple countries. The trend of increased user customization (or should I say customisation) drives this point further.

Having a product image is pivotal if you want to sell something without showing the real product face-to-face with the customer or other end user. A missing product image is a sign of a broken business process for collecting product data as pondered in the post Image Coming Soon.

Level 4, Self-service Data

At the fourth level, we have three main sorts of product information: Product attributes, basic product relations and standard digital assets. These data supports when customers makes buying decisions within eCommerce and other self-service scenarios.

Product attributes are also sometimes called product properties or product features. These are up to thousands of different data elements that describes a product. Some are very common for most products like height, length, weight and colour. Some are very specific to the product category. This challenge is the reason of being for dedicated Product Information Management (PIM) solutions as told in the post MDM Tools Revealed.

Basic product relations are the links between a product and other products like a product that have several different accessories that goes with the product or a product being a successor of another now decommissioned product. Product relations is described further in the post Related Products: The Often Overlooked Facet of PIM.

Standard digital assets are documents like installation guides, line drawings and data sheets as examined in the post Digital Assets and Product MDM.

Level 5, Competitive Data

As the fifth level we find elements like on the fourth level, but usually these are elements that you won’t necessarily apply to all products but only to your top products where you want to stand out from the crowd and distance yourself from your competitors. If you are a reseller, you typically make these data yourself, where level 4 hard facts are delivered from the manufacturer, as examined in the post Using Internal and External Product Information to Win.

Special content are descriptions of and stories about the product above the hard features. Here you tell about why the product is better than other products and in which circumstances the product can to be used. A common aim with these descriptions is also Search Engine Optimization.

X-sell (cross-sell) and up-sell product relations applies to your particular mix of products and may be made subjective as for example to look at up-sell from a profit margin point of view. X-sell and up-sell relations may be defined from upstream by you or your upstream trading partners but also dripping down on the roof from the behaviour of your downstream trading partners / customers as manifested in the classic webshop message: “Those who bought product A also bought / looked at product B”.

Advanced digital assets are broader and more lively material than the hard fact line drawings and other documents. Increasingly newer digital media types as video are used for this purpose.

Product Classification, Product Pricing and Product Lifecycle Status

All of the above-mentioned levels of product information is supported by product classification. Usually we see product classification handled as a reference data type across Product Information Management (PIM), ERP and Product Lifecycle Management (PLM) where applicable.

Product pricing is usually also a subject mainly belonging to the ERP side of things.

Product Lifecycle Status again goes across Product Information Management (PIM), ERP and not at least Product Lifecycle Management (PLM) where applicable.

Master Data Management (MDM) is the discipline that connects the dots between these topics.

Take the processes to next level:

You can take your Product Information Management (PIM) and Product Master Data Management (Product MDM) to a higher level by following the processes as described in the post Using Pull or Push to Get to the Next Level in Product Information Management.