Data Pool vs Data Lake

Within Product Information Management (PIM) – or Product Master Data Management if you like – there is a concept of a data pool.

Recently Justine Rodian of Stibo Systems made a nice blog post with the title Master Data Management Definitions: The Complete A-Z of MDM. Herein Justine explains a lot of terms within Master Data Management (MDM). A data pool is described as this:

“A data pool is a centralized repository of data where trading partners (e.g., retailers, distributors or suppliers) can obtain, maintain and exchange information about products in a standard format. Suppliers can, for instance, upload data to a data pool that cooperating retailers can then receive through their data pool.”

Now, during the last couple of year I have been working on the concept of applying the data lake approach to product information exchange between trading partners. Justine describes a data lake this way:

“A data lake is a place to store your data, usually in its raw form without changing it. The idea of the data lake is to provide a place for the unaltered data in its native format until it’s needed…..” 

Product Data Lake
MacRitchie Reservoir in Singapore

For a provider of product information, typically a manufacturer, the benefit of interacting via a data lake opposite to a data pool is that they do not have to go through standardization before uploading and thus have to shoehorn the data into a specific form and thereby almost certainly leave out important information and being depending on consensus between competing manufacturers.

For a receiver of information, typically a merchant as a retailer and B2B dealer, the benefit of interacting via a data lake opposite to a data pool is that they can request the data in the form they will use to be most competitive and thereby sell more and reduce costs in product information sharing. This will be further accelerated if the merchant uses several data pools.

In Product Data Lake we even combine the best of the two approaches by encompassing data pools in our reservoir concept – to stay in the water body lingo. Here data pools are refreshed with modern data management technology and less rigid incoming and outgoing streams as announced in the post Product Data Lake Version 1.3 is Live.

Seven Flavors of MDM

Master Data Management (MDM) can take many forms. An exciting side of being involved in MDM implementations is that every implementation is a little bit different which also makes room for a lot of different technology options. There is no best MDM solution out there. There are a lot of options where some will be the best fit for a given MDM implementation.

The available solutions also change over the years – typically by spreading to cover more land in the MDM space.

In the following I will shortly introduce the basic stuff with seven flavours of MDM. A given MDM implementation will typically be focused on one of these flavours with some elements of the other flavors and a given piece of technology will have an origin in one of these flavours and in more or less degree encompass some more flavors.

7 flavours

The traditional MDM platform

A traditional MDM solution is a hub for master data aiming at delivering a single source of truth (or trust) for master data within a given organization either enterprise wide or within a portion of an enterprise. The first MDM solutions were aimed at Customer Data Integration (CDI), because having multiple and inconsistent data stores for customer data with varying data quality is a well-known pain point almost everywhere. Besides that, similar pain points exist around vendor data and other party roles, product data, assets, locations and other master data domains and dedicated solutions for that are available.

Product Information Management (PIM)

Special breed of solutions for Product Information Management aimed at having consistent product specifications across the enterprise to be published in multiple sales channels have been around for years and we have seen a continuously integration of the market for such solutions into the traditional MDM space as many of these solutions have morphed into being a kind of MDM solution.

Digital Asset Management (DAM)

Not at least in relation to PIM we have a distinct discipline around handling digital assets as text documents, audio files, video and other rich media data that are different from the structured and granular data we can manage in data models in common database technologies. A post on this blog examines How MDM, PIM and DAM Stick Together.

Big Data Integration

The rise of big data is having a considerable influence on how MDM solutions will look like in the future. You may handle big data directly inside MDM og link to big data outside MDM as told in the post about The Intersection of MDM and Big Data.

Application Data Management (ADM)

Another area where you have to decide where master data stops and handling other data starts is when it comes to transactional data and other forms data handled in dedicated applications as ERP, CRM, PLM (Product Lifecycle Management) and plenty of other industry specific applications. This conundrum was touched in a recent post called MDM vs ADM.

Multi-Domain MDM

Many MDM implementations focus on a single master data domain as customer, vendor or product or you see MDM programs that have a multi-domain vision, overall project management but quite separate tracks for each domain. We have though seen many technology vendors preparing for the multi-domain future either by:

  • Being born in the multi-domain age as for example Semarchy
  • Acquiring the stuff as for example Informatica and IBM
  • Extend from PIM as for example Riversand and Stibo Systems

MDM in the cloud

MDM follows the source applications up into the cloud. New MDM solutions naturally come as a cloud solution. The traditional vendors introduce cloud alternatives to or based on their proven on-promise solutions. There is only one direction here: More and more cloud MDM – also as customer as business partner engagement will take place in the cloud.

Ecosystem wide MDM

Doing MDM enterprise wide is hard enough. But it does not stop there. Increasingly every organization will be an integrated part of a business ecosystem where collaboration with business partners will be a part of digitalization and thus we will have a need for working on the same foundation around master data as reported in the post Ecosystem Wide MDM.

Ecosystem Wide MDM

Doing Master Data Management (MDM) enterprise wide is hard enough. The ability to control master data across your organization is essential to enable digitalization initiatives and ensure the competitiveness of your organization in the future.

But it does not stop there. Increasingly every organization will be an integrated part of a business ecosystem where collaboration with business partners will be a part of digitalization and thus we will have a need for working on the same foundation around master data.

The different master data domains will have different roles to play in such endeavors. Party master will be shared in some degree but there are both competitive factors, data protection and privacy factors to be observed as well.

MDM Ecosystem

Product master data – or product information if you like – is an obvious master data domain where you can gain business benefits from extending master data management to be ecosystem wide. This includes:

  • Working with the same product classifications or being able to continuously map between different classifications used by trading partners
  • Utilizing the same attribute definitions (metadata around products) or being able to continuously map between different attribute taxonomies in use by trading partners
  • Sharing data on product relationships (available accessories, relevant spare parts, updated succession for products, cross-sell information and up-sell opportunities)
  • Having access to latest versions of digital assets (text, audio, video) associated with products

The concept of ecosystem wide Multi-Domain MDM is explored further is the article about Master Data Share.

Spreadsheets, Business Process Re-engineering and Robots

Product information is the data a potential buyer of a product needs to make a purchasing decision. Today purchasing is more and more made by self-services as in e-commerce. The product information is usually obtained through a supply chain between trading partners stretching from the manufacturer to the end merchant.

The most common way of exchanging product information between trading partners is using spreadsheets. Spreadsheets are marvellous, because you can do almost anything you want with them. However, spreadsheets are also horrendous, because you can do almost anything you want with them. Therefore, trading partners are often stuck with manual, cumbersome and error prone processes on both the providing and receiving end.

At Product Data Lake we have developed a new mechanism that enables a whole new process for exchanging product information between trading partners. We have kept the flexibility of spreadsheets when it comes to choosing the data standards on the providing and receiving end but at the same time introduced automation and correctness when it comes to transferring, translating and transforming the data.

When telling about our service I am often asked if we have a nice feature for on-boarding spreadsheets. We don’t. Because the process is designed to omit the spreadsheets and transfer directly from the providers in-house product information data store(s) to the receiving in-house product information data store.

RobotThis reminds me of when we talk about using robots to substitute human labor. Then we often think about a machine that looks like a human. But effective industrial robots do not look like humans. They a designed to do a specific process much more effective than a human and will therefore not look like a human. The same is true in digitalization. When we redesign business processes to be much more effective they should not include spreadsheets.

Product Information on Demand

Video on demand has become a popular way to watch television series, films and other entertainment and Netflix is probably the most known brand for delivering that.

The great thing about watching video on demand is that you do not have to enjoy the service at the exact same time as everyone else, as it was the case back in the days when watching TV or going to the movies were the options available.

At Product Data Lake we will bring that convenience to business ecosystems, as the situation today with broadcasting product information in supply chains very much resembles the situation we had before video on demand came around in the TV/Movie world.

As a provider of product information (being a manufacturer or upstream distributor), you will push your product information into Product Data lake, when you have the information available. Moreover, you will only do that once for each product and piece of information. No more coming to each theatre near your audience and extensive reruns of old stuff.

As a receiver of product information (being a downstream distributor, reseller or large end user), you will pull product information when you need it. That will be when you take a new product into your range or do a special product sale as well as when you start to deal with a new piece of information. No more having to be home at a certain time when your supplier does the show or waiting in ages for a rerun when you missed it.

Learn more about how Product Data Lake makes your life in Product Information Management (PIM) easier by following us here on LinkedIn.

Product Data Lake

 

Three Major Sectors within Product Information Exchange

When working with Product Information Management (PIM) and not at least with product information exchange between trading partners, I have noticed three major sectors where the requirements and means differs quite a bit.

These sectors are:

  • Food, beverage at pharmaceuticals: These are highly regulated sectors where the rules for taxonomy, completeness and exchange formats are advanced. Exchange standards and underpinning services as GS1/GDSN are well penetrated at least for basic data elements among major players. This sector counts for circa 1/6 of the world trade.
  • Fashion, books and mainstream electronics: The products within this sector can be described with common accepted taxonomies and do not differ that much though there certainly are room for more common adhered standards in some areas. The trade here is becoming more penetrated by marketplaces with their specific product information requirements. This sector counts for circa 1/6 of the world trade.
  • The rest (including building materials, special electronics, machinery, homeware): This is a diverse segment of products groups and the product groups themselves are diverse. The requirements for product information completeness and other data quality dimensions are overwhelming and the choice of standards are many, so most often two trading partners will be on different pages. This sector counts for circa 2/3 of the world trade.

Note: Automotive (vehicles) is a special vertical, where the main products (for example cars) resembles mainstream electronics and all the spare parts resembles special electronics. Some retailers (like department stores) covers all sectors and therefore need hybrid solutions to their product information exchange handling challenges.

The main drivers for better product information handling are compliance – not at least within food, beverage and pharmaceuticals – and self-service purchasing (as in ecommerce), where the latter has raged many years within fashion, books and mainstream electronics and now also is raising in more B2B (business-to-business) biased product groups as building materials, special electronics and machinery.

Learn more about how to tackle these diverse needs in product information exchange in the article and discussion about Product Data Lake.

segments.png

Welcome Enterworks, Contentserv and SyncForce on The Disruptive MDM List

I am happy to welcome three new entries on The Disruptive Master Data Management Solutions List.

This site is meant to be a list of available:

  • Master Data Management (MDM) solutions
  • Customer Data Integration (CDI) solutions
  • Product Information Management (PIM) solutions
  • Digital Asset Management (DAM) solutions

Organizations on the look for a solution of the above kind can use this site as an alternative to the likes of Gartner, Forrester, MDM Institute and others, not at least because this site will include the market leaders as well as smaller and disruptive solutions with specific use case, geographical, industry or other best of breed capabilities.

The new entries are:

ew

  • EnterWorks who is among the market leaders in multi-domain master data solutions for acquiring, managing and transforming a company’s multi-domain master data into persuasive and personalized content for marketing, sales, digital commerce and new market opportunities.
  • Contentserv thumbCONTENTSERV who offers a real-time Product Experience Platform. This integrated and product centric solution seamlessly combines the functionalities of multi domain Master Data Management, Product Information Management & Marketing Content Management.
  • SyncForce-plus-iconSyncForce who makes your product portfolio digitally available with a click of a button, in every shape and form, both internal and external, so you can shift your attention from fire fighting to building successful business with your trading partners.

You can visit the list here.

New logos 20180313

 If you are a vendor, you can register your solution here.

Embracing Standards versus Imposing Standards

When working with Product Information Management (PIM) and the recurring challenges in exchanging product information between trading partners the idea about everyone adhering to the same standard is a tempting idea.
This idea is also governing the many product data pools around. However, there are some serious considerations against this idea, namely:
  • Being on the same standard and not to say on the same version within your business ecosystem is quite utopic (being that within your own organization is hard enough).
  • It is not desirable to have the same product information as your competitors if you are going to compete on other factors than price.
In my eyes it is a better idea to forget about imposing a rigid standard for everyone and instead embrace the many available standards for product information where your organization utilize those being best for you at the given time and your various trading partners utilize those being best for them at a given time.
The solution for that is Product Data Lake.
Sell more Reduce costs

The Wide End-to-End Solution for Product Information Management (PIM)

The term End-to-End is used a lot in marketing jargon. Now, I will jump on that wagon too.

In reality, no solution will be an End-to-End solution for all your business needs. Therefore, my take will merely be to cast some light on an End-to-End need for which there are only very scattered solutions today.

If we look at Product Information Management (PIM) there are many good solutions for taking care of the End-to-End needs within your organisation. The aim is to gather the product information that exist within your organisation in various silos, have one trusted place for all this information and being able to publish this information in a consistent way across all sales channels – the omnichannel theme.

However, product information does in many cases not live just within your organization. In most cases, it lives in a business ecosystem of manufacturers, distributors, merchants and large end users.

Therefore we need an End-to-End solution for product information that encompasses the path from manufacturers over distributors to merchants and large end users and in some cases the way back.

Whether you are a manufacturer, distributor, merchant, large end user or a provider of tools and services for product information you can join the business ecosystem oriented End-to-End solution for product information. Please find some more information about Product Data Lake here.

As a manufacturer, you can find your benefits on the Product Data Push site here.

As a merchant, you can find your benefits on the Product Data Pull site here.

If you are a vendor in the Product Information Management space, you can join forces with us a explained here.

pdl-top

 

Why it is not a Product Data Warehouse, but a Product Data Lake

There is a need for a new solution to sharing product information between trading partners. Product Data Lake is that new solution. Using the term data lake as a part of the name for the solution is very deliberate. Here is why:

Volume

When setting up a warehouse, and a data warehouse, you have to estimate the storing size and the throughput. There will be a limit to how much data you can store and how much data you can upload and download within a given period.

Our vision is that Product Data Lake will be the process driven key service for exchanging any sort of product information within business ecosystems all over the world, with the aim of optimally assist self-service purchase of every kind of product.

In order to achieve that vision, we need to be able to scale up drastically. Therefore, we use a document-oriented database called MongoDB to store product information.

Even if you choose to implement a Product Data Lake instance for a single business ecosystem, you will benefit from the high scalability.

Velocity

Business ecosystems changes all the time. You need to rapidly be able to adapt your data management, not at least when it comes to exchanging product information.

Swapping trading partners is one thing. That often means dealing with other product information requirements and opportunities and adhering to other standards.

We will also see business ecosystems in new shapes in the future. There will be fewer nodes between manufacturers and point-of-sales and point-of-sales will more likely be online marketplaces.

However, the changes will not happen as a big bang but in varying pace for each industry, geography and organization.

The rigid consensus structure of a data warehouse, and product information exchange solutions that resembles a data warehouse, will not cope with that change. The data lake concept, in the form of Product Data Lake, will.

In Product Data Lake you as a provider upload product information in your structure and format and you as a receiver download in your structure and format. The linking and transformation takes place inside Product Data Lake using linked metadata.

Variety

While everyone agrees that a common standard for all product information is the best answer we must on the other hand accept, that using a common standard for every kind of product and every piece of information needed is quite utopic. We haven’t even a common uniquely spelled term in English for standardization/standarisation.

Also, we must foresee that one organization will mature in a different pace than another organisation in the same business ecosystem.

These observations are the reasons behind the launch of Product Data Lake. In Product Data Lake we encompass the use of (in prioritized order):

  • The same standard in the same version
  • The same standard in different versions
  • Different standards
  • No standards
Learn about some of these standards in the post Five Product Classification Standards.
big data pdl.png