Product Data Lake Behind the Scenes

Product Data Lake is a cloud service for exchanging product information between manufacturers, distributors and merchants. When telling about the service I usually concentrate on the business benefits and how the service will make you sell more and reduce costs.

However, there will always be one or two persons in the audience who wants to know about the technology behind. And for sure, this is important too.

The service is built using some of the newest and best-of-breed technologies available for this purpose today. This includes Amazon Elastic Computing Cloud for hosting the public cloud version, MongoDB for storing data, RabbitMQ for handling data streams and ElasticSearch for finding data.

PDL Architecture

You can dive into the geeky parts in this PDF document: Product Data Lake Architecture.

MDM in The Cloud, On-Premise or Both

One of the forms of Master Data Management (MDM) is the rising cloud deployment model as touched in the Disruptive MDM List blog post about 8 Forms of Master Data Management.

If we look at the MDM solution vendors, they may in that sense be divided into three kinds:

  • Cloud only, which are vendors born in the cloud age and who are delivering their service in the cloud only. Reltio is an example of that kind of MDM vendor.
  • Cloud or on-premise, which are vendors that can deliver both in the cloud and on premise, but where it makes most sense that you as a customer chooses the one that fits you the best. An example is Semarchy.
  • Cloud and on-premise. Informatica is the example of an MDM vendor that embraces both deployment models (together with other data management disciplines) at the same time (called hybrid) as told in an article by Kristin Nicole of SiliconANGLE. The title goes like this: Balancing act: Informatica straddles on-prem needs with cloud data at Informatica World 2018

Cloud MDM

A Business Oriented Data Mind Map

You can look at data in many ways.

Below is a mind map embracing some of the ways you can make a picture of data within your business.

data mind map

Data is often seen as the raw material that will be processed into information, which can be used to gather knowledge and thereby over time emerge as business wisdom.

When working with processing data we may distinguish between structured data that is already pre-processed into a workable format and unstructured data that is not easily ingested as information yet.

The main forms of structured data are:

  • Reference data that often is defined and maintained in a wider scope than in your organization but where you still may consider be more knowledgeable inside your organisation as touched in the post The World of Reference Data.
  • Master data that describes the who, where and what in your business transactions. You can drill further down into this in the post A Master Data Mind Map.
  • Transactions that holds the details of the ongoing production events, about when we make purchases and sales and the financials related to all activities in the business.

Unstructured data will in the end hold much more information than our structured data. This includes communication data, digital assets and big data. Some structured data sources are though also big as examined in the post Five Flavors of Big Data.

We may also store the data in different places. For historical reasons within computer technology we have stored our data on premise, but organizations are, in different pace, increasingly depolying new data stores in the cloud.

In organisations with activities in multiple geographies and/or other organizational splits an ongoing consideration is whether a chunk of data is to be handled locally for each unit or to be handled globally (within the organization).

I am sure there are a lot of other ways in which you can look at data. What is on your mind?

 

Seven Flavors of MDM

Master Data Management (MDM) can take many forms. An exciting side of being involved in MDM implementations is that every implementation is a little bit different which also makes room for a lot of different technology options. There is no best MDM solution out there. There are a lot of options where some will be the best fit for a given MDM implementation.

The available solutions also change over the years – typically by spreading to cover more land in the MDM space.

In the following I will shortly introduce the basic stuff with seven flavours of MDM. A given MDM implementation will typically be focused on one of these flavours with some elements of the other flavors and a given piece of technology will have an origin in one of these flavours and in more or less degree encompass some more flavors.

7 flavours

The traditional MDM platform

A traditional MDM solution is a hub for master data aiming at delivering a single source of truth (or trust) for master data within a given organization either enterprise wide or within a portion of an enterprise. The first MDM solutions were aimed at Customer Data Integration (CDI), because having multiple and inconsistent data stores for customer data with varying data quality is a well-known pain point almost everywhere. Besides that, similar pain points exist around vendor data and other party roles, product data, assets, locations and other master data domains and dedicated solutions for that are available.

Product Information Management (PIM)

Special breed of solutions for Product Information Management aimed at having consistent product specifications across the enterprise to be published in multiple sales channels have been around for years and we have seen a continuously integration of the market for such solutions into the traditional MDM space as many of these solutions have morphed into being a kind of MDM solution.

Digital Asset Management (DAM)

Not at least in relation to PIM we have a distinct discipline around handling digital assets as text documents, audio files, video and other rich media data that are different from the structured and granular data we can manage in data models in common database technologies. A post on this blog examines How MDM, PIM and DAM Stick Together.

Big Data Integration

The rise of big data is having a considerable influence on how MDM solutions will look like in the future. You may handle big data directly inside MDM og link to big data outside MDM as told in the post about The Intersection of MDM and Big Data.

Application Data Management (ADM)

Another area where you have to decide where master data stops and handling other data starts is when it comes to transactional data and other forms data handled in dedicated applications as ERP, CRM, PLM (Product Lifecycle Management) and plenty of other industry specific applications. This conundrum was touched in a recent post called MDM vs ADM.

Multi-Domain MDM

Many MDM implementations focus on a single master data domain as customer, vendor or product or you see MDM programs that have a multi-domain vision, overall project management but quite separate tracks for each domain. We have though seen many technology vendors preparing for the multi-domain future either by:

  • Being born in the multi-domain age as for example Semarchy
  • Acquiring the stuff as for example Informatica and IBM
  • Extend from PIM as for example Riversand and Stibo Systems

MDM in the cloud

MDM follows the source applications up into the cloud. New MDM solutions naturally come as a cloud solution. The traditional vendors introduce cloud alternatives to or based on their proven on-promise solutions. There is only one direction here: More and more cloud MDM – also as customer as business partner engagement will take place in the cloud.

Ecosystem wide MDM

Doing MDM enterprise wide is hard enough. But it does not stop there. Increasingly every organization will be an integrated part of a business ecosystem where collaboration with business partners will be a part of digitalization and thus we will have a need for working on the same foundation around master data as reported in the post Ecosystem Wide MDM.

Product Data Completeness

Completeness is one of the most frequently mentioned data quality dimensions as touched in the post How to Improve Completeness of Data.

ChecklistWhile every data quality dimension applies to all domains of Master Data Management (MDM), some different dimensions apply a bit more to one of the domains or the intersections of the domains as explained in the post Multi-Domain MDM and Data Quality Dimensions.

With product master data (or product information if you like) completeness is often a big pain. One reason is that completeness means different requirements for different categories of products as pondered in the post Hierarchical Completeness within Product Information Management.

At Product Data Lake we develop a range of cloud service offerings that will help you improve completeness of product data. These are namely:

  • Measuring completeness against these industry standards that have attribute requirements such as eClass and ETIM
  • For manufacturers measuring completeness against downstream trading partner requirements (if not fully governed by an industry standard).
  • For merchants measuring incoming completeness when pulling from merchants.
  • Measuring against completeness required by marketplaces.
  • Transforming product information to meet conformity and thereby ability to populate according to requirements
  • Translating product information in order to populate attributes in more languages
  • Transferring product information by letting manufacturers push it in their way and letting merchants pull it their way as described in the post Using Pull or Push to Get to the Next Level in Product Information Management.

To the Cloud and Beyond

Over at the Informatica blog Joe McKendrick recently wrote about When It’s Time to Give Data Warehouse a Digital Makeover.

In here Joe examines how data warehouses can be modernized to augment architectures supporting data lakes and Mater Data Management and the case for moving data warehouses to the cloud.

In my view, a lot of data management disciplines will eventually move to the cloud as one follows the other. By adding “beyond” I suggest, that cloud solutions will not only be something that is supported company by company. Eventually you will be able to get business outcome by sharing data management burdens within your business ecosystem.

My current venture called Product Data Lake is an example of such a solution. It modernizes the data warehouse thinking within product information sharing by using a data lake concept in the cloud ready-to-use by trading partners within business ecosystems:

  • If you are a provider of product information, typically as a manufacturer of goods, you can harvest your business outcome by using us for Product Data Push
  • If you are a receiver of product information, you can harvest your business outcome by using us for Product Data Pull

pdl-top

MDM, Reltio, Gartner and Business Outcome

A recent well commented blog post by Andrew White of Gartner, the analyst firm, debates What’s Happening in Master Data Management (MDM) Land?

The post is an answer to a much liked and commented LinkedIn status post by Ramon Chen, Chief Product Officer of Reltio.

In his post Andrew connects the classic dots: How does technology lead to business outcome? Especially the use of cloud solutions and the multi-tenant aspect is in the focus. Andrew asks: What do you see “out there”?

My view is that multi-tenant is not just about offering the same subscription based cloud solutions to a range of clients. It is about making clients sharing the same business ecosystem work in the same MDM realm. This is the platform described in Master Data Share.

Gartner Digital Platforms 2
Source: Gartner

Oh, and what does that have to do with business outcome? A lot. Organizations will not win the future the race by optimizing there inhouse MDM capabilities alone. With the rise of digitalization, they need to connect with and understand their customers, which I believe is something Reltio is good at. Furthermore, organisations need to be much better at working with their business partners in a modern way, including at the master data level. The business outcome of this is:

  • Having complete, accurate and timely data assets needed for understanding and connecting with customers. You will sell more.
  • Having a fast and seamless flow of data assets, not at least product information, to and from your trading partners. You will reduce costs.
  • Having a holistic view of internal and external data needed for decision making. You will mitigate risks.

MDM Will Go Cloud

How cloud is changing MDM (Master Data Management) is a subject examined in a very read worthy article by Julie Hunt published recently. The article is called How Does Technology Enable Effective MDM?

In here Julie says: “Adoption of cloud-based MDM or MDM-as-a-Service is on the rise, opening up new dimensions for how organizations take advantage of MDM and data governance.”

Julie’s article is part 3 of a six part series on the “New Age of Master Data Management”, so I may touch on a dimension that is covered in the upcoming articles. This dimension is how business ecosystems must be a part of your organizations MDM roadmap, and that dimension is, according to Gartner, the analyst firm, covering 8 underlying dimensions as told in the post From Business Ecosystem Strategy to PIM Technology.

Working with MDM in a business ecosystem context does require MDM in the cloud of some sort. Inhouse Mater Data Management and Product Information Management (PIM), which may be on premise or in the cloud or perhaps a hybrid, is only the beginning. Collaboration with business partners in a sophisticated environment will be controlled by a cloud solution.

More on this concept is explained in this piece about Master Data Share.

greatbeltbridge.jpg

MDM as Managed Service

This month I am going to London to attend the Master Data Management Summit Europe 2017.

As a teaser before the conference Aaron Zornes made a post called MDM Market 2017-18: Facts vs. Beliefs (with apologies to current political affairs fans!).

In his article, Aaron Zornes looks at the slow intake of multi-domain MDM, proactive data governance, graph technology and Microsoft stuff ending with stating that MDM as MANAGED SERVICE = HOT:

“Just as business users increasingly gave up on IT to deliver modest CRM in a timely, cost effective fashion (remember all the Siebel CRM debacles), so too are marketing and sales teams especially looking to improve the quality of their customer data… and pay for it as a “service” rather than as a complex, long-time-to-value capital expenditure that IT manages”.

Master Data ShareI second that, having been working with the iDQ™ service years ago, and will add, that the same will be true for product data as well and then eventually also multi-domain MDM.

How that is going to look like is explained here on Master Data Share.