This idea is also governing the many product data pools around. However, there are some serious considerations against this idea, namely:
- Being on the same standard and not to say on the same version within your business ecosystem is quite utopic (being that within your own organization is hard enough).
- It is not desirable to have the same product information as your competitors if you are going to compete on other factors than price.
The solution for that is Product Data Lake.
There is a need for a new solution to sharing product information between trading partners. Product Data Lake is that new solution. Using the term data lake as a part of the name for the solution is very deliberate. Here is why:
When setting up a warehouse, and a data warehouse, you have to estimate the storing size and the throughput. There will be a limit to how much data you can store and how much data you can upload and download within a given period.
Our vision is that Product Data Lake will be the process driven key service for exchanging any sort of product information within business ecosystems all over the world, with the aim of optimally assist self-service purchase of every kind of product.
In order to achieve that vision, we need to be able to scale up drastically. Therefore, we use a document-oriented database called MongoDB to store product information.
Even if you choose to implement a Product Data Lake instance for a single business ecosystem, you will benefit from the high scalability.
Business ecosystems changes all the time. You need to rapidly be able to adapt your data management, not at least when it comes to exchanging product information.
Swapping trading partners is one thing. That often means dealing with other product information requirements and opportunities and adhering to other standards.
We will also see business ecosystems in new shapes in the future. There will be fewer nodes between manufacturers and point-of-sales and point-of-sales will more likely be online marketplaces.
However, the changes will not happen as a big bang but in varying pace for each industry, geography and organization.
The rigid consensus structure of a data warehouse, and product information exchange solutions that resembles a data warehouse, will not cope with that change. The data lake concept, in the form of Product Data Lake, will.
In Product Data Lake you as a provider upload product information in your structure and format and you as a receiver download in your structure and format. The linking and transformation takes place inside Product Data Lake using linked metadata.
While everyone agrees that a common standard for all product information is the best answer we must on the other hand accept, that using a common standard for every kind of product and every piece of information needed is quite utopic. We haven’t even a common uniquely spelled term in English for standardization/standarisation.
Also, we must foresee that one organization will mature in a different pace than another organisation in the same business ecosystem.
These observations are the reasons behind the launch of Product Data Lake. In Product Data Lake we encompass the use of (in prioritized order):
- The same standard in the same version
- The same standard in different versions
- Different standards
- No standards
Product Information Management (PIM) is challenged by the fact that product data is the kind of data that usually flows cross company. The most common route starts with that the hard facts about a product originates at the manufacturer. Then the information may be used on the brand’s own website, propagated to a marketplace (online shop-in-shop) and propagated downstream to distributors and merchants.
The challenge to the manufacturer is that this represent many ways of providing product information, not at least when it comes to distributors and merchants, as these will require different structurers and formats using various standards and not being on the same maturity level.
Looking at this from the downstream side as a merchant you have the opposite challenge. Manufacturers provide product information in different structurers and formats using various standards and are not on the same maturity level.
Supply chain participants can challenge this in a passive or an active way. Unfortunately, many have chosen – or are about to choose – the passive way. It goes like this:
- As a manufacturer, we have a product data portal where trading partners who wants to do business with us, who obviously is the best manufacturer in our field, can download the product information we have in our structure and format using the standards we have found best.
- As a merchant we have a supplier product data portal where trading partners who wants to do business with us, the leading player in our field, can upload the product information we for the time being will require in our structure and format using the standard(s) we have found best.
- As a distributor, you could take both these standpoints.
This approach seems to work if you are bigger than your trading partner. And many times, one will be bigger than the other. But unless you are very big, you will in many cases not be the biggest. And in all cases where you are the biggest, you will not be a company being easy to do business with, which eventually will decide how big you will stay.
Using (often local) industry product data hubs is a workaround, but the challenges shines through and often it leads to that everyone will only benefit from what anyone can do and still calls for many touch points when doing business internationally and across several product data groups.
The better way is the active way creating a win-win situation for all trading partners as described in the article about Product Data Lake Business Benefits.
The concept of a data lake has until now mainly been used to describe how data gathered by a given organization can be organized in order to provide for analytical purposes. The data lake concept is closely tied to the term big data, which means that a data lake caters for handling huge volumes of data, with high velocity and where data comes in heaps of varieties. A data lake is much more agile than data marts and data warehouses, where you have to determine the purpose of the use of data beforehand.
In my eyes the idea about that every organization should gather all the data of interest behind its own firewall does not make sense. Some data will for sure be only for eyes of people behind the corporate walls. Here you should indeed put your private data into your own data lake. But a lot of data are common known data that everyone should not spend time on collecting and eventually cleansing. Here you should collaborate within your industry or other sphere around data lakes for public data.
Perhaps most importantly you should share data lakes with other members of your business ecosystem. You probably already do share data within business ecosystems with your trading partners. Until now such sharing has resembled the concepts of data marts and data warehouses being very purpose specific exchanges of data build on common beforehand understood standards for data.
Right now I am working on a cloud service called the Product Data Lake. Here we host a data lake for sharing product data in the business ecosystems of manufacturers, distributors, merchants and large end users of product information. You can join our journey by following us here on LinkedIn at the Product Data Lake LinkedIn Company Page.
When Gartner, the analyst firm, today evaluates MDM solutions they measure their strengths within these use cases:
- MDM of B2C Customer Data, which is about handling master data related to individuals within households acting as buyers (and users) of the products offered by an organisation
- MDM of B2B Customer Data, which is about handling master data related to other organizations acting as buyers (and users) of the products offered by an organisation.
- MDM of Buy-Side Product Data, which is about handling product master data as they are received from other organisations.
- MDM of Sell-Side Product Data, which is about handling product master data as they are provided to other organisations and individuals.
- Multidomain MDM, where all the above master data are handled in conjunction with other party roles than customer (eg supplier) and further core objects as locations, assets and more.
- Multivector MDM, where Gartner adds industries, scenarios, structures and styles to the lingo.
The core party and product picture could look like examined in the post An Alternative Multi-Domain MDM Quadrant. Compared to the Gartner Magic Quadrant lingo (and the underlying critical capabilities) this picture is different because:
- The distinction between B2B and B2C in customer MDM is diminishing and does not today make any significant differentiation between the solutions on the market.
- Handling customer as one of several party roles will be the norm as told in the post Gravitational Waves in the MDM World.
- We need (at least) one good MDMish solution to connect the buy-sides and the sell-sides in business ecosystems as pondered in the post Gravitational Collapse in the PIM Space.
The answer to the question about who is behind with EU General Data Protection Regulation (GDPR) readiness is in short:
- A lot of companies
- A lot of governments
When following the vibe around getting prepared for GDPR, and from my own involvement at clients, there is no doubt about that time is short and that not every company (well, probably only a few companies) within the European Union will be 100 % ready on 25th May 2018 and this also counts for those outside EU who is targeting EU citizens or processing personal data from the EU.
However, most EU governments are not any better. According to a recent communication from the EU only two Member States (Germany and Austria) have adopted the necessary national legislation. And from own experience I can tell that the late incoming of the national legislation does not help in getting the details ready for 25th May.
Some areas where national legislation is important were discussed in the post Where GDPR Still Becomes National. In my eyes, the remaining governments do not set an example for companies who are struggling with this (else justified) extra work.
Gartner, the analyst firm, defines five different types of digital technology platforms:
- Information system platform — Supports the back office operations such as ERP, CRM, PIM and other core systems with associated middleware and development capabilities.
- Customer experience platform — Contains the main customer-facing elements, such as customer and citizen portals, multichannel commerce, and customer apps.
- Analytics and intelligence platform — Contains information management and analytical capabilities. Data management programs and analytical applications fuel data-driven decision making, and algorithms automate discovery and action.
- IoT platform — Connects physical assets for monitoring, optimization, control and monetization.
- Business ecosystem platform — Supports the creation of, and connection to, external ecosystems, marketplaces and communities.
As a vendor of a modern data management platform, you will probably identify yourself primarily within one of these five types.
At Product Data Lake we are first and foremost a business ecosystem platform, being a cloud service for sharing product data in the business ecosystems of manufacturers, distributors, merchants and the end users of product information. As such, we are proud to be a part of the The Rise of Business Ecosystems in Data Management.
Of course, there are ties to the other types of digital technology platforms as well. As explained in the post Adding Things to Product Data Lake, the ecosystem approach is necessary to identify and track physical assets. Analytics will encompass data, as for example product data, in the business ecosystem. Customer experience in multichannel commerce when it comes to completeness of product information will require an effective cross company digital technology platform.
An external focused business ecosystem platform will have to be easily connected to the various internal focused information system platforms at trading partners. In our case, this is What a PIM-2-PIM Solution Looks Like.
When talking about Product Master Data Management (Product MDM) Product Information Management (PIM) I like to divide the different kinds of product data into the schema below:
Level 1, Basic Data
At the first level, we find the basic product data that typically is the minimum required for creating a product in any system of record.
Here we find the primary product identification number or code that is the internal key to all other product data structures and transactions related to the product within an organization.
Then there usually is a short product description. This description helps internal employees identifying a product and distinguishing that product from other products. Most often the product is named in the official language of the company.
If an upstream trading partner produces the product, we may find the identification of that supplier here too. If the product is part of internal production, we may have a material type telling about if it is a raw material, semi-finished product, finished good or packing material.
Level 2, Trading Data
The second level has product data related to trading the product. We may have a unique Global Trade Item Number (GTIN) that may be in the form of an International – former European – Article Number (EAN) or a Universal Product Code (UPC). Here we have commodity codes and a lot of other product data that supports buying, receiving, selling and delivering the product.
Level 3, Recognition Data
On the third level, we find the two basic pieces of product information that came to existence when we started producing product catalogues and had the first ecommerce solutions online.
The extended product description is needed because the usual short product description used internally have no meaning to an outsider as told in the post Customer Friendly Product Master Data. Some good best practices for governing the extended product description is to have a common structure of how the description is written, not to use abbreviations and to have a strict vocabulary as reported in the post Toilet Seats and Data Quality.
We often see that the extended product descriptions need to be present in the range of languages covering the locations where business is done either if the business is international or done in a country with multiple countries. The trend of increased user customization (or should I say customisation) drives this point further.
Having a product image is pivotal if you want to sell something without showing the real product face-to-face with the customer or other end user. A missing product image is a sign of a broken business process for collecting product data as pondered in the post Image Coming Soon.
Level 4, Self-service Data
At the fourth level, we have three main sorts of product information: Product attributes, basic product relations and standard digital assets. These data supports when customers makes buying decisions within eCommerce and other self-service scenarios.
Product attributes are also sometimes called product properties or product features. These are up to thousands of different data elements that describes a product. Some are very common for most products like height, length, weight and colour. Some are very specific to the product category. This challenge is the reason of being for dedicated Product Information Management (PIM) solutions as told in the post MDM Tools Revealed.
Basic product relations are the links between a product and other products like a product that have several different accessories that goes with the product or a product being a successor of another now decommissioned product. Product relations is described further in the post Related Products: The Often Overlooked Facet of PIM.
Standard digital assets are documents like installation guides, line drawings and data sheets as examined in the post Digital Assets and Product MDM.
Level 5, Competitive Data
As the fifth level we find elements like on the fourth level, but usually these are elements that you won’t necessarily apply to all products but only to your top products where you want to stand out from the crowd and distance yourself from your competitors. If you are a reseller, you typically make these data yourself, where level 4 hard facts are delivered from the manufacturer, as examined in the post Using Internal and External Product Information to Win.
Special content are descriptions of and stories about the product above the hard features. Here you tell about why the product is better than other products and in which circumstances the product can to be used. A common aim with these descriptions is also Search Engine Optimization.
X-sell (cross-sell) and up-sell product relations applies to your particular mix of products and may be made subjective as for example to look at up-sell from a profit margin point of view. X-sell and up-sell relations may be defined from upstream by you or your upstream trading partners but also dripping down on the roof from the behaviour of your downstream trading partners / customers as manifested in the classic webshop message: “Those who bought product A also bought / looked at product B”.
Advanced digital assets are broader and more lively material than the hard fact line drawings and other documents. Increasingly newer digital media types as video are used for this purpose.
Product Classification, Product Pricing and Product Lifecycle Status
All of the above-mentioned levels of product information is supported by product classification. Usually we see product classification handled as a reference data type across Product Information Management (PIM), ERP and Product Lifecycle Management (PLM) where applicable.
Product pricing is usually also a subject mainly belonging to the ERP side of things.
Product Lifecycle Status again goes across Product Information Management (PIM), ERP and not at least Product Lifecycle Management (PLM) where applicable.
Master Data Management (MDM) is the discipline that connects the dots between these topics.
Take the processes to next level:
You can take your Product Information Management (PIM) and Product Master Data Management (Product MDM) to a higher level by following the processes as described in the post Using Pull or Push to Get to the Next Level in Product Information Management.
In here David explores the topics in the new era of the customer 360 degree view being encompassing all customer data, covering analytical and operational usages and improving customer experience.
The post includes this testimonial from Deotis Harris, Senior Director, MDM at Dell EMC: “We saw an opportunity to leverage AllSight’s modern technology (Customer Intelligence), coupled with our legacy systems such as Master Data Management (MDM), to provide the insight required to enable our sellers, marketers and customer service reps to create better experiences for our customers.”
By the way: Being a MDM practitioner who have spent many years with customer 360 and now spending equal chunks of time with product 360, I find the forward-looking topics being very similar between customer 360 and product 360. In short:
- The span of product data to handle has increased dramatically in recent years as told in the post Self-service Ready Product Data.
- We can use the same data architecture for analytical and operational purposes as mentioned in the post The Intersection of MDM and Big Data.
- It is all about creating better experiences for your customers.
Product Data Lake is a cloud service for sharing product data in the business ecosystems of manufacturers, distributors, merchants, marketplaces and large end users of product information.
Yesterday some new capabilities have been included in the service.
As a manufacturer you can now utilize Product Data Lake even more as a cloud based lightweight Product Information Management (PIM) system. We have added better views of uploaded product information and better means of managing product data within the service. This will be of benefit for manufacturers who already handles product data in ERP and Product Lifecycle Management (PLM) solutions and needs a cost-effective solution to share these data with trading partners. Learn more about this option on our Product Data Push site.
Also, independent providers of hubs of product information within a given industry and/or geography can now self-register as a reservoir inside Product Data Lake and thus add a modern and generic way of collecting and distributing product information to existing specialized product data pools.
But we do not stop there. The next version 1.4 will be live just before our Far East development team takes some time off for the Lunar New Year. This version adds new possibilities for pushing product information through Product Data Lake. We already support file drops via FTP domains, traditional interactive upload from network drives and direct data entry. Next option is APIs.
Further versions during the coming months covers deeper integration of popular product information standards such as ETIM, eClass and UNSPSC. Learn more about these standards in the post Five Product Classification Standards.
If you want a presentation of current and future capabilities within Product Data Lake, please make contact below: