5 Data Management Mistakes to Avoid during Data Integration Projects

mistake-876597_1920

I am very pleased to welcome today’s guest blogger. Canada based Maira Bay de Souza of Product Data Lake Technologies shares her view on data integration and the mistakes to avoid doing that:

Throughout my 5 years of working with Data Integration, Data Migration and Data Architecture, I’ve noticed some common (but sometimes serious) mistakes related to Data Management and Software Quality Management. I hope that by reading about them you will be able to avoid them in your future Data Integration projects.

 1 Ignoring Data Architecture

Defining the Data Architecture in a Data Integration project is the equivalent of defining the Requirements in a normal (non-data-oriented) software project. A normal software application is (most of the times) defined by its actions and interactions with the user. That’s why, in the first phase of software development (the Requirements Phase), one of the key steps is creating Use-Cases (or User Stories). On the other hand, a Data Integration application is defined by its operations on datasets. Interacting with data structures is at the core of its functionality. Therefore, we need to have a clear picture of what these data structures look like in order to define what operations we will do on them.

 It is widely accepted in normal software development that having well-defined requirements is key to success. The common saying “If you don’t know where you’re going, any road will get you there” also applies for Data Integration applications. When ETL developers don’t have a clear definition of the Data Architecture they’re working with, they will inevitably make assumptions. Those assumptions might not always be the same as the ones you, or worse, your customer made.

(see here and here for more examples on the consequences of not finding software bugs early in the process due to by badly defined requirements)

 Simple but detailed questions like “can this field be null or not?” need to be answered. If the wrong decision is made, it can have serious consequences. Most senior Java programmers like me are well aware of the infamous “Null Pointer Exception“. If you feed a null value to a variable that doesn’t accept null (but you don’t know that that’s the case because you’ve never seen any architecture specification), you will get that error message. Because it is a vague message, it can be time-consuming to debug and find the root cause (especially for junior programmers): you have to open your ETL in the IDE, go to the code view, find the line of code that is causing the problem (sometimes you might even have to run the ETL yourself), then find where that variable is located in the design view of your IDE, add a fix there, test it to make sure it’s working and then deploy it in production again. That also means that normally, this error causes an ETL application to stop functioning altogether (unless there is some sort of error handling). Depending on your domain that can have serious, life-threatening consequences (for example, healthcare or aviation), or lead to major financial losses (for example, e-commerce).

 Knowing the format, boundaries, constraints, relationships and other information about your data is imperative to developing a high quality Data Integration application. Taking the time to define the Data Architecture will prevent a lot of problems down the road.

2 Doing Shallow Data Profiling

Data profiling is another key element to developing good Data Integration applications.

 When doing data profiling, most ETL developers look at the current dataset in front of them, and develop the ETL to clean and process the data in that dataset. But unfortunately that is not enough. It is important to also think about how the dataset might change over time.

 For example, let’s say we find  a customer in our dataset with the postal code in the city field. We then add an instruction in the ETL for when we find that specific customer’s data, to extract the postal code from the city field and put it in the postal code field. That works well for the current dataset. But what if next time we run the ETL another customer has the same problem? (it could be because the postal code field only accepts numbers and now we are starting to have Canadian customers, who have numbers and letters in the postal code, so the user started putting the postal code in the city field)

Not thinking about future datasets means your ETL will only work for the current dataset. However, we all know that data can change over time (as seen in the example above) – and if it is inputted by the user, it can change unpredictably. If you don’t want to be making updates to your ETL every week or month, you need to make it flexible enough to handle changes in the dataset. You should use data profiling not only to analise current data, but also to deduce how it might change over time.

Doing deep data profiling in the beginning of your project means you will spend less time making updates to the Data Cleaning portion of your ETL in the future.

 3 Ignoring Data Governance

 This point goes hand-in-hand with my last one.

 A good software quality professional will always think about the “what if” situations when designing their tests (as opposed to writing tests just to “make sure it works”). In my 9 years of software testing experience, I can’t tell you how many times I asked a requirements analyst “what if the user does/enters [insert strange combination of actions/inputs here]?” and the answer was almost always “the user will never do that“. But the reality is that users are unpredictable, and there have been several times when the user did what they “would never do” with the applications I’ve tested.

The same applies to data being inputted into an ETL. Thinking that “data will never come this way” is similar to saying “the user will never do that“. It’s better to be prepared for unexpected changes in the dataset instead of leaving it to be fixed later on, when the problem has already spread across several different systems and data stores. For example, it’s better to add validation steps to make sure that a postal code is in the right format, instead of making no validation and later finding provinces in the postal code field. Depending on your data structures, how dirty the data is and how widespread the problem is, the cost to clean it can be prohibitive.

This also relates to my first point: a well-defined Data Architecture is the starting point to implementing Data Governance controls.

 When designing a high quality Data Integration application, it’s important to think of what might go wrong, and imagine how data (especially if it’s inputted by a human) might be completely different than you expect. As demonstrated in the example above, designing a robust ETL can save hours of expensive manual data cleaning in the future.

 4 Confusing Agile with Code-And-Fix

 A classic mistake in startups and small software companies (especially those ran by people without a comprehensive education or background in Software Engineering) is rushing into coding and leaving design and documentation behind. That’s why the US Military and CMU created the CMMI: to measure how (dis)organized a software company is, and help them move from amateur to professional software development. However, the compliance requirements for a high maturity organization are impractical for small teams. So things like XP, Agile, Scrum, Lean, etc have been used to make small software teams more organized without getting slowed down by compliance paperwork.

Those techniques, along with iterative development, proved to be great for startups and innovative projects due to their flexibility. However, they can also be a slippery slope, especially if managers don’t understand the importance of things like design and documentation. When the deadlines are hanging over a team’s head, the tendency is always to jump into coding and leave everything else behind. With time, managers start confusing agile and iterative development with code-and-fix.

 Throughout my 16 years of experience in the Software Industry, I have been in teams where Agile development worked very well. But I have also been in teams where it didn’t work well at all – because it was code-and-fix disguised as Agile. Doing things efficiently is not the same as skipping steps.

Unfortunately, in my experience this is no different in ETL development. Because it is such a new and unpopular discipline (as opposed to, for example, web development), there aren’t a lot of software engineering tools and techniques around it. ETL design patterns are still in their infancy, still being researched and perfected in the academic world. So the slippery slope from Agile to code-and-fix is even more tempting.

 What is the solution then? My recommendation is to use the proven, existing software engineering tools and techniques (like design patterns, UML, etc) and adapt them to ETL development. The key here is to do something. The fact that there is a gap in the industry’s body of knowledge is no excuse for skipping requirements, design, or testing, and jumping into “code-and-fix disguised as Agile“. Experiment, adapt and find out which tools, methodologies and techniques (normally used in other types of software development) will work for your ETL projects and teams.

5 Not Paying Down Your Technical Debt

The idea of postponing parts of your to-do list until later because you only have time to complete a portion of them now is not new. But unfortunately, with the popularization of agile methodologies and incremental development, Technical Debt has become an easy way out of running behind schedule or budget (and masking the root cause of the problem which was an unrealistic estimate).

As you might have guessed, I am not the world’s biggest fan of Technical Debt. But I understand that there are time and money constraints in every project. And even the best estimates can sometimes be very far from reality – especially when you’re dealing with a technology that is new for your team. So I am ok with Technical Debt, when it makes sense.

However, some managers seem to think that technical debt is a magic box where we can place all our complex bugs, and somehow they will get less complex with time. Unfortunately, in my experience, what happens is the exact opposite: the longer you owe technical debt (and the more you keep adding to it), the more complex and patchy the application becomes. If you keep developing on top of – or even around – an application that has a complex flaw, it is very likely that you will only increase the complexity of the problem. Even worse, if you keep adding other complex flaws on top of – or again, even around – it, the application becomes exponentially complex. Your developers will want to run away each time they need to maintain it. Pretty soon you end up with a piece of software that looks more like a Frankenstein monster than a clean, cohesive, elegant solution to a real-world problem. It is then only a matter of time (usually very short time) before it stops working altogether and you have no choice but to redesign it from scratch.

This (unfortunately) frequent scenario in software development is already a nightmare in regular (non-data-oriented) software applications. But when you are dealing with Data Integration applications, the impact of dirty data or ever-changing data (especially if it’s inputted by a human), combined with the other 4 Data Management mistakes I mentioned above, can quickly escalate this scenario into a catastrophe of epic proportions.

So how do you prevent that from happening? First of all, you need to have a plan for when you will pay your technical debt (especially if it is a complex bug). The more complex the required change or bug is, the sooner it should be dealt with. If it impacts a lot of other modules in your application or ecosystem, it is also important to pay it off sooner rather than later. Secondly, you need to understand why you had to go into technical debt, so that you can prevent it from happening again. For example, if you had to postpone features because you didn’t get to them, then you need to look at why that happened. Did you under-estimate another feature’s complexity? Did you fail to account for unknown unknowns in your estimate? Did sales or your superior impose an unrealistic estimate on your team? The key is to stop the problem on its tracks and make sure it doesn’t happen again. Technical Debt can be helpful at times, but you need to manage it wisely.

 I hope you learned something from this list, and will try to avoid these 5 Data Management and Software Quality Management mistakes on your next projects. If you need help with Data Management or Software Quality Management, please contact me for a free 15-min consultation.

Maira holds a Bsc in Computer Science, 2 software quality certifications and over 16 years of experience in the Software Industry. Her open-mindedness and adaptability have allowed her to thrive in a multidisciplinary career that includes Software Development, Quality Assurance and Project Management. She has taken senior and consultant roles at Fortune 20 companies (IBM and HP), as well as medium and small businesses. She has spent the last 5 years helping clients manage and develop software for Data Migration, Data Integration, Data Quality and Data Consistency. She is a Product Data Lake Ambassador & Technology Integrator through her startup Product Data Lake Technologies.

The Real Reason Why Your Business Needs a PIM Tool

Today’s guest blog post is the second one from Dan O’Connor, a United States based product data taxonomy guru. Here are Dan’s thoughts on why you should have a Product Information Management (PIM) tool:

Over the past year I have moved from a position of watching a Product Information Management tool, or PIM, being installed, to working for a PIM vendor, to working through the process of installing a PIM tool from the client side. In the same way that I justified buying a sports car to my wife based on the utilitarian value of having 350 horsepower at my disposal, I’ve seen many different justifications for installing a PIM tool. From “Micro Moments” to “collaborative data collection” and “syndication”, terms are tossed around that attempt to add to the value of a PIM installation.

The simple truth is there is only one reason you need a PIM tool. Every justification is solving a symptom of a data problem in a business, not the core problem. Every good management executive knowns that solving symptoms is a rabbit hole that can cost time and money at an incredible rate, so understanding what the core problem that requires a PIM in your business is vital to your business growth.

PIM messageControlling your Messaging

That core problem your business needs to solve is product messaging. Simply put, without a central hub for your data your business has a lack of control over how your product messaging is spread both internally and externally.  If you are still working in spread sheets or collecting data multiple times for a single product for different channels you have lost most of your product messaging structure.

PIM is a tool that solves that problem, and the symptomology that comes with it. Does your business spend too much time assembling data to meet downstream partner needs? You have a product messaging problem. Is your business’ ability to ingest data limited by spread sheets transferred over network folders or email? You have a product messaging problem.

All the benefits of PIM can be summed up into a simple statement: If you want to be in control of your product brand and your product data quality your business needs a PIM tool. Do you want to reduce product data setup costs? You need a central location for all your product messaging to do so. Does your business have product data quality issues that occur due to poor adherence to best practices? Poor data quality affects your product messaging, and can be solved by a PIM tool. Is your business spending too much time chasing down emails with product specs and spread sheets full of setup data? These bad workflow practices affect your ability to provide a consistent message downstream to your business partners, whether your business is B2B or B2C. They are a symptom of your poor product messaging control.

The True PIM ROI Story

The central premise of a PIM tool is to standardize and normalize your product data collection and setup workflows and processes. If your business looks at a PIM tool only for this metric your vision for PIM is limited. Syndication, the distribution of data to consuming internal and external systems, is another huge benefit to PIM. However, if the product messaging your PIM system is sending or receiving is not well controlled within your PIM your vision is incomplete. There is not a single benefit to PIM that you cannot add the terms “with a consistent approach to your product messaging” to the end of.

Why is product messaging so important? In previous blogs I have demonstrated how failures in product messaging lead to odd product experiences, especially when you look at the messaging across platforms. If your web store shows a length for a product and your channel partner shows a different length you have a product messaging problem. If that product data came from a central source that issue would not exist. It might be as simple as the downstream partner swapped length for depth and there isn’t a true data issue, but to your customers there is an inconsistent product data message.

Extrapolating this out to something as simple as web descriptions actually validates this business case. If you provide a basic web description for a product based on an individual manually typing in marketing copy into a web portal you have lost control of your product messaging. That same person may be responsible for typing that web description in 4 different places, and without a central repository for that data the chances that those 4 messages will complement each other is slim. Add to that the fact that many major retailers edit web descriptions to conform to their standards after your business has completed product setup and you are less in control of your product messaging than you imagined.

Having a PIM tool solves this. You have a single source for web descriptions that you know will be represented in a singular repeatable fashion downstream. You can map your dimension attributes to your downstream channel partner dimensions, ensuring that the appropriate data appears in each field. You can customize web descriptions in a controlled and normalized environment so that you have more control over how those descriptions are customized by your channel partners.

The Importance of Product Messaging

Product messaging is your voice to your customers. As B2B ecommerce follows the path blazed by B2C it has become more important to have a consistent and controlled message for your products to all your customers. Spread sheets are not capable of that task, and email is not a mechanism for maintaining product data quality. Automated systems with proper workflows and data quality checks are paramount to ensuring the voice you expect your customers to hear is your business’ voice.

Reducing catalog printing costs, syndication of product data to channel partners, and reducing product setup headcount are valid reasons to install a PIM tool. However, they all should be part of a greater goal to control your voice to your customers. Those benefits are symptoms of a need in your business to have a unifying voice, and not including product messaging control as the overriding goal of your PIM installation is a strategic error.

In having performed many PIM installations here is the impact of not seeing product messaging control as the overarching goal. A company I worked with went through the process of installing a PIM tool, and we reached the point of remediating their existing product data to fit the new model. This company, who had invested heavily in this project, decided they did not want to perform any data remediation. They simply added back into their PIM tool every attribute that had existed in their old system. There was vision to improve the data they were displaying to their customers: They simply wanted to speed up product setup.

That business has spent the last 6 months undoing the benefits on controlled product messaging. It was less costly to them in the short term to simply replicate their existing data issues in a new system. Their old product data was unwieldly, hyper-specific to channel, and involved writing product titles and web descriptions manually for each channel. There is no common theme to the product messaging they are creating, and their ability to reduce product setup costs has been hampered by these decisions.

In Summary: Product Data is Your Product Messaging

Micro moments and product experience management is just fancy terminology for what is simply an understanding of the importance of your product data. If your vision is to control your product messaging, you have to start with your product data. A PIM tool is the only functional approach that meets that goal, but has to be looked at as a foundational piece of that product messaging. Attempting to reduce product setup costs or speed product data transfer is a valid business goal and a justification for a PIM project, but the true visionary approach has to include an overall product messaging approach. Otherwise, your business is limiting the return on investment it will achieve from any attempt to solve your product data setup and distribution problems.

Dan O’Connor is a Product Taxonomy, Product Information Management (PIM), and Product Data Consultant and an avid blogger on taxonomy topics. He has developed taxonomies for major retails as well as manufacturers and distributors, and assists with the development of product data models for large and small companies. See his LinkedIn bio for more information.

The Rise of Business Ecosystems in Data Management

There are many signs showing that we are entering the age of business ecosystems. A recent example is an article from Digital McKinsey. This read worthy article is called Adopting an ecosystem view of business technology.

In here, the authors emphasizes on the need to adapt traditional IT functions to the opportunities and challenges of emerging technologies that embraces business ecosystems. I fully support that sentiment.

In my eyes, some of the emerging technologies we see are in large misunderstood as something meant for being behind the corporate walls. My favorite example is the data lake concept. I do not think a data lake will be an often seen success solely within a single company as explained in the post Data Lakes in Business Ecosystems.

The raise of technology for business ecosystems will also affect the data management roles we know today. For example, a data steward will be a lot more focused towards external data than before as elaborated in the post The Future of Data Stewardship.

Encompassing business ecosystems in data management is of course a huge challenge we have to face while most enterprises still have not reached an acceptable maturity when it comes internal data and information governance. However, letting the outside in will also help in getting data and information right as told in the post Data Sharing Is The Answer To A Single Version Of The Truth.

biz-eco

Alternatives to Product Data Lake

Within Product Information Management (PIM) there is a growing awareness about that sharing product information between trading partners is a very important issue.

So, how do we do that? We could do that, on a global scale, by using:

  • 1,234,567,890 spreadsheets
  • 2,345,678 customer data portals
  • 901,234 supplier data portals

Spreadsheets is the most common mean to exchange product information between trading partners today. The typical scenario is that a receiver of product information, being a downstream distributor, retailer or large end user, will have a spreadsheet for each product group that is sent to be filled by each supplier each time a new range of products is to be on-boarded (and potentially each time you need a new piece of information). As a provider of product information, being a manufacturer or upstream distributor, you will receive a different spreadsheet to be filled from each trading partner each time you are to deliver a new range of products (and potentially each time they need a new piece of information).

Customer data portals is a concept a provider of product information may have, plan to have or dream about. The idea is that each downstream trading partner can go to your customer data portal, structured in your way and format, when they need product information from you. Your trading partner will then only have to deal with your customer data portal – and the 1,234 other customer data portals in their supplier range.

Supplier data portals is a concept a receiver of product information may have, plan to have or dream about. The idea is that each upstream trading partner can go to your supplier data portal, structured in your way and format, when they have to deliver product information to you. Your trading partner will then only have to deal with your supplier data portal – and the 567 other supplier data portals in their business-to-business customer range.

Product Data Lake is the sound alternative to the above options. Hailstorms of spreadsheets does not work. If everyone has either a passive customer data portal or a passive supplier data portal, no one will exchange anything. The solution is that you as a provider of product information will push your data in your structure and format into Product Data Lake each time you have a new product or a new piece of product information. As a receiver you will set up pull requests, that will give you data in your structure and format each time you have a new range of products, need a new piece of information or each time your trading partner has a new piece of information.

Learn more about how that works in Product Data Lake Documentation and Data Governance.

alternatives
Potential number of solutions / degree of dissatisfaction / total cost of ownership

 

Is blockchain technology useful within MDM?

This question was raised on this blog back in January this year in the post Tough Questions About MDM.

Since then the use of the term blockchain has been used more and more in general and related to Master Data Management (MDM). As you know, we love new fancy terms in our else boring industry.

blockchainHowever, there are good reasons to consider using the blockchain approach when it comes to master data. A blockchain approach can be coined as centralized consensus, which can be seen as opposite to centralized registry. After the MDM discipline has been around for more than a decade, most practitioners agree that the single source of truth is not practically achievable within a given organization of a certain size. Moreover, in the age of business ecosystems, it will be even harder to achieve that between trading partners.

This way of thinking is at the backbone of the MDM venture called Product Data Lake I’m working with right now. Yes, we love buzzwords. As if cloud computing, social network thinking, big data architecture and preparing for Internet of Things wasn’t enough, we can add blockchain approach as a predicate too.

In Product Data Lake this approach is used to establish consensus about the information and digital assets related to a given product and each instance of that product (physical asset or thing) where it makes sense. If you are interested in how that develops, why not follow Product Data Lake on LinkedIn.

Bookmark and Share

Approaches to Sharing Product Information in Business Ecosystems

One of the most promising aspects of digitalization is sharing information in business ecosystems. In the Master Data Management (MDM) realm, we will in my eyes see a dramatic increase in sharing product information between trading partners as touched in the post Data Quality 3.0 as a stepping-stone on the path to Industry 4.0.

Standardization (or standardisation)

A challenge in doing that is how we link the different ways of handling product information within each organization in business ecosystems. While everyone agrees that a common standard is the best answer we must on the other hand accept, that using a common standard for every kind of product and every piece of information needed is quite utopic. We haven’t even a common uniquely spelled term in English.

Also, we must foresee that one organization will mature in a different pace than another organisation in the same business ecosystem.

Product Data Lake

These observations are the reasons behind the launch of Product Data Lake. In Product Data Lake we encompass the use of (in prioritized order):

  • The same standard in the same version
  • The same standard in different versions
  • Different standards
  • No standards

In order to link the product information and the formats and structures at two trading partners, we support the following approaches:

  • Automation based on product information tagged with a standard as explained in the post Connecting Product Information.
  • Ambassadorship, which is a role taken by a product information professional, who collaborates with the upstream and downstream trading partner in linking the product information. Read more about becoming a Product Data Lake ambassador here.
  • Upstream responsibility. Here the upstream trading partner makes the linking in Product Data Lake.
  • Downstream responsibility. Here the downstream trading partner makes the linking in Product Data Lake.

cross-company-data-governanceData Governance

Regardless of the mix of the above approaches, you will need a cross company data governance framework to control the standards used and the rules that applies to the exchange of product information with your trading partners. Product Data Lake have established a partnership with one of the most recommended authorities in data governance: Nicola Askham – the Data Governance Coach.

For a quick overview please have a look at the Cross Company Data Governance Framework.

Please request more information here.

Bookmark and Share