The Practitioner's Guide to Data Quality Improvement

David Loshin

• Publisher: Morgan Kaufmann
• Year: 2010
• Print ISBN: 9780123737175
• eBook ISBN: 9780080920344
• Topic: Computer Science
• Subtopic: Data Processing
• Index: Computer Science

The Practitioner's Guide to Data Quality Improvement offers a comprehensive look at data quality for business and IT, encompassing people, process, and technology. It shares the fundamentals for understanding the impacts of poor data quality, and guides practitioners and managers alike in socializing, gaining sponsorship for, planning, and establishing a data quality program.It demonstrates how to institute and run a data quality program, from first thoughts and justifications to maintenance and ongoing metrics. It includes an in-depth look at the use of data quality tools, including business case templates, and tools for analysis, reporting, and strategic planning.This book is recommended for data management practitioners, including database analysts, information analysts, data administrators, data architects, enterprise architects, data warehouse engineers, and systems analysts, and their managers.- Offers a comprehensive look at data quality for business and IT, encompassing people, process, and technology.- Shows how to institute and run a data quality program, from first thoughts and justifications to maintenance and ongoing metrics.- Includes an in-depth look at the use of data quality tools, including business case templates, and tools for analysis, reporting, and strategic planning.

Chapter 1 Business Impacts of Poor Data Quality

Chapter outline

• 1.1 Information Value and Data Quality Improvement 3
• 1.2 Business Expectations and Data Quality 4
• 1.3 Qualifying Impacts 5
• 1.4 Some Examples 7
• 1.5 More on Impact Classification 11
• 1.6 Business Impact Analysis 13
• 1.7 Additional Impact Categories 14
• 1.8 Impact Taxonomies and Iterative Refinement 15
• 1.9 Summary: Translating Impact into Performance 16

Most organizations today depend on the use of data in two general ways. Standard business processes use data for executing transactions, as well as supporting operational activities. Business analysts review data captured as a result of day-to-day operations through reports and analysis engines as a way of identifying new opportunities for efficiency or growth. In other words, data is used to both run and improve the ways that organizations achieve their business objectives. If that is true, then there must be processes in place to ensure that data is of sufficient quality to meet the business needs. Therefore, it is of great value to any enterprise risk management program to incorporate a program that includes processes for assessing, measuring, reporting, reacting to, and controlling the risks associated with poor data quality.

Flaws in any process are bound to introduce risks to successfully achieving the objectives that drive your organization's daily activities. If the flaws are introduced in a typical manufacturing process that takes raw input and generates a single output, the risks of significant impact might be mitigated by closely controlling the quality of the process, overseeing the activities from end to end, and making sure that any imperfections can be identified as early as possible. Information, however, is an asset that is generated through numerous processes, with multiple feeds of raw data that are combined, processed, and fed out to multiple customers both inside and outside your organization. Because data is of a much more dynamic nature, created and used across the different operational and analytical applications, there are additional challenges in establishing ways to assess the risks related to data failures as well as ways to monitor conformance to business user expectations.

This uncovers a deeper question: to what extent does the introduction of flawed data impact the way that your organization does business? While it is probably easy to point to specific examples of where unexpected data led to business problems, there is bound to be real evidence of hard impacts that can be directly associated with poor quality data. Anecdotes are strong motivators in that they raise awareness of data quality as an issue, but our intention is to develop a performance management framework that helps to identify, isolate, measure, and improve the value of data within the environment. The problem is that the magnitude and challenge of correlating business impacts with data failures appear to be too large to be able to manage – thus the reliance on anecdotes to justify an investment in good data management practices.

But we can compare the job of characterizing the impacts of poor data quality to eating an elephant: it seems pretty big, but if we can carve it down into small enough chunks, it can be done one bite at a time. To be able to communicate the value of data quality improvement, it is necessary to be able to characterize the loss of value that is attributable to poor data quality.

This requires some exploration into assembling the business case, namely:

• • Reviewing the types of risks relating to the use of information,
• • Considering ways to specify data quality expectations,
• • Developing processes and tools for clarifying what data quality means,
• • Defining data validity constraints,
• • Measuring data quality, and
• • Reporting and tracking data issues,

all contributing to performance management reporting using a data quality scorecard, to support the objectives of instituting data governance and data quality control.

Many business issues can be tied, usually directly, to a situation where data quality is below user expectations. Given some basic understanding of data use, information value, and the ways that information value degrades when data does not meet quality expectations, we can explore different categories of business impacts attributable to poor information quality, and discuss ways to facilitate identification and classification of cost impacts related to poor data quality. In this chapter we look at the types of risks that are attributable to poor data quality as well as an approach to correlating business impacts to data flaws.

1.1 Information Value and Data Quality Improvement

Is information an organizational asset? Certainly, if all a company does is accumulate and store data, there is some cost associated with the ongoing management of that data – the costs of storage, maintenance, office space, support staff, and so on – and this could show up on the balance sheet as a liability. Though it is unlikely that any corporation lists its data as a line item as either an asset or a liability on its balance sheet, there is no doubt that, because of a significant dependence on data to both run and improve the business, senior managers at most organizations certainly rely on their data as much as any other asset.

We can view data as an asset, since data can be used to provide benefits to the company, it is controlled by the organization, it is the result of a sequence of transactions (either as the result of internal data creation internally or external data acquisition), it incurs costs for acquisition and management, and it is used to create value. Data is not treated as an asset, though; for example, there is no depreciation schedule for purchased data.

On the other hand, the dependence of automated operational systems on data for processing clearly shows how data is used to create value. Transaction systems that manage the daily operations enable “business as usual.” And when analytic systems are used for reporting, performance management, and discovery of new business opportunities, the value of that information is shown yet again. But it can be a challenge to assign a direct monetary value to any specific data value. For example, while a computing system may expect to see a complete record to be processed, a transaction may still be complete even in the absence of some of the data elements. Does this imply that those data elements have no value? Of course not, otherwise there would not have been an expectation for those elements to be populated in the first place.

There are different ways of looking at information value. The simplest approaches consider the cost of acquisition (i.e., the data is worth what we paid for it) or its market value (i.e., what someone is willing to pay for it). But in an environment where data is created, stored, processed, exchanged, shared, aggregated, and reused, perhaps the best approach for understanding information value is its utility – the expected value to be derived from the information.

That value can grow as a function of different aspects of the business, ranging from strictly operational to strategic. Sales transactions are necessary to complete the sales process, and therefore part of your sales revenues are related to the data used to process the transaction. Daily performa...

Table of contents

• Cover
• Title Page
• Copyright
• Table of Contents
• Foreword
• Preface
• Acknowledgments
• About the Author
• 1: Business Impacts of Poor Data Quality
• 2: The Organizational Data Quality Program
• 3: Data Quality Maturity
• 4: Enterprise Initiative Integration
• 5: Developing A Business Case and A Data Quality Road Map
• 6: Metrics and Performance Improvement
• 7: Data Governance
• 8: Dimensions of Data Quality
• 9: Data Requirements Analysis
• 10: Metadata and Data Standards
• 11: Data Quality Assessment
• 12: Remediation and Improvement Planning
• 13: Data Quality Service Level Agreements
• 14: Data Profiling
• 15: Parsing and Standardization
• 16: Entity Identity Resolution
• 17: Inspection, Monitoring, Auditing, and Tracking
• 18: Data Enhancement
• 19: Master Data Management and Data Quality
• 20: Bringing It All Together
• Index