‘Dirty Data’ is a Business Problem, Not an IT Problem, Says Gartner September 9, 2010
This Resource was created by: Gartner Plc
More than 25 Percent of Critical Data in the World’s Top Companies is Flawed
SYDNEY, Australia, March 2, 2007 —
Over the next two years, more than 25 percent of critical data in Fortune 1000 companies will continue to be flawed, that is, the information will be inaccurate, incomplete or duplicated, according to research and advisory firm Gartner, Inc. Gartner expects that three-quarters of large enterprises will make little to no progress towards improving data quality until 2010. To gain competitive advantage from information, organisations need to identify ‘data stewards’ in the business and manage information as a corporate asset.
Speaking at the company’s inaugural Business Intelligence and Information Management Summit in Sydney this week, Gartner Research vice president Andreas Bitterer said that “dirty data” or poor data quality is an often-overlooked business issue and it can have a large negative impact on a business.
“There is not a company on the planet that does not have a data quality problem,” said Mr Bitterer. “And where a company does recognise they have a problem, they often underestimate the size of it. It’s also important to realise that data quality is not static, as various forces cause it to change over time. So addressing this issue is not a one-time project, but an ongoing program that requires commitment and often a cultural shift as well.”
The cost of poor data quality
Gartner research shows that poor-quality customer data leads to significant costs, such as higher customer turnover, excessive expenses from customer contact processes like mail-outs and missed sales opportunities. But companies are now discovering that data quality has a significant impact on their most strategic business initiatives, not only sales and marketing. Other back-office functions like budgeting, manufacturing and distribution are also affected.
Compliance and transparency are now at the top of the list of most companies’ data concerns, according to Gartner. Legislation such as the US Sarbanes-Oxley Act, the European Basel Accords and the Privacy Act in Australia now demands that information is accurate and managed appropriately.
“By introducing data quality initiatives, some companies have added millions of dollars to their bottom line as they gain benefits such as increased sales, lower distribution costs and better compliance,” said Mr Bitterer.
Fixing the problem of ‘dirty data’
“Data quality is not an IT problem. IT can help fix it, but the business must own the problem,” said Mr Bitterer. “For example, company culture can have a significant influence. Organisations need ‘data stewards’, people within the business who are responsible for the quality of the information. However, technology will play a role in fixing many data quality issues, and organisations need to invest in a portfolio of data quality solutions such as profiling, cleansing, matching and enrichment.”
According to Gartner, data quality has many facets, including:
- Existence (whether the organisation has the data)
- Validity (whether the data values fall within an acceptable range or domain).
- Consistency (for example, whether the same piece of data stored in multiple locations contains the same values)
- Integrity (the completeness of relationships between data elements and across data sets)
- Accuracy (whether the data describes the properties of the object it is meant to model)
- Relevance (whether the data is the appropriate data to support the business objectives)
The first two of these aspects of quality represent a good staring point for improvement, as they can be measured fairly easily and will help identify major gaps in the data. From there, organisations should look at including other attributes in their definition of data quality.
“Many companies are trying to improve data quality ‘by hand’, writing their own correction or cleansing routines, when instead they should look at tools already available which can help them,” said Mr Bitterer.
According to Gartner, the market for data quality tools is currently small (approximately US$300 million in annual license revenue) but growing. Large vendors, such as Business Objects, IBM and Pitney Bowes, have entered via acquisitions, while numerous small vendors have developed their own data quality technology.
In Gartner’s annual survey of 1400 CIOs worldwide released this month, business intelligence was again ranked the number one technology priority, as organisations increasingly rely on data todrive growth and innovation.
For more information about Gartner’s Business Intelligence and Information Management Summit, please visit www.gartner.com/ap/bi
+61 2 9459 4692
Gartner, Inc. (NYSE: IT) is the world’s leading information technology research and advisory company. Gartner deliver the technology-related insight necessary for its clients to make the right decisions, every day. From CIOs and senior IT leaders in corporations and government agencies, to business leaders in high-tech and telecom enterprises and professional services firms, to technology investors, Gartner is the indispensable partner to approximately 60,000 clients in 10,000 distinct organizations. Through the resources of Gartner Research, Gartner Executive Programs, Gartner Consulting and Gartner Events, Gartner works with every client to research, analyze and interpret the business of IT within the context of their individual role. Founded in 1979, Gartner is headquartered in Stamford, Connecticut, U.S.A., and has approximately 4,300 associates, including approximately 1,200 research analysts and consultants serving clients in 80 countries. For more information, visit http://www.gartner.com/.