{"id":4265,"date":"2022-08-30T14:29:34","date_gmt":"2022-08-30T18:29:34","guid":{"rendered":"https:\/\/solutionsreview.com\/data-management\/?p=4265"},"modified":"2022-09-16T10:47:00","modified_gmt":"2022-09-16T14:47:00","slug":"what-is-data-observability-and-how-does-it-improve-data-quality","status":"publish","type":"post","link":"https:\/\/solutionsreview.com\/data-management\/what-is-data-observability-and-how-does-it-improve-data-quality\/","title":{"rendered":"What is Data Observability and How Does it Improve Data Quality?"},"content":{"rendered":"

\"What<\/p>\n

This is part of Solutions Review\u2019s Premium Content Series, a collection of contributed columns written by industry experts in maturing software categories. In this submission, Kensu<\/a> Founder and CPO Andy Petrella answers the question “What is Data Observability?” and offers some key ways it helps to improve data quality.<\/em><\/strong><\/p>\n

\"SRData quality issues significantly impact thousands of organizations every day. Among others, low data quality can break automated reports<\/a><\/strong><\/span> with figures that do not add up, or they can lead to ML models producing poor recommendations. These result in poor user experiences or wrong business decisions including investment in the wrong sectors, reducing production or capacity rather than increasing it, and a multitude of other issues that come with high price tags.<\/p>\n

Gartner research in 2018 found organizations estimated poor data quality<\/a><\/strong><\/span> to be responsible for $15 million per year in losses. A recent D&B survey found that 1 in 5 businesses lose revenue and customers due to incomplete data. As businesses become increasingly data-driven today, and they are collecting and processing more and more data, this will only lead to more frequent data quality<\/a><\/strong><\/span> issues and costly missteps.<\/p>\n

Over time, these issues also lead to a drop in confidence in the data, causing the decision-makers to be cautious when acting on it and show less enthusiasm when deciding the budget allocated to the data department.<\/p>\n

When these data issues and their consequences rise to the surface, sometimes, after having impacted the business for weeks or months, data teams must locate and troubleshoot them. This exercise can take hours, days, and sometimes even weeks as multiple stakeholders might be involved: those who collect the data, those who process it, and finally, those who consume it.<\/p>\n

Patiently the data team will have to dig in and explore the pipelines and their applications, wasting their time while enduring some backfire from the business users. Sometimes, they will be able to find the root cause and adequately fix the problem; sometimes, they will run out of time and implement some quick fixes, which will probably cause more problems in the future.<\/p>\n

\t\t\t
\"Download<\/a><\/div>\n\t\t<\/div><\/div>\n

Know Your Data at its Source<\/strong><\/h3>\n

A solution to these problems is to observe the data directly into the pipelines, where and when it is being processed. This approach, called data observability<\/a><\/strong><\/span>, is achieved by automatically:<\/p>\n

    \n
  1. Recording where the data comes from<\/li>\n
  2. Monitoring the quality of data (e.g., distribution, volume, average)<\/li>\n
  3. Logging how the applications process it<\/li>\n
  4. Documenting where the data goes<\/li>\n<\/ol>\n

    This information helps data teams understand their data pipeline health and address, in no particular order, the leading causes of data incidents:<\/p>\n