{"id":4072,"date":"2022-06-24T12:40:42","date_gmt":"2022-06-24T16:40:42","guid":{"rendered":"https:\/\/solutionsreview.com\/data-management\/?p=4072"},"modified":"2022-06-24T13:36:16","modified_gmt":"2022-06-24T17:36:16","slug":"data-quality-vs-data-observability-whats-the-difference","status":"publish","type":"post","link":"https:\/\/solutionsreview.com\/data-management\/data-quality-vs-data-observability-whats-the-difference\/","title":{"rendered":"Data Quality vs. Data Observability; What’s the Difference?"},"content":{"rendered":"

\"Data<\/p>\n

This is part of Solutions Review\u2019s Premium Content Series, a collection of contributed columns written by industry experts in maturing software categories. In this submission, Datafold<\/a> founding CEO Gleb Mezhanskiy answers the question: “data quality vs. data observability; what’s the difference?”<\/em><\/strong><\/p>\n

\"SRAs a general rule, data quality problems have always been handled reactively. In the past, data quality issues were discovered in production by business users and were then fixed by a trained data engineer<\/a><\/strong><\/span>. This meant that these issues may have existed for a long time, and many insights and decisions based on that data could have been inaccurate.<\/p>\n

As data becomes more and more central to how a business operates, we can learn from the software world how to reduce quality issues. In software, a key principle for maintaining quality is that defects are most easily addressed when they\u2019re discovered while writing code \u2014 before anything is merged into production. We should aim for the same in data to shift the testing from business users left to the data engineers who are modeling the data.<\/p>\n

\t\t\t
\"Download<\/a><\/div>\n\t\t<\/div><\/div>\n

Today, there are a handful of data quality solutions<\/a><\/strong><\/span> that focus on shortening the interval between the initial appearance of a data quality issue and its eventual resolution. This typically takes the form of a rules-based monitoring system that tracks live data within a data warehouse, then sends an alert if it sees anything that falls outside of acceptable bounds. Some products use artificial intelligence (AI) to scan for anomalies, fine-tuning their selection criteria along the way. In either case, the process is very much the same: A problem is spotted in the data, then comes an intervention, things get fixed, and life goes on until the next issue shows up.<\/p>\n

That\u2019s data observability. It\u2019s better than the business user letting you know about the problem. The question is: Can we do better? Monitoring for data quality problems is useful, but it doesn\u2019t actually prevent problems from cropping up in the first place. In order for these tools to do their job, something must already have broken.<\/p>\n

In an era of big data analytics, AI\/ML, and automation at scale, that\u2019s a problem. The data stored in warehouses is now being used to power a wide variety of reports, analytics and applications. That includes things like email marketing segmentation, fraud detection, or data science applications. Dashboards, data apps, notebooks, and data activation are consuming data at scale, very often in ways that may not have been planned or predicted in advance. In some cases, the data engineers overseeing a warehouse might not even be aware of all the ways their data is being used.<\/p>\n

The key point here is that data quality issues can have very real (and far-reaching) consequences. Detecting them post-factum may very often be too late. Alerting systems may also be incomplete because some anomalies may fly under the radar, not easily detectable by algorithms.<\/p>\n

When poor data quality is allowed to persist, it drives bad decisions, with potentially catastrophic consequences. As algorithms drive more and more business processes based on data, poor data quality can lead to poor automation. That means payments being denied or processed improperly, email workflows driving the wrong messages to the wrong people, and ultimately, poor customer experiences.<\/p>\n

The term \u201cdata quality,\u201d broadly speaking, defines a problem domain. Data observability describes one potential solution to the problem of poor data quality, but it\u2019s not the only one \u2014 and it\u2019s not necessarily the best one. There is a second (and better) solution, which we might refer to as preemptive data testing.<\/p>\n

Avora co-founder David Jayatillake recently pointed out that while shift-left testing has been a driving force in software engineering for quite some time, it\u2019s a relatively new idea within the context of data engineering and data governance. By taking a proactive approach to data quality, we can prevent a great many data quality problems from ever emerging at all. In other cases, we can at least identify and fix issues earlier, at a much lower cost.<\/p>\n

This suggests a fundamental shift in how data stewards approach the problem of poor data quality. It calls for making data testing an integral element of the data engineering process. In other words, we should look for conditions that will result in poor data quality. Although many vendors in the data quality space<\/a><\/strong><\/span> are pursuing data observability, they\u2019re missing out on this opportunity.<\/p>\n

By providing data engineers with a data quality impact report, we can systematically identify potential data quality problems up front within their pull request workflow. That will allow data engineers to make better design choices and reduce the number of quality incidents that get introduced into the data.<\/p>\n

This is a preemptive approach, aimed at prevention, so it shows up at the far left extreme of the data flow diagram. Deal with data quality in the pull request by understanding the impact that changes will have on the underlying data as well as any connected data assets and products. Armed with that information, data engineers can prevent their newly merged code from adversely impacting data quality.<\/p>\n

<\/div>\n
\t\t\t

\"Download<\/a><\/p>\n<\/div>\n\t\t<\/div><\/div>\n","protected":false},"excerpt":{"rendered":"

This is part of Solutions Review\u2019s Premium Content Series, a collection of contributed columns written by industry experts in maturing software categories. In this submission, Datafold founding CEO Gleb Mezhanskiy answers the question: “data quality vs. data observability; what’s the difference?” As a general rule, data quality problems have always been handled reactively. In the […]<\/p>\n","protected":false},"author":206,"featured_media":4081,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"inline_featured_image":false,"footnotes":""},"categories":[3],"tags":[1285,1282],"acf":[],"yoast_head":"\nData Quality vs. Data Observability; What's the Difference?<\/title>\n<meta name=\"description\" content=\"In this submission, Datafold founding CEO Gleb Mezhanskiy answers the question: "data quality vs. data observability; what's the difference?"\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/solutionsreview.com\/data-management\/data-quality-vs-data-observability-whats-the-difference\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Data Quality vs. Data Observability; What's the Difference?\" \/>\n<meta property=\"og:description\" content=\"In this submission, Datafold founding CEO Gleb Mezhanskiy answers the question: "data quality vs. data observability; what's the difference?"\" \/>\n<meta property=\"og:url\" content=\"https:\/\/solutionsreview.com\/data-management\/data-quality-vs-data-observability-whats-the-difference\/\" \/>\n<meta property=\"og:site_name\" content=\"Data Management Software & Infrastructure | Solutions Review\" \/>\n<meta property=\"article:published_time\" content=\"2022-06-24T16:40:42+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2022-06-24T17:36:16+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/solutionsreview.com\/data-management\/files\/2022\/06\/MicrosoftTeams-image-2.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"800\" \/>\n\t<meta property=\"og:image:height\" content=\"400\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Gleb Mezhanskiy\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Gleb Mezhanskiy\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"5 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/solutionsreview.com\/data-management\/data-quality-vs-data-observability-whats-the-difference\/\",\"url\":\"https:\/\/solutionsreview.com\/data-management\/data-quality-vs-data-observability-whats-the-difference\/\",\"name\":\"Data Quality vs. Data Observability; What's the Difference?\",\"isPartOf\":{\"@id\":\"https:\/\/solutionsreview.com\/data-management\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/solutionsreview.com\/data-management\/data-quality-vs-data-observability-whats-the-difference\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/solutionsreview.com\/data-management\/data-quality-vs-data-observability-whats-the-difference\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/solutionsreview.com\/data-management\/files\/2022\/06\/MicrosoftTeams-image-2.jpg\",\"datePublished\":\"2022-06-24T16:40:42+00:00\",\"dateModified\":\"2022-06-24T17:36:16+00:00\",\"author\":{\"@id\":\"https:\/\/solutionsreview.com\/data-management\/#\/schema\/person\/055d982cb2f58cbbe37b406c3454200a\"},\"description\":\"In this submission, Datafold founding CEO Gleb Mezhanskiy answers the question: \\\"data quality vs. data observability; what's the difference?\\\"\",\"breadcrumb\":{\"@id\":\"https:\/\/solutionsreview.com\/data-management\/data-quality-vs-data-observability-whats-the-difference\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/solutionsreview.com\/data-management\/data-quality-vs-data-observability-whats-the-difference\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/solutionsreview.com\/data-management\/data-quality-vs-data-observability-whats-the-difference\/#primaryimage\",\"url\":\"https:\/\/solutionsreview.com\/data-management\/files\/2022\/06\/MicrosoftTeams-image-2.jpg\",\"contentUrl\":\"https:\/\/solutionsreview.com\/data-management\/files\/2022\/06\/MicrosoftTeams-image-2.jpg\",\"width\":800,\"height\":400,\"caption\":\"Data Quality vs. Data Observability; What's the Difference?\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/solutionsreview.com\/data-management\/data-quality-vs-data-observability-whats-the-difference\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/solutionsreview.com\/data-management\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Data Quality vs. Data Observability; What’s the Difference?\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/solutionsreview.com\/data-management\/#website\",\"url\":\"https:\/\/solutionsreview.com\/data-management\/\",\"name\":\"Data Management Software & Infrastructure | Solutions Review\",\"description\":\"Evaluating Enterprise Master Data Management, Data Governance & Storage Tools.\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/solutionsreview.com\/data-management\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/solutionsreview.com\/data-management\/#\/schema\/person\/055d982cb2f58cbbe37b406c3454200a\",\"name\":\"Gleb Mezhanskiy\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/solutionsreview.com\/data-management\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/af3c1c4ba1405783c63bb4a746ca9ec5?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/af3c1c4ba1405783c63bb4a746ca9ec5?s=96&d=mm&r=g\",\"caption\":\"Gleb Mezhanskiy\"},\"description\":\"Gleb Mezhanskiy is the founding CEO of Datafold. As a founding member of data teams at Lyft and Autodesk and head of product at Phantom Auto, Gleb built some of the world\u2019s largest and most sophisticated data platforms, including essential tools for data discovery, ETL development, forecasting, and anomaly detection.\",\"url\":\"https:\/\/solutionsreview.com\/data-management\/author\/gmezhanskiy\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Data Quality vs. Data Observability; What's the Difference?","description":"In this submission, Datafold founding CEO Gleb Mezhanskiy answers the question: \"data quality vs. data observability; what's the difference?\"","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/solutionsreview.com\/data-management\/data-quality-vs-data-observability-whats-the-difference\/","og_locale":"en_US","og_type":"article","og_title":"Data Quality vs. Data Observability; What's the Difference?","og_description":"In this submission, Datafold founding CEO Gleb Mezhanskiy answers the question: \"data quality vs. data observability; what's the difference?\"","og_url":"https:\/\/solutionsreview.com\/data-management\/data-quality-vs-data-observability-whats-the-difference\/","og_site_name":"Data Management Software & Infrastructure | Solutions Review","article_published_time":"2022-06-24T16:40:42+00:00","article_modified_time":"2022-06-24T17:36:16+00:00","og_image":[{"width":800,"height":400,"url":"https:\/\/solutionsreview.com\/data-management\/files\/2022\/06\/MicrosoftTeams-image-2.jpg","type":"image\/jpeg"}],"author":"Gleb Mezhanskiy","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Gleb Mezhanskiy","Est. reading time":"5 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/solutionsreview.com\/data-management\/data-quality-vs-data-observability-whats-the-difference\/","url":"https:\/\/solutionsreview.com\/data-management\/data-quality-vs-data-observability-whats-the-difference\/","name":"Data Quality vs. Data Observability; What's the Difference?","isPartOf":{"@id":"https:\/\/solutionsreview.com\/data-management\/#website"},"primaryImageOfPage":{"@id":"https:\/\/solutionsreview.com\/data-management\/data-quality-vs-data-observability-whats-the-difference\/#primaryimage"},"image":{"@id":"https:\/\/solutionsreview.com\/data-management\/data-quality-vs-data-observability-whats-the-difference\/#primaryimage"},"thumbnailUrl":"https:\/\/solutionsreview.com\/data-management\/files\/2022\/06\/MicrosoftTeams-image-2.jpg","datePublished":"2022-06-24T16:40:42+00:00","dateModified":"2022-06-24T17:36:16+00:00","author":{"@id":"https:\/\/solutionsreview.com\/data-management\/#\/schema\/person\/055d982cb2f58cbbe37b406c3454200a"},"description":"In this submission, Datafold founding CEO Gleb Mezhanskiy answers the question: \"data quality vs. data observability; what's the difference?\"","breadcrumb":{"@id":"https:\/\/solutionsreview.com\/data-management\/data-quality-vs-data-observability-whats-the-difference\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/solutionsreview.com\/data-management\/data-quality-vs-data-observability-whats-the-difference\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/solutionsreview.com\/data-management\/data-quality-vs-data-observability-whats-the-difference\/#primaryimage","url":"https:\/\/solutionsreview.com\/data-management\/files\/2022\/06\/MicrosoftTeams-image-2.jpg","contentUrl":"https:\/\/solutionsreview.com\/data-management\/files\/2022\/06\/MicrosoftTeams-image-2.jpg","width":800,"height":400,"caption":"Data Quality vs. Data Observability; What's the Difference?"},{"@type":"BreadcrumbList","@id":"https:\/\/solutionsreview.com\/data-management\/data-quality-vs-data-observability-whats-the-difference\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/solutionsreview.com\/data-management\/"},{"@type":"ListItem","position":2,"name":"Data Quality vs. Data Observability; What’s the Difference?"}]},{"@type":"WebSite","@id":"https:\/\/solutionsreview.com\/data-management\/#website","url":"https:\/\/solutionsreview.com\/data-management\/","name":"Data Management Software & Infrastructure | Solutions Review","description":"Evaluating Enterprise Master Data Management, Data Governance & Storage Tools.","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/solutionsreview.com\/data-management\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/solutionsreview.com\/data-management\/#\/schema\/person\/055d982cb2f58cbbe37b406c3454200a","name":"Gleb Mezhanskiy","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/solutionsreview.com\/data-management\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/af3c1c4ba1405783c63bb4a746ca9ec5?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/af3c1c4ba1405783c63bb4a746ca9ec5?s=96&d=mm&r=g","caption":"Gleb Mezhanskiy"},"description":"Gleb Mezhanskiy is the founding CEO of Datafold. As a founding member of data teams at Lyft and Autodesk and head of product at Phantom Auto, Gleb built some of the world\u2019s largest and most sophisticated data platforms, including essential tools for data discovery, ETL development, forecasting, and anomaly detection.","url":"https:\/\/solutionsreview.com\/data-management\/author\/gmezhanskiy\/"}]}},"_links":{"self":[{"href":"https:\/\/solutionsreview.com\/data-management\/wp-json\/wp\/v2\/posts\/4072"}],"collection":[{"href":"https:\/\/solutionsreview.com\/data-management\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/solutionsreview.com\/data-management\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/solutionsreview.com\/data-management\/wp-json\/wp\/v2\/users\/206"}],"replies":[{"embeddable":true,"href":"https:\/\/solutionsreview.com\/data-management\/wp-json\/wp\/v2\/comments?post=4072"}],"version-history":[{"count":0,"href":"https:\/\/solutionsreview.com\/data-management\/wp-json\/wp\/v2\/posts\/4072\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/solutionsreview.com\/data-management\/wp-json\/wp\/v2\/media\/4081"}],"wp:attachment":[{"href":"https:\/\/solutionsreview.com\/data-management\/wp-json\/wp\/v2\/media?parent=4072"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/solutionsreview.com\/data-management\/wp-json\/wp\/v2\/categories?post=4072"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/solutionsreview.com\/data-management\/wp-json\/wp\/v2\/tags?post=4072"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}