{"id":532,"date":"2024-01-01T14:51:31","date_gmt":"2024-01-01T14:51:31","guid":{"rendered":"https:\/\/solutionsreview.com\/expert\/?p=532"},"modified":"2024-02-02T14:40:48","modified_gmt":"2024-02-02T14:40:48","slug":"why-data-analytics-is-heavy-on-data-engineering","status":"publish","type":"post","link":"https:\/\/solutionsreview.com\/thought-leaders\/why-data-analytics-is-heavy-on-data-engineering\/","title":{"rendered":"Why Data Analytics is Heavy on Data Engineering?"},"content":{"rendered":"<p style=\"text-align: justify;\">While many companies have embarked on data analytics initiatives, only a few have been successful. Studies have shown that over 70% of data analytics programs fail to realize their full potential and over 80% of the digital transformation initiatives fail. While there are many reasons that affect successful deployment of data analytics, one fundamental reason is lack of good quality data. However, many business enterprises realize this and invest considerable time and effort in data cleansing and remediation; technically known as data engineering. It is estimated that about 60 to 70% of the effort in data analytics is on data engineering. Given that data quality is an essential requirement for analytics, there are 5 key reasons on why\u00a0<strong>data<\/strong>\u00a0<strong>analytics is heavy on data engineering<\/strong>.<\/p>\n<p style=\"text-align: justify;\"><span style=\"font-size: medium;\"><strong>1.<\/strong><strong>Different systems and technology mechanisms to integrate data.<\/strong><\/span><\/p>\n<p style=\"text-align: justify;\">Business systems are designed and implemented for a purpose; mainly for recording business transactions. The mechanisms for data capture in Business systems such as ERP is batch\/discrete data while in the SCADA\/IoT Field Systems it is for continuous\/time-series data. This means that these business systems store diverse data\u00a0types caused by the velocity, volume, and variety dimensions in the data. Hence the technology (including the database itself) to capture data is varied and complex. \u00a0And when you are trying to integrate data from these diverse systems from different vendors, the metadata model varies resulting in data integration challenges.<\/p>\n<p style=\"text-align: justify;\"><strong><span style=\"font-size: medium;\">2. Different time frames of data capture<\/span><\/strong><\/p>\n<p style=\"text-align: justify;\">The timeframes for data ingestion during data capture varies. For example, in ERP\/transactional systems the data ingestion is typically batch\/discrete\/manual, while in SCADA\/IoT\/Field Systems, the data ingestion is usually automatic and real-time. For example, when the product delivery to the customer is done, the shipment details are normally captured in real-time by the hand-held devices. But the invoicing cannot be immediately processed as invoices are issued from the ERP systems to the customer. This creates a delay in Delivery-Invoicing compliance reporting.<\/p>\n<p style=\"text-align: justify;\"><strong><span style=\"font-size: medium;\">3. Different user value-propositions<\/span><\/strong><\/p>\n<p style=\"text-align: justify;\">In business, the same data is created and consumed by different stakeholders (inside the company) in different ways as their value-propositions vary. For example, vendor payment terms for Finance is a cost object, while for Procurement the same data element is a risk element (as\u00a0<strong>longer payment terms generally result in longer deliveries).<\/strong><\/p>\n<p style=\"text-align: justify;\"><span style=\"font-size: medium;\"><strong>4.<\/strong>\u00a0<strong>Different business processes<\/strong><\/span><\/p>\n<p style=\"text-align: justify;\">The same data element can be different due to differences in business processes based on geographies, laws, regulations, market conditions, etc. For example, the Date-of-Birth data element in Canada is subject to data privacy regulations, while Date-of-Birth data element in most developing countries is generally not part of the data privacy regulations. So, getting customer buying habit report based on age for a developing market is much \u201ceasier\u201d than getting the same report in Canada.<\/p>\n<p style=\"text-align: justify;\"><b><span style=\"font-size: medium;\">5. Different aggregations driven by organizational structures<\/span><\/b><\/p>\n<p style=\"text-align: justify;\">One data element can be viewed differently based on differences in granularities or aggregations driven by organizational structures.\u00a0 For example, the VP of procurement might need a spend report based on item categories (an aggregation of items procured), while the procurement manager needs the spend report based on individual items procured.\u00a0 That aggregation might vary based on the item type, supplier type, delivery location, etc.<\/p>\n<p style=\"text-align: justify;\">Good analytics relies on good quality data. So, if you are embarking on the analytics journey by looking at technologies, tools, and hiring data scientists, pause for a minute. Challenge your assumption and ask one basic question \u2013\u00a0<strong>is diversity of my business operations affecting a good quality data for analytics? \u00a0<\/strong>If the answer is yes, get ready for a long and a complex data engineering effort.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>While many companies have embarked on data analytics initiatives, only a few have been successful. Studies have shown that over 70% of data analytics programs fail to realize their full potential and over 80% of the digital transformation initiatives fail. While there are many reasons that affect successful deployment of data analytics, one fundamental reason [&hellip;]<\/p>\n","protected":false},"author":432,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"inline_featured_image":false,"footnotes":""},"categories":[10],"tags":[],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v23.5 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Why Data Analytics is Heavy on Data Engineering?<\/title>\n<meta name=\"robots\" content=\"noindex, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Why Data Analytics is Heavy on Data Engineering?\" \/>\n<meta property=\"og:description\" content=\"While many companies have embarked on data analytics initiatives, only a few have been successful. Studies have shown that over 70% of data analytics programs fail to realize their full potential and over 80% of the digital transformation initiatives fail. While there are many reasons that affect successful deployment of data analytics, one fundamental reason [&hellip;]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/solutionsreview.com\/thought-leaders\/why-data-analytics-is-heavy-on-data-engineering\/\" \/>\n<meta property=\"og:site_name\" content=\"Solutions Review Thought Leaders\" \/>\n<meta property=\"article:published_time\" content=\"2024-01-01T14:51:31+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2024-02-02T14:40:48+00:00\" \/>\n<meta name=\"author\" content=\"Prashanth Southekal\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Prashanth Southekal\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/solutionsreview.com\/thought-leaders\/why-data-analytics-is-heavy-on-data-engineering\/\",\"url\":\"https:\/\/solutionsreview.com\/thought-leaders\/why-data-analytics-is-heavy-on-data-engineering\/\",\"name\":\"Why Data Analytics is Heavy on Data Engineering?\",\"isPartOf\":{\"@id\":\"https:\/\/solutionsreview.com\/thought-leaders\/#website\"},\"datePublished\":\"2024-01-01T14:51:31+00:00\",\"dateModified\":\"2024-02-02T14:40:48+00:00\",\"author\":{\"@id\":\"https:\/\/solutionsreview.com\/thought-leaders\/#\/schema\/person\/ecca132a165bddd87f1c093e77981b70\"},\"breadcrumb\":{\"@id\":\"https:\/\/solutionsreview.com\/thought-leaders\/why-data-analytics-is-heavy-on-data-engineering\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/solutionsreview.com\/thought-leaders\/why-data-analytics-is-heavy-on-data-engineering\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/solutionsreview.com\/thought-leaders\/why-data-analytics-is-heavy-on-data-engineering\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/solutionsreview.com\/thought-leaders\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Why Data Analytics is Heavy on Data Engineering?\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/solutionsreview.com\/thought-leaders\/#website\",\"url\":\"https:\/\/solutionsreview.com\/thought-leaders\/\",\"name\":\"Solutions Review Thought Leaders\",\"description\":\"\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/solutionsreview.com\/thought-leaders\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/solutionsreview.com\/thought-leaders\/#\/schema\/person\/ecca132a165bddd87f1c093e77981b70\",\"name\":\"Prashanth Southekal\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/solutionsreview.com\/thought-leaders\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/47cc283c07825713f7f9d9dc91d653e3?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/47cc283c07825713f7f9d9dc91d653e3?s=96&d=mm&r=g\",\"caption\":\"Prashanth Southekal\"},\"description\":\"Prashanth Southekal is the Managing Principal of DBP-Institute, a data and analytics consulting and education firm. He has consulted for over 75 organizations. Dr. Southekal is a published author of two books and a contributing writer on data, analytics, and machine learning in Forbes and CFO.University. Apart from his consulting pursuits, he has trained over 2,500 professionals worldwide.\",\"sameAs\":[\"https:\/\/www.dbp-institute.com\",\"www.linkedin.com\/in\/prashanthsouthekal\/\"],\"url\":\"https:\/\/solutionsreview.com\/thought-leaders\/author\/prashanth-southekal\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Why Data Analytics is Heavy on Data Engineering?","robots":{"index":"noindex","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"og_locale":"en_US","og_type":"article","og_title":"Why Data Analytics is Heavy on Data Engineering?","og_description":"While many companies have embarked on data analytics initiatives, only a few have been successful. Studies have shown that over 70% of data analytics programs fail to realize their full potential and over 80% of the digital transformation initiatives fail. While there are many reasons that affect successful deployment of data analytics, one fundamental reason [&hellip;]","og_url":"https:\/\/solutionsreview.com\/thought-leaders\/why-data-analytics-is-heavy-on-data-engineering\/","og_site_name":"Solutions Review Thought Leaders","article_published_time":"2024-01-01T14:51:31+00:00","article_modified_time":"2024-02-02T14:40:48+00:00","author":"Prashanth Southekal","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Prashanth Southekal","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/solutionsreview.com\/thought-leaders\/why-data-analytics-is-heavy-on-data-engineering\/","url":"https:\/\/solutionsreview.com\/thought-leaders\/why-data-analytics-is-heavy-on-data-engineering\/","name":"Why Data Analytics is Heavy on Data Engineering?","isPartOf":{"@id":"https:\/\/solutionsreview.com\/thought-leaders\/#website"},"datePublished":"2024-01-01T14:51:31+00:00","dateModified":"2024-02-02T14:40:48+00:00","author":{"@id":"https:\/\/solutionsreview.com\/thought-leaders\/#\/schema\/person\/ecca132a165bddd87f1c093e77981b70"},"breadcrumb":{"@id":"https:\/\/solutionsreview.com\/thought-leaders\/why-data-analytics-is-heavy-on-data-engineering\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/solutionsreview.com\/thought-leaders\/why-data-analytics-is-heavy-on-data-engineering\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/solutionsreview.com\/thought-leaders\/why-data-analytics-is-heavy-on-data-engineering\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/solutionsreview.com\/thought-leaders\/"},{"@type":"ListItem","position":2,"name":"Why Data Analytics is Heavy on Data Engineering?"}]},{"@type":"WebSite","@id":"https:\/\/solutionsreview.com\/thought-leaders\/#website","url":"https:\/\/solutionsreview.com\/thought-leaders\/","name":"Solutions Review Thought Leaders","description":"","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/solutionsreview.com\/thought-leaders\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/solutionsreview.com\/thought-leaders\/#\/schema\/person\/ecca132a165bddd87f1c093e77981b70","name":"Prashanth Southekal","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/solutionsreview.com\/thought-leaders\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/47cc283c07825713f7f9d9dc91d653e3?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/47cc283c07825713f7f9d9dc91d653e3?s=96&d=mm&r=g","caption":"Prashanth Southekal"},"description":"Prashanth Southekal is the Managing Principal of DBP-Institute, a data and analytics consulting and education firm. He has consulted for over 75 organizations. Dr. Southekal is a published author of two books and a contributing writer on data, analytics, and machine learning in Forbes and CFO.University. Apart from his consulting pursuits, he has trained over 2,500 professionals worldwide.","sameAs":["https:\/\/www.dbp-institute.com","www.linkedin.com\/in\/prashanthsouthekal\/"],"url":"https:\/\/solutionsreview.com\/thought-leaders\/author\/prashanth-southekal\/"}]}},"_links":{"self":[{"href":"https:\/\/solutionsreview.com\/thought-leaders\/wp-json\/wp\/v2\/posts\/532"}],"collection":[{"href":"https:\/\/solutionsreview.com\/thought-leaders\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/solutionsreview.com\/thought-leaders\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/solutionsreview.com\/thought-leaders\/wp-json\/wp\/v2\/users\/432"}],"replies":[{"embeddable":true,"href":"https:\/\/solutionsreview.com\/thought-leaders\/wp-json\/wp\/v2\/comments?post=532"}],"version-history":[{"count":0,"href":"https:\/\/solutionsreview.com\/thought-leaders\/wp-json\/wp\/v2\/posts\/532\/revisions"}],"wp:attachment":[{"href":"https:\/\/solutionsreview.com\/thought-leaders\/wp-json\/wp\/v2\/media?parent=532"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/solutionsreview.com\/thought-leaders\/wp-json\/wp\/v2\/categories?post=532"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/solutionsreview.com\/thought-leaders\/wp-json\/wp\/v2\/tags?post=532"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}