{"id":586,"date":"2015-09-25T02:52:37","date_gmt":"2015-09-25T06:52:37","guid":{"rendered":"https:\/\/solutionsreview.com\/data-integration\/?p=586"},"modified":"2016-10-04T15:31:10","modified_gmt":"2016-10-04T19:31:10","slug":"is-etl-on-life-support","status":"publish","type":"post","link":"https:\/\/solutionsreview.com\/data-integration\/is-etl-on-life-support\/","title":{"rendered":"Is ETL on Life Support?"},"content":{"rendered":"<p><img loading=\"lazy\" decoding=\"async\" class=\"size-full wp-image-592 aligncenter\" src=\"https:\/\/solutionsreview.com\/data-integration\/files\/2014\/06\/Is-ETL-on-Life-Support.jpg\" alt=\"Is ETL on Life Support\" width=\"600\" height=\"270\" srcset=\"https:\/\/solutionsreview.com\/data-integration\/files\/2014\/06\/Is-ETL-on-Life-Support.jpg 600w, https:\/\/solutionsreview.com\/data-integration\/files\/2014\/06\/Is-ETL-on-Life-Support-300x135.jpg 300w\" sizes=\"(max-width: 600px) 100vw, 600px\" \/><\/p>\n<p>Three years ago, Doug Henschen, Executive Editor of InformationWeek, wrote an interesting article called \u201c<a href=\"https:\/\/www.informationweek.com\/big-data\/big-data-analytics\/big-data-debate-end-near-for-etl\/d\/d-id\/1107641?\" target=\"_blank\">Big Data Debate: End Near for ETL?<\/a>&#8221;\u00a0which covers a debate about Hadoop replacing ETL.\u00a0In this article, he quotes\u00a0Phil Shelley, the former Chief Technology Officer of Sears Holdings and CEO of Metascale, who debates that ETL will become obsolete. Shelley states,<\/p>\n<blockquote><p>\u201cThe growth of ETL has been alarming, as data volumes escalate year after year. Companies have significant investment in people, skills, software and hardware to do nothing but ETL. Some consider ETL to be a bottleneck in IT operations: ETL takes time as, by definition, data has to be moved. Reading from one system, copying over a network and writing all take time &#8212; ever growing blocks of time, causing latency in the data before it can be used. ETL is expensive in terms of people, software licensing and hardware. ETL is a non-value-added activity too, as the data is unusable until it lands in the destination system.&#8221;<\/p><\/blockquote>\n<p>Shelley made some very good points on why\u00a0\u201cETL\u2019s days are numbered.\u201d\u00a0Shelley also talked\u00a0about how Hadoop will eventually replace ETL because it is a data hub that can store, transform, and use data without ETL tools. The author described how it works in the bullets below,<\/p>\n<ul>\n<li>Systems generate data, just as they always have.<\/li>\n<li>As near to real-time as possible, data is loaded into Hadoop &#8211; yes, this is still &#8220;E&#8221; from traditional ETL, but that is where the similarity ends.<\/li>\n<li>Now we can aggregate, sort, transform and analyze the data inside Hadoop. This is the &#8220;T&#8221; and the &#8220;L&#8221; from traditional ETL.<\/li>\n<li>Data latency is reduced to minutes instead of hours because the data never leaves Hadoop. There is no network copying time, no licenses for ETL software, and no additional ETL hardware.<\/li>\n<li>Now data can be consumed in place without moving it. There are a number of graphic analytic and reporting options to consume data without moving large amounts of data out of Hadoop.<\/li>\n<li>Some subsets of data do have to be moved out of Hadoop into other systems for specific purposes. However, with a strong and coherent enterprise data architecture, this can be managed to be the exception.<\/li>\n<\/ul>\n<p>Now, three years later. Let\u2019s take a look at what has happened to ETL.<\/p>\n<p>In a VentureBeat article called, &#8220;<a href=\"https:\/\/venturebeat.com\/2014\/05\/11\/the-state-of-big-data-in-2014-chart\/\" target=\"_blank\">The state of big data in 2014 (chart)<\/a>&#8220;, Matt Turck presents and describes his chart on the landscape of big data and how it has changed over the last three years. He speaks to what has happened with infrastructure: \u201cHadoop seems to have solidified its position as the cornerstone of the entire ecosystem, but there are still a number of competing distributions \u2014 this will probably need to evolve.&#8221;<\/p>\n<p>In a different article titled,\u00a0&#8220;<a href=\"https:\/\/venturebeat.com\/2014\/05\/19\/more-data-needs-more-data-scientists-they-need-more-data-tools\/\" target=\"_blank\">More data means more data scientists, &amp; they need more data tools<\/a>&#8220;, by Jordan Novet, the author refers to a book written by Tom Davenport, called \u201cBig Data @ Work: Dispelling the Myths, Uncovering the Opportunities.\u201d Novet says that Davenport believes that the industry needs more data integration tools to transfer and move data to essentially remove a great deal of the \u201cdirty work\u201d that is involved in migrating\u00a0data.<\/p>\n<p>So is ETL on life support?<\/p>\n<p>No, far from it actually. Shelley&#8217;s prediction was correct in his belief that Hadoop would prosper. However, he was completely wrong about\u00a0the &#8220;death of ETL&#8221; as more and more ETL tools have emerged and others have evolved. ETL tools that work without the weaknesses that Shelley presents in the article such as slow speeds and high costs are easy to find. Common in the market today\u00a0are ETL tools that have automation features which decrease the amount of time spent coding for ETL and are user-friendly, which decreases the time required for programming. Change data capture (CDC) tools are available now which migrates only changed data, shaving off process time. And then there are data replication tools that can do the work in \u201creal time\u201d speeds flooding the market.<\/p>\n<p>Why are ETL tools evolving and flooding the market? My take is that these tools are in demand by corporations that don\u2019t want to replace\u00a0all of their current systems for something like Hadoop because of the costs and efforts of such an endeavor. \u00a0Instead, they are buying other less costly solutions and using them as add-ons for their enterprise systems that can do what Hadoop can do.<\/p>\n<p>What&#8217;s your take?<br \/>\n<br \/>Widget not in any sidebars<br \/><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Three years ago, Doug Henschen, Executive Editor of InformationWeek, wrote an interesting article called \u201cBig Data Debate: End Near for ETL?&#8221;\u00a0which covers a debate about Hadoop replacing ETL.\u00a0In this article, he quotes\u00a0Phil Shelley, the former Chief Technology Officer of Sears Holdings and CEO of Metascale, who debates that ETL will become obsolete. Shelley states, \u201cThe [&hellip;]<\/p>\n","protected":false},"author":23,"featured_media":592,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"inline_featured_image":false,"footnotes":""},"categories":[1],"tags":[19,194,55],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v23.5 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Is ETL on Life Support?<\/title>\n<meta name=\"description\" content=\"Is ETL on Life Support? The answer may surprise you.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/solutionsreview.com\/data-integration\/is-etl-on-life-support\/\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Tim King\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"4 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/solutionsreview.com\/data-integration\/is-etl-on-life-support\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/solutionsreview.com\/data-integration\/is-etl-on-life-support\/\"},\"author\":{\"name\":\"Tim King\",\"@id\":\"https:\/\/solutionsreview.com\/data-integration\/#\/schema\/person\/154e152a275103e373e24ada7f2feb5c\"},\"headline\":\"Is ETL on Life Support?\",\"datePublished\":\"2015-09-25T06:52:37+00:00\",\"dateModified\":\"2016-10-04T19:31:10+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/solutionsreview.com\/data-integration\/is-etl-on-life-support\/\"},\"wordCount\":749,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/solutionsreview.com\/data-integration\/#organization\"},\"image\":{\"@id\":\"https:\/\/solutionsreview.com\/data-integration\/is-etl-on-life-support\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/solutionsreview.com\/data-integration\/files\/2014\/06\/Is-ETL-on-Life-Support.jpg\",\"keywords\":[\"Big Data\",\"ETL\",\"Hadoop\"],\"articleSection\":[\"Best Practices\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/solutionsreview.com\/data-integration\/is-etl-on-life-support\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/solutionsreview.com\/data-integration\/is-etl-on-life-support\/\",\"url\":\"https:\/\/solutionsreview.com\/data-integration\/is-etl-on-life-support\/\",\"name\":\"Is ETL on Life Support?\",\"isPartOf\":{\"@id\":\"https:\/\/solutionsreview.com\/data-integration\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/solutionsreview.com\/data-integration\/is-etl-on-life-support\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/solutionsreview.com\/data-integration\/is-etl-on-life-support\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/solutionsreview.com\/data-integration\/files\/2014\/06\/Is-ETL-on-Life-Support.jpg\",\"datePublished\":\"2015-09-25T06:52:37+00:00\",\"dateModified\":\"2016-10-04T19:31:10+00:00\",\"description\":\"Is ETL on Life Support? The answer may surprise you.\",\"breadcrumb\":{\"@id\":\"https:\/\/solutionsreview.com\/data-integration\/is-etl-on-life-support\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/solutionsreview.com\/data-integration\/is-etl-on-life-support\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/solutionsreview.com\/data-integration\/is-etl-on-life-support\/#primaryimage\",\"url\":\"https:\/\/solutionsreview.com\/data-integration\/files\/2014\/06\/Is-ETL-on-Life-Support.jpg\",\"contentUrl\":\"https:\/\/solutionsreview.com\/data-integration\/files\/2014\/06\/Is-ETL-on-Life-Support.jpg\",\"width\":600,\"height\":270,\"caption\":\"Is ETL on Life Support\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/solutionsreview.com\/data-integration\/is-etl-on-life-support\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/solutionsreview.com\/data-integration\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Is ETL on Life Support?\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/solutionsreview.com\/data-integration\/#website\",\"url\":\"https:\/\/solutionsreview.com\/data-integration\/\",\"name\":\"Best Data Integration Vendors, News &amp; Reviews for Big Data, Applications, ETL and Hadoop\",\"description\":\"Data Integration Buyers Guide and Best Practices\",\"publisher\":{\"@id\":\"https:\/\/solutionsreview.com\/data-integration\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/solutionsreview.com\/data-integration\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/solutionsreview.com\/data-integration\/#organization\",\"name\":\"Solutions Review\",\"url\":\"https:\/\/solutionsreview.com\/data-integration\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/solutionsreview.com\/data-integration\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/solutionsreview.com\/data-integration\/files\/2016\/02\/Solutions_Review_Header_Data_Integration_225.png\",\"contentUrl\":\"https:\/\/solutionsreview.com\/data-integration\/files\/2016\/02\/Solutions_Review_Header_Data_Integration_225.png\",\"width\":225,\"height\":90,\"caption\":\"Solutions Review\"},\"image\":{\"@id\":\"https:\/\/solutionsreview.com\/data-integration\/#\/schema\/logo\/image\/\"}},{\"@type\":\"Person\",\"@id\":\"https:\/\/solutionsreview.com\/data-integration\/#\/schema\/person\/154e152a275103e373e24ada7f2feb5c\",\"name\":\"Tim King\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/solutionsreview.com\/data-integration\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/solutionsreview.com\/data-integration\/files\/2023\/12\/tk.jpg\",\"contentUrl\":\"https:\/\/solutionsreview.com\/data-integration\/files\/2023\/12\/tk.jpg\",\"caption\":\"Tim King\"},\"description\":\"Tim is Solutions Review's Executive Editor covering the human impact of AI on the future of work and learning. He is also the Media Strategist behind Insight Jam (1M+ on YouTube) events and programming. A 2017 and 2018 Most Influential Business Journalist and 2021 \\\"Who's Who\\\" in multiple categories, Tim is a recognized thought leader in enterprise tech and AI.\",\"url\":\"https:\/\/solutionsreview.com\/data-integration\/author\/timking\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Is ETL on Life Support?","description":"Is ETL on Life Support? The answer may surprise you.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/solutionsreview.com\/data-integration\/is-etl-on-life-support\/","twitter_misc":{"Written by":"Tim King","Est. reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/solutionsreview.com\/data-integration\/is-etl-on-life-support\/#article","isPartOf":{"@id":"https:\/\/solutionsreview.com\/data-integration\/is-etl-on-life-support\/"},"author":{"name":"Tim King","@id":"https:\/\/solutionsreview.com\/data-integration\/#\/schema\/person\/154e152a275103e373e24ada7f2feb5c"},"headline":"Is ETL on Life Support?","datePublished":"2015-09-25T06:52:37+00:00","dateModified":"2016-10-04T19:31:10+00:00","mainEntityOfPage":{"@id":"https:\/\/solutionsreview.com\/data-integration\/is-etl-on-life-support\/"},"wordCount":749,"commentCount":0,"publisher":{"@id":"https:\/\/solutionsreview.com\/data-integration\/#organization"},"image":{"@id":"https:\/\/solutionsreview.com\/data-integration\/is-etl-on-life-support\/#primaryimage"},"thumbnailUrl":"https:\/\/solutionsreview.com\/data-integration\/files\/2014\/06\/Is-ETL-on-Life-Support.jpg","keywords":["Big Data","ETL","Hadoop"],"articleSection":["Best Practices"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/solutionsreview.com\/data-integration\/is-etl-on-life-support\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/solutionsreview.com\/data-integration\/is-etl-on-life-support\/","url":"https:\/\/solutionsreview.com\/data-integration\/is-etl-on-life-support\/","name":"Is ETL on Life Support?","isPartOf":{"@id":"https:\/\/solutionsreview.com\/data-integration\/#website"},"primaryImageOfPage":{"@id":"https:\/\/solutionsreview.com\/data-integration\/is-etl-on-life-support\/#primaryimage"},"image":{"@id":"https:\/\/solutionsreview.com\/data-integration\/is-etl-on-life-support\/#primaryimage"},"thumbnailUrl":"https:\/\/solutionsreview.com\/data-integration\/files\/2014\/06\/Is-ETL-on-Life-Support.jpg","datePublished":"2015-09-25T06:52:37+00:00","dateModified":"2016-10-04T19:31:10+00:00","description":"Is ETL on Life Support? The answer may surprise you.","breadcrumb":{"@id":"https:\/\/solutionsreview.com\/data-integration\/is-etl-on-life-support\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/solutionsreview.com\/data-integration\/is-etl-on-life-support\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/solutionsreview.com\/data-integration\/is-etl-on-life-support\/#primaryimage","url":"https:\/\/solutionsreview.com\/data-integration\/files\/2014\/06\/Is-ETL-on-Life-Support.jpg","contentUrl":"https:\/\/solutionsreview.com\/data-integration\/files\/2014\/06\/Is-ETL-on-Life-Support.jpg","width":600,"height":270,"caption":"Is ETL on Life Support"},{"@type":"BreadcrumbList","@id":"https:\/\/solutionsreview.com\/data-integration\/is-etl-on-life-support\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/solutionsreview.com\/data-integration\/"},{"@type":"ListItem","position":2,"name":"Is ETL on Life Support?"}]},{"@type":"WebSite","@id":"https:\/\/solutionsreview.com\/data-integration\/#website","url":"https:\/\/solutionsreview.com\/data-integration\/","name":"Best Data Integration Vendors, News &amp; Reviews for Big Data, Applications, ETL and Hadoop","description":"Data Integration Buyers Guide and Best Practices","publisher":{"@id":"https:\/\/solutionsreview.com\/data-integration\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/solutionsreview.com\/data-integration\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/solutionsreview.com\/data-integration\/#organization","name":"Solutions Review","url":"https:\/\/solutionsreview.com\/data-integration\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/solutionsreview.com\/data-integration\/#\/schema\/logo\/image\/","url":"https:\/\/solutionsreview.com\/data-integration\/files\/2016\/02\/Solutions_Review_Header_Data_Integration_225.png","contentUrl":"https:\/\/solutionsreview.com\/data-integration\/files\/2016\/02\/Solutions_Review_Header_Data_Integration_225.png","width":225,"height":90,"caption":"Solutions Review"},"image":{"@id":"https:\/\/solutionsreview.com\/data-integration\/#\/schema\/logo\/image\/"}},{"@type":"Person","@id":"https:\/\/solutionsreview.com\/data-integration\/#\/schema\/person\/154e152a275103e373e24ada7f2feb5c","name":"Tim King","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/solutionsreview.com\/data-integration\/#\/schema\/person\/image\/","url":"https:\/\/solutionsreview.com\/data-integration\/files\/2023\/12\/tk.jpg","contentUrl":"https:\/\/solutionsreview.com\/data-integration\/files\/2023\/12\/tk.jpg","caption":"Tim King"},"description":"Tim is Solutions Review's Executive Editor covering the human impact of AI on the future of work and learning. He is also the Media Strategist behind Insight Jam (1M+ on YouTube) events and programming. A 2017 and 2018 Most Influential Business Journalist and 2021 \"Who's Who\" in multiple categories, Tim is a recognized thought leader in enterprise tech and AI.","url":"https:\/\/solutionsreview.com\/data-integration\/author\/timking\/"}]}},"_links":{"self":[{"href":"https:\/\/solutionsreview.com\/data-integration\/wp-json\/wp\/v2\/posts\/586"}],"collection":[{"href":"https:\/\/solutionsreview.com\/data-integration\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/solutionsreview.com\/data-integration\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/solutionsreview.com\/data-integration\/wp-json\/wp\/v2\/users\/23"}],"replies":[{"embeddable":true,"href":"https:\/\/solutionsreview.com\/data-integration\/wp-json\/wp\/v2\/comments?post=586"}],"version-history":[{"count":0,"href":"https:\/\/solutionsreview.com\/data-integration\/wp-json\/wp\/v2\/posts\/586\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/solutionsreview.com\/data-integration\/wp-json\/wp\/v2\/media\/592"}],"wp:attachment":[{"href":"https:\/\/solutionsreview.com\/data-integration\/wp-json\/wp\/v2\/media?parent=586"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/solutionsreview.com\/data-integration\/wp-json\/wp\/v2\/categories?post=586"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/solutionsreview.com\/data-integration\/wp-json\/wp\/v2\/tags?post=586"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}