{"id":1188,"date":"2016-02-16T04:50:06","date_gmt":"2016-02-16T09:50:06","guid":{"rendered":"https:\/\/solutionsreview.com\/data-integration\/?p=1188"},"modified":"2016-10-05T09:51:24","modified_gmt":"2016-10-05T13:51:24","slug":"bernd-harzogs-2016-big-data-market-predictions-part-2","status":"publish","type":"post","link":"https:\/\/solutionsreview.com\/data-integration\/bernd-harzogs-2016-big-data-market-predictions-part-2\/","title":{"rendered":"Bernd Harzog’s 2016 Big Data Market Predictions: Part 2"},"content":{"rendered":"

\"Prediction<\/p>\n

By\u00a0Bernd Harzog<\/em><\/strong><\/p>\n

6. Relationships between sources of data are crucial:\u00a0<\/strong>Let\u2019s consider some retail sales data, where we have sales data by product, store and geography. The customer has an account with us so we know some basic demographics like gender and age range. All of this data comes to us from the online business system. Now we need to understand if the operation of the site affects revenue. This means that we need to combine end-user experience data, application performance data and IT Operations data with business data, as all of these disparate streams of data arrive. To analyze these disparate streams of data in real-time, ETL no longer works. There is no time to do ETL, so batch ETL needs to be replaced with real-time and continuous discovery of the relationships between streams of data from disparate sources as these sources of data arrive.<\/p>\n

7. Statistical relationships are only valuable against deterministically related data:\u00a0<\/strong>Much hope has been placed in \u201cBig Data analytics\u201d and \u201cmachine learning.\u201d Both are powerful techniques, which are benefiting from enormous innovations created by very smart people doing world-class work in self-learning algorithms. To the earlier point of \u201cgarbage in, garbage out\u201d \u2013 the best algorithms will produce mediocre or worthless results when applied against data where the items have no inherent relationship.<\/p>\n

8. The graph database is a crucial innovation:\u00a0<\/strong>The graph database is an important part of the solution to this problem. LinkedIn can show you who you are connected to and show you your entire \u201csocial graph\u201d (who is connected to whom). We need to connect our business, IT Operations and IoT data in the same manner.<\/p>\n
Widget not in any sidebars
\n

\n

9. The Hadoop stack is not the only future of Big Data:\u00a0<\/strong>The Hadoop stack has provided an exponential improvement upon the capabilities of the previous and now legacy data warehouse class of solutions. But when it comes to real-time and continuous stream processing, with the requirement to ingest data continuously and process it for immediate consumption in real time, Hadoop is itself a legacy solution. The modern innovations in this area include Cassandra, InfluxDB, FiloDB, Scylladb, Spark, Kafka, various graph databases like NEO4J and Titan, etc. In fact, there is even an acronym for a real-time Big Data stack which is SMACK \u2013 Spark, Mesos, Akka, Cassandra and Kafka. If you want to run it at scale then run SMACK HARD, where HARD is \u2013 High, Availability, Redundancy, and Distributed.<\/p>\n

10. Real-time Big Data will bind the entire online enterprise together:\u00a0<\/strong>Every part of the modern online enterprise is producing valuable streams of data. Each stream constitutes \u201cBig Data\u201d on its own. Together, they constitute a real-time deluge of data that must be collectively related, processed and made useful in seconds after ingest. Successful online enterprises will use their ability to take action upon real-time Big Data to achieve\u00a0advantage over slow moving rivals. In today\u2019s world, it is not about the big vs. the small. It is about the fast vs. the slow.<\/p>\n

Click here for part one of Harzog’s 2016 Big Data market predictions.<\/a><\/p>\n

\"BerndBernd Harzog is the CEO and Founder of OpsDataStore Inc.<\/a>, where he is responsible for the strategy, execution and financing activities of the company. Before Bernd founded OpsDataStore, he was the CEO and founder of APM Experts, CEO of RTO Software, Inc., founding VP of Products at Netuitive, a general manager at XcelleNet, and a Research Director for the Gartner Group focusing upon the Windows Server Operating family of products. Connect with him on LinkedIn.<\/a><\/em><\/p>\n","protected":false},"excerpt":{"rendered":"

By\u00a0Bernd Harzog 6. Relationships between sources of data are crucial:\u00a0Let\u2019s consider some retail sales data, where we have sales data by product, store and geography. The customer has an account with us so we know some basic demographics like gender and age range. All of this data comes to us from the online business system. […]<\/p>\n","protected":false},"author":23,"featured_media":1352,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"inline_featured_image":false,"footnotes":""},"categories":[1],"tags":[249,19,55,250,253],"acf":[],"yoast_head":"\nBernd Harzog's 2016 Big Data Market Predictions: Part 2<\/title>\n<meta name=\"description\" content=\"Part two of Bernd Harzog's 2016 enterprise Big Data market predictions.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/solutionsreview.com\/data-integration\/bernd-harzogs-2016-big-data-market-predictions-part-2\/\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Tim King\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/solutionsreview.com\/data-integration\/bernd-harzogs-2016-big-data-market-predictions-part-2\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/solutionsreview.com\/data-integration\/bernd-harzogs-2016-big-data-market-predictions-part-2\/\"},\"author\":{\"name\":\"Tim King\",\"@id\":\"https:\/\/solutionsreview.com\/data-integration\/#\/schema\/person\/154e152a275103e373e24ada7f2feb5c\"},\"headline\":\"Bernd Harzog’s 2016 Big Data Market Predictions: Part 2\",\"datePublished\":\"2016-02-16T09:50:06+00:00\",\"dateModified\":\"2016-10-05T13:51:24+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/solutionsreview.com\/data-integration\/bernd-harzogs-2016-big-data-market-predictions-part-2\/\"},\"wordCount\":596,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/solutionsreview.com\/data-integration\/#organization\"},\"image\":{\"@id\":\"https:\/\/solutionsreview.com\/data-integration\/bernd-harzogs-2016-big-data-market-predictions-part-2\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/solutionsreview.com\/data-integration\/files\/2016\/02\/oie_MxVuaHOEDfES.jpg\",\"keywords\":[\"Bernd Harzog\",\"Big Data\",\"Hadoop\",\"OpsDataStore\",\"Predictions\"],\"articleSection\":[\"Best Practices\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/solutionsreview.com\/data-integration\/bernd-harzogs-2016-big-data-market-predictions-part-2\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/solutionsreview.com\/data-integration\/bernd-harzogs-2016-big-data-market-predictions-part-2\/\",\"url\":\"https:\/\/solutionsreview.com\/data-integration\/bernd-harzogs-2016-big-data-market-predictions-part-2\/\",\"name\":\"Bernd Harzog's 2016 Big Data Market Predictions: Part 2\",\"isPartOf\":{\"@id\":\"https:\/\/solutionsreview.com\/data-integration\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/solutionsreview.com\/data-integration\/bernd-harzogs-2016-big-data-market-predictions-part-2\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/solutionsreview.com\/data-integration\/bernd-harzogs-2016-big-data-market-predictions-part-2\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/solutionsreview.com\/data-integration\/files\/2016\/02\/oie_MxVuaHOEDfES.jpg\",\"datePublished\":\"2016-02-16T09:50:06+00:00\",\"dateModified\":\"2016-10-05T13:51:24+00:00\",\"description\":\"Part two of Bernd Harzog's 2016 enterprise Big Data market predictions.\",\"breadcrumb\":{\"@id\":\"https:\/\/solutionsreview.com\/data-integration\/bernd-harzogs-2016-big-data-market-predictions-part-2\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/solutionsreview.com\/data-integration\/bernd-harzogs-2016-big-data-market-predictions-part-2\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/solutionsreview.com\/data-integration\/bernd-harzogs-2016-big-data-market-predictions-part-2\/#primaryimage\",\"url\":\"https:\/\/solutionsreview.com\/data-integration\/files\/2016\/02\/oie_MxVuaHOEDfES.jpg\",\"contentUrl\":\"https:\/\/solutionsreview.com\/data-integration\/files\/2016\/02\/oie_MxVuaHOEDfES.jpg\",\"width\":800,\"height\":350},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/solutionsreview.com\/data-integration\/bernd-harzogs-2016-big-data-market-predictions-part-2\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/solutionsreview.com\/data-integration\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Bernd Harzog’s 2016 Big Data Market Predictions: Part 2\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/solutionsreview.com\/data-integration\/#website\",\"url\":\"https:\/\/solutionsreview.com\/data-integration\/\",\"name\":\"Data Integration Tools & ETL Software | Solutions Review\",\"description\":\"Evaluating Enterprise ETL Tools, Cloud iPaaS & Real-Time Data Streaming Platforms.\",\"publisher\":{\"@id\":\"https:\/\/solutionsreview.com\/data-integration\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/solutionsreview.com\/data-integration\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/solutionsreview.com\/data-integration\/#organization\",\"name\":\"Solutions Review\",\"url\":\"https:\/\/solutionsreview.com\/data-integration\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/solutionsreview.com\/data-integration\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/solutionsreview.com\/data-integration\/files\/2016\/02\/Solutions_Review_Header_Data_Integration_225.png\",\"contentUrl\":\"https:\/\/solutionsreview.com\/data-integration\/files\/2016\/02\/Solutions_Review_Header_Data_Integration_225.png\",\"width\":225,\"height\":90,\"caption\":\"Solutions Review\"},\"image\":{\"@id\":\"https:\/\/solutionsreview.com\/data-integration\/#\/schema\/logo\/image\/\"}},{\"@type\":\"Person\",\"@id\":\"https:\/\/solutionsreview.com\/data-integration\/#\/schema\/person\/154e152a275103e373e24ada7f2feb5c\",\"name\":\"Tim King\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/solutionsreview.com\/data-integration\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/solutionsreview.com\/data-integration\/files\/2023\/12\/tk.jpg\",\"contentUrl\":\"https:\/\/solutionsreview.com\/data-integration\/files\/2023\/12\/tk.jpg\",\"caption\":\"Tim King\"},\"description\":\"Tim is Solutions Review's Executive Editor covering the human impact of AI on the future of work and learning. He is also the Media Strategist behind Insight Jam (1M+ on YouTube) events and programming. A 2017 and 2018 Most Influential Business Journalist and 2021 \\\"Who's Who\\\" in multiple categories, Tim is a recognized thought leader in enterprise tech and AI.\",\"url\":\"https:\/\/solutionsreview.com\/data-integration\/author\/timking\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Bernd Harzog's 2016 Big Data Market Predictions: Part 2","description":"Part two of Bernd Harzog's 2016 enterprise Big Data market predictions.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/solutionsreview.com\/data-integration\/bernd-harzogs-2016-big-data-market-predictions-part-2\/","twitter_misc":{"Written by":"Tim King","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/solutionsreview.com\/data-integration\/bernd-harzogs-2016-big-data-market-predictions-part-2\/#article","isPartOf":{"@id":"https:\/\/solutionsreview.com\/data-integration\/bernd-harzogs-2016-big-data-market-predictions-part-2\/"},"author":{"name":"Tim King","@id":"https:\/\/solutionsreview.com\/data-integration\/#\/schema\/person\/154e152a275103e373e24ada7f2feb5c"},"headline":"Bernd Harzog’s 2016 Big Data Market Predictions: Part 2","datePublished":"2016-02-16T09:50:06+00:00","dateModified":"2016-10-05T13:51:24+00:00","mainEntityOfPage":{"@id":"https:\/\/solutionsreview.com\/data-integration\/bernd-harzogs-2016-big-data-market-predictions-part-2\/"},"wordCount":596,"commentCount":0,"publisher":{"@id":"https:\/\/solutionsreview.com\/data-integration\/#organization"},"image":{"@id":"https:\/\/solutionsreview.com\/data-integration\/bernd-harzogs-2016-big-data-market-predictions-part-2\/#primaryimage"},"thumbnailUrl":"https:\/\/solutionsreview.com\/data-integration\/files\/2016\/02\/oie_MxVuaHOEDfES.jpg","keywords":["Bernd Harzog","Big Data","Hadoop","OpsDataStore","Predictions"],"articleSection":["Best Practices"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/solutionsreview.com\/data-integration\/bernd-harzogs-2016-big-data-market-predictions-part-2\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/solutionsreview.com\/data-integration\/bernd-harzogs-2016-big-data-market-predictions-part-2\/","url":"https:\/\/solutionsreview.com\/data-integration\/bernd-harzogs-2016-big-data-market-predictions-part-2\/","name":"Bernd Harzog's 2016 Big Data Market Predictions: Part 2","isPartOf":{"@id":"https:\/\/solutionsreview.com\/data-integration\/#website"},"primaryImageOfPage":{"@id":"https:\/\/solutionsreview.com\/data-integration\/bernd-harzogs-2016-big-data-market-predictions-part-2\/#primaryimage"},"image":{"@id":"https:\/\/solutionsreview.com\/data-integration\/bernd-harzogs-2016-big-data-market-predictions-part-2\/#primaryimage"},"thumbnailUrl":"https:\/\/solutionsreview.com\/data-integration\/files\/2016\/02\/oie_MxVuaHOEDfES.jpg","datePublished":"2016-02-16T09:50:06+00:00","dateModified":"2016-10-05T13:51:24+00:00","description":"Part two of Bernd Harzog's 2016 enterprise Big Data market predictions.","breadcrumb":{"@id":"https:\/\/solutionsreview.com\/data-integration\/bernd-harzogs-2016-big-data-market-predictions-part-2\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/solutionsreview.com\/data-integration\/bernd-harzogs-2016-big-data-market-predictions-part-2\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/solutionsreview.com\/data-integration\/bernd-harzogs-2016-big-data-market-predictions-part-2\/#primaryimage","url":"https:\/\/solutionsreview.com\/data-integration\/files\/2016\/02\/oie_MxVuaHOEDfES.jpg","contentUrl":"https:\/\/solutionsreview.com\/data-integration\/files\/2016\/02\/oie_MxVuaHOEDfES.jpg","width":800,"height":350},{"@type":"BreadcrumbList","@id":"https:\/\/solutionsreview.com\/data-integration\/bernd-harzogs-2016-big-data-market-predictions-part-2\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/solutionsreview.com\/data-integration\/"},{"@type":"ListItem","position":2,"name":"Bernd Harzog’s 2016 Big Data Market Predictions: Part 2"}]},{"@type":"WebSite","@id":"https:\/\/solutionsreview.com\/data-integration\/#website","url":"https:\/\/solutionsreview.com\/data-integration\/","name":"Data Integration Tools & ETL Software | Solutions Review","description":"Evaluating Enterprise ETL Tools, Cloud iPaaS & Real-Time Data Streaming Platforms.","publisher":{"@id":"https:\/\/solutionsreview.com\/data-integration\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/solutionsreview.com\/data-integration\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/solutionsreview.com\/data-integration\/#organization","name":"Solutions Review","url":"https:\/\/solutionsreview.com\/data-integration\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/solutionsreview.com\/data-integration\/#\/schema\/logo\/image\/","url":"https:\/\/solutionsreview.com\/data-integration\/files\/2016\/02\/Solutions_Review_Header_Data_Integration_225.png","contentUrl":"https:\/\/solutionsreview.com\/data-integration\/files\/2016\/02\/Solutions_Review_Header_Data_Integration_225.png","width":225,"height":90,"caption":"Solutions Review"},"image":{"@id":"https:\/\/solutionsreview.com\/data-integration\/#\/schema\/logo\/image\/"}},{"@type":"Person","@id":"https:\/\/solutionsreview.com\/data-integration\/#\/schema\/person\/154e152a275103e373e24ada7f2feb5c","name":"Tim King","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/solutionsreview.com\/data-integration\/#\/schema\/person\/image\/","url":"https:\/\/solutionsreview.com\/data-integration\/files\/2023\/12\/tk.jpg","contentUrl":"https:\/\/solutionsreview.com\/data-integration\/files\/2023\/12\/tk.jpg","caption":"Tim King"},"description":"Tim is Solutions Review's Executive Editor covering the human impact of AI on the future of work and learning. He is also the Media Strategist behind Insight Jam (1M+ on YouTube) events and programming. A 2017 and 2018 Most Influential Business Journalist and 2021 \"Who's Who\" in multiple categories, Tim is a recognized thought leader in enterprise tech and AI.","url":"https:\/\/solutionsreview.com\/data-integration\/author\/timking\/"}]}},"_links":{"self":[{"href":"https:\/\/solutionsreview.com\/data-integration\/wp-json\/wp\/v2\/posts\/1188"}],"collection":[{"href":"https:\/\/solutionsreview.com\/data-integration\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/solutionsreview.com\/data-integration\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/solutionsreview.com\/data-integration\/wp-json\/wp\/v2\/users\/23"}],"replies":[{"embeddable":true,"href":"https:\/\/solutionsreview.com\/data-integration\/wp-json\/wp\/v2\/comments?post=1188"}],"version-history":[{"count":0,"href":"https:\/\/solutionsreview.com\/data-integration\/wp-json\/wp\/v2\/posts\/1188\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/solutionsreview.com\/data-integration\/wp-json\/wp\/v2\/media\/1352"}],"wp:attachment":[{"href":"https:\/\/solutionsreview.com\/data-integration\/wp-json\/wp\/v2\/media?parent=1188"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/solutionsreview.com\/data-integration\/wp-json\/wp\/v2\/categories?post=1188"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/solutionsreview.com\/data-integration\/wp-json\/wp\/v2\/tags?post=1188"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}