{"id":4074,"date":"2022-06-24T17:39:44","date_gmt":"2022-06-24T21:39:44","guid":{"rendered":"https:\/\/solutionsreview.com\/data-management\/?p=4074"},"modified":"2022-06-30T13:02:57","modified_gmt":"2022-06-30T17:02:57","slug":"how-to-reinvent-your-data-streaming-architecture-a-brief","status":"publish","type":"post","link":"https:\/\/solutionsreview.com\/data-management\/how-to-reinvent-your-data-streaming-architecture-a-brief\/","title":{"rendered":"How to Reinvent Your Data Streaming Architecture: A Brief"},"content":{"rendered":"

\"How<\/p>\n

This is part of Solutions Review\u2019s Premium Content Series, a collection of contributed columns written by industry experts in maturing software categories. In this submission, Upsolver<\/a> Co-Founder and CEO Ori Rafael outlines key ways to reinvent your data streaming architecture with examples.<\/em><\/strong><\/p>\n

\"SRData volumes are exploding, driven mostly by tech-led companies (e.g. not just the big ones, but emerging players in fintech, adtech, edtech, etc.). However, old-school companies have jumped in as well, by adding IoT sensors to automobiles, factory lines, and oil pipelines, plus collecting and analyzing customer interactions from their websites, digital products, and customer support. Data is used for everything from automating highway toll payments to measuring earthquakes to monitoring assembly lines, as The Wall Street Journal reports. But most data never leaves the nest. IDC estimates nearly two-thirds of 2020\u2019s data existed only briefly. Of the other third, much of it sits in storage for years, unused.<\/p>\n

This is because data is born as a new event in some source system, and the value of these data events degrades very quickly. So, you have to utilize new data quickly so you can act while it still matters. And to understand and act on real-time events you have to implement streaming data technologies.<\/p>\n

Make no mistake, exploiting streaming data requires a paradigm shift. We\u2019ve spent decades taking data events and batch processing them on an hourly, daily or weekly basis. To deliver real-time action requires a stream processing mindset where new data is continually compared to historical data to identify interesting changes that can be acted upon by the business.<\/p>\n

This means that your organization can\u2019t continue to rely on the data infrastructure it has deployed over the last three decades. Unless it is updated to support a streaming process, your analytics will fare poorly relative to what is possible.<\/p>\n

In short, you need to change your mindset and reinvent your architecture. Here is how you can do that.<\/p>\n

\t\t\t
\"Download<\/a><\/div>\n\t\t<\/div><\/div>\n

Architect for Events<\/strong><\/h3>\n

While data is born as an event, these events are usually batched because that\u2019s the process supported by traditional data infrastructure. A traditional data pipeline typically involves batch processing with defined start and end times (say a batch-processed at the end of each hour).<\/p>\n

Obviously, you can\u2019t enable real-time analytics using batch. Batch processing creates delay, and that delay equates to missed opportunities. To get to fresher analytics, you must adopt a streaming data pipeline approach for which logic runs on each new event as it arrives and allows you to detect changes as they happen.<\/p>\n

Besides providing fresher data, stream processing has the added benefit of being backward traceable. A well-designed streaming architecture employs an \u201cevent-source\u201d approach that keeps a log of every change made to the data set since its inception. This allows you to make changes to your logic and rerun the new logic on old data. So, if you discover a bug, or an unexpected change to your source data, you can rerun your pipeline. This makes your data operations more flexible and resilient.<\/p>\n

Reengineer for Freshness at Scale<\/strong><\/h3>\n

Imagine you want to implement a \u201cnext best offer\u201d system, which combines real-time behavioral data from an app user with all sorts of contextual data about them (e.g. browsing history, location, demographics) to determine which offers make the most sense for that individual. This kind of in-the-moment action relies on data freshness enforced by strict service level agreements (SLAs).<\/p>\n

As these SLAs tighten, and the amount of data you source and process increases, and the number of pipelines you run increases (as the data is used by more data consumers), you will need to scale your infrastructure flexibly to maintain your data freshness.<\/p>\n

As your requirements change and your demand grows, make sure that you can maintain real- time performance by using cloud data processing services with elastic scaling. Ensure that you don\u2019t have any long-running operations that will slow down your process, or excessive memory utilization that can lead to service level inconsistencies.<\/p>\n

Implement a Real-Time Data Lake<\/strong><\/h3>\n

Streaming data necessitates a number of new technologies. You need a way to ingest events in a stream, store them affordably, process them efficiently and distribute the transformed data to various analytics systems. The good news is that there are technologies available and proven in the marketplace. In total, these technologies constitute a real-time data lake.<\/p>\n

Streaming ingestion: You need to install and manage a message queue such as Apache Kafka, Amazon Kinesis or Azure Event Hub that can collect events into streams.<\/p>\n

Data lake storage: Streaming data can accumulate to a huge size, so a cloud data lake based on object storage such as Amazon S3 or Azure Data Lake Service (ADLS) is the most economical way to handle it. Many tools will allow you to connect message queues to a data lake.<\/p>\n

Processing platform: Once data is stored in the data lake, you can either use Apache Spark \u2013 jobs are written in Python, Java or Scala \u2013 or a SQL-based tool such as Upsolver to blend recent real-time data with historical data to feed downstream systems. This blending occurs continuously as new data arrives.<\/p>\n

Data lake optimization tools: Data lakes are very affordable, but they require performance optimization, something we call PipelineOps. Here are some examples. First, a compaction process is required to turn millions of single-event files into bigger files that are efficiently processed. Second, blending streaming and batch data requires orchestration of the tasks required to execute the processing job. Third, a state store is required for stateful processing that joins batches with streams.<\/p>\n

These processes can be accomplished through a set of specific tools such as Airflow for orchestration or RocksDB, Redis or Cassandra as a state store, with these tools glued to the processing engine via code. Alternatively, you can implement a declarative data pipeline platform such as Upsolver, which automates these PipelineOps functions.<\/p>\n

Analytics systems: Once you\u2019ve successfully optimized your data lake, you will be able to output your streaming data as \u201clive tables\u201d (they auto-update as new events arrive) that any analytics system can use.<\/p>\n

Streaming Is the Way Forward \u2013 and It\u2019s Time to Get Moving<\/strong><\/h3>\n

As you can see, streaming requires a mindset shift, thoughtful planning and new infrastructure. However, if you invest the time and effort to implement modern streaming processes and systems, you will be able to leverage fresh data and timely analytics insights.<\/p>\n

<\/div>\n
\t\t\t

\"Download<\/a><\/p>\n<\/div>\n\t\t<\/div><\/div>\n","protected":false},"excerpt":{"rendered":"

This is part of Solutions Review\u2019s Premium Content Series, a collection of contributed columns written by industry experts in maturing software categories. In this submission, Upsolver Co-Founder and CEO Ori Rafael outlines key ways to reinvent your data streaming architecture with examples. Data volumes are exploding, driven mostly by tech-led companies (e.g. not just the […]<\/p>\n","protected":false},"author":207,"featured_media":4084,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"inline_featured_image":false,"footnotes":""},"categories":[3],"tags":[1014,1013],"acf":[],"yoast_head":"\nHow to Reinvent Your Data Streaming Architecture: A Brief<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/solutionsreview.com\/data-management\/how-to-reinvent-your-data-streaming-architecture-a-brief\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"How to Reinvent Your Data Streaming Architecture: A Brief\" \/>\n<meta property=\"og:description\" content=\"This is part of Solutions Review\u2019s Premium Content Series, a collection of contributed columns written by industry experts in maturing software categories. In this submission, Upsolver Co-Founder and CEO Ori Rafael outlines key ways to reinvent your data streaming architecture with examples. Data volumes are exploding, driven mostly by tech-led companies (e.g. not just the […]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/solutionsreview.com\/data-management\/how-to-reinvent-your-data-streaming-architecture-a-brief\/\" \/>\n<meta property=\"og:site_name\" content=\"Data Management Software & Infrastructure | Solutions Review\" \/>\n<meta property=\"article:published_time\" content=\"2022-06-24T21:39:44+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2022-06-30T17:02:57+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/solutionsreview.com\/data-management\/files\/2022\/06\/MicrosoftTeams-image-12.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"800\" \/>\n\t<meta property=\"og:image:height\" content=\"400\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Ori Rafael\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Ori Rafael\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"5 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/solutionsreview.com\/data-management\/how-to-reinvent-your-data-streaming-architecture-a-brief\/\",\"url\":\"https:\/\/solutionsreview.com\/data-management\/how-to-reinvent-your-data-streaming-architecture-a-brief\/\",\"name\":\"How to Reinvent Your Data Streaming Architecture: A Brief\",\"isPartOf\":{\"@id\":\"https:\/\/solutionsreview.com\/data-management\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/solutionsreview.com\/data-management\/how-to-reinvent-your-data-streaming-architecture-a-brief\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/solutionsreview.com\/data-management\/how-to-reinvent-your-data-streaming-architecture-a-brief\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/solutionsreview.com\/data-management\/files\/2022\/06\/MicrosoftTeams-image-12.jpg\",\"datePublished\":\"2022-06-24T21:39:44+00:00\",\"dateModified\":\"2022-06-30T17:02:57+00:00\",\"author\":{\"@id\":\"https:\/\/solutionsreview.com\/data-management\/#\/schema\/person\/5f1a04c09c1d3ef8f4086f3246eabc2d\"},\"breadcrumb\":{\"@id\":\"https:\/\/solutionsreview.com\/data-management\/how-to-reinvent-your-data-streaming-architecture-a-brief\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/solutionsreview.com\/data-management\/how-to-reinvent-your-data-streaming-architecture-a-brief\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/solutionsreview.com\/data-management\/how-to-reinvent-your-data-streaming-architecture-a-brief\/#primaryimage\",\"url\":\"https:\/\/solutionsreview.com\/data-management\/files\/2022\/06\/MicrosoftTeams-image-12.jpg\",\"contentUrl\":\"https:\/\/solutionsreview.com\/data-management\/files\/2022\/06\/MicrosoftTeams-image-12.jpg\",\"width\":800,\"height\":400,\"caption\":\"How to Reinvent Your Data Streaming Architecture: A Brief\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/solutionsreview.com\/data-management\/how-to-reinvent-your-data-streaming-architecture-a-brief\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/solutionsreview.com\/data-management\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"How to Reinvent Your Data Streaming Architecture: A Brief\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/solutionsreview.com\/data-management\/#website\",\"url\":\"https:\/\/solutionsreview.com\/data-management\/\",\"name\":\"Data Management Software & Infrastructure | Solutions Review\",\"description\":\"Evaluating Enterprise Master Data Management, Data Governance & Storage Tools.\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/solutionsreview.com\/data-management\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/solutionsreview.com\/data-management\/#\/schema\/person\/5f1a04c09c1d3ef8f4086f3246eabc2d\",\"name\":\"Ori Rafael\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/solutionsreview.com\/data-management\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/4b56cd22234a508249f5bde459930088?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/4b56cd22234a508249f5bde459930088?s=96&d=mm&r=g\",\"caption\":\"Ori Rafael\"},\"description\":\"Ori Rafael is the CEO and Co-Founder of Upsolver, the only no-code data lake engineering platform for agile cloud analytics. Before founding Upsolver, he held various technology management roles at IDF\u2019s elite technology intelligence unit, followed by corporate roles. Rafael has a BA in computer science and an MBA.\",\"url\":\"https:\/\/solutionsreview.com\/data-management\/author\/orafael\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"How to Reinvent Your Data Streaming Architecture: A Brief","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/solutionsreview.com\/data-management\/how-to-reinvent-your-data-streaming-architecture-a-brief\/","og_locale":"en_US","og_type":"article","og_title":"How to Reinvent Your Data Streaming Architecture: A Brief","og_description":"This is part of Solutions Review\u2019s Premium Content Series, a collection of contributed columns written by industry experts in maturing software categories. In this submission, Upsolver Co-Founder and CEO Ori Rafael outlines key ways to reinvent your data streaming architecture with examples. Data volumes are exploding, driven mostly by tech-led companies (e.g. not just the […]","og_url":"https:\/\/solutionsreview.com\/data-management\/how-to-reinvent-your-data-streaming-architecture-a-brief\/","og_site_name":"Data Management Software & Infrastructure | Solutions Review","article_published_time":"2022-06-24T21:39:44+00:00","article_modified_time":"2022-06-30T17:02:57+00:00","og_image":[{"width":800,"height":400,"url":"https:\/\/solutionsreview.com\/data-management\/files\/2022\/06\/MicrosoftTeams-image-12.jpg","type":"image\/jpeg"}],"author":"Ori Rafael","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Ori Rafael","Est. reading time":"5 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/solutionsreview.com\/data-management\/how-to-reinvent-your-data-streaming-architecture-a-brief\/","url":"https:\/\/solutionsreview.com\/data-management\/how-to-reinvent-your-data-streaming-architecture-a-brief\/","name":"How to Reinvent Your Data Streaming Architecture: A Brief","isPartOf":{"@id":"https:\/\/solutionsreview.com\/data-management\/#website"},"primaryImageOfPage":{"@id":"https:\/\/solutionsreview.com\/data-management\/how-to-reinvent-your-data-streaming-architecture-a-brief\/#primaryimage"},"image":{"@id":"https:\/\/solutionsreview.com\/data-management\/how-to-reinvent-your-data-streaming-architecture-a-brief\/#primaryimage"},"thumbnailUrl":"https:\/\/solutionsreview.com\/data-management\/files\/2022\/06\/MicrosoftTeams-image-12.jpg","datePublished":"2022-06-24T21:39:44+00:00","dateModified":"2022-06-30T17:02:57+00:00","author":{"@id":"https:\/\/solutionsreview.com\/data-management\/#\/schema\/person\/5f1a04c09c1d3ef8f4086f3246eabc2d"},"breadcrumb":{"@id":"https:\/\/solutionsreview.com\/data-management\/how-to-reinvent-your-data-streaming-architecture-a-brief\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/solutionsreview.com\/data-management\/how-to-reinvent-your-data-streaming-architecture-a-brief\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/solutionsreview.com\/data-management\/how-to-reinvent-your-data-streaming-architecture-a-brief\/#primaryimage","url":"https:\/\/solutionsreview.com\/data-management\/files\/2022\/06\/MicrosoftTeams-image-12.jpg","contentUrl":"https:\/\/solutionsreview.com\/data-management\/files\/2022\/06\/MicrosoftTeams-image-12.jpg","width":800,"height":400,"caption":"How to Reinvent Your Data Streaming Architecture: A Brief"},{"@type":"BreadcrumbList","@id":"https:\/\/solutionsreview.com\/data-management\/how-to-reinvent-your-data-streaming-architecture-a-brief\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/solutionsreview.com\/data-management\/"},{"@type":"ListItem","position":2,"name":"How to Reinvent Your Data Streaming Architecture: A Brief"}]},{"@type":"WebSite","@id":"https:\/\/solutionsreview.com\/data-management\/#website","url":"https:\/\/solutionsreview.com\/data-management\/","name":"Data Management Software & Infrastructure | Solutions Review","description":"Evaluating Enterprise Master Data Management, Data Governance & Storage Tools.","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/solutionsreview.com\/data-management\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/solutionsreview.com\/data-management\/#\/schema\/person\/5f1a04c09c1d3ef8f4086f3246eabc2d","name":"Ori Rafael","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/solutionsreview.com\/data-management\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/4b56cd22234a508249f5bde459930088?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/4b56cd22234a508249f5bde459930088?s=96&d=mm&r=g","caption":"Ori Rafael"},"description":"Ori Rafael is the CEO and Co-Founder of Upsolver, the only no-code data lake engineering platform for agile cloud analytics. Before founding Upsolver, he held various technology management roles at IDF\u2019s elite technology intelligence unit, followed by corporate roles. Rafael has a BA in computer science and an MBA.","url":"https:\/\/solutionsreview.com\/data-management\/author\/orafael\/"}]}},"_links":{"self":[{"href":"https:\/\/solutionsreview.com\/data-management\/wp-json\/wp\/v2\/posts\/4074"}],"collection":[{"href":"https:\/\/solutionsreview.com\/data-management\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/solutionsreview.com\/data-management\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/solutionsreview.com\/data-management\/wp-json\/wp\/v2\/users\/207"}],"replies":[{"embeddable":true,"href":"https:\/\/solutionsreview.com\/data-management\/wp-json\/wp\/v2\/comments?post=4074"}],"version-history":[{"count":0,"href":"https:\/\/solutionsreview.com\/data-management\/wp-json\/wp\/v2\/posts\/4074\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/solutionsreview.com\/data-management\/wp-json\/wp\/v2\/media\/4084"}],"wp:attachment":[{"href":"https:\/\/solutionsreview.com\/data-management\/wp-json\/wp\/v2\/media?parent=4074"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/solutionsreview.com\/data-management\/wp-json\/wp\/v2\/categories?post=4074"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/solutionsreview.com\/data-management\/wp-json\/wp\/v2\/tags?post=4074"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}