The data mesh is an attempt by software engineers to remake the data industry in their image. There is a lot of goodness in the data mesh but will it work?<\/p>\n
Every decade or so, a segment of the information technology industry \u201cdiscovers data\u201d and makes big investments that don\u2019t pan out as expected and sometimes fail spectacularly.<\/p>\n
App Vendors.<\/strong>\u00a0In the mid 2000s, enterprise application providers, including\u00a0SAP<\/a>,\u00a0IBM<\/a>, and\u00a0Oracle<\/a>, who paid billions to buy into the maturing business intelligence market through acquisitions of Business Objects, Cognos, and Hyperion. Those products were soon eclipsed by new ad hoc visualization tools (e.g.,\u00a0Qlik<\/a>,\u00a0Tableau<\/a>,\u00a0PowerBI<\/a>) and more recently cloud native BI tools. It\u2019s unlikely the Big Three ever realized a smart return on their money.<\/p>\n Open Source.<\/strong>\u00a0In the mid 2010s, the open source community coined the term \u201cBig Data,\u201d which was all the rage until its flagship codebase \u2013 Hadoop \u2013 collapsed under the weight of its complexity, unreliability, and, yes, cost. Spark, then cloud data platforms, such as\u00a0Snowflake<\/a>\u00a0and\u00a0Databricks<\/a>, buried Hadoop and its dreams of making data warehouses obsolete.<\/p>\n Software Engineering.<\/strong>\u00a0Today, software engineers have discovered data and don\u2019t like what they see. The data mesh is a software engineer\u2019s model for how to reinvent data so it functions more like modern object-oriented code and microservices. Among other things, it specifies a peer-to-peer architecture that eliminates the need for data warehouses and data lakes. Zhamak Dheghani, author of \u201cData Mesh: Delivering Data-Driven Value at Scale<\/a>,\u201d writes:<\/p>\n \u201cThe principles of data mesh are a generalization and adaptation of practices that have evolved over the last two decades and proved to solve our last complexity challenge: scale of software complexity led by the mass digitization of organizations\u2026. They are an adaptation of what formulated the previous paradigm shift in software: the microservices and APIs revolution, platform-based Team Topologies, computational governance models such as Zero Trust Architecture, and\u00a0operating distributed solutions securely and across multiple clouds and hosting environments.\u201d<\/em><\/p>\n I give Dheghani credit for accurately cataloging the ailments of most data implementations today: the endless data bottlenecks, rigid data models, poor data quality, disenfranchised data users, and lack of return on data investments. All true\u2014guilty as charged.<\/p>\n I give her more credit for coining a buzzword that represents the holistic changes that organizations need to make to optimize their use of data. Yes, organizations need data governance; yes, they need a self-service platform; yes, they need a distributed architecture; yes, they need to make it easy for people and teams to find and use data; yes, they need to build solutions closer to the business; yes, people and teams need to create their own data-driven solutions and share them with others. Data consultants and experts have counseled organizations for years about doing these things.<\/p>\n But like the big data movement, Dheghani is eager to ditch the old in favor of the new. The\u00a0data mesh<\/a>\u00a0has no place for a data warehouse, a data lake, or the data pipelines that feed them and the enterprise data engineers who create them. There is no place for an enterprise data team except to build and maintain the data sharing platform and (perhaps) facilitate the governance work that keeps the data domains from spinning into irreconcilable data silos. In other words, the data mesh is all about decentralizing data. That sounds good in theory, but will it work?<\/p>\n In a true data mesh, all data is created, managed, and shared by domain teams in the business. This is a true peer-to-peer data sharing environment in which domain teams create and share atomic data products through standard interfaces. Welcome to a microservices architecture for data! The question is: Will the lessons of software engineers who\u2019ve mastered how to code complex solutions in large geographically dispersed organizations apply to data?<\/p>\n Welcome to a microservices architecture for data!\u00a0<\/em><\/p>\n In my last\u00a0article<\/a>\u00a0on the data mesh, I wrote about some of its obvious problems. Most business teams don\u2019t have the time, staff, skills, budget, or interest to hire their own data experts to build and manage data products for themselves, let alone others in the company. A self-service data platform will make it easier for them to do this\u2014and certain groups\u2014typically finance and marketing that have already invested in their own data resources\u2014will benefit immensely. But most other groups will have to rely on enterprise data teams to do this work.<\/p>\n In addition, a big challenge with the mesh is keeping data domains from spinning off into irreconcilable data silos. A shared data platform is important, but a robust data governance program is critical. Unfortunately, data governance is notoriously hard to do right, even with a strong enterprise data team facilitating the work. Enforcing consistency on dimensions, metrics, master and reference data, and security is key to ensuring that a data mesh doesn\u2019t descend into data chaos.<\/p>\n Nonetheless, the data mesh is gaining momentum. Five new technologies are transforming the way organizations use data, and propelling adoption of the data mesh. It provides a suitable, if not coincidental, methodology to tie these technologies together. Many companies already use them to support modern data architectures, whether or not they are implementing a \u201cdata mesh\u201d.<\/p>\n Cloud data platforms.<\/strong>\u00a0These global, multi-tenant databases give each data domain its own space on a shared platform that makes it easy to share data across domains and access a common repository of enterprise data.<\/p>\n<\/li>\n DataOps.<\/strong>\u00a0Another carryover from software engineering,\u00a0DataOps<\/a>\u00a0makes it easy for domain teams to quickly build reliable data pipelines on a cloud data platform, especially ones that support zero-copy cloning and time travel.<\/p>\n<\/li>\n Data Catalog.<\/strong>\u00a0Finally, we have tools that index and tag all data in an enterprise. Catalogs make it easy for business users to find that data and understand its meaning and context.<\/p>\n<\/li>\n Data Marketplace.<\/strong>\u00a0An extension to a data catalog, the data marketplace makes it easy for individuals and teams to publish and consume data with each other, including external data.<\/p>\n<\/li>\n Data Virtualization.<\/strong>\u00a0These modern SQL engines make it easy to find and query distributed data at scale. They federate queries across multiple systems behind a common business model.<\/p>\n<\/li>\n<\/ol>\n Roche, a global biotech company, began implementing a data mesh a year-and-a-half ago to replace an on-premises data warehousing architecture afflicted with delays and bottlenecks. The migration has been a success. To date, Roche onboarded 40 data product teams which have created 50 data products. The data team has increased its delivery cadence from one release every three months to an astounding 120 releases in one month. (See \u201cWhat Can Data Mesh and DataOps Do For You? Ask Roche<\/a>\u201d):<\/p>\nThe Reckoning<\/strong><\/h4>\n
Stars Are Aligned<\/strong><\/h4>\n
\n