{"id":501,"date":"2024-01-01T18:25:00","date_gmt":"2024-01-01T18:25:00","guid":{"rendered":"https:\/\/solutionsreview.com\/expert\/?p=501"},"modified":"2024-02-06T13:58:48","modified_gmt":"2024-02-06T13:58:48","slug":"rethinking-the-data-mesh-apply-it-piecemeal","status":"publish","type":"post","link":"https:\/\/solutionsreview.com\/thought-leaders\/rethinking-the-data-mesh-apply-it-piecemeal\/","title":{"rendered":"Rethinking the Data Mesh: Apply it Piecemeal"},"content":{"rendered":"

The data mesh is gaining popularity as a distributed architecture that mirrors reality. But its practical limitations mean it should be applied piecemeal. Our clients now ask us about the \u201cdata mesh\u201d- an emerging distributed architecture that addresses the shortcomings of traditional centralized data architectures which power most enterprises today.<\/p>\n

For those of us who have been around this field for a while (almost 30 years for me), the term \u201cdata mesh\u201d conjures up a\u00a0d\u00e9j\u00e0 vu of past distributed technologies, whose monikers long ago entered the dustbin of analytics history: virtual data warehouse, enterprise information intelligence, logical data warehouse, data virtualization, and others. Whatever the name, these technologies all have provided a global semantic view of distributed data and a mechanism to federate queries across systems in real time.<\/p>\n

To be fair, the data mesh as conceived by\u00a0Zhamak Dehghani<\/a>\u00a0is broader than a traditional data virtualization tool. It embeds the technology within a decentralized ownership model, distributed data development, and federated governance. It also requires a common data platform that abstracts the complexity of building and managing data products and semantic models so ordinary business users can publish data to the mesh.<\/p>\n

For companies exploring the architectural possibilities of a data mesh, it\u2019s noteworthy that no one seems to understand it. There is confusion among people who have read deeply on the subject[1]. And it doesn\u2019t help that Zhamak\u2019s writing style is a bit opaque, which prompts skeptics to wonder if the concept is simply \u201cold wine in new wineskins\u201d and gives vendors license to proclaim that their technology powers it. If nothing else, the data mesh is now a huge marketing bonanza for vendors who want to hitch themselves to a hot industry buzzword.<\/p>\n

If nothing else, the data mesh is now a huge marketing bonanza for vendors who want to hitch themselves to a hot industry buzzword.<\/em><\/p>\n

There are two issues here: 1) how seriously should we take the data mesh? Is there any redeeming value in this concept? and 2) what is the nature of the data mesh technology that vendors are hawking today? We\u2019ll tackle the first question in this article and the second next month.<\/p>\n

Data Mesh Perspectives<\/strong><\/h4>\n

Centralization versus decentralization.<\/strong> The data mesh is a reaction to centralized data organizations, architectures, and governance models that are prone to rigidity, inflexibility, and backlogs. To address these shortcomings, the data mesh pivots 180-degrees and embraces a decentralized approach. The data mesh gives data domain owners (e.g., department or subject-area heads) complete jurisdiction over data in their domains. With a data mesh, data never moves; anyone who wants the data goes to the department to get it. That puts the onus on departments to model and publish data in a form that others can use.<\/p>\n

Unfortunately, decentralized organizations and architectures foster a hornet\u2019s nest of problems: they proliferate data silos and fragment data, making it impossible for executives to get quick answers to simple questions, like \u201cHow many customers do we have?\u201d To me, the data mesh reinforces bad organizational habits and renegade behavior. If you want to modernize your data environment, do you double-down on data dysfunction and reinforce data silos? This is akin to paving the cow paths. But that is what the data mesh approach advocates.<\/p>\n

To me, the data mesh reinforces bad organizational habits and renegade behavior. If you want to modernize your data environment, do you double-down on data dysfunction?<\/em><\/p>\n

Importance of Federation.<\/strong>\u00a0The only route to enduring value is to balance the imperatives of centralization and decentralization. Central teams deliver standards, scale, and technical expertise, while distributed teams deliver agility, adaptability, and domain knowledge. One without the other creates huge problems. The best way to design data organizations, architectures, and governance approaches is to federate them: align central and local resources and activity so you get the best of both worlds with few drawbacks.<\/p>\n

The best way to design data organizations, architectures, and governance approaches is to federate them.<\/em><\/p>\n

Bottom-up federation.<\/strong>\u00a0Fortunately, Zhamak recognizes this, at least partially, and builds federation into the data mesh concept. For example, each domain needs a product owner who must understand the needs of enterprise users; the self-service platform is built centrally (presumably) to help domain owners manage and publish their own data; a central team (presumably) needs to build cross-functional data products that no single domain can or will build; and perhaps most importantly, cross-functional teams of product owners and developers must convene to hammer out global models, governance standards, and application interfaces.<\/p>\n

Top-down federation.<\/strong>\u00a0This sounds a lot like what organizations with centralized data architectures do. The best ones align central and local resources by assigning technical specialists to data domains, establishing cross-functional governance boards, and scheduling regular strategic planning meetings with department leaders. In addition, central data teams build platforms and tools that empower business units to meet their own data and analytics needs and deliver cross-functional applications that no department has the time, money, or interest in building.<\/p>\n

In the end, it doesn\u2019t matter whether you federate from below or from the top as long as you do! While many centralized data teams have learned to federate best practices, standards, and development, the reality is that some departments (e.g., finance, sales, marketing) have their own data engineers and data analysts and don\u2019t want to give them up. But with the right incentives, these departments can be coaxed to outsource costly IT-related tasks, like maintaining brittle data systems managed by part-time data engineers that provide a colloquial view of data, to a central team without forfeiting local control.<\/p>\n

It doesn\u2019t matter whether you federate from below or from the top as long as you do!<\/em><\/p>\n

Data Mesh Drawbacks.<\/strong>\u00a0Smarter people than me have evaluated the data mesh in depth and see shortcomings. Many of the drawbacks focus on the lack of expertise and interest within departments to publish and manage data on behalf of themselves and others. Here is a sampling of 30+ areas of concern mentioned by James Serra and followers of his\u00a0blog<\/a>. Serra is a well-known data architect who was a data evangelist for Microsoft and now is a data platform architecture lead at EY.<\/p>\n

Some Challenges Facing Data Mesh Implementations<\/strong><\/p>\n