In Data Mesh, the ITdepartment acts as a platform provider for analyticstools and standardsthat departments can use to quickly and flexibly individual dataviews. Find out when the effort involved in a data mesh is worthwhile and what role data fabric plays here in our blog article..
Data Mesh – Why, how, and what for?
If you observe the world of analytics over a period of time, you will recognize a pattern. After a period of increased and growing freedom for end users, chaos quickly ensues in the form of uncoordinated analytics islands, where departments can flexibly and very quickly implement "their" view of the data. Unfortunately, the data worlds, technology stacks, and processes do not fit together, making comprehensive evaluations virtually impossible and compromising data quality. After a while, the total costs and expenses also become significant, while reliability and usefulness decline.
At some point, this approach will be recaptured and centralized and standardized. Unfortunately, this often goes too far. An IT structure with lengthy processes and a high degree of division of labor between a wide range of specialists ensures optimal quality, but quickly leads to bottlenecks and overload in IT. The needs of the departments are no longer met in a timely manner, and they start building their own solutions again. The cycle starts all over again...
The analytics market and companies went through several hype cycles with changing governance models and corresponding technical tools. Buzzwords that emerged included spreadsheet hell, EIS/MIS, DWH, hub-and-spoke, self-service BI, data platform, data lake, BICC, and many more. Fortunately, however, the market learned from experience and tended toward compromise solutions that were somewhere in the middle between the two extremes of "central DWH" and self-service BI. Technical innovations such as data virtualization and data catalogs now support tailor-made solutions, as do new process and organizational models based on agile software development (e.g., Scrum) and organization (e.g., the Spotify model).
A data mesh now represents the latest approach to finding a compromise in the eternal conflict between time, quality, and cost.
Data Mesh – what is it?
A data mesh is actually a governance construct based on modern principles for agile organization and software development. The central concept is the domain (from the domain-driven design approach), which can be roughly interpreted as a technical subject area. The content and technical responsibility for such a domain lies with the department that works most in the domain concerned. The data in this domain is offered as a product by the domain owner in the company-wide data exchange market. This role as product owner requires skills that are not necessarily available in specialist departments. For this reason, cross-functional teams are often formed to take responsibility for technical definition and documentation, implementation of data preparation and storage, and visualization and distribution of the data.
In this approach, IT assumes the role of platform provider. It provides the tools and standards necessary to enable departments without in-depth technical knowledge to offer data products. However, the process of defining and implementing the data products takes place without IT's involvement, meaning that IT no longer constitutes a bottleneck and the overall model is highly scalable.
Which is better? Data mesh or data fabric?
Another current approach in BI architecture is the data fabric. However, this is not a competing approach to the data mesh.
Data fabric is an approach to building a data platform and operating it transparently and securely. Data fabric is therefore a suitable approach that IT can use to provide the necessary data platform on which the data mesh approach is based. The right question is therefore not "data fabric or data mesh," but rather whether one of the approaches or both in combination seems suitable for a given organization. A data mesh places high demands on the business departments and is only worthwhile if the business departments themselves have high requirements for flexibility and speed that a central IT solution cannot adequately meet.
Want more detailed information about Data Mesh with SAP?
We would be happy to examine the suitability of the above approaches for your company as part of an analytics strategy consultation. We are currently developing a maturity model with a corresponding audit procedure to classify the necessary digital skills of the specialist departments.
Feel free to contact me!
Silvio Bergmann
Senior Manager
Analytics & Insights
silvio.bergmann@isr.de
+49(0)151 422 05 418
