Data Fabric vs Data Mesh: Key Differences and Similarities

Data structure and data mesh both strive to organize data that is spread across databases or data lakes. The data structure is very technology driven and the data mesh focuses on organizational changes.

Every data-driven enterprise is striving to adopt or is already adopting a self-service business intelligence model. Many of these companies are still unable to make their data fully accessible on their platform and scale it to all of their users across different verticals. For these companies, data that sits in silos in a data warehouse or data lake with zero or limited facilitation capabilities depending on the needs of the teams. This is where data technologies such as data mesh and data structure come into play.

Viewed superficially, the two may seem fundamentally similar. After all, knits are born from fabrics. Given their impact on any IT system, it can be helpful to know the difference between these two offerings in order to identify the right product for your organization. In many cases, finding the best of both worlds, an entity-centric data structure can incorporate the data product concepts of the data mesh, and decentralizing data engineering might just be what the organization needs.

See also: Business Leaders Must Make Data Fabrics a Priority in 2022

Data structure

Noel Yuhanna, analyst at Forrester, was one of the first people to define the data fabric. Data management tools have come a long way from databases to data warehouses to data lakes depending on the complexity of enterprise solutions. The data structure can be considered the next logical step in the data management process.

Data fabric is a metadata-driven process that aims to connect a wide range of data sources and tools in a unified, self-service way. As the size of data stored in organizations continues to grow, the number of silos that also hold this data increases. The type of data also varies greatly as it can be transactional or operational data.

With the Data Fabric deployed on these repositories, data lakes, or warehouses, it brings clarity in terms of centralizing data across the organization. It facilitates data provisioning for downstream consumers, be they data engineers, quality assurance management engineers, or analysts. It should however be noted that if the management of this data is centralized, the places of access remain the same.

Data Fabric is Gartner’s #1 technology trend for 2022, due to its ability to reclaim nearly 70% of developers’ work in the data lifecycle. Unlike manual integration processes, the data fabric offers a significant advantage with the benefits derived from processing this data.

See also: Data Fabric Success: Is Query Response the Missing Point?

Data mesh

Zhamak Dehghani, a consultant at ThoughtWorks, first defined the concept of data mesh. Basically, it tries to solve the same problem that data structure aims at – managing data that is siled in the organization. But it’s different in how in a data mesh, distributed teams can have the control and access to manage their data in their silos at their discretion.

The reason for the push towards data mesh is to solve synchronization issues between data lakes and data warehouses. The logical architecture proposed by Dehghani focuses on filtered data based on data that is commonly shared between users and data sources instead of hard-coding it for transformation. In a data mesh, the data is kept in roughly the same format as the source, and that data is then taken by domain-specific teams to turn it into a data product as they see fit.

The primary benefit of data mesh is that the infrastructure as a self-service platform provides teams requesting data along with monitoring, logging, alerting, and normalization, all with a standard process that is the same in all areas. and which is also domain independent.

Data Meshes Vs. Data Fabrics

To summarize, data fabrics and meshes are data management architectures. The difference between them is that data structure is a framework which is technology independent and can provide data products as one of its many outputs whereas data mesh is an architecture which only produces business domain specific data products.

Data structure and data mesh both strive to organize data distributed in databases or data lakes, data structure is very technology-centric and data mesh focuses on organizational changes. Mesh depends on people and teams for change in organizational changes, and fabric is an architectural approach to managing complex data and metadata.

In terms of design, data structure uses metadata and centralized data engineering based on the overall experience of data consumers in the organization, while data mesh uses the expertise that teams have in various areas to create and design its deliverable: a business-oriented data product.

In the words of Yuhanna, “A data mesh is essentially an API driven [solution] for developers, unlike [data] Fabric,” [data fabric] is the opposite of data mesh, where you write code for interface APIs. On the other hand, the data structure is low-code, no-code, because the API integration takes place inside the structure. »

Similarities Between Data Meshes and Data Fabrics

It is important to know how these two offers are similar. Both stem from nearly five decades of data management expertise. Both can leverage each other and use each other’s data practices. In many cases, the costs of implementing and maintaining the two frameworks are also similar. The similarities in architecture principles relate to the business domain foundation, data product output, continuous data discovery, and data behavior graph.

Conclusion

Although the use cases and architecture of the data structure and data mesh may vary, they are, ultimately, always architecture frameworks and not architectures per se. The architecture intervenes when the needs are correctly defined, the data understood and the processes of the organization taken into account. It is even practical to include the best data structure and the best data mesh in the final architecture. It would be prudent to find which of these two architecture frameworks works best for your system.

Comments are closed.