Recently we co-authored the book Data Mesh For Dummies, along with our colleague Adrian Estala. We greatly enjoyed writing the book and collaborating to succinctly deliver the ideas around Data Mesh in a straightforward format. However, like many authors we were asked to trim our book’s word count to satisfy a strict page limit.
Fortunately, there’s a very popular medium known as blogs and we decided to salvage the text on Data Mesh roles and responsibilities and repurpose it here. Below, we are thrilled to present one of our lost chapters on Data Mesh roles and responsibilities.
Roles and Responsibilities in a Data Mesh
Embarking on a Data Mesh journey will result in organizational changes and adjustments to employees’ roles. Existing workers will be critical to the success of adopting a Data Mesh, as they have invaluable tacit knowledge to contribute to the Data Mesh journey. Therefore, the transition of data ownership from a central data team to decentralized domains should be approached as a reallocation of existing data-focused employees.
With that in mind, here are some of the roles and related responsibilities that will likely be involved:
- Executive Sponsor: Responsible for overall corporate data strategy and ensuring alignment with the business strategy. This is likely to be a member of the C-suite due to the significance of the organizational change required.
- Domain Owner: Represents the needs of the domain when defining federated computational governance standards.
- Data Product Manager: Responsible for the scope,delivery, and lifecycle of one or more data products within a domain.
- Domain Data Architect: Designs how data flows from source to products, defines data product data models, and ensures adherence to architecture standards within a domain.
- Data Product Developer (Data Engineer): Creates and maintains data products.
- Infrastructure Engineer: Responsible for the operation and maintenance of the self-serve data platform.
- Domain SME / Analyst: Experts in the business area that relates to the domain, and understands the consumers’ needs, to ensure that data products meet their business objectives. Typically the first consumer of a data product.
- Legal / InfoSec / Audit: A set of expert resources available to the domains to provide guidance on areas such as corporate risk and compliance standards.
It’s important to note that these are roles, and not necessarily individual people. In some organizations, especially in the early adoption phase of Data Mesh, some employees will take on multiple roles. In addition, the choice of technologies within a self-serve data platform can dramatically affect the workload and efficiency of the employees in these roles.
Designing the decentralized organization for success
The design for your Data Mesh organization should focus on efficiency through empowerment. Data engineering skills will be embedded within business-focused domain teams. This organizational change will take time and will evolve as you move along your Data Mesh journey.
Here are some key design principles to consider for your organization:
Distribute data engineering responsibilities to the domains
Data engineers are the experts in shaping data, and over time they will become subject matter experts in the data related to the domain in which they operate. Giving them autonomy and control will accelerate results.
Design the domain to manage its data with autonomy
A federated computational governance model enables the domains to make salient and relevant decisions within their domain, while ensuring that they follow the cross-domain data governance rules.
Use external advisory teams
Domains should be autonomous, but not all domains will require full-time support for every specialist role. For example, each domain may need access to a legal advisor to ensure that they stay within GDPR rules when dealing with personal data.
As organizations move along their Data Mesh journey it’s likely that new domains will be developed and the number of data products will increase. Data product developers and data product architects will manage the data lifecycle within the domain, and will play a key role in creating new data products.
Data Mesh For Dummies
Want to discuss or learn more? Contact us
