People and Resources

Once the sponsors and use cases have been scoped, the initiative will need to identify the suitable persons that could fill in the various roles needed. It is necessary to staff a project with the appropriate persons and roles to ensure its success. For a data engineering initiative, there could be a myriad of teams and roles that would aid in various aspects of the data engineering lifecycle. These teams and roles could include:

• Data Owners
• Data Stewards
• Data Custodians
• Data Architects
• Data Governance Teams
• Data Scientists and Analysts
• Data Engineers

The availability of the role listed may vary between each organisation, depending on its maturity and organisational structure. Each team/role bears a unique responsibility that considers data ownership, data storage, data usability, and data accountability. The diagram below describes several data management roles in greater detail. These roles would provide greater clarity on one’s responsibility, and that of others within the initiative.

For both new and existing data engineering initiatives, it is highly recommended to identify a sufficiently sized team, staffed with the necessary roles and persons, to ensure an efficient data engineering strategy. For instance, an initiative consisting of only data architects and engineers may design an efficient data pipeline. However, data usage and standards within said initiative are not properly defined due to the lack of data owners.

The core skills and capabilities required may differ between each initiative. As such, data roles are defined according to the requirements gathered for the initiative, and individual contributors will need to be sourced accordingly.

This highlights several different roles (and accompanying responsibility) in a data-related project. It should be noted that the responsibilities and skills may overlap between each role. Depending on the data system, you might require different types of data roles. For example, a data engineer with some knowledge of data analytics can be described as a Data Analytics Engineer. This list is not exhaustive, but some other examples of the types of data engineers are as shown below:

When forming the team, it is paramount to determine the data roles required to satisfy the project requirements. While not impossible, it is rare to identify a “Data Unicorn” – a person who has the skills to fulfil all roles in the project. As such, an assessment is recommended to identify the most suitable person to aid in the project.

The subsequent sections provide artefacts and elaborations to aid readers in determining the appropriate people and resources for a data engineering initiative.