Components of a Data Engineering Strategy
As part of GovTech’s Data Engineering team, we had the opportunity to observe how various data projects across the government were conducted. The genesis of this data engineering strategy came about as we observed what worked best in a typical data engineering life cycle. Fundamentally, such a life cycle would involve these typical stages of (1) Requirement Gathering -> (2) Pipeline Design -> (3) Pipeline Implementation -> (4) Pipeline Orchestration -> (5) Data Quality Monitoring.
As the diagram above illustrates, success at any stage of the data engineering lifecycle depends on a multitude of factors, which may or may not be technical in nature. Based on our observations across multiple government projects, if insufficient attention is given to any of the following five components, a project is highly unlikely to succeed.
Sponsors and Use Cases
We have found that projects with strong use cases clearly identified how the desired data would be used and the resulting benefits or cost savings that would be gained when the data engineering pipelines are built. This made it easier to showcase the results back to management after project completion assuring them on their sponsorship. Additionally, this also prepares the users of the data to also have a clear understanding on how to use their data.
People and Resources
It is important for organisations to know the different roles in a project and the contributions made by each role. Projects that understand this concept can run more effectively as people with the relevant skillsets are identified and have been assigned suitable roles. On the flipside, projects should avoid giving individuals too many roles as overall effectiveness is reduced.
Data and Processes
Lack of formal processes being defined can greatly impact the maintainability of a system. For instance, if code changes are not tracked or if change implementation is not reviewed, it becomes much more complex to troubleshoot live cases. Defined processes also have the additional benefit of allowing users or stakeholders to have a common understanding, as well as identify functional gaps that are not handled, which can then be added to the developer’s backlog.
Architecture and Infrstructure
Badly designed architectures could limit scalability (or even be time-bombs) if their flaws or vulnerabilities are not quickly handled in later releases. As such, most government projects allocate a considerable amount of time to ensure systems are designed to be scalable and future-proof in terms of functionality. Thus, system architectures are thoroughly reviewed to ensure that both system and data are secured, without compromising performance when put through heavy loads.
Tools and Technology
There are a variety of tools available in the data world. Therefore, we advise data project teams to evaluate user requirements and map them to the functions of tools or technologies. By taking the requirements of users and sponsors into account, everyone in the organisation will have a better appreciation of why certain tools and technologies are selected.
Summary
Each component has varying degrees of influence during the different phases of the data engineering life cycle. As illustrated in the diagram below, “Sponsors & Use Cases”, “People & Resources”, and “Process & Data” have a greater influence on the Requirements Gathering phase (colour coded as purple), while “Infrastructure & Architecture” and “Tools & Technology” typically influence the later parts of the data engineering life cycle (colour coded as blue).
