Why is a data engineering strategy important?

Becoming data-driven brings forth a new era of fact-based decisions.

Data has long existed in many forms, and will continue to exist ad infinitum. However, organisations have only recently started to look at data in a different light to understand its potential for making informed decisions.

In 2021, a blog post by AWS Vice President and Chief Evangelist Jeff Barr mentions that “Amazon S3 stores over 100 trillion objects, and regularly peaks at tens of millions of requests per second”. This is exponentially larger than the 2.9 billion objects stored since its launch in 2006. Given the sudden growth in the amount and variety of data collected, it is no wonder that some industry experts have started use the term “data tsunami” to explain this phenomenon.

In addition, there have been a myriad of data analytics tools launched, ranging from AWS Athena to AWS QuickSight. This proliferation of analytical tools signals a clear trend towards a increasingly data-driven world. As organisations dive deeper into this world, they sometimes find themselves surrounded by an ocean of data, trying to make sense of what they are collecting and to what end. Over time, you may encounter several familiar scenarios or questions like:

• We have lots of data to manage, and more will be coming
• We have a bulging web of data providers and consumers
• No one knows how useful the data we currently hold are
• Are we effectively making use of the data that we have?

In today’s reality, data often exists in varying forms across different silos, making it a challenge to access them efficiently. This greatly increases the time taken to consolidate different datasets, potentially reducing the effectiveness of the insights gained from these datasets. In addition, data is rapidly growing in terms of volume and complexity – coupled with the increasing need for data pipelines. If pipelines are built without data testing, data validation and data quality monitoring, there is a risk of unmaintainable data pipelines that produce dirty and unreliable data.

In the Data Engineering team, we are convinced that to navigate this data tsunami, project teams should adopt an effective strategy towards approaching data engineering. This is where data engineering initiatives come into play. We define a data engineering initiative as one that designs and implements systems that collect and ingest, validate, transform, store, and analyse data at scale consisting of activities like data modelling, data lake and data design, data pipelines building, data integration, data testing and orchestration. Data engineering is a critical component in driving data and analytics success, comprising most of the data work done. As such, an effective strategy oversees the implementation of an endless cycle of data engineering initiatives which are aimed at connecting new sets of data and improving overall data quality. Having a good strategy will enable projects and agencies to keep up with the tidal wave of data, allowing the power of said data to be unleashed.