Data transformation
Data transformation is the process of converting raw data into clean, standardised, and quality data for analysis, dashboarding, and reporting. This is the next step after data ingestion.
The typical process for data transformation in StackOps Elastic involves the following steps:
- Data profiling and discovery — Identify the source data and understand its structure and fields. This helps to determine what needs to be sanitised, identified or enriched.
- Data sanitisation — Sanitise the data, removing unneccessary or sensitive information.
- Identity resolution — Resolve discrepancies in data to ensure that different records referring to the same entity (e.g., a cluster, or a user) are treated as one. This improves data quality and eliminates fragmentation.
- Data enrichment — Enrich the data with relevant information to improve data quality and usability. These new fields can be from internal or external sources.
Common Elastic concepts
Here are some common Elastic features for processing data:
- Elastic Agent processors—You can use Elastic Agent processors to sanitise or enrich raw data at the source. Use Elastic Agent processors if you need to control what data is sent across the wire, or if you need to enrich the raw data with information available on the host. Learn more in Elastic documentation
- Elasticsearch ingest pipelines—You can use Elasticsearch ingest pipelines to enrich incoming data or normalise field data before the data is indexed. Elasticsearch ingest pipelines enable you to manipulate the data as it comes in. This approach helps you avoid adding processing overhead to the hosts from which you’re collecting data. Learn more in Ingest pipelines.
What you can do
Learn how to transform your data with StackOps Elastic in the following topics:
