A Singapore Government Agency Website

Choosing a data ingestion method

To pipe data into Elasticsearch, you can use a number of different methods - each with its own benefits and constraints. In this page, we will discuss each method so that you can choose the right ingestion method for your needs.

Use the StackOps Observability Accelerator (SOA) for simpler onboarding to StackOps. Refer to the SOA guide for more details.

- Our links to Elastic documentation may not align with your specific Elastic deployment version. Please select your version from the dropdown menu on the Elastic documentation page for accurate information.
- For other external documentation shared with you, always refer to the latest version provided to ensure up-to-date information.Links to Elastic documentation may not match your exact Elastic version.

Recommendation criteria

StackOps recommends data ingestion methods based on the following criteria:

  • Adherence to Elastic Common Schema (ECS): Used by SIEM and other security tools to ease data ingestion and correlation from various sources for analysis. ECS is converging with open monitoring standards like OpenTelemetry.
  • Cloud-agnosticity: Supports different Cloud service providers and on-premises environments.
  • Low operation overhead costs: Frees up IT resources for other tasks.
  • Higher out-of-the-box coverage with minimal work: Requires minimal configuration, saving time and effort.

For more information, refer to the following links:

Things to note

  • Meets the minimum requirements for Elastic Agent
  • You should have a configured NAT Gateway or Forward Proxy for controlling public internet access.
  • Elastic Agent should be configured in an environment with controlled access to the internet.
  • Resource utilisation is dependent on:
    • Amount of data
    • Integrations enabled
  • Configure a Elastic Agent Processor (Recommended if you intend to sanitise data before it reaches Elastic Cloud)

In this section, we have ranked the available data ingestion methods according to the recommendation criteria, from most to least recommended.

Click to expand and learn more about each method.

We’ve summarised the recommended methods in this table for comparison. For more details, refer to the respective accordion tabs below.

Category Fleet-managed Elastic Agent on server Fleet-managed Elastic Agent on server for external integrations Elastic Agent on Kubernetes (K8) as DaemonSet
Data type(s) collected Logs and metrics Logs, metrics, and events Logs and metrics
Supported outputs - Elasticsearch
- Logstash
- Elasticsearch
- Logstash
- Elasticsearch
- Logstash
Limitation(s) One installed Elastic Agent per host. One installed Elastic Agent per host. No access to several data sources.
Elastic Common Schema (ECS) support warning Only for K8 component metrics and logs
Elastic Stack component supported versions Same version or higher Same version or higher Same version or higher
Additional capabilities

This method is recommended for Fleet-managed individual EC2 instances on servers and VMs that might require additional capabilities like XDR/EDR and OSQuery.

Category Details
Architecture diagram Installed Elastic Agent to Elasticsearch
Data type(s) collected Logs and metrics
Supported outputs - Elasticsearch
- Logstash
Limitation(s) Only one installed Elastic Agent per host, but Elastic Agent policy allows multiple integrations. Needs root user (or Administrator on Windows) to run installation/enrollment commands.
Elastic Common Schema (ECS) support For supported integrations, you need out-of-the-box data parsing to align with ECS. For general custom integrations e.g. Custom Logs, you need additional parsing to align with ECS.
Elastic Stack component supported versions Generally compatible with stack components that are the same version or higher. Refer to Elastic’s support matrix.
Additional capabilities ✅ XDR/EDR (with Elastic Defend/Endpoint Integration)
✅OSQuery (with Osquery Integration)

These capabilities will be disabled due to security requirements.
Other considerations Costs:
- Egress from CSP to ESS.
- CSP API invocation cost if integration retrieves data via CSP API. Refer to the relevant Integration documentation for details.
- Cost of compute is already incurred for the hosts required for tenant’s applications/ use cases.

Operations: ✅ Able to upgrade Elastic Agents through Fleet UI.

Implementation: For manageability of Elastic Agent Policies, it is recommended to streamline policies with specific integrations based on category of hosts. For example, MySQL Servers will have Elastic Agent Policies with System and MySQL integrations.
Relevant guides Install Fleet-managed Elastic Agent

This method is recommended for Function-as-a-Service (FaaS), Software-as-a-Service (SaaS), and managed services that have Fleet-managed Elastic Agents deployed on servers or containers to receive data from external integrations.

Category Details
Architecture diagram Elastic Agent to Elasticsearch: APIs for collection
Data type(s) collected Logs, metrics, and events
Supported outputs - Elasticsearch
- Logstash
Limitation(s) Only one installed Elastic Agent per host, but Elastic Agent policy allows multiple integrations. Needs root user (or Administrator on Windows) to run installation/enrollment commands.
Elastic Common Schema (ECS) support For supported integrations, you need out-of-the-box data parsing to align with ECS. For general custom integrations e.g. Custom TCP Logs, Custom UDP Logs, Custom HTTP Endpoint Logs, you need additional parsing to align with ECS.
Elastic Stack component supported versions Generally compatible with stack components that are the same version or higher. Refer to Elastic’s support matrix.
Additional capabilities None
Other considerations Costs:
- Egress from CSP to ESS
- SaaS/CSP API invocation costs if integration retrieves data via API, refer to relevant Integration documentation for details.
- Additional Compute to host Elastic Agent.

Operations: ✅ Able to upgrade Elastic Agents through Fleet UI.

Implementation: High Availability is dependent on the data source and integration(s) used.
Relevant guides Install Fleet-managed Elastic Agent

This method is recommended for self-managed Kubernetes/Openshift and CSP-managed Kubernetes like GKE, EKS, and AKS.

Category Details
Data type(s) collected Logs and metrics. Logs from applications deployed on K8 will also be collected since the Elastic Agent picks up container logs as well.
Supported outputs - Elasticsearch
- Logstash
Limitation(s) On managed Kubernetes solutions like EKS, GKE and AKS, Elastic Agent has no access to several data sources, meaning logs and metrics from certain control plane components cannot be collected by Elastic Agent.
Refer to Elastic Agent documentation for the respective CSP managed K8 solutions for details.
Elastic Common Schema (ECS) support warning Only for K8 component metrics and logs. COTs and custom application logs will only be in generic ECS format and will not be fully parsed down to log message contents.
Elastic Stack component supported versions Generally compatible with stack components that are the same version or higher. Refer to Elastic’s support matrix.
Additional capabilities None
Other considerations Costs: Egress from CSP to ESS

Implementation: warning Managed control-plane data is not accessible if SaaS, such as EKS, AKS and GKE.
Relevant guides - Getting started with Kubernetes
- Install Elastic Agent in a containerized environment

Rank 2 - Restricted for specific hosting environment

This method is recommended if you are using Elastic Serverless Forwarder, which is an Amazon Web Services (AWS) Lambda function that ships logs from your AWS environment to Elastic.

Category Details
Architecture diagram Elastic Serverless Forwarder for AWS
Data type(s) collected Logs, metrics, and events
Supported outputs - Elasticsearch
- Logstash (preview)
Limitation(s) warning AWS only.
Elastic Common Schema (ECS) support For supported integrations, you need out-of-the-box data parsing to align with ECS. For general custom integrations e.g. AWS CloudWatch, Custom AWS Logs (beta), you need additional parsing to align with ECS.
Elastic Stack component supported versions Elastic Serverless Forwarder works with Elastic Stack 7.16 and later.
Additional capabilities None
Other considerations Costs:
- Egress from CSP to ESS.
- Firehose usage cost and cost for routing data from other AWS services to Firehose.
- SaaS / CSP API invocation cost if Integration retrieves data via API e.g. AWS CW.

Operations: warning Refer to Deploy Elastic Serverless Forwarder for prerequisites, deployment options, configuration options and updating of publishing script of Elastic Serverless Forwarder.

Implementation: Only supports the following inputs:
- Amazon S3 (via SQS event notifications)
- Amazon Kinesis Data Streams
- Amazon CloudWatch Logs subscription filters
- Amazon SQS message payload
Relevant guides Elastic Serverless Forwarder for AWS

This method is recommended if you are instrumenting applications with Elastic Agents to collect application performance monitoring data.

Category Details
Data type(s) collected Logs, metrics, and events
Supported outputs Only sends to APM server but can be configured to send output to Elasticsearch or Logstash.
Limitation(s) warning May require code changes at the application level for language/technologies/frameworks without auto-instrumentation support and complex instrumentation scenarios.
warning Only error logs, e.g. exceptions and stacktrace captured by the APM agent during runtime application logs on other log levels e.g. info needs to be picked up by Elastic Agent or beats from the application log directory
warning Only APM Server standalone (not managed by Elastic Agent) supports output to Logstash.
Elastic Common Schema (ECS) support Yes.
Elastic Stack component supported versions Refer to APM agent compatibility.
Additional capabilities ✅ 1. Tracing
✅ 2. APM
Other considerations Costs: Egress from CSP to ESS

Operations: Auto-instrumentation available for supported technologies/framework.

Implementation: Elastic APM agents automatically pick up basic host-level metrics and agent-specific metrics, like JVM metrics in the Java Agent, and Go runtime metrics in the Go Agent.

Note: If APM is implemented within a software running as a container (host in this case is the container), it will not pick up EC2/VM metrics.
Relevant guides Guides for APM agents

Using Firehose to send your data to Elastic means you have no agents to deploy, no Lambdas to configure, and no Beats to manage.

Category Details
Data type(s) collected Logs, metrics, and events
Supported outputs Elasticsearch on ESS only
Limitation(s) warning AWS only
❌ Only able to integrate with public internet (StackOps implements Traffic Filter by default)
warning Compatible only with Elastic Stack version 7.17 or later versions, and it can only run on Elastic Cloud.
Elastic Common Schema (ECS) support For supported integrations, you need out-of-the-box data parsing to align with ECS. For general custom integrations e.g. AWS CloudWatch, Custom AWS Logs (beta), you need additional parsing to align with ECS.
Elastic Stack component supported versions Generally compatible with stack components that are the same version or higher.
Additional capabilities Configure multiple log sources to send data to Firehouse delivery streams
Stream data from Firehose to Elasticsearch
✅ Has built-in auto-scaling capability and is designed to handle high-volume use cases.
Other considerations Costs:
- Egress from CSP to ESS
- Firehose usage cost and cost for routing data from other AWS services to Firehose
- SaaS/CSP API invocation cost if integration retrieves data via API, e.g. AWS CW.
Relevant guides Quickstart: Collect data with AWS Firehose

For Elastic Agents deployed on servers or containers that are not managed by Fleet.

Category Details
Data type(s) collected Logs and metrics
Supported outputs - Elasticsearch
- Logstash
Limitation(s) Only one installed Elastic Agent per host, but Elastic Agent policy allows multiple integrations. Needs root user (or Administrator on Windows) to run installation/enrollment commands.
Elastic Common Schema (ECS) support For supported integrations, you need out-of-the-box data parsing to align with ECS. For general custom integrations e.g. Custom TCP Logs, Custom UDP Logs, Custom HTTP Endpoint Logs, you need additional parsing to align with ECS.
Elastic Stack component supported versions Generally compatible with stack components that are the same version or higher. Refer to Elastic’s support matrix.
Additional capabilities Configure multiple log sources to send data to Firehouse delivery streams.
Stream data from Firehose to Elasticsearch
✅ Auto-scaling capability is built in, and the service is designed to handle high-volume use cases.
Other considerations Costs:
- Egress from CSP to ESS
- SaaS/CSP API invocation costs if integration retrieves data via API, refer to relevant Integration documentation for details
- Additional Compute to host Elastic Agent.

Operations: ❌ Updating integrations, management of Agent Policies and upgrading Elastic Agent needs to be done manually via SSH access to server.
Relevant guides Install standalone Elastic Agents

Deploys Filebeat to collect logs and Metricbeat to collect metrics from K8s, usually for Elastic Agents on Kubernetes as DaemonSet with certain environment limitations e.g. Elastic Agents on Kubernetes has Openshift with CoreOS and Metricbeat on Cloud Foundry.

Category Details
Data type(s) collected Logs and metrics
Supported outputs A variety of outputs: Elasticsearch, Logstash, Kafka, Redis, File, Console. Refer to Filebeat output and Metricbeat output documentation.
Limitation(s) warning Changes to data collection and output configuration for beats requires redeployment of manifest.
Elastic Common Schema (ECS) support warning Hints based on autodiscover can be configured to faciliate the collection and parsing of data to align with ECS if a supported module for the COTs application is available.
Refer to Filebeat modules and Metricbeat modules for list of modules and hints for autodiscover.
Elastic Stack component supported versions Elastic recommends running beats on the same version as Elastic Stack components e.g. Elasticsearch and Logstash.

Prior to version 7.4, Functionbeat is only compatible with Elasticsearch as an output destination. Logstash and other outputs are not supported. Starting with version 7.4, Functionbeat supports Elasticsearch and Logstash outputs. Refer to the respective support matrix for Beats, Logstash, and Elasticsearch.
Additional capabilities None
Other considerations Costs: Egress from CSP to ESS

Implementation: warning Managed control-plane data is not accessible if it’s SaaS, such as EKS, AKS and GKE.
Relevant guides Run Metricbeat on Kubernetes

Deploys Logstash on server/container to collect from a source that Elastic Agent or Beats cannot read, e.g. databases or as an ingestion engine for the specified additional capabilities.

Category Details
Architecture diagram Logstash to Elasticsearch
Data type(s) collected Logs, metrics, and events
Supported outputs A variety of outputs. Refer to Logstash Output Plugins.
Limitation(s) warning Generally requires effort for parsing data at source or at the Logstash pipelines to align with the desired format i.e. ECS.
Elastic Common Schema (ECS) support ❌ Generally not supported out-of-the-box. Incoming data source must already be in ECS format or processing must be done at the filter phase of Logstash pipelines to align with ECS format.
Elastic Stack component supported versions Refer to Elastic’s support matrix.
Additional capabilities ✅ Data enrichment between Elastic Agent and Elasticsearch
✅ Persistent queue (PQ) buffering to accommodate network issues and downstream unavailability
✅ Proxying in cases where Elastic Agents have network restrictions i.e. connecting outside of the Elastic Agent network
✅ Ability to route data to multiple Elasticsearch clusters and other destinations depending on the content
✅ Centralised pipeline management to manage Logstash pipelines via Kibana UI
Other considerations Costs:
- Egress from CSP to ESS
- PaaS/SaaS/Faas/CSP API invocation costs if utilising Logstash Input/Filter/Output plugins.
- Additional Compute to host Logstash

Implementation: Deployment and scaling considerations
Relevant guides Installing Logstash

Deploys Beats as agents on servers/containers to collect a variety of data.

Category Details
Data type(s) collected Logs, metrics, and events
Supported outputs A variety of outputs: Elasticsearch, Logstash, Kafka, Redis, File, Console. Refer to the respective Beats output documentation.
Limitation(s) ❌ Beats need to be configured at server level.
Elastic Common Schema (ECS) support For supported modules, there is out-of-the-box data parsing to align with ECS. Refer to the respective Beats’ Module documentation.
Elastic Stack component supported versions warning Elastic recommends running beats on the same version as Elastic Stack components e.g. Elasticsearch and Logstash.

Prior to version 7.4, Functionbeat is only compatible with Elasticsearch as an output destination. Logstash and other outputs are not supported. Starting with version 7.4, Functionbeat supports Elasticsearch and Logstash outputs.

Refer to the respective support matrix for Beats, Logstash, and Elasticsearch.
Additional capabilities ✅ Data enrichment between Elastic Agent and Elasticsearch
✅ Persistent queue (PQ) buffering to accommodate network issues and downstream unavailability
✅ Proxying in cases where Elastic Agents have network restrictions i.e. connecting outside of the Elastic Agent network
✅ Ability to route data to multiple Elasticsearch clusters and other destinations depending on the content
✅ Centralised pipeline management to manage Logstash pipelines via Kibana UI
Other considerations Costs:
- Egress from CSP to ESS
- PaaS/SaaS/Faas/CSP API invocation costs if Module retrieves data via API.
- Additional Compute to host Beats
Relevant guides Refer to the respective Beats documentation for installation and configuration.

For custom applications using Elasticsearch language clients to ingest directly to Elasticsearch.

Category Details
Data type(s) collected Logs, metrics, and events
Supported outputs Elasticsearch
Limitation(s) warning Requires implementation at software level
Elastic Common Schema (ECS) support ❌ ECS support not available. Has to be done programmatically when indexing data to Elasticsearch.
Elastic Stack component supported versions Language clients are forward compatible i.e. clients support communicating with the same or newer minor versions of Elasticsearch.

Elasticsearch language clients are only backwards compatible with default distributions and without guarantees made. Refer to the respective language client’s documentation for details.
Additional capabilities -
Other considerations Costs:
- Egress from CSP to ESS
- Any other costs that might be incurred by custom application code to retrieve data, e.g. CloudWatch API.
Relevant guides Ingesting data from applications
Chat with me!
Bot Launcher