Choosing a data ingestion method
To pipe data into Elasticsearch, you can use a number of different methods - each with its own benefits and constraints. In this page, we will discuss each method so that you can choose the right ingestion method for your needs.
Use the StackOps Observability Accelerator (SOA) for simpler onboarding to StackOps. Refer to the SOA guide for more details.
- Our links to Elastic documentation may not align with your specific Elastic deployment version. Please select your version from the dropdown menu on the Elastic documentation page for accurate information.
- For other external documentation shared with you, always refer to the latest version provided to ensure up-to-date information.Links to Elastic documentation may not match your exact Elastic version.
Recommendation criteria
StackOps recommends data ingestion methods based on the following criteria:
- Adherence to Elastic Common Schema (ECS): Used by SIEM and other security tools to ease data ingestion and correlation from various sources for analysis. ECS is converging with open monitoring standards like OpenTelemetry.
- Cloud-agnosticity: Supports different Cloud service providers and on-premises environments.
- Low operation overhead costs: Frees up IT resources for other tasks.
- Higher out-of-the-box coverage with minimal work: Requires minimal configuration, saving time and effort.
For more information, refer to the following links:
Things to note
- Meets the minimum requirements for Elastic Agent
- You should have a configured NAT Gateway or Forward Proxy for controlling public internet access.
- Elastic Agent should be configured in an environment with controlled access to the internet.
- Resource utilisation is dependent on:
- Amount of data
- Integrations enabled
- Configure a Elastic Agent Processor (Recommended if you intend to sanitise data before it reaches Elastic Cloud)
Recommended data ingestion methods
In this section, we have ranked the available data ingestion methods according to the recommendation criteria, from most to least recommended.
Click to expand and learn more about each method.
Rank 1 - Recommended
We’ve summarised the recommended methods in this table for comparison. For more details, refer to the respective accordion tabs below.
Category | Fleet-managed Elastic Agent on server | Fleet-managed Elastic Agent on server for external integrations | Elastic Agent on Kubernetes (K8) as DaemonSet |
---|---|---|---|
Data type(s) collected | Logs and metrics | Logs, metrics, and events | Logs and metrics |
Supported outputs | - Elasticsearch - Logstash |
- Elasticsearch - Logstash |
- Elasticsearch - Logstash |
Limitation(s) | One installed Elastic Agent per host. | One installed Elastic Agent per host. | No access to several data sources. |
Elastic Common Schema (ECS) support | ✅ | ✅ | ![]() |
Elastic Stack component supported versions | ✅ Same version or higher | ✅ Same version or higher | ✅ Same version or higher |
Additional capabilities | ✅ | ❌ | ❌ |
This method is recommended for Fleet-managed individual EC2 instances on servers and VMs that might require additional capabilities like XDR/EDR and OSQuery.
Category | Details |
---|---|
Architecture diagram | Installed Elastic Agent to Elasticsearch |
Data type(s) collected | Logs and metrics |
Supported outputs | - Elasticsearch - Logstash |
Limitation(s) | Only one installed Elastic Agent per host, but Elastic Agent policy allows multiple integrations. Needs root user (or Administrator on Windows) to run installation/enrollment commands. |
Elastic Common Schema (ECS) support | For supported integrations, you need out-of-the-box data parsing to align with ECS. For general custom integrations e.g. Custom Logs, you need additional parsing to align with ECS. |
Elastic Stack component supported versions | Generally compatible with stack components that are the same version or higher. Refer to Elastic’s support matrix. |
Additional capabilities | ✅ XDR/EDR (with Elastic Defend/Endpoint Integration) ✅OSQuery (with Osquery Integration) These capabilities will be disabled due to security requirements. |
Other considerations | Costs: - Egress from CSP to ESS. - CSP API invocation cost if integration retrieves data via CSP API. Refer to the relevant Integration documentation for details. - Cost of compute is already incurred for the hosts required for tenant’s applications/ use cases. Operations: ✅ Able to upgrade Elastic Agents through Fleet UI. Implementation: For manageability of Elastic Agent Policies, it is recommended to streamline policies with specific integrations based on category of hosts. For example, MySQL Servers will have Elastic Agent Policies with System and MySQL integrations. |
Relevant guides | Install Fleet-managed Elastic Agent |
This method is recommended for Function-as-a-Service (FaaS), Software-as-a-Service (SaaS), and managed services that have Fleet-managed Elastic Agents deployed on servers or containers to receive data from external integrations.
Category | Details |
---|---|
Architecture diagram | Elastic Agent to Elasticsearch: APIs for collection |
Data type(s) collected | Logs, metrics, and events |
Supported outputs | - Elasticsearch - Logstash |
Limitation(s) | Only one installed Elastic Agent per host, but Elastic Agent policy allows multiple integrations. Needs root user (or Administrator on Windows) to run installation/enrollment commands. |
Elastic Common Schema (ECS) support | For supported integrations, you need out-of-the-box data parsing to align with ECS. For general custom integrations e.g. Custom TCP Logs, Custom UDP Logs, Custom HTTP Endpoint Logs, you need additional parsing to align with ECS. |
Elastic Stack component supported versions | Generally compatible with stack components that are the same version or higher. Refer to Elastic’s support matrix. |
Additional capabilities | None |
Other considerations | Costs: - Egress from CSP to ESS - SaaS/CSP API invocation costs if integration retrieves data via API, refer to relevant Integration documentation for details. - Additional Compute to host Elastic Agent. Operations: ✅ Able to upgrade Elastic Agents through Fleet UI. Implementation: High Availability is dependent on the data source and integration(s) used. |
Relevant guides | Install Fleet-managed Elastic Agent |
This method is recommended for self-managed Kubernetes/Openshift and CSP-managed Kubernetes like GKE, EKS, and AKS.
Category | Details |
---|---|
Data type(s) collected | Logs and metrics. Logs from applications deployed on K8 will also be collected since the Elastic Agent picks up container logs as well. |
Supported outputs | - Elasticsearch - Logstash |
Limitation(s) | On managed Kubernetes solutions like EKS, GKE and AKS, Elastic Agent has no access to several data sources, meaning logs and metrics from certain control plane components cannot be collected by Elastic Agent. Refer to Elastic Agent documentation for the respective CSP managed K8 solutions for details. |
Elastic Common Schema (ECS) support | ![]() |
Elastic Stack component supported versions | Generally compatible with stack components that are the same version or higher. Refer to Elastic’s support matrix. |
Additional capabilities | None |
Other considerations | Costs: Egress from CSP to ESS Implementation: ![]() |
Relevant guides | - Getting started with Kubernetes - Install Elastic Agent in a containerized environment |
Rank 2 - Restricted for specific hosting environment
This method is recommended if you are using Elastic Serverless Forwarder, which is an Amazon Web Services (AWS) Lambda function that ships logs from your AWS environment to Elastic.
Category | Details |
---|---|
Architecture diagram | Elastic Serverless Forwarder for AWS |
Data type(s) collected | Logs, metrics, and events |
Supported outputs | - Elasticsearch - Logstash (preview) |
Limitation(s) | ![]() |
Elastic Common Schema (ECS) support | For supported integrations, you need out-of-the-box data parsing to align with ECS. For general custom integrations e.g. AWS CloudWatch, Custom AWS Logs (beta), you need additional parsing to align with ECS. |
Elastic Stack component supported versions | Elastic Serverless Forwarder works with Elastic Stack 7.16 and later. |
Additional capabilities | None |
Other considerations | Costs: - Egress from CSP to ESS. - Firehose usage cost and cost for routing data from other AWS services to Firehose. - SaaS / CSP API invocation cost if Integration retrieves data via API e.g. AWS CW. Operations: ![]() Implementation: Only supports the following inputs: - Amazon S3 (via SQS event notifications) - Amazon Kinesis Data Streams - Amazon CloudWatch Logs subscription filters - Amazon SQS message payload |
Relevant guides | Elastic Serverless Forwarder for AWS |
Rank 0 - Recommended for Application Performance Monitoring
This method is recommended if you are instrumenting applications with Elastic Agents to collect application performance monitoring data.
Category | Details |
---|---|
Data type(s) collected | Logs, metrics, and events |
Supported outputs | Only sends to APM server but can be configured to send output to Elasticsearch or Logstash. |
Limitation(s) | ![]() ![]() ![]() |
Elastic Common Schema (ECS) support | Yes. |
Elastic Stack component supported versions | Refer to APM agent compatibility. |
Additional capabilities | ✅ 1. Tracing ✅ 2. APM |
Other considerations | Costs: Egress from CSP to ESS Operations: Auto-instrumentation available for supported technologies/framework. Implementation: Elastic APM agents automatically pick up basic host-level metrics and agent-specific metrics, like JVM metrics in the Java Agent, and Go runtime metrics in the Go Agent. Note: If APM is implemented within a software running as a container (host in this case is the container), it will not pick up EC2/VM metrics. |
Relevant guides | Guides for APM agents |
(Not recommended) Other data ingestion methods
Using Firehose to send your data to Elastic means you have no agents to deploy, no Lambdas to configure, and no Beats to manage.
Category | Details |
---|---|
Data type(s) collected | Logs, metrics, and events |
Supported outputs | Elasticsearch on ESS only |
Limitation(s) | ![]() ❌ Only able to integrate with public internet (StackOps implements Traffic Filter by default) ![]() |
Elastic Common Schema (ECS) support | For supported integrations, you need out-of-the-box data parsing to align with ECS. For general custom integrations e.g. AWS CloudWatch, Custom AWS Logs (beta), you need additional parsing to align with ECS. |
Elastic Stack component supported versions | Generally compatible with stack components that are the same version or higher. |
Additional capabilities | ✅ Configure multiple log sources to send data to Firehouse delivery streams ✅ Stream data from Firehose to Elasticsearch ✅ Has built-in auto-scaling capability and is designed to handle high-volume use cases. |
Other considerations | Costs: - Egress from CSP to ESS - Firehose usage cost and cost for routing data from other AWS services to Firehose - SaaS/CSP API invocation cost if integration retrieves data via API, e.g. AWS CW. |
Relevant guides | Quickstart: Collect data with AWS Firehose |
For Elastic Agents deployed on servers or containers that are not managed by Fleet.
Category | Details |
---|---|
Data type(s) collected | Logs and metrics |
Supported outputs | - Elasticsearch - Logstash |
Limitation(s) | Only one installed Elastic Agent per host, but Elastic Agent policy allows multiple integrations. Needs root user (or Administrator on Windows) to run installation/enrollment commands. |
Elastic Common Schema (ECS) support | For supported integrations, you need out-of-the-box data parsing to align with ECS. For general custom integrations e.g. Custom TCP Logs, Custom UDP Logs, Custom HTTP Endpoint Logs, you need additional parsing to align with ECS. |
Elastic Stack component supported versions | Generally compatible with stack components that are the same version or higher. Refer to Elastic’s support matrix. |
Additional capabilities | ✅ Configure multiple log sources to send data to Firehouse delivery streams. ✅ Stream data from Firehose to Elasticsearch ✅ Auto-scaling capability is built in, and the service is designed to handle high-volume use cases. |
Other considerations | Costs: - Egress from CSP to ESS - SaaS/CSP API invocation costs if integration retrieves data via API, refer to relevant Integration documentation for details - Additional Compute to host Elastic Agent. Operations: ❌ Updating integrations, management of Agent Policies and upgrading Elastic Agent needs to be done manually via SSH access to server. |
Relevant guides | Install standalone Elastic Agents |
Deploys Filebeat to collect logs and Metricbeat to collect metrics from K8s, usually for Elastic Agents on Kubernetes as DaemonSet with certain environment limitations e.g. Elastic Agents on Kubernetes has Openshift with CoreOS and Metricbeat on Cloud Foundry.
Category | Details |
---|---|
Data type(s) collected | Logs and metrics |
Supported outputs | A variety of outputs: Elasticsearch, Logstash, Kafka, Redis, File, Console. Refer to Filebeat output and Metricbeat output documentation. |
Limitation(s) | ![]() |
Elastic Common Schema (ECS) support | ![]() autodiscover can be configured to faciliate the collection and parsing of data to align with ECS if a supported module for the COTs application is available.Refer to Filebeat modules and Metricbeat modules for list of modules and hints for autodiscover . |
Elastic Stack component supported versions | Elastic recommends running beats on the same version as Elastic Stack components e.g. Elasticsearch and Logstash. Prior to version 7.4, Functionbeat is only compatible with Elasticsearch as an output destination. Logstash and other outputs are not supported. Starting with version 7.4, Functionbeat supports Elasticsearch and Logstash outputs. Refer to the respective support matrix for Beats, Logstash, and Elasticsearch. |
Additional capabilities | None |
Other considerations | Costs: Egress from CSP to ESS Implementation: ![]() |
Relevant guides | Run Metricbeat on Kubernetes |
Deploys Logstash on server/container to collect from a source that Elastic Agent or Beats cannot read, e.g. databases or as an ingestion engine for the specified additional capabilities.
Category | Details |
---|---|
Architecture diagram | Logstash to Elasticsearch |
Data type(s) collected | Logs, metrics, and events |
Supported outputs | A variety of outputs. Refer to Logstash Output Plugins. |
Limitation(s) | ![]() |
Elastic Common Schema (ECS) support | ❌ Generally not supported out-of-the-box. Incoming data source must already be in ECS format or processing must be done at the filter phase of Logstash pipelines to align with ECS format. |
Elastic Stack component supported versions | Refer to Elastic’s support matrix. |
Additional capabilities | ✅ Data enrichment between Elastic Agent and Elasticsearch ✅ Persistent queue (PQ) buffering to accommodate network issues and downstream unavailability ✅ Proxying in cases where Elastic Agents have network restrictions i.e. connecting outside of the Elastic Agent network ✅ Ability to route data to multiple Elasticsearch clusters and other destinations depending on the content ✅ Centralised pipeline management to manage Logstash pipelines via Kibana UI |
Other considerations | Costs: - Egress from CSP to ESS - PaaS/SaaS/Faas/CSP API invocation costs if utilising Logstash Input/Filter/Output plugins. - Additional Compute to host Logstash Implementation: Deployment and scaling considerations |
Relevant guides | Installing Logstash |
Deploys Beats as agents on servers/containers to collect a variety of data.
Category | Details |
---|---|
Data type(s) collected | Logs, metrics, and events |
Supported outputs | A variety of outputs: Elasticsearch, Logstash, Kafka, Redis, File, Console. Refer to the respective Beats output documentation. |
Limitation(s) | ❌ Beats need to be configured at server level. |
Elastic Common Schema (ECS) support | For supported modules, there is out-of-the-box data parsing to align with ECS. Refer to the respective Beats’ Module documentation. |
Elastic Stack component supported versions | ![]() Prior to version 7.4, Functionbeat is only compatible with Elasticsearch as an output destination. Logstash and other outputs are not supported. Starting with version 7.4, Functionbeat supports Elasticsearch and Logstash outputs. Refer to the respective support matrix for Beats, Logstash, and Elasticsearch. |
Additional capabilities | ✅ Data enrichment between Elastic Agent and Elasticsearch ✅ Persistent queue (PQ) buffering to accommodate network issues and downstream unavailability ✅ Proxying in cases where Elastic Agents have network restrictions i.e. connecting outside of the Elastic Agent network ✅ Ability to route data to multiple Elasticsearch clusters and other destinations depending on the content ✅ Centralised pipeline management to manage Logstash pipelines via Kibana UI |
Other considerations | Costs: - Egress from CSP to ESS - PaaS/SaaS/Faas/CSP API invocation costs if Module retrieves data via API. - Additional Compute to host Beats |
Relevant guides | Refer to the respective Beats documentation for installation and configuration. |
For custom applications using Elasticsearch language clients to ingest directly to Elasticsearch.
Category | Details |
---|---|
Data type(s) collected | Logs, metrics, and events |
Supported outputs | Elasticsearch |
Limitation(s) | ![]() |
Elastic Common Schema (ECS) support | ❌ ECS support not available. Has to be done programmatically when indexing data to Elasticsearch. |
Elastic Stack component supported versions | Language clients are forward compatible i.e. clients support communicating with the same or newer minor versions of Elasticsearch. Elasticsearch language clients are only backwards compatible with default distributions and without guarantees made. Refer to the respective language client’s documentation for details. |
Additional capabilities | - |
Other considerations | Costs: - Egress from CSP to ESS - Any other costs that might be incurred by custom application code to retrieve data, e.g. CloudWatch API. |
Relevant guides | Ingesting data from applications |