Choosing a data ingestion method
In StackOps Elastic, each data ingestion method comes with its own benefits, implementation requirements, and constraints. In this guide, we will discuss each data ingestion method so that you can choose the right method for your needs.
Recommendation criteria
StackOps recommends data ingestion methods that meet the following criteria:
- Cloud agnostic — Supports different cloud service providers and on-premises environments.
- High out-of-the-box coverage — Requires minimal configuration, saving time and effort to set up.
- Adherence to Elastic Common Schema (ECS) — Enables data normalisation, as ECS is used by SIEM and other security tools to ease data ingestion and correlation from various sources for analysis. ECS is converging with open monitoring standards like OpenTelemetry.
- Low overhead costs — Requires minimal operational oversight, freeing up IT resources for other tasks.
Note: Only recommended methods fall within the scope of StackOps support.
Recommended data ingestion methods
Here are the available data ingestion methods, ranked based on the recommendation criteria above.
Rank 1 - Recommended
The Rank 1 methods generally make use of Elastic Agents, which can be installed on a variety of hosts. The table below summarises the different Elastic Agent architectures for comparison. For more details, refer to the respective accordion tabs below.
| Fleet-managed Elastic Agent | Fleet-managed Elastic Agent (for API data collection) | Fleet-managed Elastic Agent as Kubernetes DaemonSet | |
|---|---|---|---|
| Description | Elastic Agent is directly installed onto the host running the software you want to monitor. | Elastic Agent installed on a host that acts as an intermediary to fetch or receive data from an external platform via APIs. | Elastic Agent is installed on every node in a Kubernetes cluster. |
| Data type(s) collected | Logs and metrics | Logs, metrics, and events | Logs and metrics |
| Supported outputs | - Elasticsearch - Logstash |
- Elasticsearch - Logstash |
- Elasticsearch - Logstash |
| Elastic Common Schema (ECS) support | ✓ | ✓ |  Only for Kubernetes component metrics and logs. |
| Supported versions for Elastic Stack components | Same version or higher. | Same version or higher. | Same version or higher. |
Fleet-managed Elastic Agent
This method is recommended for directly deploying Elastic Agents on individual EC2 instances, VMs, or servers that are running the software you want to monitor. For more details on its architecture, refer to Elastic documentation.
Details
| Details | |
|---|---|
| Data types collected | Logs and metrics. |
| Supported outputs | - Elasticsearch - Logstash |
| Elastic Common Schema (ECS) support | Automatically included if the agent is used with out-of-the-box integrations. |
| Supported versions for Elastic Stack components | Generally compatible with stack components that are the same version or higher. Refer to Elastic’s support matrix. |
| Additional capabilities | ✓ XDR/EDR (with Elastic Defend/Endpoint Integration) ✓ OSQuery (with Osquery Integration) By default, these capabilities are disabled due to security requirements. If required, tenants can raise a service request to enable these capabilities. You must be responsible for maintaining the security posture. |
| Implementation considerations | - You can only install one Elastic Agent per host, but Elastic Agent policy allows multiple integrations. - You need root user on Mac (or Administrator on Windows) to run installation/enrollment commands. |
| Cost considerations | - Egress from your cloud provider to Elastic. - Your cloud provider API invocation cost if data is retrieved via your cloud provider API. - Cost of compute is already incurred for the hosts required for tenant’s applications/use cases. |
| Relevant setup guides | - StackOps Resiliency Accelerator (SORA) (recommended) - StackOps guide for manual setup |
Fleet-managed Elastic Agent (for API data collection)
This method is recommended for deploying Elastic Agents to collect data from Function-as-a-Service (FaaS), Software-as-a-Service (SaaS), and managed services. For more details on its architecture, refer to Elastic documentation.
Details
| Details | |
|---|---|
| Data types collected | Logs, metrics, and events. |
| Supported outputs | - Elasticsearch - Logstash |
| Elastic Common Schema (ECS) support | Automatically included if the agent is used with out-of-the-box integrations. |
| Supported versions for Elastic Stack components | Generally compatible with stack components that are the same version or higher. Refer to Elastic’s support matrix. |
| Additional capabilities | None. |
| Implementation considerations | - You can only install one Elastic Agent per host, but Elastic Agent policy allows multiple integrations. - You need root user on Mac (or Administrator on Windows) to run installation/enrollment commands. |
| Cost considerations | - Egress from your cloud provider to Elastic. - SaaS/your cloud provider API invocation costs if data is retrieved via API. - Additional compute costs to host Elastic Agent. |
| Relevant setup guides | - StackOps Resiliency Accelerator (SORA) (recommended) - StackOps guide for manual setup |
Fleet-managed Elastic Agent as Kubernetes DaemonSet
This method is recommended for self-managed Kubernetes/Openshift and managed Kubernetes like GKE, EKS, and AKS.
Details
| Details | |
|---|---|
| Data types collected | Logs and metrics. Logs from applications deployed on Kubernetes will also be collected since the Elastic Agent will pick up container logs. |
| Supported outputs | - Elasticsearch - Logstash |
| Elastic Common Schema (ECS) support | Only for Kubernetes component metrics and logs. Commercial and custom application logs will only be in generic ECS format and will not be fully parsed down to log message contents. |
| Supported versions for Elastic Stack components | Generally compatible with stack components that are the same version or higher. Refer to Elastic’s support matrix. |
| Additional capabilities | None. |
| Limitations | On managed Kubernetes solutions, Elastic Agent has no access to several data sources. This means that logs and metrics from certain control plane components cannot be collected by Elastic Agent. |
| Implementation considerations | - You can only install one Elastic Agent per host, but Elastic Agent policy allows multiple integrations. - You need root user on Mac (or Administrator on Windows) to run installation/enrollment commands. |
| Cost considerations | - Egress from your cloud provider to Elastic. |
| Relevant setup guides | - StackOps Resiliency Accelerator (SORA) (recommended) - Elastic documentation for manual setup - For EKS, use the AWS add-on for easier setup: Elastic documentation |
Elastic APM Agent
This method is recommended for application performance monitoring. The Elastic APM Agent is instrumented in application code to collect application performance monitoring data. Auto-instrumentation available for supported languages or frameworks.
Details
| Details | |
|---|---|
| Data types collected | Logs, metrics, and events. |
| Supported outputs | Only sends to APM server but can be configured to send output to Elasticsearch or Logstash. |
| Elastic Common Schema (ECS) support | Yes. |
| Supported versions for agent compatibility | Refer to Elastic documentation for more information. |
| Additional capabilities | ✓Tracing ✓APM |
| Limitations | - May require code changes at the application level for complex instrumentation scenarios, or for languages and frameworks without auto-instrumentation support. - Only error logs (e.g., exceptions and stacktrace) will be captured during runtime, and other ingestion methods have to be used to capture application logs on other log levels. - Only APM Server standalone (not managed by Elastic Agent) supports output to Logstash. |
| Implementation considerations | - Elastic APM agents automatically pick up basic host-level metrics and agent-specific metrics, like JVM metrics in the Java Agent, and Go runtime metrics in the Go Agent. Note: If APM is implemented within a software running as a container, it will not pick up EC2/VM metrics. |
| Cost considerations | - Egress from your cloud provider to Elastic. |
| Relevant setup guides | - Elastic documentation |
Rank 2 - Restricted to specific hosting environments
Elastic Serverless Forwarder
Elastic Serverless Forwarder is an AWS Lambda function that ships logs from your AWS environment to Elastic. It only supports the following inputs:
- Amazon S3 (via SQS event notifications)
- Amazon Kinesis Data Streams
- Amazon CloudWatch Logs subscription filters
- Amazon SQS message payload
This method recommended if you are monitoring a serverless AWS workload. For more details on its architecture, refer to Elastic documentation.
Details
| Details | |
|---|---|
| Data types collected | Logs, metrics, and events. |
| Supported outputs | - Elasticsearch - Logstash (preview) |
| Elastic Common Schema (ECS) support | Automatically included if used with out-of-the-box integrations. |
| Supported versions for Elastic Stack components | Compatible with Elastic Stack 7.16 and later. |
| Additional capabilities | None. |
| Limitations | - Applicable to AWS environments only. |
| Cost considerations | - Egress from your cloud provider to Elastic. - Firehose usage cost and cost for routing data from other AWS services to Firehose. - API invocation cost if data is retrieved via API (e.g., AWS CloudWatch API). |
| Relevant setup guides | - StackOps guide for manual setup |
Not recommended: Other methods
Note: Methods that are not recommended are not covered by StackOps support.
Agentless (AWS Firehose)
Using Firehose to send your data to Elastic means you have no agents to deploy, no Lambdas to configure, and no Beats to manage.
| Details | |
|---|---|
| Data types collected | Logs, metrics, and events. |
| Supported outputs | Elasticsearch on Elastic Cloud only. |
| Elastic Common Schema (ECS) support | Automatically included if used with out-of-the-box integrations. |
| Supported versions for Elastic Stack components | Generally compatible with stack components that are the same version or higher. Refer to Elastic’s support matrix. |
| Additional capabilities | ✓ Configure multiple log sources to send data to Firehouse delivery streams. Refer to AWS documentation for more information. ✓ Has built-in auto-scaling capability and is designed to handle high-volume use cases. |
| Limitations | - Applicable to AWS environments only. - Only able to integrate with public internet; in contrast, StackOps implements traffic filters by default. - Compatible only with Elastic Stack version 7.17 or later versions, and it can only run on Elastic Cloud. |
| Implementation considerations | None. |
| Cost considerations | - Egress from your cloud provider to Elastic. - Firehose usage cost and cost for routing data from other AWS services to Firehose. - API invocation cost if data is retrieved via API (e.g., AWS CloudWatch API). |
| Relevant setup guides | - Elastic quickstart Note: Methods that are not recommended are not covered by StackOps support. |
Standalone Elastic Agent
Standalone Elastic Agents are not centrally managed by Fleet.
| Details | |
|---|---|
| Data types collected | Logs and metrics. |
| Supported outputs | - Elasticsearch - Logstash |
| Elastic Common Schema (ECS) support | Automatically included if used with out-of-the-box integrations. |
| Supported versions for Elastic Stack components | Generally compatible with stack components that are the same version or higher. Refer to Elastic’s support matrix. |
| Additional capabilities | None. |
| Implementation considerations | - You can only install one Elastic Agent per host, but Elastic Agent policy allows multiple integrations. - You need root user on Mac (or Administrator on Windows) to run installation/enrollment commands. - Management of agent policies and Elastic Agents must be done manually via SSH access to server. |
| Cost considerations | - Egress from your cloud provider to Elastic. - API invocation cost if data is retrieved via API. - Additional compute costs to host Elastic Agent. |
| Relevant setup guides | - Elastic documentation Note: Methods that are not recommended are not covered by StackOps support. |
Beats on Kubernetes as Daemonset/Deployment
This method deploys Filebeat to collect logs and Metricbeat to collect metrics from Kubernetes. It serves as an alternative to using Elastic Agents on Kubernetes as DaemonSet due to certain environment limitations (e.g., Openshift with CoreOS, or Metricbeat on Cloud Foundry).
| Details | |
|---|---|
| Data types collected | Logs and metrics. |
| Supported outputs | A variety of outputs, including Elasticsearch, Logstash, Kafka, Redis, File. Refer to Filebeat output and Metricbeat output documentation. |
| Elastic Common Schema (ECS) support | Hints based on autodiscover can be configured to faciliate the collection and parsing of data to align with ECS if a supported module is available. Refer to Filebeat modules and Metricbeat modules for list of modules and hints for autodiscover. |
| Supported versions for Elastic Stack components | Elastic recommends running Beats on the same version as Elastic Stack components. Prior to version 7.4, Functionbeat is only compatible with Elasticsearch as an output destination. Logstash and other outputs are not supported. Starting with version 7.4, Functionbeat supports Elasticsearch and Logstash outputs. For more information, refer to the respective support matrix for Beats, Logstash, and Elasticsearch. |
| Additional capabilities | None. |
| Limitations | On managed Kubernetes solutions, managed control plane data will not be available. |
| Implementation considerations | - Changes to data collection and output configuration for Beats require redeployment of manifest. |
| Cost considerations | - Egress from your cloud provider to Elastic. |
| Relevant setup guides | - Elastic documentation Note: Methods that are not recommended are not covered by StackOps support. |
Standalone Logstash
This method deploys Logstash on server/container to collect data from sources that Elastic Agent or Beats cannot read.
| Details | |
|---|---|
| Data types collected | Logs, metrics, and events. |
| Supported outputs | A variety of outputs. Refer to Elastic documentation for details on the Logstash output plugins. |
| Elastic Common Schema (ECS) support | Generally not supported out-of-the-box. Incoming data source must already be in ECS format or processing must be done at the filter phase of Logstash pipelines to align with ECS format. |
| Supported versions for Elastic Stack components | Refer to Elastic’s support matrix. |
| Additional capabilities | ✓ Data enrichment between Elastic Agent and Elasticsearch. ✓ Persistent queue (PQ) buffering to accommodate network issues and downstream unavailability. ✓ Proxying in cases where Elastic Agents have network restrictions. ✓ Ability to route data to multiple Elasticsearch clusters and other destinations depending on the content. ✓ Centralised pipeline management to manage Logstash pipelines via Kibana UI. |
| Limitations | - Generally requires additional effort to parse data at source or at the Logstash pipelines, in order to align with ECS format. |
| Implementation considerations | For deployment and scaling considerations, refer to Elastic documentation. |
| Cost considerations | - Egress from your cloud provider to Elastic. - API invocation costs if you are utilising Logstash Input/Filter/Output plugins. - Additional compute costs to host Logstash. |
| Relevant setup guides | - Elastic documentation for Logstash Note: Methods that are not recommended are not covered by StackOps support. |
Standalone Beats
This method deploys Beats as agents on servers/containers to collect a variety of data.
| Details | |
|---|---|
| Data types collected | Logs, metrics, and events. |
| Supported outputs | A variety of outputs, including Elasticsearch, Logstash, Kafka, Redis, File. Refer to the respective Beats output documentation for more information. |
| Elastic Common Schema (ECS) support | Automatically included if used with supported modules. Refer to the respective Beats’ Module documentation for more information. |
| Supported versions for Elastic Stack components | Elastic recommends running Beats on the same version as Elastic Stack components. Prior to version 7.4, Functionbeat is only compatible with Elasticsearch as an output destination. Logstash and other outputs are not supported. Starting with version 7.4, Functionbeat supports Elasticsearch and Logstash outputs. Refer to the respective support matrix for Beats, Logstash, and Elasticsearch. |
| Additional capabilities | ✓ Data enrichment between Elastic Agent and Elasticsearch. ✓ Persistent queue (PQ) buffering to accommodate network issues and downstream unavailability. ✓ Proxying in cases where Elastic Agents have network restrictions. ✓ Ability to route data to multiple Elasticsearch clusters and other destinations depending on the content. ✓ Centralised pipeline management to manage Logstash pipelines via Kibana UI. |
| Implementation considerations | Beats need to be configured at server level. |
| Cost considerations | - Egress from your cloud provider to Elastic. - API invocation cost if data is retrieved via API. - Additional compute costs to host Beats. |
| Relevant setup guides | - Elastic documentation for Beats Note: Methods that are not recommended are not covered by StackOps support. |
Custom application using Elasticsearch language clients
This method is for custom applications using Elasticsearch language clients to ingest directly to Elasticsearch.
| Details | |
|---|---|
| Data types collected | Logs, metrics, and events. |
| Supported outputs | Elasticsearch. |
| Elastic Common Schema (ECS) support | ECS support not available. ECS parsing must be done programmatically when indexing data to Elasticsearch. |
| Supported versions for Elastic Stack components | Language clients are forward compatible and therefore support communicating with the same or newer minor versions of Elasticsearch. Language clients are only backwards compatible with default distributions and without guarantees made. Refer to the respective language client’s documentation for details. |
| Additional capabilities | None. |
| Implementation considerations | Requires implementation at software level. |
| Cost considerations | - Egress from your cloud provider to Elastic. - Any other costs that might be incurred by your custom application code when retrieving data (e.g., CloudWatch API). |
| Relevant setup guides | - Elastic documentation Note: Methods that are not recommended are not covered by StackOps support. |
