ML anomaly detection (MLAD)

Anomaly detection is the main feature of Elastic ML, designed for continuous, real-time analysis of time-series data for observability or security purposes. It identifies unusual patterns in data that could signal problems, threats, or opportunities across various fields.

How it works

ML anomaly detection (MLAD) jobs work by creating a baseline of normal behaviour, then identifying data points that deviate significantly from that baseline.

Over time, the algorithm learns the rhythms, trends, and seasonality of your data. For example, it learns that high website traffic on a Monday morning is normal, but the same level of traffic at 3 AM on a Sunday is an anomaly. Furthermore, these jobs will automatically identify influencers, the specific entities (e.g., user, host, or IP) contributing most to an anomaly.

By catching these outliers early, organizations can take proactive action to prevent issues from escalating, leading to improved security, operational efficiency, and informed decision-making.

Configuration

In Elastic, MLAD uses a two-tier architecture:

• ML jobs—Create anomaly scores by modelling the selected data based on the selected metric, statisical function, and time interval. This value indicates the severity of an anomaly compared to past events.
• ML rules—Perform scheduled checks on your anomaly detection jobs and automatically send alerts when an anomaly is detected or when there is an issue affecting the job performance.
  • Anomaly detection alert—Monitor MLAD job results and generate alerts if an anomaly score exceeds the set threshold.
  • Anomaly detection jobs health—Monitors the status of an MLAD job and alerts you if issues occur (such as memory limit errors or a stopped datafeed). Each ML rule is associated with a set of ML jobs.

Ways to set up MLAD

You can create MLAD jobs using one of the following methods:

• Use Elastic’s pre-configured jobs
• Create a custom job
• Import StackOps’ preconfigured jobs for SIEM purposes

Structure of ML alert rules

Each MLAD alert rule connects a specific ML job to an alerting system. The key components are:

• name: Human-readable rule identifier following StackOps naming convention.
• risk_score: Numerical risk assessment (0-100) for alert prioritisation.
• severity: Alert severity level (low, medium, high, critical).
• machine_learning_job_id: Exact ML job identifier being monitored.
• anomaly_threshold: Minimum anomaly score required to trigger an alert.

Example ML alert rule:

{
  "name": "[StackOps][ML][AWS] IAM Rare Activities",
  "risk_score": 74,
  "severity": "high", 
  "machine_learning_job_id": "stackops_aws_iam_rare_activities_event",
  "anomaly_threshold": 50
}Copy to clipboardErrorCopied

Best practices

Elastic ML jobs will take time to provide accurate results as they learn the unique patterns and seasonality of your data. To avoid alert fatigue, you should refrain from creating alerts immediately. The best practice is to create an ML alert rule only after a few weeks, once there has been enough time to analyse the initial ML results. This “burn-in” period allows the model to validate the findings and understand your system’s true baseline, so that when an alert does fire, it represents a meaningful, actionable event rather than statistical noise.