Overview
You can use this document to learn about the configuration parameters available in a configuration file, specifically for Thresholds.
A threshold defines alerting conditions at the agent level. Each agent locally evaluates the thresholds and then triggers an alert if the threshold is met. Alert destinations, such as Slack, PagerDuty, and email, can be added to the same workflow to receive these alerts.
Note
By default, alerts are saved in the Edge Delta SaaS backend, which populates the Insights page.
- To learn more about the Insights page, see Insights.
Note
There are 2 ways to define a threshold:
- Processor-level thresholds
- Most processors support the trigger_thresholds parameter to define thresholds.
- To learn more, see Processors Overview.
- Workflow-level thresholds
- This method is the newer and more flexible way to define thresholds with various operators and regex-based metric name matching.
- This document focuses on this method to explain how to configure thresholds.
Review Parameters for Thresholds
Review the following parameters that you can configure in the Edge Delta App:
| YAML | Description |
| name |
Enter a descriptive name for the threshold, which will be used to map this threshold to a workflow. This parameter is required. Review the following example: name: consecutive-bytes-threshold |
| type |
This parameter only applies to thresholds with multiple conditions. Enter and to use multiple conditions within a single threshold. This parameter is optional. Review the following example: type: and |
| interval |
This parameter only applies to thresholds with multiple conditions.
Enter a length of time to flush conditional states (triggered and not triggered). This parameter is optional. Review the following example: interval: 1m |
| metric_name |
This parameter is the exact name of the metric to be evaluated. Metric names are generated based on processor names.
You must enter a Metric Name (metric_name) or Metric Name Pattern (metric_name_pattern), but not both. This parameter is optional. Review the following example: metric_name: incoming_lines.anomaly1 |
| metric_name_pattern |
This parameter is the regular expression that will be used to match the metric names. You must enter a Metric Name (metric_name) or Metric Name Pattern (metric_name_pattern), but not both. This parameter is optional. Review the following example: metric_name_pattern: http_request_method_.*_latency\.p95 |
| operator |
This parameter supports the following operators:
This parameter is optional. Review the following example: operator: ">" |
| value |
This parameter is the threshold value used to compare with the metric value, based on the specified operator. This parameter is optional. Review the following example: value: 90 |
| consecutive |
This parameter is the number of times in a row that a threshold condition must be met to trigger an alert. For example, the default value is 0, which means that any threshold condition met will cause an alert to trigger. This parameter is required. Review the following example: consecutive: 5 |
| conditions |
This parameter only applies to thresholds with multiple conditions. Specifically, this parameter allows you to add multiple conditions to a single threshold. This parameter is optional. Review the following example: conditions:
- metric_name: http_request_method_updateconfig_latency.avg
operator: ">="
value: 100
- metric_name: http_request_method_deleteconfig_latency.max
operator: ">"
value: 125
consecutive: 5
To learn more, see Learn About Multiple Conditions in a Threshold. |
Review Sample Configuration:
Review the following sample configuration:
thresholds:
- name: http-latencyp95-threshold
metric_name_pattern: http_request_method_.*_latency\.p95
operator: ">"
value: 120
- name: http-avg-threshold
metric_name: http_request_method_getconfig_latency.avg
operator: ">="
value: 50
- name: cluster-errors-threshold
metric_name: error.anomaly1
operator: ">"
value: 80
- name: incoming-lines-threshold
metric_name: incoming_lines.anomaly1
operator: ">"
value: 90
- name: incoming-bytes-threshold
metric_name: incoming_bytes.anomaly2
operator: ">"
value: 90
- name: consecutive-bytes-threshold
metric_name: incoming_bytes.anomaly2
operator: ">"
value: 90
consecutive: 5
Learn About Multiple Conditions in a Threshold
You can add multiple conditions to a single threshold. With this action, an alert will only trigger when all conditions in the threshold are met.
At a high level, to add multiple conditions in a single threshold, you will seemingly create "child" thresholds that will live under a single "parent" threshold.
For example, a typical threshold contains the following parameters:
- metric_name (or metric_name_pattern)
- operator
- value
- consecutive
You can use these same parameters to create a condition ("child" threshold) within the "parent" threshold. Then, you will use the parameters below to combine and add the conditions into a single "parent" threshold.
Review the following parameters:
| Parameter | Description |
| type |
Enter and to use multiple conditions within a single threshold. Review the following example: type: and |
| interval |
Enter a length of time to flush conditional states (triggered and not triggered). Review the following example: interval: 1m |
| conditions |
This parameter allows you to create and add specific conditions to a single threshold:
Review the following example: conditions:
- metric_name: http_request_method_updateconfig_latency.avg
operator: ">="
value: 100
- metric_name: http_request_method_deleteconfig_latency.max
operator: ">"
value: 125
consecutive: 5
|
Review Sample Configuration for Multi-Conditions
Review the following sample configuration:
- name: cluster-errors-multi-threshold
type: and
interval: 1m
conditions:
- metric_name: http_request_method_updateconfig_latency.avg
operator: ">="
value: 100
- metric_name: http_request_method_deleteconfig_latency.max
operator: ">"
value: 125
consecutive: 5
Create and Manage a Threshold
To create and manage a threshold, you must populate a YAML file.
To access the YAML file for a new configuration:
-
In the Edge Delta App, on the left-side navigation, click Data Pipeline, and then click Agent Settings.
-
Click Create Configuration.
-
Click YAML.
- Enter your desired parameters, and then click Save.
- To understand the parameters to configure a threshold, see Review Parameters for Thresholds.
To access the YAML file for an existing configuration:
-
In the Edge Delta App, on the left-side navigation, click Data Pipeline, and then click Agent Settings.
-
Locate the desired configuration, then under Actions, click the vertical ellipses, and then click Edit.
-
Review the YAML file, make your changes, and then click Save.
- To understand the parameters to configure a threshold, see Review Parameters for Thresholds.
Quick links