Overview
You can use this document to learn about the configuration parameters available in a configuration file, specifically for Thresholds.
A threshold defines alerting conditions at the agent level. Each agent locally evaluates the thresholds and then triggers an alert if the threshold is met. Alert destinations, such as Slack, PagerDuty, and email, can be added to the same workflow to receive these alerts.
Note
By default, alerts are saved in the Edge Delta SaaS backend, which populates the Insights page.
- To learn more about the Insights page, see Insights.
Note
There are 2 ways to define a threshold:
- Processor-level thresholds
- Most processors support the trigger_thresholds parameter to define thresholds.
- To learn more, see Processors Overview.
- Workflow-level thresholds
- This method is the newer and more flexible way to define thresholds with various operators and regex-based metric name matching.
- This document focuses on this method to explain how to configure thresholds.
Review Sample Configuration:
Review the following sample configuration:
thresholds: - name: http-latencyp95-threshold metric_name_pattern: http_request_method_.*_latency\.p95 operator: ">" value: 120 - name: http-avg-threshold metric_name: http_request_method_getconfig_latency.avg operator: ">=" value: 50 - name: cluster-errors-threshold metric_name: error.anomaly1 operator: ">" value: 80 - name: incoming-lines-threshold metric_name: incoming_lines.anomaly1 operator: ">" value: 90 - name: incoming-bytes-threshold metric_name: incoming_bytes.anomaly2 operator: ">" value: 90 - name: consecutive-bytes-threshold metric_name: incoming_bytes.anomaly2 operator: ">" value: 90 consecutive: 5
Review Parameters for Thresholds
Review the following parameters that you can configure in the Edge Delta App.
name
Required
Enter a descriptive name for the threshold, which will be used to map this threshold to a workflow.
Review the following example:
name: consecutive-bytes-threshold
type
Optional
This parameter only applies to thresholds with multiple conditions.
Enter and to use multiple conditions within a single threshold.
Review the following example:
type: and
interval
Optional
This parameter only applies to thresholds with multiple conditions.
- To learn more, see Learn About Multiple Conditions in a Threshold.
Enter a length of time to flush conditional states (triggered and not triggered).
Review the following example:
interval: 1m
metric_name
Optional
This parameter is the exact name of the metric to be evaluated.
Metric names are generated based on processor names.
- To learn more, see Processors Overview.
You must enter a Metric Name (metric_name) or Metric Name Pattern (metric_name_pattern), but not both.
Review the following example:
metric_name: incoming_lines.anomaly1
metric_name_pattern
Optional
This parameter is the regular expression that will be used to match the metric names.
You must enter a Metric Name (metric_name) or Metric Name Pattern (metric_name_pattern), but not both.
Review the following example:
metric_name_pattern: http_request_method_.*_latency\.p95
operator
Optional
This parameter supports the following operators:
- ==
- >
- >=
- <
- <=
Review the following example:
operator: ">"
value
Optional
This parameter is the threshold value used to compare with the metric value, based on the specified operator.
Review the following example:
value: 90
consecutive
Required
This parameter is the number of times in a row that a threshold condition must be met to trigger an alert.
For example, the default value is 0, which means that any threshold condition met will cause an alert to trigger.
Review the following example:
consecutive: 5
conditions
Optional
This parameter only applies to thresholds with multiple conditions.
Specifically, this parameter allows you to add multiple conditions to a single threshold.
- To learn more, see Learn About Multiple Conditions in a Threshold.
Review the following example:
conditions: - metric_name: http_request_method_updateconfig_latency.avg operator: ">=" value: 100 - metric_name: http_request_method_deleteconfig_latency.max operator: ">" value: 125 consecutive: 5
Review Sample Configuration for a Multi-Condition Threshold
Review the following sample configuration:
- name: cluster-errors-multi-threshold type: and interval: 1m conditions: - metric_name: http_request_method_updateconfig_latency.avg operator: ">=" value: 100 - metric_name: http_request_method_deleteconfig_latency.max operator: ">" value: 125 consecutive: 5
Review Parameters for a Multi-Condition Threshold
You can add multiple conditions to a single threshold. With this action, an alert will only trigger when all conditions in the threshold are met.
At a high level, to add multiple conditions in a single threshold, you will seemingly create "child" thresholds that will live under a single "parent" threshold.
For example, a typical threshold contains the following parameters:
- metric_name (or metric_name_pattern)
- operator
- value
- consecutive
You can use these same parameters to create a condition ("child" threshold) within the "parent" threshold. Then, you will use the parameters below to combine and add the conditions into a single "parent" threshold.
type
Optional
Enter and to use multiple conditions within a single threshold.
Review the following example:
type: and
interval
Optional
Enter a length of time to flush conditional states (triggered and not triggered).
Review the following example:
interval: 1m
conditions
Optional
This parameter allows you to create and add specific conditions to a single threshold:
- metric_name (or metric_name_pattern)
- operator
- value
- consecutive
Review the following example:
conditions: - metric_name: http_request_method_updateconfig_latency.avg operator: ">=" value: 100 - metric_name: http_request_method_deleteconfig_latency.max operator: ">" value: 125 consecutive: 5
Create and Manage a Threshold
To create and manage a threshold, you must populate a YAML file.
To access the YAML file for a new configuration:
-
In the Edge Delta App, on the left-side navigation, click Data Pipeline, and then click Agent Settings.
-
Click Create Configuration.
-
Click YAML.
- Enter your desired parameters, and then click Save.
- To understand the parameters to configure a threshold, see Review Parameters for Thresholds.
To access the YAML file for an existing configuration:
-
In the Edge Delta App, on the left-side navigation, click Data Pipeline, and then click Agent Settings.
-
Locate the desired configuration, then under Actions, click the vertical ellipses, and then click Edit.
-
Review the YAML file, make your changes, and then click Save.
- To understand the parameters to configure a threshold, see Review Parameters for Thresholds.