Thresholds

Overview

You can use this document to learn about the configuration parameters available in a configuration file, specifically for Thresholds.

A threshold defines alerting conditions at the agent level. Each agent locally evaluates the thresholds and then triggers an alert if the threshold is met. Alert destinations, such as Slack, PagerDuty, and email, can be added to the same workflow to receive these alerts.

Note

By default, alerts are saved in the Edge Delta SaaS backend, which populates the Insights page. 

• To learn more about the Insights page, see Insights.

Note

There are 2 ways to define a threshold:

• Processor-level thresholds
  • Most processors support the trigger_thresholds parameter to define thresholds.
  • To learn more, see Processors Overview.
• Workflow-level thresholds
  • This method is the newer and more flexible way to define thresholds with various operators and regex-based metric name matching.
  • This document focuses on this method to explain how to configure thresholds.

Review Sample Configuration: 

Review the following sample configuration: 

thresholds:
  - name: http-latencyp95-threshold
    metric_name_pattern: http_request_method_.*_latency\.p95  
    operator: ">"
    value: 120
  - name: http-avg-threshold
    metric_name: http_request_method_getconfig_latency.avg    
    operator: ">="
    value: 50
  - name: cluster-errors-threshold
    metric_name: error.anomaly1
    operator: ">"
    value: 80
  - name: incoming-lines-threshold
    metric_name: incoming_lines.anomaly1
    operator: ">"
    value: 90
  - name: incoming-bytes-threshold
    metric_name: incoming_bytes.anomaly2
    operator: ">"
    value: 90
  - name: consecutive-bytes-threshold
    metric_name: incoming_bytes.anomaly2
    operator: ">"
    value: 90
    consecutive: 5

Review Parameters for Thresholds

Review the following parameters that you can configure in the Edge Delta App. 

name

Required

Enter a descriptive name for the threshold, which will be used to map this threshold to a workflow.

Review the following example: 

name: consecutive-bytes-threshold

type

Optional

This parameter only applies to thresholds with multiple conditions. 

Enter and to use multiple conditions within a single threshold.

Review the following example: 

type: and

interval 

Optional 

This parameter only applies to thresholds with multiple conditions. 

• To learn more, see Learn About Multiple Conditions in a Threshold.

Enter a length of time to flush conditional states (triggered and not triggered). 

Review the following example: 

interval: 1m

metric_name

Optional

This parameter is the exact name of the metric to be evaluated.

Metric names are generated based on processor names.

• To learn more, see Processors Overview.

You must enter a Metric Name (metric_name) or Metric Name Pattern (metric_name_pattern), but not both.

Review the following example: 

metric_name: incoming_lines.anomaly1

metric_name_pattern

Optional

This parameter is the regular expression that will be used to match the metric names.

You must enter a Metric Name (metric_name) or Metric Name Pattern (metric_name_pattern), but not both.

Review the following example: 

metric_name_pattern: http_request_method_.*_latency\.p95

operator

Optional 

This parameter supports the following operators:

• ==
• >
• >=
• <
• <=

Review the following example: 

operator: ">"

value

Optional 

This parameter is the threshold value used to compare with the metric value, based on the specified operator. 

Review the following example: 

value: 90

consecutive

Required

This parameter is the number of times in a row that a threshold condition must be met to trigger an alert.

For example, the default value is 0, which means that any threshold condition met will cause an alert to trigger.

Review the following example: 

consecutive: 5

conditions

Optional

This parameter only applies to thresholds with multiple conditions. 

Specifically, this parameter allows you to add multiple conditions to a single threshold. 

• To learn more, see Learn About Multiple Conditions in a Threshold.

Review the following example: 

    conditions:
    - metric_name: http_request_method_updateconfig_latency.avg
      operator: ">="
      value: 100
    - metric_name: http_request_method_deleteconfig_latency.max
      operator: ">"
      value: 125
      consecutive: 5

Review Sample Configuration for a Multi-Condition Threshold

Review the following sample configuration: 

  - name: cluster-errors-multi-threshold
    type: and
    interval: 1m
    conditions: 
    - metric_name: http_request_method_updateconfig_latency.avg
      operator: ">="
      value: 100
    - metric_name: http_request_method_deleteconfig_latency.max
      operator: ">"
      value: 125
      consecutive: 5 

Review Parameters for a Multi-Condition Threshold

You can add multiple conditions to a single threshold. With this action, an alert will only trigger when all conditions in the threshold are met.  

At a high level, to add multiple conditions in a single threshold, you will seemingly create "child" thresholds that will live under a single "parent" threshold. 

For example, a typical threshold contains the following parameters: 

• metric_name (or metric_name_pattern)
• operator
• value
• consecutive

You can use these same parameters to create a condition ("child" threshold) within the "parent" threshold. Then, you will use the parameters below to combine and add the conditions into a single "parent" threshold. 

type

Optional

Enter and to use multiple conditions within a single threshold.

Review the following example: 

type: and

interval

Optional 

Enter a length of time to flush conditional states (triggered and not triggered). 

Review the following example: 

interval: 1m

conditions

Optional 

This parameter allows you to create and add specific conditions to a single threshold:

• metric_name (or metric_name_pattern)
• operator
• value
• consecutive

Review the following example: 

    conditions:
    - metric_name: http_request_method_updateconfig_latency.avg
      operator: ">="
      value: 100
    - metric_name: http_request_method_deleteconfig_latency.max
      operator: ">"
      value: 125
      consecutive: 5

Create and Manage a Threshold 

To create and manage a threshold, you must populate a YAML file.

To access the YAML file for a new configuration:

1. In the Edge Delta App, on the left-side navigation, click Data Pipeline, and then click Agent Settings.

2. Click Create Configuration.

3. Click YAML.

4. Enter your desired parameters, and then click Save.  
   • To understand the parameters to configure a threshold, see Review Parameters for Thresholds.

To access the YAML file for an existing configuration:

1. In the Edge Delta App, on the left-side navigation, click Data Pipeline, and then click Agent Settings.

2. Locate the desired configuration, then under Actions, click the vertical ellipses, and then click Edit. 

3. Review the YAML file, make your changes, and then click Save.

   • To understand the parameters to configure a threshold, see Review Parameters for Thresholds.