Thresholds

George Alpizar
George Alpizar
  • Updated

Overview

You can use this document to learn about the configuration parameters available in a configuration file, specifically for Thresholds.

A threshold defines alerting conditions at the agent level. Each agent locally evaluates the thresholds and then triggers an alert if the threshold is met. Alert destinations, such as Slack, PagerDuty, and email, can be added to the same workflow to receive these alerts.

Note

By default, alerts are saved in the Edge Delta SaaS backend, which populates the Insights page. 

  • To learn more about the Insights page, see Insights.

Note

There are 2 ways to define a threshold:

  • Processor-level thresholds
    • Most processors support the trigger_thresholds parameter to define thresholds.
    • To learn more, see Processors Overview.
  • Workflow-level thresholds
    • This method is the newer and more flexible way to define thresholds with various operators and regex-based metric name matching.
    • This document focuses on this method to explain how to configure thresholds.

Review Parameters for Thresholds

Review the following parameters that you can configure in the Edge Delta App:

YAML Description
name

Enter a descriptive name for the threshold, which will be used to map this threshold to a workflow.

This parameter is required. 

Review the following example: 

name: consecutive-bytes-threshold
type

This parameter only applies to thresholds with multiple conditions. 

Enter and to use multiple conditions within a single threshold.

This parameter is optional. 

Review the following example: 

type: and
interval

This parameter only applies to thresholds with multiple conditions. 

Enter a length of time to flush conditional states (triggered and not triggered). 

This parameter is optional. 

Review the following example: 

interval: 1m
metric_name

This parameter is the exact name of the metric to be evaluated.

Metric names are generated based on processor names.

You must enter a Metric Name (metric_name) or Metric Name Pattern (metric_name_pattern), but not both.

This parameter is optional. 

Review the following example: 

metric_name: incoming_lines.anomaly1
metric_name_pattern

This parameter is the regular expression that will be used to match the metric names.

You must enter a Metric Name (metric_name) or Metric Name Pattern (metric_name_pattern), but not both.

This parameter is optional. 

Review the following example: 

metric_name_pattern: http_request_method_.*_latency\.p95
operator

This parameter supports the following operators:

  • ==
  • >
  • >=
  • <
  • <=

This parameter is optional. 

Review the following example: 

operator: ">"
value

This parameter is the threshold value used to compare with the metric value, based on the specified operator. 

This parameter is optional. 

Review the following example: 

value: 90
consecutive

This parameter is the number of times in a row that a threshold condition must be met to trigger an alert.

For example, the default value is 0, which means that any threshold condition met will cause an alert to trigger.

This parameter is required. 

Review the following example: 

consecutive: 5
conditions

This parameter only applies to thresholds with multiple conditions. 

Specifically, this parameter allows you to add multiple conditions to a single threshold. 

This parameter is optional. 

Review the following example: 

    conditions:
    - metric_name: http_request_method_updateconfig_latency.avg
      operator: ">="
      value: 100
    - metric_name: http_request_method_deleteconfig_latency.max
      operator: ">"
      value: 125
      consecutive: 5

To learn more, see Learn About Multiple Conditions in a Threshold.

 


Review Sample Configuration: 

Review the following sample configuration: 

thresholds:
  - name: http-latencyp95-threshold
    metric_name_pattern: http_request_method_.*_latency\.p95  
    operator: ">"
    value: 120
  - name: http-avg-threshold
    metric_name: http_request_method_getconfig_latency.avg    
    operator: ">="
    value: 50
  - name: cluster-errors-threshold
    metric_name: error.anomaly1
    operator: ">"
    value: 80
  - name: incoming-lines-threshold
    metric_name: incoming_lines.anomaly1
    operator: ">"
    value: 90
  - name: incoming-bytes-threshold
    metric_name: incoming_bytes.anomaly2
    operator: ">"
    value: 90
  - name: consecutive-bytes-threshold
    metric_name: incoming_bytes.anomaly2
    operator: ">"
    value: 90
    consecutive: 5

 


Learn About Multiple Conditions in a Threshold 

You can add multiple conditions to a single threshold. With this action, an alert will only trigger when all conditions in the threshold are met.  

At a high level, to add multiple conditions in a single threshold, you will seemingly create "child" thresholds that will live under a single "parent" threshold. 

For example, a typical threshold contains the following parameters: 

  • metric_name (or metric_name_pattern)
  • operator
  • value
  • consecutive

You can use these same parameters to create a condition ("child" threshold) within the "parent" threshold. Then, you will use the parameters below to combine and add the conditions into a single "parent" threshold. 

Review the following parameters: 

Parameter Description
type

Enter and to use multiple conditions within a single threshold.

Review the following example: 

type: and
interval

Enter a length of time to flush conditional states (triggered and not triggered). 

Review the following example: 

interval: 1m
conditions

This parameter allows you to create and add specific conditions to a single threshold:

  • metric_name (or metric_name_pattern)
  • operator
  • value
  • consecutive

Review the following example: 

    conditions:
    - metric_name: http_request_method_updateconfig_latency.avg
      operator: ">="
      value: 100
    - metric_name: http_request_method_deleteconfig_latency.max
      operator: ">"
      value: 125
      consecutive: 5

Review Sample Configuration for Multi-Conditions

Review the following sample configuration: 

  - name: cluster-errors-multi-threshold
    type: and
    interval: 1m
    conditions: 
    - metric_name: http_request_method_updateconfig_latency.avg
      operator: ">="
      value: 100
    - metric_name: http_request_method_deleteconfig_latency.max
      operator: ">"
      value: 125
      consecutive: 5  

Create and Manage a Threshold 

To create and manage a threshold, you must populate a YAML file.

To access the YAML file for a new configuration:

  1. In the Edge Delta App, on the left-side navigation, click Data Pipeline, and then click Agent Settings.

  2. Click Create Configuration.

  3. Click YAML.

  4. Enter your desired parameters, and then click Save.  

To access the YAML file for an existing configuration:

  1. In the Edge Delta App, on the left-side navigation, click Data Pipeline, and then click Agent Settings.

  2. Locate the desired configuration, then under Actions, click the vertical ellipses, and then click Edit

  3. Review the YAML file, make your changes, and then click Save.


 

Share this document