Numeric Capture (Regexes) Processors

George Alpizar
George Alpizar
  • Updated

Overview

This processor type:

  • Checks logs for matches for regular expression with numeric capture group, then
  • Counts the matching logs, and then 
  • Generates anomaly scores.

This processor type is used for numerical statistics, such as response duration, payload size, etc.


Review Parameters

Review the following parameters that you can configure in the Edge Delta App:

Visual Editor YAML Description
Name name

Enter a descriptive label for this processor. 

When you create a workflow, you will use this label to enter your processor into the workflow in the visual editor.

This parameter is required. 

Review the following example:

name: "flog"
Pattern pattern

Enter a regular expression pattern with a named group.

The matching part of the log will be extracted and converted to a floating point. 

Named capture groups must follow Golang regex protocol, such as:

  • "(?P<latency>\\d+)ms"

This parameter is required. 

Review the following examples:

pattern: " (?P<statuscode>\\d+) (?P<responsesize>\\d+)$"
pattern: " (\\d+)$"
Interval interval

This parameter is a golang duration string that represents the reporting (or rollup) interval for the generated statistics.

The default value is 1m.

This parameter is optional. 

Review the following example:

internval: 2m
Retention retention

golang duration string that represents how far back the agent should look when generating anomaly scores.

The default value is 3h.

This parameter is optional. 

Review the following example:

retention: 4h
Enabled Stats  enabled_stats

This parameter specifies the data generated from a regex rule. 

This parameter is optional. 

Review the following example:

enabled_stats: ["count", "anomalymin"]
Not applicable trigger_thresholds

This parameter defines threshold limits, based on calculated metrics.

When a threshold is reached, the agent notifies the corresponding trigger destinations in the same workflow.

You can configure the following trigger threshold types:

  • anomaly_probability_percentage
  • upper_limit_per_interval
  • lower_limit_per_interval
  • consecutive

This parameter is optional. 

Review the following example:

trigger_thresholds: 
anomaly_probability_percentage: 90
upper_limit_per_interval: 250
consecutive: 5
Anomaly Probability Percentage anomaly_probability_percentage (trigger_thresholds)

This parameter sets the confidence level / probability of an anomaly that needs to be reached to trigger an alert. 

For example, if you enter 90, then an alert will trigger when there is a 90% probability that the detected pattern is an anomaly. 

Enter a number between 0 and 100.

There is no default value. 

This parameter is optional.

Review the following example:  

trigger_thresholds: 
anomaly_probability_percentage: 90
Upper Limit Per Interval  upper_limit_per_interval (trigger_thresholds)

This parameter sets a static threshold to trigger an alert.  

If the number of events that match the given pattern for the most recent reporting interval is greater than the limit, then an alert will be triggered.

There is no default value. 

This parameter is optional.

Review the following example:  

trigger_thresholds: 
upper_limit_per_interval: 250
Lower Limit Per Interval  lower_limit_per_interval (trigger_thresholds)

This parameter sets a static threshold to trigger an alert.

If the number of events that match the given pattern for the most recent reporting interval is less than the limit, then an alert will trigger.

There is no default value. 

This parameter is optional.

Review the following example: 

trigger_thresholds: 
lower_limit_per_interval: 10
Consecutive consecutive (trigger_thresholds)

This parameter sets how many consecutive times a threshold must be exceeded to trigger an alert.  

The default value is 0, which means that any condition that is met will trigger an alert. 

This parameter is optional.

Review the following example:

trigger_thresholds: 
consecutive: 5
Filters filters

Select an existing filter to add to this processor. 

To learn how to create a filter, see Filters.

This parameter is optional. 

Review the following example:

filters:
- extract_severity
Not applicable value_adjustment_rules

This parameter allows you to create a multiplication or division rule per capture group. 

The rule must follow the "(*|/)" format where:

  • An asterisk ( * ) represents multiplication
  • A slash ( / ) represents denotes division

This parameter is optional. 

Review the following example:

value_adjustment_rules:
  responsesize:
    operator: "/"
    operand: 1000.0 

Review Sample Configuration

For the example configuration below, the following metrics are generated: 

  • response_time_latency.count
    • This metric is the total count of matches in the current interval.
  • response_time_latency.avg
    • This metric is the average of captured numeric values, such as average response time.
  • response_time_latency.min
    • This metric is the minimum of the captured numeric values.
  • response_time_latency.max
    • This metric is the maximum of the captured numeric values.
  • response_time_latency.anomaly1
    • This metric is the anomaly score, based on history of average values.
    - name: "flog"  
      pattern: " (?P\\d+) (?P\\d+)$" 
      value_adjustment_rules:
        responsesize:
          operator: "/"
          operand: 1000.0 
      trigger_thresholds:
        anomaly_probability_percentage: 1
    - name: "http-response-size"  
      pattern: " (\\d+)$"
      trigger_thresholds:
        anomaly_probability_percentage: 1

 

Share this document