Simple Keyword Match (Regex) Processors

George Alpizar
George Alpizar
  • Updated

Overview

This processor type:

  • Checks for basic regex matches in logs, then
  • Counts the matching logs, and then 
  • Generates anomaly scores.

The counts are reported by the agent, based on the specified internal and emitted as metrics.

Note

You can use thresholds to receive a notification when anomalies occur, such as spikes in metric value. 

To learn more about thresholds, see Thresholds.


Review Sample Configuration

For the sample configuration below, the following metrics are generated: 

  • error.count
    • This metric is the total count of matches within an interval.
  • error.anomaly1
    • This metric is the anomaly score of the current interval, based on the total count history.
    • This metric represents how anomalous the current error count is compared to its history.
    • This metric can range from 0 to 100. 
regexes:
  - name: "error"
    pattern: "error|err|ERROR|ERR"
    trigger_thresholds:
      anomaly_probability_percentage: 90

    - name: "high_severity"
      interval: 2m
      retention: 4h
      anomaly_confidence_period: 1h
      anomaly_tolerance: 0.2
      only_report_nonzeros: true
      description: "Counts of messages including error per 2 minutes."
      trigger_thresholds:
        anomaly_probability_percentage: 90 
        upper_limit_per_interval: 250 
        consecutive: 5 
      pattern: "HIGH|high"
      filters:
        - extract_severity  

Review Parameters

Review the following parameters that you can configure in the Edge Delta App.


name

Required

Enter a descriptive label for this processor. 

When you create a workflow, you will use this label to enter your processor into the workflow.

Review the following example:  

name: "error-regex"

pattern

Required

Enter a regular expression to match patterns in a string.

The regular expression pattern must follow Golang regex protocol, such as error|ERROR.

Review the following example: 

pattern: "error|ERROR|problem|ERR|Err"

dimensions

Optional

This parameter lists fields of named capture groups to use as dynamic dimensions (to group by).

For each dimension that you specify, you must have a corresponding named capture group in the pattern field for the processor.

Review the following example:  

dimensions: ["method"]

dimensions_as_attributes

Optional

Enter true or false to to send dimension key/value pairs as attributes.

If you enter false, then the dimension key/value pairs will be appended to the metric name. 

Note

If you enable this parameter, then you can specify the Dimensions Groups  parameter.

Review the following example:

dimensions_as_attributes: true

dimensions_groups

Optional

Define the attributes that you want to group together for metrics. 

Note

To define this parameter, you must set the dimensions_as_attributes parameter to true.

Review the following example: 

dimensions_groups:
  - selected_dimensions: ["method", "code"]
  - selected_dimensions: ["method", "httpversion"]

enabled_stats

Optional 

This parameter specifies the data generated from a regex rule. 

You can obtain the following values:

  • count
  • min
  • max
  • avg
  • anomaly1
  • anomaly2 

Review the following YAML example: 

enabled_stats: ["count", "anomalymin"]

Review the following PCT example: 

func allStats() []StatType {
    return []StatType{
	Anomaly1, Anomaly2, Avg, Count, Max, Min, P25, P75, P95, P99, StdDev, Sum,

interval

Optional

This parameter is a golang duration string that represents the reporting (or rollup) interval for the generated statistics.

The default value is 1m.

Review the following example:  

interval: 2m

retention

Optional

This parameter is a golang duration string that represents how far back the agent should look when generating anomaly scores.

The default value is 3h.

Review the following example:

retention: 4h

filters

Optional

Enter an existing filter to add to this processor. 

To learn how to create a filter, see Filters.

Review the following example:  

filters:
- extract_severity

trigger_thresholds

Optional

This parameter defines threshold limits, based on calculated metrics.

When a threshold is reached, the agent will notify the corresponding trigger destinations in the same workflow.

You can configure the following threshold types:

  • anomaly_probability_percentage
  • upper_limit_per_interval
  • lower_limit_per_interval
  • consecutive

Review the following example: 

trigger_thresholds:
  anomaly_probability_percentage: 90
upper_limit_per_interval: 250
consecutive: 5

anomaly_probability_percentage (trigger_thresholds)

Optional

This parameter sets the confidence level / probability of an anomaly that needs to be reached to trigger an alert. 

For example, if you enter 90, then an alert will trigger when there is a 90% probability that the detected pattern is an anomaly. 

Enter a number between 0 and 100.

There is no default value. 

Review the following example:  

trigger_thresholds: 
anomaly_probability_percentage: 90

upper_limit_per_interval (trigger_thresholds)

Optional

This parameter sets a static threshold to trigger an alert.  

If the number of events that match the given pattern for the most recent reporting interval is greater than the limit, then an alert will be triggered.

There is no default value. 

Review the following example:  

trigger_thresholds: 
upper_limit_per_interval: 250

lower_limit_per_interval (trigger_thresholds)

Optional

This parameter sets a static threshold to trigger an alert.

If the number of events that match the given pattern for the most recent reporting interval is less than the limit, then an alert will trigger.

There is no default value. 

Review the following example: 

trigger_thresholds: 
lower_limit_per_interval: 10

consecutive (trigger_thresholds)

Optional

This parameter sets how many consecutive times a threshold must be exceeded to trigger an alert.  

The default value is 0, which means that any condition that is met will trigger an alert. 

Review the following example:

trigger_thresholds: 
consecutive: 5

 


 

Share this document