Top-K Processors

George Alpizar
George Alpizar
  • Updated

Overview

This processor monitors top K records, such as k=10, where the records are identified with one or more named regex group values combined together

This processor reports the top k items as a string value. 

Review Parameters

Review the following parameters that you can configure in the Edge Delta App:

Visual Editor YAML Description
Name name

Enter a descriptive label for this processor. 

When you create a workflow, you will use this label to enter your processor into the workflow in the visual editor. 

This parameter is required. 

Review the following example:

name: top-api-requests
Pattern pattern

Enter a regular expression to match patterns in a string.

Logs that match this pattern will be selected, and the named groups combined together will be the key of the record.  

This parameter is required. 

Review the following example:

pattern: (?P<ip>\d+\.\d+\.\d+\.\d+) - \w+ \[.*\] "(?P<method>\w+) (?P<path>.+) HTTP\/\d.0" (?P<code>.+) \d+
Interval interval

This parameter is a golang duration string that represents the reporting (or rollup) interval for the generated statistics.

The default value is 1m.

This parameter is required. 

Review the following example:

interval: 30s
K k

Enter an integer value that specifies how many top records to monitor at every interval. 

Records are ordered by their count

Records are ordered by their count in a descending order, and the top k items are picked for reporting.

This parameter is required. 

Review the following example:

k: 10
Lower Limit lower_limit

If you enter a lower limit, then only the records with counts larger than the limit will become a top k.

This parameter is required. 

Review the following example:

lower_limit: 1
Separator separator

Enter a separator type to combine together the named group values to form a record key. 

The default setting is a comma ( , ). 

This parameter is required. 

Review the following example:

separator: ","
Filters filters

Select an existing filter to add to this processor. 

To learn how to create a filter, see Filters.

This parameter is optional. 

Review the following example:

filters:
  - extract_severity

Review Sample Configuration

Review the following sample configuration: 

  top_ks:
    - name: top-api-requests
      # logs matching this pattern will be selected and named groups combined together will be the key of the record for which we keep counter.
      # example log to match below rule: "12.195.88.88 - joe [08/Aug/2020:05:57:49 +0000] "GET /optimize/engage HTTP/1.0" 200 19092"
      pattern: (?P<ip>\d+\.\d+\.\d+\.\d+) - \w+ \[.*\] "(?P<method>\w+) (?P<path>.+) HTTP\/\d.0" (?P<code>.+) \d+
      # every interval the top records will be reported and they will be removed locally
      interval: 30s
      # records are ordered by their count in a descending order and top k items are picked for reporting.
      k: 10
      # if a lower limit is provided only records whose count is greater than the limit will be able to make it to top k.
      lower_limit: 1
      # separator is used to combine the named group values together to form a record key. Default is comma ','.
      # For example,
      #   the pattern above would generate a record key like this "12.195.88.88,GET,/optimize/engage,200"
      #   let's say this record has been seen 5 times in last period and it was one of the top k items. then below log will be reported:
      #   "12.195.88.88,GET,/optimize/engage,200=5"
      separator: ","

Review the following sample log: 

"12.195.88.88 - joe [08/Aug/2020:05:57:49 +0000] "GET /optimize/engage HTTP/1.0" 200 19092"

The pattern above would generate the following record key:

  • "12.195.88.88,GET,/optimize/engage,200"

As an example, if this record was seen 5 times in the last period and the record was a top k item, then this processor would have generated the following log and send the log to the workflow's destinations:  

  • "12.195.88.88,GET,/optimize/engage,200=5"

 

Share this document