Top-K Processors

George Alpizar
George Alpizar
  • Updated

Overview

This processor monitors top K records, such as k=10, where the records are identified with one or more named regex group values combined together

This processor reports the top k items as a string value. 


Review Sample Configuration

Review the following sample configuration: 

  top_ks:
    - name: top-api-requests
      pattern: (?P<ip>\d+\.\d+\.\d+\.\d+) - \w+ \[.*\] "(?P<method>\w+) (?P<path>.+) HTTP\/\d.0" (?P<code>.+) \d+
      interval: 30s
      k: 10
      lower_limit: 1
      separator: ","

Review the following sample log: 

"12.195.88.88 - joe [08/Aug/2020:05:57:49 +0000] "GET /optimize/engage HTTP/1.0" 200 19092"

The pattern above would generate the following record key:

  • "12.195.88.88,GET,/optimize/engage,200"

As an example, if this record was seen 5 times in the last period and the record was a top k item, then this processor would have generated the following log and send the log to the workflow's destinations:  

  • "12.195.88.88,GET,/optimize/engage,200=5"

Review Parameters

Review the following parameters that you can configure in the Edge Delta App.


name

Required

Enter a descriptive label for this processor. 

When you create a workflow, you will use this label to enter your processor into the workflow. 

Review the following example:

name: top-api-requests

pattern

Required

Enter a regular expression to match patterns in a string.

Logs that match this pattern will be selected, and the named groups combined together will be the key of the record.  

Review the following example:

pattern: (?P<ip>\d+\.\d+\.\d+\.\d+) - \w+ \[.*\] "(?P<method>\w+) (?P<path>.+) HTTP\/\d.0" (?P<code>.+) \d+

interval

Required

This parameter is a golang duration string that represents the reporting (or rollup) interval for the generated statistics.

The default value is 1m.

Review the following example:

interval: 30s

k

Required

Enter an integer value that specifies how many top records to monitor at every interval. 

Records are ordered by their count

Records are ordered by their count in a descending order, and the top k items are picked for reporting.

Review the following example:

k: 10

lower_limit

Required

If you enter a lower limit, then only the records with counts larger than the limit will become a top k.

Review the following example:

lower_limit: 1

separator

Required

Enter a separator type to combine together the named group values to form a record key. 

The default setting is a comma ( , ). 

Review the following example:

separator: ","

filters

Optional

Enter an existing filter to add to this processor. 

To learn how to create a filter, see Filters.

Review the following example:

filters:
- extract_severity

 


 

Share this document