Overview
This processor monitors top K records, such as k=10, where the records are identified with one or more named regex group values combined together
This processor reports the top k items as a string value.
Review Parameters
Review the following parameters that you can configure in the Edge Delta App:
Visual Editor | YAML | Description |
Name | name |
Enter a descriptive label for this processor. When you create a workflow, you will use this label to enter your processor into the workflow in the visual editor. This parameter is required. Review the following example: name: top-api-requests |
Pattern | pattern |
Enter a regular expression to match patterns in a string. Logs that match this pattern will be selected, and the named groups combined together will be the key of the record. This parameter is required. Review the following example: pattern: (?P<ip>\d+\.\d+\.\d+\.\d+) - \w+ \[.*\] "(?P<method>\w+) (?P<path>.+) HTTP\/\d.0" (?P<code>.+) \d+ |
Interval | interval |
This parameter is a golang duration string that represents the reporting (or rollup) interval for the generated statistics. The default value is 1m. This parameter is required. Review the following example: interval: 30s |
K | k |
Enter an integer value that specifies how many top records to monitor at every interval. Records are ordered by their count Records are ordered by their count in a descending order, and the top k items are picked for reporting. This parameter is required. Review the following example: k: 10 |
Lower Limit | lower_limit |
If you enter a lower limit, then only the records with counts larger than the limit will become a top k. This parameter is required. Review the following example: lower_limit: 1 |
Separator | separator |
Enter a separator type to combine together the named group values to form a record key. The default setting is a comma ( , ). This parameter is required. Review the following example: separator: "," |
Filters | filters |
Select an existing filter to add to this processor. To learn how to create a filter, see Filters. This parameter is optional. Review the following example: filters: |
Review Sample Configuration
Review the following sample configuration:
top_ks:
- name: top-api-requests
# logs matching this pattern will be selected and named groups combined together will be the key of the record for which we keep counter.
# example log to match below rule: "12.195.88.88 - joe [08/Aug/2020:05:57:49 +0000] "GET /optimize/engage HTTP/1.0" 200 19092"
pattern: (?P<ip>\d+\.\d+\.\d+\.\d+) - \w+ \[.*\] "(?P<method>\w+) (?P<path>.+) HTTP\/\d.0" (?P<code>.+) \d+
# every interval the top records will be reported and they will be removed locally
interval: 30s
# records are ordered by their count in a descending order and top k items are picked for reporting.
k: 10
# if a lower limit is provided only records whose count is greater than the limit will be able to make it to top k.
lower_limit: 1
# separator is used to combine the named group values together to form a record key. Default is comma ','.
# For example,
# the pattern above would generate a record key like this "12.195.88.88,GET,/optimize/engage,200"
# let's say this record has been seen 5 times in last period and it was one of the top k items. then below log will be reported:
# "12.195.88.88,GET,/optimize/engage,200=5"
separator: ","
Review the following sample log:
"12.195.88.88 - joe [08/Aug/2020:05:57:49 +0000] "GET /optimize/engage HTTP/1.0" 200 19092"
The pattern above would generate the following record key:
- "12.195.88.88,GET,/optimize/engage,200"
As an example, if this record was seen 5 times in the last period and the record was a top k item, then this processor would have generated the following log and send the log to the workflow's destinations:
- "12.195.88.88,GET,/optimize/engage,200=5"