Overview
You can use this document to learn about the configuration parameters available in a configuration file, specifically for Filters.
You can use a filter to discard unnecessary logs or protect sensitive data. In other words, you can use a filter to refine and transform collected logs before additional processing takes place. As a result, filters can help reduce the agent's resource load because of the reduced log ingestion.
Note
Some input types offer additional filtering options that you can use to complement the generic filters in this page.
For example, when you create a Kubernetes input, you can add Kubernetes-specific filters to the input, such as filter for namespaces or pods. Additionally, you can also use the filters listed in this page to add to the Kubernetes input.
To learn more about inputs, see Inputs.
Review Supported Filters Types
The Edge Delta App supports the following filter types:
APM |
This filter type can be used to process Elastic APM logs. To learn more, see APM Filters. |
Base64 |
This filter type can be used to decode base64 encoding. To learn more, see Base64 Filters. |
Buffered Trace |
This filter type handles trace logs.
To learn more, see Buffered Trace Filters. |
Combination |
This filter type allows you to combine already-existing filters into a single filter. To learn more, see Combination Filters. |
Custom Attributes |
This filter type can filter for custom attributes. To learn more, see Custom Attribute Filters. |
JSON Field Drop |
This filter type can filter and drop specified JSON fields from the incoming logs. To learn more, see JSON Field Drop Filters. |
JSON Field Extractor |
This filter type extracts a field's value and replaces the whole JSON content with the field's value. To learn more, see JSON Field Extractor Filters. |
Mask |
This filter type hides (or masks) specific data, based on the configured regex pattern. To lean more, see Mask Filters. |
OTLP (Open Telemetry) |
This filter type can be used to process OTLP (Open Telemetry) logs. To learn more, see OTLP (Open Telemetry) Filters. |
Regex |
This filter type passes all log lines that match the specified regular expression. All unmatched logs are discarded. To learn more, see Regex Filters. |
Split Lines |
This filter type can be used to match, then split a single log into multiple logs. To learn more, see Split Lines Filters. |
Create and Manage a Filter
To create and manage a filter, you must populate a YAML file.
To access the YAML file for a new configuration:
-
In the Edge Delta App, on the left-side navigation, click Data Pipeline, and then click Agent Settings.
-
Click Create Configuration.
-
Click YAML.
- Enter your desired parameters, and then click Save.
- To learn about these configurations, see Review Filter Types.
- In a YAML file, filters are defined at the top level. Review the following example:
filters:
- name: error
type: regex
pattern: "error"
To access the YAML file for an existing configuration:
-
In the Edge Delta App, on the left-side navigation, click Data Pipeline, and then click Agent Settings.
-
Locate the desired configuration, then under Actions, click the vertical ellipses, and then click Edit.
-
Review the YAML file, make your changes, and then click Save.
- To learn about these configurations, see Review Filter Types.
- In a YAML file, filters are defined at the top level. Review the following example:
filters:
- name: error
type: regex
pattern: "error"
Understand the Workflow of a Filter
After you define a filter, filters can be referenced at different places in the YAML file:
- Input filters apply right after the data ingestion from the input, but before running the workflows associated with the input.
- Workflow filters apply before the processor runs within the workflow.
- Processor filters apply before the processor runs, regardless of which workflow the processor is running within.
Inputs
The following example displays a file input with error and mask_card filters:
inputs:
files:
- labels: "billing"
path: "/var/log/billing/*.log"
filters:
- error
- mask_card
To learn how inputs can be filtered, see Inputs.
Workflows
The following example displays a workflow with the error filter:
workflows:
application_workflow:
input_labels:
- system_stats
- agent_stats
- application_logs
filters:
- error
processors:
- error-check
- fail-check
- success-check
destinations:
- sumo-logic-devops-integration
- slack-devops-integration
To learn how workflows can be filtered, see Workflows.
Processors
The following example displays the Dimension Counter Processor with the not_debug filter.
regexes:
- name: "log"
pattern: "level=(?P<level>\\w+) "
dimensions: ["level"]
trigger_thresholds:
anomaly_probability_percentage: 90
filters:
- not_debug
To learn more, see Processors Overview.