Filters Overview

Overview

You can use this document to learn about the configuration parameters available in a configuration file, specifically for Filters.

You can use a filter to discard unnecessary logs or protect sensitive data. In other words, you can use a filter to refine and transform collected logs before additional processing takes place.  As a result, filters can help reduce the agent's resource load because of the reduced log ingestion.

Note

Some input types offer additional filtering options that you can use to complement the generic filters in this page. 

For example, when you create a Kubernetes input, you can add Kubernetes-specific filters to the input, such as filter for namespaces or pods. Additionally, you can also use the filters listed in this page to add to the Kubernetes input. 

To learn more about inputs, see Inputs.

Review Supported Filters Types

The Edge Delta App supports the following filter types:

Create and Manage a Filter

To create and manage a filter, you must populate a YAML file.

To access the YAML file for a new configuration:

1. In the Edge Delta App, on the left-side navigation, click Data Pipeline, and then click Agent Settings.

2. Click Create Configuration.

3. Click YAML.

4. Enter your desired parameters, and then click Save.  
   • To learn about these configurations, see Review Filter Types.
   • In a YAML file, filters are defined at the top level. Review the following example:

filters:
  - name: error
    type: regex
    pattern: "error"

To access the YAML file for an existing configuration:

1. In the Edge Delta App, on the left-side navigation, click Data Pipeline, and then click Agent Settings.

2. Locate the desired configuration, then under Actions, click the vertical ellipses, and then click Edit. 

3. Review the YAML file, make your changes, and then click Save.

   • To learn about these configurations, see Review Filter Types.
   • In a YAML file, filters are defined at the top level. Review the following example:

filters:
  - name: error
    type: regex
    pattern: "error"

Understand the Workflow of a Filter

After you define a filter, filters can be referenced at different places in the YAML file:

• Input filters apply right after the data ingestion from the input, but before running the workflows associated with the input.
• Workflow filters apply before the processor runs within the workflow.
• Processor filters apply before the processor runs, regardless of which workflow the processor is running within.

Inputs

The following example displays a file input with error and mask_card filters:

  inputs:
    files:
      - labels: "billing"
        path: "/var/log/billing/*.log"
        filters:
          - error
          - mask_card

To learn how inputs can be filtered, see Inputs.

Workflows

The following example displays a workflow with the error filter:

workflows:
  application_workflow:
    input_labels:
      - system_stats
      - agent_stats
      - application_logs
    filters:
      - error
    processors:
      - error-check
      - fail-check
      - success-check
    destinations:
      - sumo-logic-devops-integration
      - slack-devops-integration

To learn how workflows can be filtered, see Workflows.

Processors

The following example displays the Dimension Counter Processor with the not_debug filter.

regexes:
  - name: "log"
    pattern: "level=(?P<level>\\w+) "
    dimensions: ["level"]
    trigger_thresholds:
      anomaly_probability_percentage: 90
    filters:
      - not_debug

To learn more, see Processors Overview.