Files Inputs

George Alpizar
George Alpizar
  • Updated

Overview

This input type allows you to specify a set of files for Edge Delta to monitor.


Review Parameters

Review the following parameters that you can configure in the Edge Delta App:

Visual Editor YAML Description
File Labels labels

Enter a descriptive label for this input. 

When you create a workflow, you will use this label to enter your input into the workflow. 

billing,errorcheck is the default label. 

This parameter is required. 

Review the following example: 

labels: "billing,errorcheck"
File Path

path

Enter the full path to the file (or files) that you want Edge Delta to monitor. 

  • Wildcards are supported.
  • If you want the agent to process lines for a specific line separation rule (not for New Line("\n")), then you need to define a "line_pattern" regex rule.

This parameter is required. 

Review the following example: 

path: "/etc/systemd/system/billingservice/*.log"
Not applicable 

Exclude

Enter a path (in regex) to exclude matched patterns.

This parameter is optional. 

Review the following example: 

    - labels: "billing,errorcheck"
      path: "/etc/systemd/system/billingservice/*.log"
      exclude:
        - "/etc/systemd/system/billingservice/test.log"
        - "/etc/systemd/system/billingservice/dev.log"
Auto Line Detection 

auto_detect_line_pattern

Enter true (or false) to enable auto line detection for log messages with multiple lines. 

This parameter is optional. 

Review the following example: 

auto_detect_line_pattern: true
Boost Stacktrace Detection 

boost_stracktrace_detection

Enter true (or false) to fallback (troubleshoot) missing content that resulted from the Auto Line Detection parameter. 

This parameter is optional. 

Review the following example: 

boost_stacktrace_detection: true
Separate Source 

separate_source

Enter true (or false) to have the agent treat each file that matches the glob expression as a separate source. 

By default, this option is disabled; all files that match the glob path will be treated as a single source. 

This parameter is optional. 

Review the following example: 

separate_source: true 
Filters

filters

Select (or enter) an existing filter to add to this input. 

To learn how to create a filter, see Filters.

This parameter is optional. 

Review the following example: 

filters:
  - info
  - not_trace
  - mask_card
  - mask_password
Enrichments

enrichments

You can use this parameter to enrich data with specified extracted fields. 

To learn how to enrich data from inputs, see Enrich Input Data.

This parameter is optional. 

Review the following example: 

   - labels: k8s_log
      path: /var/logs/anyDir/MyApp/users/MyPodID/transaction.log
      enrichments:
        from_path:
          field_mappings:
            - field_name: application
              pattern: /var/logs/anyDir/(?:(.+)/)?users/.*
Add Ingestion Timestamp 

add_ingestion_time

Mark (or enter) true or false to ingest a timestamp if the input format is in JSON.

This parameter is optional. 

Review the following example: 

- labels: "billing,errorcheck"
      path: "/billing/logfolder1/*.log"
      add_ingestion_time: true
      skip_ingestion_time_on_failure: true 
Skip Ingestion Timestamp On Failure 

skip_ingestion_time_on_failure

Mark (or enter) true or false to skip the ingestion of the timestamp when the input is broken or in an invalid format. 

This parameter is optional. 

Review the following example: 

skip_ingestion_time_on_failure: true
Metric Late Arrival Settings 

late_arrival_handling

You can use this parameter to configure how to accommodate delayed data. 

Specifically, you can use the ignore_after parameter to configure when to ignore metrics or pattern logs.

For example, for ignore_after, if you enter 15m, then the input will ignore logs whose timestamp is older than 15 minutes. 

This parameter is optional. 

Review the following example:  

late_arrival_handling:
  rule_metrics:
    ignore_after: 15m
  patterns:
    ignore_after: 4h
    report_with_original_timestamp: true
Pattern Late Arrival Settings 

late_arrival_handling

You can use this parameter to configure how to accommodate delayed data. 

Specifically, you can use the ignore_after parameter to configure when to ignore metrics or pattern logs.

For example, for ignore_after, if you enter 15m, then the input will ignore logs whose timestamp is older than 15 minutes. 

This parameter is optional. 

Review the following example:

      late_arrival_handling:
        rule_metrics:
          ignore_after: 15m
        patterns:
          ignore_after: 4h
          report_with_original_timestamp: true
Source Type 

source_type

Select (or enter) a source type.

Within inputs, a source type tells the Edge Delta agent which specific stream to monitor and extra logs from. 

You can select (or enter) Docker, ECS, File, K8s. or Custom

This parameter is optional. 

Review the following example: 

source_detection:
  source_type: "Docker"
  optional: false
  field_mappings:
    docker_container_id: "docker.id"
    docker_container_image: "docker.image"
Optional

optional

Enter true or false to ingest (or discard) logs with a failed source detection. 

Enter true to ingest logs with the original source information, despite a failed source detection.

Enter false to discard logs with a failed source detection.

This parameter is optional. 

Review the following example: 

optional: false
Not applicable 

source_detection:

  processing_mode

There are 2 types of processing modes:

  • json
  • regex

If you enter json, then you must enter the JSON path as a value for each field mapping. 

If you enter regex, then you must enter a regex pattern with one capturing group named field, such as

  • "path (?P<field>\w+)" 

This parameter is optional. 

Review the following example:

source_detection:
        source_type: "Custom"
        optional: false
        processing_mode: regex
        field_mappings:
          namespace: namespace (?P<field>\w+)
          serviceName: service (?P<field>\w+)
          roleName: user_role (?P<field>\w+)
          systemType: system (?P<field>\w+)

 

Not applicable

docker_mode

Enter true (or false) to collect Docker container standard output logs on a file with the JSON file logging driver.

To learn more, review this document from Docker.

This parameter is optional. 

Review the following example: 

    - labels: "docker,my_container"
      path: "/var/lib/docker/my_container/*.log"
      docker_mode: true

Review Sample Configuration

Review the following sample configuration:

  files:
    - labels: "billing,errorcheck"
      path: "/billing/logfolder1/*.log"
    - labels: "billing,errorcheck"
      path: "/etc/systemd/system/billingservice/*.log"
      auto_detect_line_pattern: true
      boost_stacktrace_detection: true
      enable_persisting_cursor: true
      filters:
        - info
        - not_trace
        - mask_card
        - mask_password
    - labels: "docker,my_container"
      path: "/var/lib/docker/my_container/*.log"
      docker_mode: true
    - labels: "app,service_a"
      path: "/var/log/service_a.log"
      line_pattern: "^MMM dd, yyyy hh:mm:ss"
      late_arrival_handling:
        rule_metrics:
          ignore_after: 15m
        patterns:
          ignore_after: 4h
          report_with_original_timestamp: true
      source_detection:
        source_type: "Docker"
        optional: false
        field_mappings:
          docker_container_id: "docker.id"
          docker_container_image: "docker.image"

 

Share this document