Files Inputs

Overview

This input type allows you to specify a set of files for Edge Delta to monitor.

Review Sample Configuration

Review the following sample configuration:

 files:
    - labels: "billing,errorcheck"
      path: "/billing/logfolder1/*.log"
    - labels: "billing,errorcheck"
      path: "/etc/systemd/system/billingservice/*.log"
      exclude:
        - "/etc/systemd/system/billingservice/test.log"
        - "/etc/systemd/system/billingservice/dev.log"
      auto_detect_line_pattern: true
      boost_stacktrace_detection: true
      enable_persisting_cursor: true
      filters:
        - info
        - not_trace
        - mask_card
        - mask_password
    - labels: "docker,my_container"
      path: "/var/lib/docker/my_container/*.log"
      docker_mode: true
    - labels: "app,service_a"
      path: "/var/log/service_a.log"
      line_pattern: "^MMM dd, yyyy hh:mm:ss"
      late_arrival_handling:
        rule_metrics:
          ignore_after: 15m
        patterns:
          ignore_after: 4h
          report_with_original_timestamp: true
      filters:
      - source-detection-docker
    - labels: k8s_log
      path: /var/logs/anyDir/MyApp/users/MyPodID/transaction.log
      filters:
      - enrichment-full
    - labels: "billing,errorcheck"
      path: "/billing/logfolder1/*.log"
      separate_source: true  
    - labels: "billing,errorcheck"
      path: "/billing/logfolder1/*.log"
      add_ingestion_time: true
      skip_ingestion_time_on_failure: true 

Review Parameters

Review the following parameters that you can configure in the Edge Delta App.

labels

Required

Enter a descriptive label for this input. 

When you create a workflow, you will use this label to enter your input into the workflow. 

billing,errorcheck is the default label. 

Review the following example: 

labels: "billing,errorcheck"

path

Required

Enter the full path to the file (or files) that you want Edge Delta to monitor. 

• Wildcards are supported.
• If you want the agent to process lines for a specific line separation rule (not for New Line("\n")), then you need to define a "line_pattern" regex rule.

Review the following example: 

path: "/etc/systemd/system/billingservice/*.log"

exclude

Optional 

Enter a path (in regex) to exclude matched patterns.

Review the following example: 

    - labels: "billing,errorcheck"
      path: "/etc/systemd/system/billingservice/*.log"
      exclude:
        - "/etc/systemd/system/billingservice/test.log"
        - "/etc/systemd/system/billingservice/dev.log"

auto_detect_line_pattern

Optional 

Enter true (or false) to enable auto line detection for log messages with multiple lines. 

Review the following example: 

auto_detect_line_pattern: true

boost_stracktrace_detection

Optional 

Enter true (or false) to fallback (troubleshoot) missing content that resulted from the Auto Line Detection parameter. 

Review the following example: 

boost_stacktrace_detection: true

separate_source

Optional 

Enter true (or false) to have the agent treat each file that matches the glob expression as a separate source. 

By default, this option is disabled; all files that match the glob path will be treated as a single source. 

Review the following example: 

separate_source: true 

filters

Optional 

Enter an existing filter to add to this input. 

To learn how to create a filter, see Filters.

Review the following example: 

filters:
  - info
  - not_trace
  - mask_card
  - mask_password

add_ingestion_time

Optional

Enter true (or false) to ingest a timestamp if the input format is in JSON.

Review the following example: 

add_ingestion_time: true

skip_ingestion_time_on_failure

Optional 

Enter true or false to skip the ingestion of the timestamp when the input is broken or in an invalid format. 

Review the following example: 

skip_ingestion_time_on_failure: true

late_arrival_handling

Optional 

You can use this parameter to configure how to accommodate delayed data. 

Specifically, you can use the ignore_after parameter to configure when to ignore metrics or pattern logs.

For example, for ignore_after, if you enter 15m, then the input will ignore logs whose timestamp is older than 15 minutes. 

Review the following example:  

late_arrival_handling:
  rule_metrics:
    ignore_after: 15m
  patterns:
    ignore_after: 4h
    report_with_original_timestamp: true

docker_mode

Optional 

Enter true (or false) to collect Docker container standard output logs on a file with the JSON file logging driver.

To learn more, review this document from Docker.

Review the following example: 

docker_mode: true