EDPort Collector Inputs

George Alpizar
George Alpizar
  • Updated

Overview

This input type allows you to specify a set of ports and protocols for the agent to listen on for incoming traffic.


Review Parameters

Review the following parameters that you can configure in the Edge Delta App:

Visual Editor YAML Description
Labels labels

Enter a descriptive name for this input. 

When you create a workflow, you will use this label to enter your input into the workflow. 

This parameter is required. 

Review the following example: 

labels: "error-counts-per-node"
Port port

Enter the port that the agent should listen for. 

This parameter is required. 

Review the following example: 

port: 4545
Protocol protocol

Select (or enter):

  • tcp
  • http
  • https

This parameter is required. 

Review the following example: 

protocol: http
Listen listen

Enter a network interface where the agent can listen for data. 

This parameter is optional. 

Review the following example: 

listen: localhost
Read Timeout  read_timeout

Enter a maximum time to wait and listen for data. 

This parameter only applies if you select tcp as the protocol

This parameter is required. 

Review the following example: 

read_timeout: 30s
Schema schema

You can only select (or enter) FlattenedObservation.

This parameter is required. 

Review the following example: 

schema: FlattenedObservation
Read Size  read_size

Enter the number of lines to read from the incoming data. 

If you do not enter a value, then 1 will be used. 

This parameter only applies if you select tcp as the protocol

This parameter is optional. 

Review the following example: 

read_size: 10000
Enable Incoming Line Anomalies 
enable_incoming_line_anomalies

Mark (or enter) true or false to generate anomaly scores. 

This parameter is optional. 

Review the following example: 

enable_incoming_line_anomalies: true
Enrichments

enrichments

You can use this parameter to enrich data with specified extracted fields. 

To learn how to enrich data from inputs, see Enrich Input Data.

This parameter is optional. 

Review the following example: 

enrichments:
from_logs: field_mappings: - field_name: podname pattern: "podname: (\\w+)" - field_name: component json_path: fields.[1].component
Add Ingestion Timestamp 

add_ingestion_time

Mark (or enter) true or false to ingest a timestamp if the input format is in JSON.

This parameter is optional. 

Review the following example: 

add_ingestion_time: true
Skip Ingestion Timestamp On Failure 
skip_ingestion_time_on_failure

Mark (or enter) true or false to skip the ingestion of the timestamp when the input is broken or in an invalid format.  

This parameter is optional. 

Review the following example: 

skip_ingestion_time_on_failure: true
Filters filters

Select an existing filter to add to this input. 

To learn more, see Filters.

This parameter is optional. 

Review the following example: 

filters: 
- info
Disable Verify 

tls:

  disable_verify

To disable a TLS verification of a certificate:

  • In the visual editor, select True.
  • In the YAML file, enter: disable_verify: true.

To enable a TLS verification of the certificate:

  • In the visual editor, select False.
  • In the YAML file, you can enter disable_verify: false or you can remove this line entirely. 

This parameter is optional. 

Review the following example: 

tls:
  disable_verify: true
CA File Path 

tls:

  ca_file_path

Enter the absolute file path to the CA certificate file.

This parameter is optional. 

Review the following example: 

tls:
  ca_path: /var/etc/kafka
CA Path 

tls:

  ca_path

Enter the absolute path to scan the CA certificate file.

This parameter is optional. 

Review the following example: 

tls:
  ca_file: /certs/ca.pem
CRT File 

tls:

  crt_path

Enter the absolute path to the certificate file. 

This parameter is optional. 

Review the following example: 

tls:
  crt_file: /certs/server-cert.pem
Key File 

tls:

  key_file

Enter the absolute path to the private key file. 

This parameter is optional. 

Review the following example: 

tls:
  key_file: /certs/server-key.pem
Key Password 

tls:

  key_password

Enter the password for the key file.

This parameter is optional. 

Review the following example: 

tls:
  key_password: p@ssword123
Client Auth Type 

tls:

  client_auth_type

Select a client authorization type. 

You can select (or enter)

  • noclientcert
  • requestclientcert
  • requireanyclientcert
  • verifyclientcertifgiven
  • requireandverifyclientcert

The default setting is noclientcert.

This parameter is optional. 

Review the following example: 

tls:
client_auth_type: noclientcert
Minimum Version

tls:

  min_version

Enter the minimum version of TLS to accept. 

This parameter is optional. 

Review the following example: 

tls:
min_version: TLSv1_1
Maximum Version

tls:

  max_version

Enter the maximum version of TLS to accept. 

This parameter is optional. 

Review the following example: 

tls:
max_version: TLSv1_3
Source Type

source_detection:

   source_type

Enter a source type.

Within inputs, a source type tells the Edge Delta agent which specific stream to monitor and extra logs from.

You can select (or enter) Docker, ECS, File, K8s. or Custom

This parameter is optional. 

Review the following example: 

source_detection:
  source_type: "K8s"
Optional

source_detection:

   optional

Enter true or false to ingest (or discard) logs with a failed source detection. 

Enter true to ingest logs with the original source information, despite a failed source detection.

Enter false to discard logs with a failed source detection.

This parameter is optional. 

Review the following example: 

optional: false
Not applicable 

source_detection:

  processing_mode

There are 2 types of processing modes:

  • json
  • regex

If you enter json, then you must enter the JSON path as a value for each field mapping. 

If you enter regex, then you must enter a regex pattern with one capturing group named field, such as

  • "path (?P<field>\w+)" 

This parameter is optional. 

Review the following example:

source_detection:
        source_type: "Custom"
        optional: false
        processing_mode: regex
        field_mappings:
          namespace: namespace (?P<field>\w+)
          serviceName: service (?P<field>\w+)
          roleName: user_role (?P<field>\w+)
          systemType: system (?P<field>\w+)

 

Field Mappings

source_detection:

   field_mappings

This parameter is optional. 

Review the following example: 

source_detection:
  field_mappings:
    k8s_namespace: "kubernetes.namespace"
    k8s_pod_name: "kubernetes.pod.name"
    k8s_container_name: "kubernetes.container.name"
    k8s_container_image: "kubernetes.container.image"

Review Sample Configuration 

Review the following sample configuration: 

  ed_ports:
    - labels: "error-counts-per-node"
      port: 4545
      protocol: http
      schema: FlattenedObservation
    - labels: "errorcheck"
      port: 9000
      protocol: tcp
      read_size: 10000
      read_timeout: 30s
      source_detection:
        source_type: "K8s" 
        optional: true
        field_mappings:
          k8s_namespace: "kubernetes.namespace"
          k8s_pod_name: "kubernetes.pod.name"
          k8s_container_name: "kubernetes.container.name"
          k8s_container_image: "kubernetes.container.image"
    - labels: "ed-port-with-auto-detect-line-pattern"
      port: 5656
      protocol: tcp
      auto_detect_line_pattern: true
    - labels: "ed-port-with-given-line-pattern"
      port: 9091
      protocol: http
      line_pattern: '^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}'
      late_arrival_handling:
        rule_metrics:
          ignore_after: 10m
        patterns:
          ignore_after: 3h
          report_with_original_timestamp: true
    - labels: "ed-port-with-network-interface"
      port: 4545
      protocol: tcp
      listen: 127.0.0,1
    - labels: "ed-port-tcp-with-tls"
      port: 4545
      protocol: tcp
      tls:
        crt_file: /certs/server-cert.pem
        key_file: /certs/server-key.pem
        ca_file: /certs/ca.pem
    - labels: "ed-port-https-with-tls"
      protocol: https
      listen: localhost
      port: 443
      tls:
        crt_file: /certs/server-cert.pem
        key_file: /certs/server-key.pem
        ca_file: /certs/ca.pem
    - labels: "ed-port-with-add-ingestion-time"
      port: 4545
      protocol: tcp
      add_ingestion_time: true
      skip_ingestion_time_on_failure: true # skip ingestion time when the input is broken or invalid format.

 

Share this document