Kafka Inputs

George Alpizar
George Alpizar
  • Updated

Overview

This input type allows Edge Delta agent will collect events from a Kafka topic.


Review Parameters

Review the following parameters that you can configure in the Edge Delta App:

Visual Editor YAML Description
Kafka Input Labels  labels

Enter a descriptive name for this input. 

When you create a workflow, you will use this label to enter your input into the workflow. 

This parameter is required.

Review the following example:

- labels: "my-kafka-events"
Kafka Endpoints  endpoint

Enter your Kafka broker address.

This parameter is required.

Review the following example:

endpoint: "something-tls"
Kafka Topic  topic

Enter your Kafka topic name.

This parameter is required.

Review the following example:

topic: "topic"
Kafka Consumer Group  group_id

Enter a consumer group within the specified topic name. 

This parameter is required.

Review the following example:

group_id: "my-group"
Filters filters

Select an existing filter to add to this input. 

To learn more, see Filters.

This parameter is optional. 

Review the following example: 

filters: 
- info
Source Type

source_detection:

   source_type

Enter a source type.

Within inputs, a source type tells the Edge Delta agent which specific stream to monitor and extra logs from.

You can select (or enter) Docker, ECS, File, K8s. or Custom

This parameter is optional. 

Review the following example: 

source_detection:
  source_type: "K8s"
Optional

source_detection:

   optional

Enter true or false to ingest (or discard) logs with a failed source detection. 

Enter true to ingest logs with the original source information, despite a failed source detection.

Enter false to discard logs with a failed source detection.

This parameter is optional. 

Review the following example: 

optional: false
Not applicable 

source_detection:

  processing_mode

There are 2 types of processing modes:

  • json
  • regex

If you enter json, then you must enter the JSON path as a value for each field mapping. 

If you enter regex, then you must enter a regex pattern with one capturing group named field, such as

  • "path (?P<field>\w+)" 

This parameter is optional. 

Review the following example:

source_detection:
        source_type: "Custom"
        optional: false
        processing_mode: regex
        field_mappings:
          namespace: namespace (?P<field>\w+)
          serviceName: service (?P<field>\w+)
          roleName: user_role (?P<field>\w+)
          systemType: system (?P<field>\w+)

 

Field Mappings

source_detection:

   field_mappings

This parameter is optional. 

Review the following example: 

source_detection:
  field_mappings:
    namespace: "kubernetes.namespace"
    serviceName: "service"
    roleName: "user.role"
    systemType: "system"
Username

sasl:

  username

Enter your Kafka SASL username.

This parameter is optional.

Review the following example:

sasl:
  username: kafka_username
Password

sasl:

  password

Enter your Kafka SASL password.

This parameter is optional.

Review the following example:

sasl:
  password: p@ssword123
Mechanism

sasl:

  mechanism

Enter a Kafka SASL mechanism type to implement a secure authentication.

You can enter: 

  • PLAIN
  • SCRAM-SHA-256
  • SCRAM-SHA-512

This parameter is optional.

Review the following example:

sasl:
  mechanism: PLAIN 
Disable Verify 

tls:

  disable_verify

To disable a TLS verification of a certificate:

  • In the visual editor, select True.
  • In the YAML file, enter: disable_verify: true.

To enable a TLS verification of the certificate:

  • In the visual editor, select False.
  • In the YAML file, you can enter disable_verify: false or you can remove this line entirely. 

This parameter is optional. 

Review the following example: 

tls:
  disable_verify: true
CA File Path 

tls:

  ca_file

Enter the absolute file path to the CA certificate file.

This parameter is optional. 

Review the following example: 

tls:
  ca_file: /var/etc/kafka
CA Path 

tls:

  ca_path

Enter the absolute path to scan the CA certificate file.

This parameter is optional. 

Review the following example: 

tls:
  ca_path: /certs/ca.pem
CRT File 

tls:

  crt_file

Enter the absolute path to the certificate file. 

This parameter is optional. 

Review the following example: 

tls:
  crt_file: /certs/server-cert.pem
Key File 

tls:

  key_file

Enter the absolute path to the private key file. 

This parameter is optional. 

Review the following example: 

tls:
  key_file: /certs/server-key.pem
Key Password 

tls:

  key_password

Enter the password for the key file.

This parameter is optional. 

Review the following example: 

tls:
  key_password: p@ssword123
Client Auth Type 

tls:

 client_auth_type

Select a client authorization type. 

You can select (or enter)

  • noclientcert
  • requestclientcert
  • requireanyclientcert
  • verifyclientcertifgiven
  • requireandverifyclientcert

The default setting is noclientcert.

This parameter is optional. 

Review the following example: 

tls:
client_auth_type: noclientcert
Minimum Version

tls:

  min_version

Enter the minimum version of TLS to accept. 

This parameter is optional. 

Review the following example: 

tls:
min_version: TLSv1_1
Maximum Version

tls:

  max_version

Enter the maximum version of TLS to accept. 

This parameter is optional. 

Review the following example: 

tls:
max_version: TLSv1_3

Review Sample Configuration

Review the following sample configuration:

  kafkas:
    - labels: "errorcheck"
      endpoint: "something"
      topic: "topic"
    - labels: "errorcheck-tls"
      endpoint: "something-tls"
      topic: "topic"
      tls:
        disable_verify: true
        ca_file: /var/etc/kafka/ca_file
        ca_path: /var/etc/kafka
        crt_file: /var/etc/kafka/crt_file
        key_file: /var/etc/kafka/keyfile
        key_password: p@ssword123
        client_auth_type: noclientcert 
        min_version: TLSv1_1
        max_version: TLSv1_3
    - labels: "my-kafka-events"
      endpoint: "something"
      topic: "topic"
      group_id: "my-group"
      sasl:
        username: kafka_username
        password: p@ssword123
        mechanism: PLAIN # possible selections: PLAIN, SCRAM-SHA-256, SCRAM-SHA-512
      source_detection:
        source_type: "Custom"
        optional: false
        field_mappings:
          namespace: "kubernetes.namespace"
          serviceName: "service"
          roleName: "user.role"
          systemType: "system"

 

Share this document