Overview
This input type allows Edge Delta agent will collect events from a Kafka topic.
Review Parameters
Review the following parameters that you can configure in the Edge Delta App:
Visual Editor | YAML | Description |
Kafka Input Labels | labels |
Enter a descriptive name for this input. When you create a workflow, you will use this label to enter your input into the workflow. This parameter is required. Review the following example: - labels: "my-kafka-events" |
Kafka Endpoints | endpoint |
Enter your Kafka broker address. This parameter is required. Review the following example: endpoint: "something-tls" |
Kafka Topic | topic |
Enter your Kafka topic name. This parameter is required. Review the following example: topic: "topic" |
Kafka Consumer Group | group_id |
Enter a consumer group within the specified topic name. This parameter is required. Review the following example: group_id: "my-group" |
Filters | filters |
Select an existing filter to add to this input. To learn more, see Filters. This parameter is optional. Review the following example: filters: |
Source Type |
source_detection: source_type |
Enter a source type. Within inputs, a source type tells the Edge Delta agent which specific stream to monitor and extra logs from. You can select (or enter) Docker, ECS, File, K8s. or Custom. This parameter is optional. Review the following example: source_detection: source_type: "K8s" |
Optional |
source_detection: optional |
Enter true or false to ingest (or discard) logs with a failed source detection. Enter true to ingest logs with the original source information, despite a failed source detection. Enter false to discard logs with a failed source detection. This parameter is optional. Review the following example: optional: false |
Not applicable |
source_detection: processing_mode |
There are 2 types of processing modes:
If you enter json, then you must enter the JSON path as a value for each field mapping. If you enter regex, then you must enter a regex pattern with one capturing group named field, such as
This parameter is optional. Review the following example: source_detection: source_type: "Custom" optional: false processing_mode: regex field_mappings: namespace: namespace (?P<field>\w+) serviceName: service (?P<field>\w+) roleName: user_role (?P<field>\w+) systemType: system (?P<field>\w+)
|
Field Mappings |
source_detection: field_mappings |
This parameter is optional. Review the following example: source_detection: field_mappings: namespace: "kubernetes.namespace" serviceName: "service" roleName: "user.role" systemType: "system" |
Username |
sasl: username |
Enter your Kafka SASL username. This parameter is optional. Review the following example: sasl: username: kafka_username |
Password |
sasl: password |
Enter your Kafka SASL password. This parameter is optional. Review the following example: sasl: password: p@ssword123 |
Mechanism |
sasl: mechanism |
Enter a Kafka SASL mechanism type to implement a secure authentication. You can enter:
This parameter is optional. Review the following example: sasl: mechanism: PLAIN |
Disable Verify |
tls: disable_verify |
To disable a TLS verification of a certificate:
To enable a TLS verification of the certificate:
This parameter is optional. Review the following example: tls: disable_verify: true |
CA File Path |
tls: ca_file |
Enter the absolute file path to the CA certificate file. This parameter is optional. Review the following example: tls: ca_file: /var/etc/kafka |
CA Path |
tls: ca_path |
Enter the absolute path to scan the CA certificate file. This parameter is optional. Review the following example: tls: ca_path: /certs/ca.pem |
CRT File |
tls: crt_file |
Enter the absolute path to the certificate file. This parameter is optional. Review the following example: tls: crt_file: /certs/server-cert.pem |
Key File |
tls: key_file |
Enter the absolute path to the private key file. This parameter is optional. Review the following example: tls: key_file: /certs/server-key.pem |
Key Password |
tls: key_password |
Enter the password for the key file. This parameter is optional. Review the following example: tls: key_password: p@ssword123 |
Client Auth Type |
tls: client_auth_type |
Select a client authorization type. You can select (or enter)
The default setting is noclientcert. This parameter is optional. Review the following example: tls: |
Minimum Version |
tls: min_version |
Enter the minimum version of TLS to accept. This parameter is optional. Review the following example: tls: |
Maximum Version |
tls: max_version |
Enter the maximum version of TLS to accept. This parameter is optional. Review the following example: tls: |
Review Sample Configuration
Review the following sample configuration:
kafkas:
- labels: "errorcheck"
endpoint: "something"
topic: "topic"
- labels: "errorcheck-tls"
endpoint: "something-tls"
topic: "topic"
tls:
disable_verify: true
ca_file: /var/etc/kafka/ca_file
ca_path: /var/etc/kafka
crt_file: /var/etc/kafka/crt_file
key_file: /var/etc/kafka/keyfile
key_password: p@ssword123
client_auth_type: noclientcert
min_version: TLSv1_1
max_version: TLSv1_3
- labels: "my-kafka-events"
endpoint: "something"
topic: "topic"
group_id: "my-group"
sasl:
username: kafka_username
password: p@ssword123
mechanism: PLAIN # possible selections: PLAIN, SCRAM-SHA-256, SCRAM-SHA-512
source_detection:
source_type: "Custom"
optional: false
field_mappings:
namespace: "kubernetes.namespace"
serviceName: "service"
roleName: "user.role"
systemType: "system"