Overview
The Kafka output will stream analytics and insights to your Kafka endpoint.
Review Sample Configuration
The following sample configuration displays an output without the name of the organization-level integration:
- name: kafka
type: kafka
endpoint: localhost:2888,localhost:3888 # brokers
topic: example_kafka_topic
required_acks: 10
batch_size: 1000
batch_bytes: 10000
batch_timeout: 1m
async: true
features: log,metric
tls:
disable_verify: true
ca_file: /var/etc/kafka/ca_file
ca_path: /var/etc/kafka
crt_file: /var/etc/kafka/crt_file
key_file: /var/etc/kafka/keyfile
key_password: p@ssword123
client_auth_type: noclientcert
sasl:
username: kafka_username
password: p@ssword123
mechanism: PLAIN
Review Parameters
Review the following parameters that you can configure in the Edge Delta App.
name
Required
Enter a descriptive name for the output or integration.
For outputs, this name will be used to map this destination to a workflow.
Review the following example:
name: kafka
integration_name
Optional
This parameter refers to the organization-level integration created in the Integrations page.
If you need to add multiple instances of the same integration into the config, then you can add a custom name to each instance via the name parameter. In this situation, the name should be used to refer to the specific instance of the destination in the workflows.
Review the following example:
integration_name: orgs-kafka
type
Required
Enter kafka.
Review the following example:
type: kafka
endpoint
Required
Enter your Kafka broker address.
Review the following example:
endpoint: localhost:2888,localhost:3888
topic
Required
Enter your Kafka topic name.
Review the following example:
topic: example_kafka_topic
required_acks
Optional
Enter the number of acknowledgments that the leader must receive before considering a request to be complete.
- To learn more, review this article from Kafka.
Review the following example:
required_acks: 10
batch_size
Optional
Enter the maximum number of messages to buffer being being sent to a partition.
The default limit is 100 messages.
Review the following example:
batch_size: 1000
batch_bytes
Optional
Enter a limit (in bytes) for the maximum size of a request before being sent to a partition.
The default value is 1048576.
Review the following example:
batch_bytes: 10000
batch_timeout
Optional
Enter a time limit for how often incomplete message batches will be flushed to Kafka.
Review the following example:
batch_timeout: 1m
async
Optional
Enter true or false to enable or disable asynchronous communication between Edge Delta and Kafka.
Review the following example:
async: true
tls: disable_verify
Optional
To disable a TLS verification of a certificate, in the YAML file, enter:
- disable_verify:true.
To enable a TLS verification of the certificate, in the YAML file, you can enter disable_verify:false or you can remove this line entirely.
Review the following example:
tls: disable_verify: true
tls: ca_file
Optional
Enter the absolute file path to the CA certificate file.
Review the following example:
tls:
ca_file: /var/etc/cribl/ca_file
tls: ca_path
Optional
Enter the absolute path to scan the CA certificate file.
Review the following example:
tls:
ca_path: /var/etc/cribl
tls: crt_file
Optional
Enter the absolute path to the certificate file.
Review the following example:
tls:
crt_file: /var/etc/kafka/crt_file
tls: key_file
Optional
Enter the absolute path to the private key file.
Review the following example:
tls: key_file: /certs/server-key.pem
tls: key_password
Optional
Enter the password for the key file.
Review the following example:
tls: key_password: p@ssword123
tls: client_auth_type
Optional
Enter a client authorization type.
You can enter:
- noclientcert
- requestclientcert
- requireanyclientcert
- verifyclientcertifgiven
- requireandverifyclientcert
The default setting is noclientcert.
Review the following example:
tls:
client_auth_type: noclientcert
tls: min_version
Optional
Enter the minimum version of TLS to accept.
Review the following example:
tls:
min_version: TLSv1_1
tls: max_version
Optional
Enter the maximum version of TLS to accept.
Review the following example:
tls:
max_version: TLSv1_3
sasl: username
Optional
Enter your Kafka SASL username.
Review the following example:
sasl: username: kafka_username
sasl: password
Optional
Enter your Kafka SASL password.
Review the following example:
sasl:
password: p@ssword123
sasl: mechanism
Optional
Enter a Kafka SASL mechanism type to implement a secure authentication.
You can enter:
- PLAIN
- SCRAM-SHA-256
- SCRAM-SHA-512
Review the following example:
sasl:
mechanism: PLAIN
features
Optional
This parameter defines which data types to stream to the destination.
To learn more, review the Review Feature Types section in Stream Outputs and Integrations Overview.
Review the following example:
features: log,edac,metric,alert
buffer_ttl
Optional
Enter a length of time to retry failed streaming data.
After this length of time is reached, the failed streaming data will no longer be tried.
Review the following example:
buffer_ttl: 2h
buffer_path
Optional
Enter a folder path to temporarily store failed streaming data.
The failed streaming data will be retried until the data reaches its destinations or until the Buffer TTL value is reached.
If you enter a path that does not exist, then the agent will create directories, as needed.
Review the following example:
buffer_path: /var/log/edgedelta/pushbuffer/
buffer_max_bytesize
Optional
Enter the maximum size of failed streaming data that you want to retry.
If the failed streaming data is larger than this size, then the failed streaming data will not be retried.
Review the following example:
buffer_max_bytesize: 100MB
Quick links