Overview
You can use this document to learn about the configuration parameters available in a configuration file, specifically for Agent Settings.
Review Parameters for Agent Settings
Visual Editor | YAML | Description | Default Value | Required or Optional |
Anomaly Capture Size | anomaly_capture_size | This parameter represents the number of log lines (buffer size) to capture during an anomaly capture. | 125 | Optional |
Anomaly Capture Byte Size | anomaly_capture_bytesize | This parameter represents the maximum buffer size (in bytes) to capture during an anomaly capture. | 0b (Disabled) | Optional |
Anomaly Capture Duration | anomaly_capture_duration | This parameter represents the maximum time span that the logs of an anomaly capture can belong to, such as logs from the last 10 minutes. | 0s (Disabled) | Optional |
Anomaly Coefficient | anomaly_coefficient | This parameter represents the anomaly coefficient used to multiply the final score to a range of 0 - 100. The higher the coefficient, the higher the anomaly score will be. For some rules types, this parameter can be set at the rule level. |
10 | Optional |
Anomaly Confidence Period | anomaly_confidence_period | 30M | Optional | |
Anomaly Tolerance | anomaly_tolerance |
When it is non-zero, anomaly scores handle edge cases better when standard deviation is too small. For some rules types, this parameter can be set at the rule level. |
0.01 | Optional |
Archive | archive_destination |
This parameter allows an existing integration to stream logs to an S3. endpoint. For example, if enabled, the AWS S3 integration will stream logs to an AWS S3 endpoint.
|
Optional | |
Attributes | attributes |
This parameter defines a user-defined, key-value pair that is used to label and distinguish different running agents. You can set keywords for app, environment, and region.
|
Optional | |
Capture Flush Mode | capture_flush_mode |
This parameter sets the behavior for flushing captured contextual log buffers. This parameter supports the following modes:
|
local_per_source | Optional |
Log | log |
This parameter contains additional subfields to configure an agent's log settings. Specifically, you can configure the following parameters: Level
agent_settings: tag: prod log: level: debug Secure Logging
|
Not applicable | Optional |
Multiline Byte Size | multiline_max_bytesize |
This parameter configures the buffer byte size for multiline accumulation. |
10 KB | Optional |
Multiline Count | multiline_max_size |
This parameter defines the buffer length size for multiline accumulation. If there is an overflow of lines, then those extra lines are dumped as a single line. |
250 | Optional |
Report Only Non-Zero Stats | only_report_nonzeros |
This parameter configures if non-zero stats should be reported or not. For some rules types, this parameter can be set at the rule level. |
False | Optional |
Skip Empty Intervals | skip_empty_intervals |
This parameter configures if empty intervals should be skipped so that anomaly scores are calculated based on non-zero intervals. For some rules types, this parameter can be set at the rule level. |
False | Optional |
Soft CPU Limit | soft_cpu_limit |
This parameter allows you to use more CPU than what is specified in the allocation. This parameter is only honored by the clustering processor at the moment. 0.5 means 50% of a core. This parameter complements the cpu_friendly parameter for Processors. To enable, in the clustering rule, set cpu_friendly=true. |
0.0 | Optional |
Tag | tag |
This parameter is a user-defined tag used to describe the environment, such as prod_us_west_2_cluster. While the default value is Edge, Edge Delta recommends that you set a value. |
Edge | Optional |
Not applicable | max_file_per_glob_path |
Enter the maximum number of files to tail, per glob path. |
100 | Optional |
Not applicable | forget_file_after |
Enter a length of time to drop files that have not been modified. |
1h | Optional |
Not applicable | total_seek_capacity |
Enter the maximum size that tailers can seek concurrently. |
5MB | Optional |
Not applicable | max_seek_size |
Enter the maximum size that a tailer can seek, per second. |
Optional | |
Not applicable | source_discovery_interval |
Enter how often the source discovery is invoked. |
5s | Optional |
Not applicable | file_tailer_buffer_size: |
Enter the maximum number of logs that a file tailer can store in its memory until the logs are ingested by the agent’s internal router. If the router is busy and cannot ingest the logs and the tailer’s buffer is reached, then the seeking will be blocked. |
1000 | Optional |
Not applicable | router_per_source_buffer_size |
Enter the maximum number of logs that an agent’s internal router can store in its memory, per source. |
1000 | Optional |
Not applicable | archive_flush_interval |
Enter a time frame to flush and send logs to a configured archiving destination.
|
5m | Optional |
Not applicable | archive_max_byte_limit |
Enter the maximum number of bytes that can be buffered (in memory) before a flush is triggered to an archive destination. |
26MB | Optional |
Review Parameters for Archive
This parameter allows an existing integration to stream logs to an S3 endpoint.
For example, if enabled, the AWS S3 integration will stream logs to an AWS S3 endpoint.
Review the following parameters that you can configure in the Edge Delta App:
Visual Editor | YAML | Description | Required or Optional |
Name | name | Enter a descriptive name for the archive destination. | Required |
Aws Key Id | aws_key_id | Enter the AWS key ID for the IAM user that has bucket access. | Required |
Aws Security Key | aws_sec_key | Enter the AWS security key for the IAM user that has bucket access. | Required |
S3 Bucket | s3_bucket | Enter the name of the S3 bucket. | Required |
S3 Region | s3_region | Enter the region of the S3 bucket. | Required |
Size | size | Enter a buffer size that should be reached before the data is sent to be archived. | Optional |
Compress | compress |
Select (or enter) a compression method. Currently, gzip is supported. |
Optional |
Path | path | Enter a path to temporarily store buffered data before the data is archived. | Optional |
Review the following YAML example of Archive:
archive: # archives all the logs to an s3 bucket with proper date/time folder structure. name: s3 aws_key_id: '{{ Env "TEST_AWS_KEY_ID" }}' aws_sec_key: "awssecret123" s3_bucket: "testbucket" s3_region: "us-east-2" size: 16MB compress: gzip buffer: # setting buffer's path will enable file buffering to reduce agent's memory usage. path: "/var/log/edgedelta/archive"
Review Parameters for Attributes
Review the following parameters that you can configure in the Edge Delta App:
Visual Editor | YAML | Description | Required or Optional |
App | app | Enter a descriptive label that will be used to enrich data generated by the agent. | Optional |
Environment | environment | Enter a descriptive label that will be used to enrich data generated by the agent. | Optional |
Region | region | Enter a descriptive label that will be used to enrich data generated by the agent. | Optional |
Review the following YAML example of Attributes:
attributes: environment: prod app: smp region: us-west
Review YAML Example of Agent Settings
agent_settings:
tag: prod_payments
log:
level: info
persisting_cursor_settings:
path: /var/lib/edgedelta/cursor_provider
file_name: cursor_provider.json
flush_interval: 5s
soft_cpu_limit: 0.5
anomaly_tolerance: 0.1
anomaly_confidence_period: 1m
skip_empty_intervals: false
only_report_nonzeros: false
anomaly_capture_size: 1000
anomaly_capture_bytesize: "10 KB"
anomaly_capture_duration: 1m
anomaly_coefficient: 10.0
grace_period: 30s
Create and Manage Agent Settings
At a high level, there are 2 ways to manage Agent Settings:
- If you need to create a new configuration, then you can use the visual editor to populate a YAML file, as well as make changes directly in the YAML file.
- If you already have an existing configuration, then you can update the configuration in the YAML file.
To access the visual editor for a new configuration:
- In the Edge Delta App, on the left-side navigation, click Data Pipeline, and then click Agent Settings.
- Click Create Configuration.
- Click Visual.
- On the right-side, select Agent Settings.
- Review the list of options.
To access the YAML file for an existing configuration:
- In the Edge Delta App, on the left-side navigation, click Data Pipeline, and then click Agent Settings.
- Locate the desired configuration, then under Actions, click the vertical ellipses, and then click Edit.
- Review the YAML file.