Agent Settings

George Alpizar
George Alpizar
  • Updated

Overview

You can use this document to learn about the configuration parameters available in a configuration file, specifically for Agent Settings.

Review Parameters for Agent Settings

Visual Editor YAML Description Default Value Required or Optional 
Anomaly Capture Size anomaly_capture_size This parameter represents the number of log lines (buffer size) to capture during an anomaly capture. 125 Optional
Anomaly Capture Byte Size anomaly_capture_bytesize This parameter represents the maximum buffer size (in bytes) to capture during an anomaly capture. 0b (Disabled) Optional
Anomaly Capture Duration anomaly_capture_duration This parameter represents the maximum time span that the logs of an anomaly capture can belong to, such as logs from the last 10 minutes. 0s (Disabled) Optional
Anomaly Coefficient anomaly_coefficient This parameter represents the anomaly coefficient used to multiply the final score to a range of 0 - 100.

The higher the coefficient, the higher the anomaly score will be.

For some rules types, this parameter can be set at the rule level.		 10 Optional
Anomaly Confidence Period anomaly_confidence_period
This parameter represents a grace period between when a configuration is updated, which will restart the agent, and when data will flow into the app. 
 
Anomaly scores will all be zero while baselines are established.
For some rules types, this parameter can be set at the rule level.
 30M Optional
Anomaly Tolerance anomaly_tolerance

When it is non-zero, anomaly scores handle edge cases better when standard deviation is too small.

For some rules types, this parameter can be set at the rule level.

 0.01 Optional
Archive archive_destination

This parameter allows an existing integration to stream logs to an S3. endpoint.

For example, if enabled, the AWS S3 integration will stream logs to an AWS S3 endpoint.

   Optional
Attributes attributes

This parameter defines a user-defined, key-value pair that is used to label and distinguish different running agents.

These key-value pairs are attached to the data that is collected and generated by the agent and sent to streaming destinations.

You can set keywords for app, environment, and region.

   Optional
Capture Flush Mode capture_flush_mode

This parameter sets the behavior for flushing captured contextual log buffers.

This parameter supports the following modes:

  • local_per_source flushes the captured buffer of a source when a local alert is triggered from the same source.
  • local_all flushes all captured buffers when a local alert is triggered, not necessarily from the same source. As a result, when an alert is triggered from an agent, all captured buffers from all active sources will be flushed.
  • tag_per_source flushes the captured buffer of a source when an alert is triggered from the same source and tag, including any agent within the current tag.
  • tag_all flushes all captured buffers for all agents within the same tag when any agent triggers an alert.
  • custom_local_per_group flushes specified custom groups that should flush together if one of the groups triggers an alert.
 local_per_source Optional
Log log

This parameter contains additional subfields to configure an agent's log settings.

Specifically, you can configure the following parameters:

Level

  • Use this parameter to organize the entires in your log file.   
  • Review the following example: 
agent_settings:
  tag: prod
  log:
    level: debug

Secure Logging 

  • Use this parameter to hide sensitive data from the specified agent logs, such as API keys, secrets, and authentication information. 

Note

When an agent runs inside a container, such as Kubernetes, the agent logs to a standard output.

When an agent runs as a Linux, Windows, or macOS service, the agent logs to a file named edgedelta.log next to the installed service location.

 Not applicable  Optional
Multiline Byte Size multiline_max_bytesize

This parameter configures the buffer byte size for multiline accumulation.

 10 KB Optional
Multiline Count multiline_max_size

This parameter defines the buffer length size for multiline accumulation.

If there is an overflow of lines, then those extra lines are dumped as a single line.

 250 Optional
Report Only Non-Zero Stats only_report_nonzeros

This parameter configures if non-zero stats should be reported or not.

For some rules types, this parameter can be set at the rule level.

 False Optional
Skip Empty Intervals skip_empty_intervals

This parameter configures if empty intervals should be skipped so that anomaly scores are calculated based on non-zero intervals.

For some rules types, this parameter can be set at the rule level.

 False Optional
Soft CPU Limit soft_cpu_limit

This parameter allows you to use more CPU than what is specified in the allocation.

This parameter is only honored by the clustering processor at the moment. 0.5 means 50% of a core.

This parameter complements the cpu_friendly parameter for Processors. 

To enable, in the clustering rule, set cpu_friendly=true.

 0.0 Optional
Tag tag

This parameter is a user-defined tag used to describe the environment, such as prod_us_west_2_cluster.

While the default value is Edge, Edge Delta recommends that you set a value.

 Edge Optional 
Not applicable max_file_per_glob_path

Enter the maximum number of files to tail, per glob path.

 100 Optional
Not applicable forget_file_after

Enter a length of time to drop files that have not been modified.

 1h Optional
Not applicable total_seek_capacity

Enter the maximum size that tailers can seek concurrently.

 5MB Optional
Not applicable max_seek_size

Enter the maximum size that a tailer can seek, per second.

   Optional
Not applicable source_discovery_interval

Enter how often the source discovery is invoked.

 5s Optional
Not applicable file_tailer_buffer_size:

Enter the maximum number of logs that a file tailer can store in its memory until the logs are ingested by the agent’s internal router.

If the router is busy and cannot ingest the logs and the tailer’s buffer is reached, then the seeking will be blocked.

 1000 Optional
Not applicable router_per_source_buffer_size

Enter the maximum number of logs that an agent’s internal router can store in its memory, per source.

 1000 Optional
Not applicable archive_flush_interval

Enter a time frame to flush and send logs to a configured archiving destination.

 

 5m Optional
Not applicable archive_max_byte_limit

Enter the maximum number of bytes that can be buffered (in memory) before a flush is triggered to an archive destination.

 26MB Optional

 

Review Parameters for Archive

This parameter allows an existing integration to stream logs to an S3 endpoint.

For example, if enabled, the AWS S3 integration will stream logs to an AWS S3 endpoint.

Review the following parameters that you can configure in the Edge Delta App:

Visual Editor YAML Description Required or Optional
Name name Enter a descriptive name for the archive destination.  Required
Aws Key Id aws_key_id Enter the AWS key ID for the IAM user that has bucket access.  Required
Aws Security Key  aws_sec_key Enter the AWS security key for the IAM user that has bucket access.  Required
S3 Bucket s3_bucket Enter the name of the S3 bucket.  Required
S3 Region  s3_region Enter the region of the S3 bucket.  Required
Size size Enter a buffer size that should be reached before the data is sent to be archived.  Optional
Compress compress

Select (or enter) a compression method. 

Currently, gzip is supported. 

 Optional
Path path Enter a path to temporarily store buffered data before the data is archived.  Optional

 

Review the following YAML example of Archive:

  archive:  # archives all the logs to an s3 bucket with proper date/time folder structure.
    name: s3
    aws_key_id: '{{ Env "TEST_AWS_KEY_ID" }}'
    aws_sec_key: "awssecret123"
    s3_bucket: "testbucket"
    s3_region: "us-east-2"
    size: 16MB
    compress: gzip
    buffer:
      # setting buffer's path will enable file buffering to reduce agent's memory usage.
      path: "/var/log/edgedelta/archive"

Review Parameters for Attributes

Review the following parameters that you can configure in the Edge Delta App:

Visual Editor YAML Description Required or Optional
App app Enter a descriptive label that will be used to enrich data generated by the agent.  Optional
Environment environment Enter a descriptive label that will be used to enrich data generated by the agent.  Optional
Region region Enter a descriptive label that will be used to enrich data generated by the agent.  Optional

Review the following YAML example of Attributes: 

  attributes:
    environment: prod
    app: smp
    region: us-west

Review YAML Example of Agent Settings

agent_settings:
  tag: prod_payments
  log:
    level: info
  persisting_cursor_settings:
    path: /var/lib/edgedelta/cursor_provider
    file_name: cursor_provider.json
    flush_interval: 5s
  soft_cpu_limit: 0.5
  anomaly_tolerance: 0.1
  anomaly_confidence_period: 1m
  skip_empty_intervals: false
  only_report_nonzeros: false
  anomaly_capture_size: 1000
  anomaly_capture_bytesize: "10 KB"
  anomaly_capture_duration: 1m
  anomaly_coefficient: 10.0
  grace_period: 30s

Create and Manage Agent Settings

At a high level, there are 2 ways to manage Agent Settings:

  • If you need to create a new configuration, then you can use the visual editor to populate a YAML file, as well as make changes directly in the YAML file.
  • If you already have an existing configuration, then you can update the configuration in the YAML file.

To access the visual editor for a new configuration:

  1. In the Edge Delta App, on the left-side navigation, click Data Pipeline, and then click Agent Settings.
  2. Click Create Configuration.
  3. Click Visual.
  4. On the right-side, select Agent Settings.
  5. Review the list of options.

To access the YAML file for an existing configuration:

  1. In the Edge Delta App, on the left-side navigation, click Data Pipeline, and then click Agent Settings.
  2. Locate the desired configuration, then under Actions, click the vertical ellipses, and then click Edit.
  3. Review the YAML file.

 

Share this document