Overview
You can add enrichment configurations to an input to extract custom attributes from the source. Afterwards, the enriched data will be sent to the destination configured in the workflow.
Review Enrichment Types
Review the following table to understand how you can enrich data from inputs:
Enrichment Type | Description | Supported Input Types |
from_logs |
from_logs is used to enrich data with fields extracted from specified logs. The enriched data will be sent to the destination defined in the workflow. Under from_logs, you must specify a field_name, as well as a pattern or json_path.
Review the following example: from_logs: field_mappings: - field_name: podname pattern: "podname: (\\w+)" - field_name: component json_path: fields.component |
|
from_path |
from_path is used to enrich data with fields extracted from a specified path. The enriched data will be sent to the destination defined in the workflow. Under from_path, you must specify a field_name and a pattern.
Review the following example: enrichments: from_path: field_mappings: - field_name: application pattern: /var/logs/anyDir/(?:(.+)/)?users/.* |
|
from_k8s |
from_k8s is used to enrich streaming data with K8s attributes. You can enter a pod, namespace, or node attributes. Review the following example: from_k8s: pod_identifier_pattern: /var/logs/anyDir/MyApp/users/(?:(.+)/)/.* field_mappings: - field_name: instance_id pod_attribute: pod transformers: # replace all "source" matches with "target" - source: "-" target: "_" type: "replace" # remove all "test" words - source: "test*" target: "" type: "regex" - field_name: namespace pod_attribute: namespace # fields from labels should have pod_attribute start with "labels." - field_name: service pod_attribute: labels.service |
|
dynamic |
dynamic is used to enrich data from another source, field, or component to dynamically enrich the data. In other words, data will be pulled externally from the raw log message. You must enter a field_name and a value. For dynamic enrichment, consider the following statements:
Review the following example of how to obtain data from a static field: enrichments: dynamic: field_mappings: - field_name: "app_name" value: "transaction_api" Review the following example of how to obtain data from another dynamic enrichment field: enrichments: dynamic: field_mappings: - field_name: "service_base_url" value: "https://api.mycompany.com" - field_name: "tansaction_api_url" value: " {{ .service_base_url }}/transaction " Review the following example of how to obtain data from another enrichment source (Kubernetes):
enrichments: dynamic: field_mappings: - field_name: "source" value: `.annotations.kubernetes.io/{{.container_name}}.logs`, Review the following example of how to obtain data from an AWS EC2 instance:
enrichments: dynamic: field_mappings: - field_name: "instance_id" value: '{{".aws-instance.instance-id"}}' - field_name: "instance_type" value: '{{".aws-instance.instance-type"}}' For additional information, please review the Retrieve instance metadata document from Amazon. |
|
Troubleshoot Mapping Failures
To troubleshoot potential mapping failures, you can configure the failure_behavior parameter.
Review the following options:
failure_behavior option | Description |
stop_enrichment | This option will terminate the enrichment. |
drop_source | This option will terminate the enrichment and also stop tailing the source. |
skip_failing_fields |
This option will skip over the detected failed mapping in the enrichment. This option is the default value. |
Additionally, you can use the fallback_value parameter to troubleshoot. Specifically, if mapping fails based on the value or json_path parameter, then the configured value for fallback_value will be used until the agent confirms that the mapping has failed.
Review the following example:
enrichments: failure_behavior: stop_enrichment dynamic: field_mappings: - field_name: "service" value: '{{".labels.service"}}' - field_name: "source" value: '.annotations.kubernetes.io/{{.container_name}}.logs' json_path: "[0].source" fallback_value: '{{".short_container_image"}}'
test