Enrich Input Data

George Alpizar
George Alpizar
  • Updated

Overview

You can add enrichment configurations to an input to extract custom attributes from the source. Afterwards, the enriched data will be sent to the destination configured in the workflow.


Review Enrichment Types

Review the following table to understand how you can enrich data from inputs: 

Enrichment Type Description Supported Input Types
from_logs

from_logs is used to enrich data with fields extracted from specified logs.

The enriched data will be sent to the destination defined in the workflow. 

Under from_logs, you must specify a field_name, as well as a pattern or json_path

  • Use field_name to define the source field in the raw data.
  • Use pattern to define how to extract the raw data. 
    • In the example below, when podname is detected, any data after the colon ( : ) will be extracted. 
    • You must enter a capture pattern.
    • You can only enter one capture group.
  • Use json_path to enrich data with fields extracted from JSON logs.

Review the following example: 

        from_logs:
          field_mappings:
            - field_name: podname
              pattern: "podname: (\\w+)"
            - field_name: component
              json_path: fields.component
  • Files
  • Containers (Docker)
  • Ports
  • EDPort Collector
from_path

from_path is used to enrich data with fields extracted from a specified path.

The enriched data will be sent to the destination defined in the workflow. 

Under from_path, you must specify a field_name and a pattern.

  • Use field_name to define the source field in the raw data.
  • Use pattern to define how to extract the raw data. 
    • You must enter a capture pattern.
    • You can only enter one capture group.

Review the following example:

      enrichments:
        from_path:
          field_mappings:
            - field_name: application
              pattern: /var/logs/anyDir/(?:(.+)/)?users/.*
  • Files
  • ECS Containers
  • Kubernetes
  • Containers (Docker)
from_k8s

from_k8s is used to enrich streaming data with K8s attributes.

You can enter a pod, namespace, or node attributes.

Review the following example: 

        from_k8s:
          pod_identifier_pattern: /var/logs/anyDir/MyApp/users/(?:(.+)/)/.*
          field_mappings:
            - field_name: instance_id
              pod_attribute: pod
              transformers:
                - source: "-"
                  target: "_"
                  type: "replace"
                - source: "test*"
                  target: ""
                  type: "regex"
            - field_name: namespace
              pod_attribute: namespace
            - field_name: service
              pod_attribute: labels.service
  • Files
  • ECS Containers
  • Kubernetes
  • Containers (Docker)
dynamic

dynamic is used to enrich data from another source, field, or component to dynamically enrich the data. In other words, data will be pulled externally from the raw log message. 

You must enter a field_name and a value

For dynamic enrichment, consider the following statements:

  • Dynamic fields can be derived from other enrichment fields, as well as static fields.
  • Dependent fields should be ordered in a particular order.
    • For example, if field2 is dependent on field1, then you must define field1 before you define field2.

Review the following example of how to obtain data from a static field:

      enrichments:
        dynamic:
          field_mappings:
            - field_name: "app_name"
              value: "transaction_api"

Review the following example of how to obtain data from another dynamic enrichment field:

      enrichments:
        dynamic:
          field_mappings:
            - field_name: "service_base_url"
              value: "https://api.mycompany.com"
            - field_name: "tansaction_api_url"
              value: " {{ .service_base_url }}/transaction "

Review the following example of how to obtain data from another enrichment source (Kubernetes): 

  • Annotation keys are specific to the container in the pod.
  • In value, enter the annotation key. 
      enrichments:
        dynamic:
          field_mappings:
            - field_name: "source"
              value: `.annotations.kubernetes.io/{{.container_name}}.logs`,

Review the following example of how to obtain data from an AWS EC2 instance: 

  • To obtain data from an AWS EC2 instance, in the value parameter, you must enter aws-instance
      enrichments:
        dynamic:
          field_mappings:
            - field_name: "instance_id"
              value: '{{".aws-instance.instance-id"}}'
            - field_name: "instance_type"
              value: '{{".aws-instance.instance-type"}}'

For additional information, please review the Retrieve instance metadata document from Amazon. 

  • Files
  • ECS Containers
  • Kubernetes
  • Containers (Docker)

 


Troubleshoot Mapping Failures

To troubleshoot potential mapping failures, you can configure the failure_behavior parameter. 

Review the following options:

failure_behavior option Description
stop_enrichment This option will terminate the enrichment.
drop_source This option will terminate the enrichment and also stop tailing the source.
skip_failing_fields

This option will skip over the detected failed mapping in the enrichment.

This option is the default value.

 

Additionally, you can use the fallback_value parameter to troubleshoot. Specifically, if mapping fails based on the value or json_path parameter, then the configured value for fallback_value will be used until the agent confirms that the mapping has failed. 

Review the following example: 

      enrichments:
        failure_behavior: stop_enrichment
        dynamic:
          field_mappings:
            - field_name: "service"
              value: '{{".labels.service"}}'
            - field_name: "source"
              value: '.annotations.kubernetes.io/{{.container_name}}.logs'
              json_path: "[0].source"
              fallback_value: '{{".short_container_image"}}'

 

Share this document