Understand Prometheus Scraping

Overview

You can use this document to learn about Edge Delta's ability to expose the metrics that will be scraped by Prometheus. This feature is useful for organizations that also want to keep their existing monitoring pipelines. 

Prometheus supports (exposes) the following metric types: 

• Counters
  • This metric type represents counts or sums that increase. Specifically, this metric type includes the rate of increase over a specified time. 
  • For example, this metric type can include errors from the last 10 seconds.
  • The Edge Delta agent exposes the _counts metric as counters.
• Gauges
  • This metric type represents sampled numeric values that may increase or decrease. 
  • For example, this metric type can include the number of processes, a temperature, or an Edge Delta anomaly score. 
  • The Edge Delta agent exposes the following metrics as gauges:
    • _avg
    • _min
    • _max
    • anomaly1

Review Examples of Prometheus Format

Exposed metrics have labels that are used in Prometheus queries to identify the metric sources.

Review the example of the anomaly1 gauge in the Prometheus format.

Specifically, for the example below: 

• provision_failure_anomaly1 is the metric name.
• metricType="stat" indicates a processor-collected metric.  
• 5 is the counter value.
• 1622638103171 is the timestamp, which is only available in gauges.
• The remaining labels indicate the metric source. In this example, the source is a K8s pod. 
• Counters also have same structure with gauges but metrics name ends with _count

In the following example, lines are intentionally split for readability:

provision_failure_anomaly1 {
  metricType="stat",
  tag="api-backend",
  workflow="api-workflow",
  sourceType="K8s",
  logicalSource="K8s,default,ReplicaSet,api-deployment,admin",
  pod_name="api-deployment-****",
  pod_id="****",
  region="us-****",
  controllerLogicalName="api-deployment",
  docker_id="d7****",
  namespace_name="default",
  controllerKind="ReplicaSet",
  labels_pod_template_hash="d4****",
  container_name="api",
  controllerName="api-deployment-d4****",
  environment="staging",
  container_image="****",
  host="****",
  labels_app="api"
} 5.000000 1622638103171

Review the example of a cluster counter in the Prometheus format.

If you enable the cluster processor in one or more of your workflows, then the clustered log pattern counts will be exposed as counters.

Specifically, for the example below: 

• metricType="cluster" indicates a cluster count.
• pattern="* DEBUG span* Operation getCachedOrgSetting latency *ms reqID * userID * X Forwarded For * X Host ID *" is the log pattern signature
• debug_span_operation_getcach... is the metric name, which is the Prometheus format compliant version of the pattern value.
• 1100 is the number of occurrences of the log pattern discovered on the source 

In the following example, lines are intentionally split for readability. Additionally, source labels are hidden:

debug_span_operation_getcachedorgsetting_latency_ms_reqid_userid_x_forwarded_for_x_host_id {
  tag="admin-api-backend",
  workflow="api-workflow",
  pattern="* DEBUG span* Operation getCachedOrgSetting latency *ms reqID * userID * X Forwarded For * X Host ID *",
  metricType="cluster",
  ...
} 1100.000000

Expose Agent Metrics

To expose metrics on the http://AGENT_IP:STORE_PORT/metrics HTTP endpoint in the Prometheus format, you must set the STORE_PORT environment variable on the Edge Delta agent.

• To learn more, see Review Environment Variables for Agent Installation.

When the metrics are exposed and accessible by Prometheus, metrics can be scraped like any other Prometheus metrics source.

Deploy Kubernetes Agent with Prometheus 

You can deploy the agent with or without Helm chart.

Optional 1: Deploy Agent Without Helm Chart

To install the Edge Delta agent as a DaemonSet on your Kubernetes cluster with Prometheus:

1. Follow the installation instructions in the Install the Agent for Kubernetes document.
2. When you reach Step 2: Advanced Installation Instructions, use the Prometheus-specific agent deployment manifest to export metrics.
   • Specifically, you will run the following command:

     kubectl apply -f https://edgedelta.github.io/k8s/edgedelta-prom-agent.yml
     

3. Continue with the rest of the the Install the Agent for Kubernetes document.

Optional 2: Deploy Agent With Helm Chart

If you use Prometheus Service Operator Helm chart to deploy Prometheus, you can also deploy the Edge Delta agent service monitor configuration to allow Prometheus to discover the agents automatically.

To install the Edge Delta agent as a DaemonSet on your Kubernetes cluster with Helm and with Prometheus:

1. Download the following file: https://edgedelta.github.io/k8s/edgedelta-prom-servicemonitor.yml
2. Based on the example below, review the Helm release name and the namespace of the Prometheus Operator deployment. Specifically, locate CHART starting with prometheus-operator:
   • In the following example, the Namespace is monitoring, and the release name is promop.

   • helm ls --all-namespaces
     Expected Output:
     
     NAME      NAMESPACE     REVISION    UPDATED                                 STATUS      CHART                        APP VERSION
     promop    monitoring    1           2020-08-28 19:49:30.516141 +0300 +03    deployed    prometheus-operator-9.3.1    0.38.1

3. If you used a different release name, then in the edgedelta-prom-servicemonitor.yml file, update promop with your Helm release name for Prometheus.

   release: promop

4. Locate the service monitor name prefix. To obtain the release name prefix for monitors, run the following command. If applicable, replace monitoring with your namespace.

   kubectl get servicemonitor -n monitoring

5. You should receive an output similar to the following example:

   NAME                                                 AGE
   promop-prometheus-operator-alertmanager              1d
   ...
   promop-prometheus-operator-prometheus                1d

6. Update the service monitor name and namespace to match the release name prefix in the edgedelta-prom-servicemonitor.yml file:

   name: promop-prometheus-operator-edgedelta
   namespace: monitoring

7. Based on the above examples, all service monitors begin with promop-prometheus-operator. As a result, the service monitor name should be promop-prometheus-operator-edgedelta.

8. Run the following command to deploy the service monitor:

   kubectl apply -f edgedelta-prom-servicemonitor.yml
   

9. You should receive an output similar to the following example: 

   servicemonitor.monitoring.coreos.com/promop-prometheus-operator-edgedelta created
   service/edgedelta-metrics created
   

10. After a few minutes, metrics should be available on Prometheus.

Related Documentation 

To learn more about metric types and how to configure a processor, see Processors.