Set Up Azure Function Monitoring

George Alpizar
George Alpizar
  • Updated

Overview

You can use this document to learn how to deploy Edge Delta components to an AKS cluster.

Azure functions generate various telemetry data, such as:

  • Traces
  • Error
  • Exceptions
  • RemoteDependencies
  • Requests

This telemetry data is ingested directly into the application insight from Azure functions.

Compared to Azure, Edge Delta provides smarter sampling options, such as collecting telemetry data specifically from failed functions and from a small percentage of successful ones. With Edge Delta, application insight telemetry ingestion costs can be reduced drastically.

Step 1: Set Up the Edge Delta Processor in an AKS Cluster

Before you begin, you must have your app insight's instrument key available. 

    1. In the Edge Delta App, on the left-side navigation, click Data Pipeline, and then click Agent Settings.

    2. Review your options: 
      • To create a new configuration, click Create Configuration, and then click YAML.
      • To update an existing configuration, locate the desired configuration, then under Actions, click the vertical ellipses, and then click Edit. 
    3. To populate the YAML file, copy and paste the content from this file.
    4. In the YAML file, replace INSTRUMENTATION_KEY with your the instrumentation key.
    5. Click Save. 
    6. Create a new AKS cluster or use an existing cluster.
    7. Add a new node pool on AKS with the following specs: 
        name: processors
    OS: linux
    size: 1
    SKU: Standard_D4s_v3
    8. When cluster is ready, connect to the cluster, and then create the Edge Delta Secret API key: 
      kubectl create namespace edgedelta
kubectl create secret generic ed-api-key \
    --namespace=edgedelta \
    --from-literal=ed-api-key="c40bafd5-xxxxxxx"
    9. Create the ingress resources described in this document from Microsoft.
      • This step requires helm. 
        kubectl create namespace ingress-basic
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm install nginx-ingress ingress-nginx/ingress-nginx \
    --namespace ingress-basic \
    --set controller.replicaCount=1 \
    --set controller.nodeSelector.agentpool=processors \
    --set defaultBackend.nodeSelector.agentpool=processors
    10. Obtain the IP address: 
      kubectl --namespace ingress-basic get services -o wide -w nginx-ingress-ingress-nginx-controller
    11. Create a DNS zone on the Azure portal, based on this document from Microsoft.
      • You will need to have a public DNS entry for your zone to be publicly accessible.
        • For example, if your DNS zone is contoso.xyz, then you need to own contoso.xyz. As a workaround, you can create a separate AKS cluster with http application routing enabled and use that DNS zone. The DNS zone will have public DNS records created by Azure.
    12. Create an A record in your DNS zone which points to the IP address of ingress controller: 
      ingest.edgedelta.<your dns zone> ->  <IP Address from above step>
    13. Install cert-manager: 
      kubectl label namespace ingress-basic cert-manager.io/disable-validation=true
helm repo add jetstack https://charts.jetstack.io
helm repo update
helm install \
    cert-manager \
    --namespace ingress-basic \
    --version v0.16.1 \
    --set installCRDs=true \
    --set nodeSelector."beta\.kubernetes\.io/os"=linux \
    jetstack/cert-manager
    14. With the DNS entry you configured earlier, update the host values in the edgedelta-ingress resource, based on ed-appinsights-trace-processor.yaml.
    15. Create the Edge Delta http recorder and agent: 
      kubectl apply -f ed-appinsights-trace-processor.yaml`
    16. Verify that edgedelta pods are running: 
      kubectl get pods -n edgedelta
    17. Verify certificate creation: 
      kubectl get certificate --namespace ingress-basic
    18. Verify public endpoint: 
      https://ingest.edgedelta.198de54f02b345ab92a8.centralus.aksapp.io/

Step 2: Set Up Azure Function With Dual Telemetry Write Mode

  1. Navigate to the Azure application folder, and then add dependencies:
dotnet add package Microsoft.Azure.Functions.Extensions
dotnet add package Microsoft.Extensions.Logging.ApplicationInsights
  1. Create a StartUp.cs file under the targeted Azure function application with the content in this StartUp.cs file.
    • Update the namespace at the top. In the example below, the custom sinker implementation called ForkingTelemetryChannel replicates telemetry data for ingestion into a secondary ingestion endpoint. Note that the dual ingestion process is parallelized to reduce the overall latency.
...
        public void Send(ITelemetry item)
        {
            var itemDup = item.DeepClone();
            itemDup.Context.InstrumentationKey = this.secondaryInstrumentationKey;
            Parallel.Invoke(
                () => { orginalChannel.Send(item); },
                () => { secondaryChannel.Send(itemDup); }
            );
        }
...
  1. Set the secondary application insight connection string.
    • Provide the secondary endpoint address created in the AKS cluster ingress endpoint.
    • Provide the secondary instrumentation key that will be used to forward the matching telemetry data to the application insight from the Edge Delta processor.
{
...
  "Values": {
...
    // Either original instrumentation key or original connection string is provided but not both
    "APPINSIGHTS_INSTRUMENTATIONKEY": "OriginalKey123";
    "APPLICATIONINSIGHTS_CONNECTION_STRING": "InstrumentationKey=OriginalKey123;IngestionEndpoint=https://dc.services.visualstudio.com",
    // Required secondary connection string for forking application insight traffic
    "APPLICATIONINSIGHTS_SECONDARY_CONNECTION_STRING": "InstrumentationKey=SecondaryKey123;IngestionEndpoint=https://ingest.edgedelta.198de54f02b345ab92a8.centralus.aksapp.io",
...
  },
...
}

Step 3: Set Up Azure Function Without Dual Writes

  1. Set the application insight connection string to point to the public endpoint previously. Use your target appinsight instrumentation key.
"APPLICATIONINSIGHTS_CONNECTION_STRING": "InstrumentationKey=***;IngestionEndpoint=https://ingest.edgedelta.198de54f02b345ab92a8.centralus.aksapp.io",

Step 4: Test the Setup

  1. Run the Azure functions and simulate failure scenarios.
  2. Visit the application insights to verify that the Edge Delta agent is forwarding failed traces.
Share this document