Set up Container Network Observability for Azure Kubernetes Service (AKS) - Azure managed Prometheus and Grafana

This article shows you how to set up Container Network Observability for Azure Kubernetes Service (AKS) using Managed Prometheus and Grafana and BYO Prometheus and Grafana and to visualize the scraped metrics

You can use Container Network Observability to collect data about the network traffic of your AKS clusters. It enables a centralized platform for monitoring application and network health. Currently, metrics are stored in Prometheus and Grafana can be used to visualize them. Container Network Observability also offers the ability to enable Hubble. These capabilities are supported for both Cilium and non-Cilium clusters.

Container Network Observability is one of the features of Advanced Container Networking Services. For more information about Advanced Container Networking Services for Azure Kubernetes Service (AKS), see What is Advanced Container Networking Services for Azure Kubernetes Service (AKS)?.

Prerequisites

• An Azure account with an active subscription. If you don't have one, create a free account before you begin.

• Use the Bash environment in Azure Cloud Shell. For more information, see Get started with Azure Cloud Shell.

  

• If you prefer to run CLI reference commands locally, install the Azure CLI. If you're running on Windows or macOS, consider running Azure CLI in a Docker container. For more information, see How to run the Azure CLI in a Docker container.

  • If you're using a local installation, sign in to the Azure CLI by using the az login command. To finish the authentication process, follow the steps displayed in your terminal. For other sign-in options, see Authenticate to Azure using Azure CLI.

  • When you're prompted, install the Azure CLI extension on first use. For more information about extensions, see Use and manage extensions with the Azure CLI.

  • Run az version to find the version and dependent libraries that are installed. To upgrade to the latest version, run az upgrade.

• The minimum version of Azure CLI required for the steps in this article is 2.56.0. Run az --version to find the version. If you need to install or upgrade, see Install Azure CLI.

Enable Advanced Container Networking Services

To proceed, you must have an AKS cluster with Advanced Container Networking Services enabled.

The az aks create command with the Advanced Container Networking Services flag, --enable-acns, creates a new AKS cluster with all Advanced Container Networking Services features. These features encompass:

• Container Network Observability: Provides insights into your network traffic. To learn more visit Container Network Observability.

• Container Network Security: Offers security features like FQDN filtering. To learn more visit Container Network Security.

# Set an environment variable for the AKS cluster name. Make sure to replace the placeholder with your own value.
export CLUSTER_NAME="<aks-cluster-name>"

# Create an AKS cluster
az aks create \
    --name $CLUSTER_NAME \
    --resource-group $RESOURCE_GROUP \
    --generate-ssh-keys \
    --___location eastus \
    --max-pods 250 \
    --network-plugin azure \
    --network-plugin-mode overlay \
    --network-dataplane cilium \
    --node-count 2 \
    --pod-cidr 192.168.0.0/16 \
    --kubernetes-version 1.29 \
    --enable-acns

# Set an environment variable for the AKS cluster name. Make sure to replace the placeholder with your own value.
export CLUSTER_NAME="<aks-cluster-name>"

# Create an AKS cluster
az aks create \
    --name $CLUSTER_NAME \
    --resource-group $RESOURCE_GROUP \
    --generate-ssh-keys \
    --network-plugin azure \
    --network-plugin-mode overlay \
    --pod-cidr 192.168.0.0/16 \
    --enable-acns

Enable Advanced Container Networking Services on an existing cluster

The az aks update command with the Advanced Container Networking Services flag, --enable-acns, updates an existing AKS cluster with all Advanced Container Networking Services features which includes Container Network Observability and the Container Network Security feature.

az aks update \
    --resource-group $RESOURCE_GROUP \
    --name $CLUSTER_NAME \
    --enable-acns

Get cluster credentials

Once you have Get your cluster credentials using the az aks get-credentials command.

az aks get-credentials --name $CLUSTER_NAME --resource-group $RESOURCE_GROUP

Azure managed Prometheus and Grafana

Skip this Section if using BYO Prometheus and Grafana

Use the following example to install and enable Prometheus and Grafana for your AKS cluster.

Create Azure Monitor resource

#Set an environment variable for the Grafana name. Make sure to replace the placeholder with your own value.
export AZURE_MONITOR_NAME="<azure-monitor-name>"

# Create Azure monitor resource
az resource create \
    --resource-group $RESOURCE_GROUP \
    --namespace microsoft.monitor \
    --resource-type accounts \
    --name $AZURE_MONITOR_NAME \
    --___location eastus \
    --properties '{}'

Create Azure Managed Grafana instance

Use az grafana create to create a Grafana instance. The name of the Grafana instance must be unique.

# Set an environment variable for the Grafana name. Make sure to replace the placeholder with your own value.
export GRAFANA_NAME="<grafana-name>"

# Create Grafana instance
az grafana create \
    --name $GRAFANA_NAME \
    --resource-group $RESOURCE_GROUP 

Place the Azure Managed Grafana and Azure Monitor resource IDs in variables

Use az grafana show to place the Grafana resource ID in a variable. Use az resource show to place the Azure Monitor resource ID in a variable. Replace myGrafana with the name of your Grafana instance.

grafanaId=$(az grafana show \
                --name $GRAFANA_NAME \
                --resource-group $RESOURCE_GROUP \
                --query id \
                --output tsv)
azuremonitorId=$(az resource show \
                    --resource-group $RESOURCE_GROUP \
                    --name $AZURE_MONITOR_NAME \
                    --resource-type "Microsoft.Monitor/accounts" \
                    --query id \
                    --output tsv)

Link Azure Monitor and Azure Managed Grafana to the AKS cluster

Use az aks update to link the Azure Monitor and Grafana resources to your AKS cluster.

az aks update \
    --name $CLUSTER_NAME \
    --resource-group $RESOURCE_GROUP \
    --enable-azure-monitor-metrics \
    --azure-monitor-workspace-resource-id $azuremonitorId \
    --grafana-resource-id $grafanaId

Visualization

Visualization using Azure Managed Grafana

Skip this step if using BYO Grafana

1. Make sure the Azure Monitor pods are running using the kubectl get pods command.

   kubectl get pods -o wide -n kube-system | grep ama-
   

   Your output should look similar to the following example output:

   ama-metrics-5bc6c6d948-zkgc9          2/2     Running   0 (21h ago)   26h
   ama-metrics-ksm-556d86b5dc-2ndkv      1/1     Running   0 (26h ago)   26h
   ama-metrics-node-lbwcj                2/2     Running   0 (21h ago)   26h
   ama-metrics-node-rzkzn                2/2     Running   0 (21h ago)   26h
   ama-metrics-win-node-gqnkw            2/2     Running   0 (26h ago)   26h
   ama-metrics-win-node-tkrm8            2/2     Running   0 (26h ago)   26h
   

2. We have created sample dashboards. They can be found under the Dashboards > Azure Managed Prometheus folder. They have names like "Kubernetes / Networking / <name>". The suite of dashboards includes:

   • Clusters: shows Node-level metrics for your clusters.
   • DNS (Cluster): shows DNS metrics on a cluster or selection of Nodes.
   • DNS (Workload): shows DNS metrics for the specified workload (e.g. Pods of a DaemonSet or Deployment such as CoreDNS).
   • Drops (Workload): shows drops to/from the specified workload (e.g. Pods of a Deployment or DaemonSet).
   • Pod Flows (Namespace): shows L4/L7 packet flows to/from the specified namespace (i.e. Pods in the Namespace).
   • Pod Flows (Workload): shows L4/L7 packet flows to/from the specified workload (e.g. Pods of a Deployment or DaemonSet).

Visualization using BYO Grafana

Skip this step if using Azure managed Grafana

1. Add the following scrape job to your existing Prometheus configuration and restart your Prometheus server:

   - job_name: networkobservability-hubble
     kubernetes_sd_configs:
       - role: pod
     relabel_configs:
       - target_label: cluster
         replacement: myAKSCluster
         action: replace
       - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_pod_label_k8s_app]
         regex: kube-system;(retina|cilium)
         action: keep
       - source_labels: [__address__]
         action: replace
         regex: ([^:]+)(?::\d+)?
         replacement: $1:9965
         target_label: __address__
       - source_labels: [__meta_kubernetes_pod_node_name]
         target_label: instance
         action: replace
     metric_relabel_configs:
       - source_labels: [__name__]
         regex: '|hubble_dns_queries_total|hubble_dns_responses_total|hubble_drop_total|hubble_tcp_flags_total' # if desired, add |hubble_flows_processed_total
         action: keep
   

2. In Targets of Prometheus, verify the network-obs-pods are present.

3. Sign in to Grafana and import following example dashboards using the following IDs:

   • Clusters: shows Node-level metrics for your clusters. (ID: 18814)
   • DNS (Cluster): shows DNS metrics on a cluster or selection of Nodes.(ID: 20925)
   • DNS (Workload): shows DNS metrics for the specified workload (e.g. Pods of a DaemonSet or Deployment such as CoreDNS). (ID: [20926] https://grafana.com/grafana/dashboards/20926-kubernetes-networking-dns-workload/)
   • Drops (Workload): shows drops to/from the specified workload (e.g. Pods of a Deployment or DaemonSet).(ID: 20927).
   • Pod Flows (Namespace): shows L4/L7 packet flows to/from the specified namespace (i.e. Pods in the Namespace). (ID: 20928)
   • Pod Flows (Workload): shows L4/L7 packet flows to/from the specified workload (e.g. Pods of a Deployment or DaemonSet).(ID: 20929)

   Note

   • Depending on your Prometheus/Grafana instances’ settings, some dashboard panels may require tweaks to display all data.
   • Cilium does not currently support DNS metrics/dashboards.

Clean up resources

If you don't plan on using this application, delete the other resources you created in this article using the az group delete command.

  az group delete --name $RESOURCE_GROUP

Next steps

In this how-to article, you learned how to install and enable Container Network Observability for your AKS cluster.

• For more information about Advanced Container Networking Services for Azure Kubernetes Service (AKS), see What is Advanced Container Networking Services for Azure Kubernetes Service (AKS)?.

• For more information on Container Network Security and its capabilities, see What is Container Network Security?.