Isovalent Enterprise for Cilium can now be installed on Azure Kubernetes Service clusters using Azure Linux as the host Operating system. In this tutorial, you will learn how to:
- Install AKS clusters running Azure CNI powered by Cilium with Azure Linux.
- Migrate your existing clusters on Azure CNI powered by Cilium from Ubuntu to Azure Linux.
- Upgrade your clusters from Azure CNI powered by Cilium running Azure Linux to Isovalent Enterprise for Cilium.
What is Isovalent Enterprise for Cilium?
Azure Kubernetes Service (AKS) uses Cilium natively, wherein AKS combines the robust control plane of Azure CNI with Cilium’s data plane to provide high-performance networking and security. Isovalent Cilium Enterprise is an enterprise-grade, hardened distribution of open-source projects Cilium, Hubble, and Tetragon, built and supported by the Cilium creators. Cilium enhances networking and security at the network layer, while Hubble ensures thorough network observability and tracing. Tetragon ties it all together with runtime enforcement and security observability, offering a well-rounded solution for connectivity, compliance, multi-cloud, and security concerns.
Why Isovalent Enterprise for Cilium?
For enterprise customers requiring support and usage of Advanced Networking, Security, and Observability features, “Isovalent Enterprise for Cilium” is recommended with the following benefits:
- Advanced network policy: Isovalent Cilium Enterprise provides advanced network policy capabilities, including DNS-aware policy, L7 policy, and deny policy, enabling fine-grained control over network traffic for micro-segmentation and improved security.
- Hubble flow observability + User Interface: Isovalent Cilium Enterprise Hubble observability feature provides real-time network traffic flow, policy visualization, and a powerful User Interface for easy troubleshooting and network management.
- Multi-cluster connectivity via Cluster Mesh: Isovalent Cilium Enterprise provides seamless networking and security across multiple clouds, including public cloud providers like AWS, Azure, and Google Cloud Platform, as well as on-premises environments.
- Advanced Security Capabilities via Tetragon: Tetragon provides advanced security capabilities such as protocol enforcement, IP and port whitelisting, and automatic application-aware policy generation to protect against the most sophisticated threats. Built on eBPF, Tetragon can easily scale to meet the needs of the most demanding cloud-native environments.
- Service Mesh: Isovalent Cilium Enterprise provides seamless service-to-service communication that’s sidecar-free and advanced load balancing, making it easy to deploy and manage complex microservices architectures.
- Enterprise-grade support: Isovalent Cilium Enterprise includes enterprise-grade support from Isovalent’s experienced team of experts, ensuring that any issues are resolved promptly and efficiently. Additionally, professional services help organizations deploy and manage Cilium in production environments.
How can you deploy Isovalent Enterprise for Cilium?
Isovalent Enterprise for Cilium is available in the Azure Marketplace. It can also be deployed using Azure Resource Manager (ARM) Templates and Azure CLI.
What is Azure Linux?
Microsoft announced the General Availability for Azure Linux Container Host in May 2023. Azure Linux is a lightweight operating system, containing only the packages needed for a cloud environment. Azure Linux can be customized through custom packages and tools, to fit the requirements of your application. Azure Kubernetes Services is one such application that uses production-grade container orchestration as an option for container hosting. The Azure Linux container host for AKS is an open-source Linux distribution created by Microsoft, and it’s available as a container host on Azure Kubernetes Service (AKS).
Why Azure Linux as the host OS?
A popular question you would ask is why choose Azure Linux as the host OS:
- Optimized to run in Azure. Built, verified, and digitally signed by Microsoft .
- Supply chain security.
- Smaller and leaner Linux to reduce footprint, surface attack area, & optimize performance.
- Operational consistency across Edge to Cloud.
- Rigorous validation and testing of packages and images on AKS infrastructure.
Pre-Requisites
The following prerequisites must be considered before you proceed with this tutorial.
- Azure CLI version 2.48.1 or later. Run az –version to see the currently installed version. If you need to install or upgrade, see Install Azure CLI.
- If using ARM templates or the REST API, the AKS API version must be 2022–09–02-preview or later.
- You should have an Azure Subscription.
- Install kubectl.
- Install Cilium CLI.
- Install Helm.
- Ensure you have enough quota resources to create an AKS cluster. Go to the Subscription blade, navigate to “Usage + Quotas,” and make sure you have enough quota for the following resources:
-Regional vCPUs
-Standard Dv4 Family vCPUs - You can choose regions where the quotas are available and not strictly follow the regions picked up during this tutorial.
Limitations with Azure Linux Container Host
- Azure Linux cannot yet be deployed through the Azure Portal.
- Azure Linux doesn’t support AppArmor. Support for SELinux can be manually configured.
- Creating an AKS cluster on Isovalent Enterprise for Cilium with Azure Linux as the host OS will be available in a future release.
Installing Azure Linux on Azure Kubernetes Service Clusters
The following combinations of installing and migrating AKS clusters with Azure Linux are supported.
Network Plugin | Default Nodepool OS (during AKS cluster creation) | Additional Nodepool OS (after AKS cluster creation) | Migration from Ubuntu to Azure Linux |
Azure CNI (Powered by Cilium)-Overlay Mode | Azure Linux | Azure Linux | N.A |
Azure CNI (Powered by Cilium)-Overlay Mode | Ubuntu | Azure Linux | Yes |
Azure CNI (Powered by Cilium)-Dynamic IP Allocation Mode | Azure Linux | Azure Linux | N.A |
Azure CNI (Powered by Cilium)-Dynamic IP Allocation Mode | Ubuntu | Azure Linux | Yes |
Azure CNI (Powered by Cilium)-Overlay Mode to Isovalent Enterprise for Cilium | Azure Linux | N.A | N.A |
Bring your own CNI (BYOCNI) | Azure Linux | Azure Linux | N.A |
Bring your own CNI (BYOCNI) | Ubuntu | Azure Linux | Yes |
- N.A= Not Applicable
- BYOCNI (Azure Linux) and BYOCNI (Ubuntu) have also been tested and validated. If you would like to get more information about them, you can get in touch with sales@isovalent.com and support@isovalent.com
Choosing the IMU for a Product?- Installation, Migration or Upgrade
You can take a look at this flowchart and then decide whether you would like to do:
- A greenfield installation of your AKS cluster with Azure Linux
- Upgrade/Migrate your existing AKS clusters from Ubuntu to Azure Linux
Scenario 1: AKS cluster on Azure CNI powered by Cilium in (Overlay mode) with Azure Linux
AKS Resource Group Creation
Create a Resource Group
AKS Cluster creation
Create a cluster with Azure CNI Powered by Cilium with network-plugin as Azure
, network-plugin-mode as Overlay
, os-sku as AzureLinux
and network-dataplane as Cilium
.
Set the Subscription
Choose the subscription you want to use if you have multiple Azure subscriptions.
- Replace SubscriptionName with your subscription name.
- You can also use your subscription ID instead of your subscription name.
Set the Kubernetes Context
Log in to the Azure portal, browse Kubernetes Services>, select the respective Kubernetes service created ( AKS Cluster), and click connect. This will help you connect to your AKS cluster and set the respective Kubernetes context.
Cluster Status Check
Check the status of the nodes and make sure they are in a ‘Ready’ state and are running ‘CBL-Mariner/Linux’ as the host OS.
Add nodepool with OS-type as AzureLinux.
Add an Azure Linux node pool to your existing cluster.
Note- When adding a new Azure Linux node pool, you need to add at least one as --mode System
. Otherwise, AKS will not allow you to delete your existing node pool.
Cluster Status Check
Check the status of the newly added nodes and make sure they are in a ‘Ready’ state and are running ‘CBL-Mariner/Linux’ as the host OS.
Scenario 2: AKS cluster on Azure CNI powered by Cilium (Overlay Mode) with Ubuntu (Migration to Azure Linux)
AKS Resource Group Creation
Create a Resource Group
AKS Cluster creation
Create a cluster with Azure CNI Powered by Cilium with network-plugin as Azure
, network-plugin-mode as Overlay
, and network-dataplane as Cilium
.
Set the Subscription
Choose the subscription you want to use if you have multiple Azure subscriptions.
- Replace SubscriptionName with your subscription name.
- You can also use your subscription ID instead of your subscription name.
Set the Kubernetes Context
Log in to the Azure portal, browse to Kubernetes Services>, select the respective Kubernetes service created ( AKS Cluster), and click connect. This will help you connect to your AKS cluster and set the respective Kubernetes context.
Cluster Status Check
Check the status of the nodes and make sure they are in a ‘Ready’ state and are running ‘Ubuntu’ as the host OS.
Add nodepool with OS-type as AzureLinux
Add an Azure Linux node pool to your existing cluster.
Note- When adding a new Azure Linux node pool, you need to add at least one as --mode System
. Otherwise, AKS will not allow you to delete your existing node pool.
Cluster Status Check
Check the status of the newly added nodes and make sure they are in a ‘Ready’ state and are running ‘CBL-Mariner/Linux’ as the host OS.
Migrate the default nodes to Azure Linux.
You can migrate the default nodes created while creating the AKS cluster and run Ubuntu as the host OS. This is optional; you can skip it if it is not required. Migration is a 3-part process:
Cordon the existing Nodes (Default)
Cordoning marks specified nodes as unschedulable and prevents any more pods from being added to the nodes.
First, obtain the names of the nodes you’d like to cordon with kubectl get nodes
:
Next, using kubectl cordon <node-names>
, specify the desired nodes in a space-separated list:
Check the status of the nodes that are being cordoned:
Drain the existing nodes (Default)
To successfully drain nodes and evict running pods, ensure that any PodDisruptionBudgets (PDBs) allow for at least 1 pod replica to be moved at a time; otherwise, the drain/evict operation will fail. To check this, you can run kubectl get pdb -A
, and make sure ALLOWED DISRUPTIONS is at least 1 or higher.
Draining nodes will cause pods running on them to be evicted and recreated on the other schedulable nodes.
To drain nodes, use kubectl drain <node-names> --ignore-daemonsets --delete-emptydir-data
, again using, a space-separated list of node names:
Note- Using --delete-emptydir-data
is required to evict the AKS-created coredns and metrics-server pods. If this flag isn’t used, an error is expected.
Remove the existing nodes (Default)
To remove the existing nodes, use the az aks delete
command. The final result is the AKS cluster having a single Azure Linux node pool with the desired SKU size and all the applications and pods properly running.
Check the status of the nodes to ensure that the default node has been deleted and the additional node running AzureLinux is in a ‘Ready’ state:
Scenario 3: AKS cluster on Azure CNI powered by Cilium (Dynamic IP mode) with Azure Linux
AKS Resource Group Creation
Create a Resource Group
AKS Network creation
Create a virtual network with a subnet for nodes and a subnet for pods and retrieve the subnetID
AKS Cluster creation
Create an AKS cluster referencing the node subnet using –vnet-subnet-id and the pod subnet using –pod-subnet-id. Make sure to use the argument –network-plugin as azure
, os-sku as AzureLinux
and network-dataplane as cilium
.
Set the Subscription
Choose the subscription you want to use if you have multiple Azure subscriptions.
- Replace SubscriptionName with your subscription name.
- You can also use your subscription ID instead of your subscription name.
Set the Kubernetes Context
Log in to the Azure portal, browse Kubernetes Services>, select the respective Kubernetes service created ( AKS Cluster), and click connect. This will help you connect to your AKS cluster and set the respective Kubernetes context.
Cluster Status Check
Check the status of the nodes and make sure they are in a ‘Ready’ state and are running ‘CBL-Mariner/Linux’ as the host OS.
Add nodepool with OS-type as AzureLinux.
Add an Azure Linux node pool to your existing cluster. In the case of Azure CNI (Dynamic IP allocation), you need to add a new subnet for pods and nodes in addition to what was created originally at the time of the AKS cluster creation.
Note- When adding a new Azure Linux node pool, you need to add at least one as --mode System
. Otherwise, AKS will not allow you to delete your existing node pool.
Cluster Status Check
Check the status of the newly added nodes and make sure they are in a ‘Ready’ state and are running ‘CBL-Mariner/Linux’ as the host OS.
Scenario 4: AKS cluster on Azure CNI powered by Cilium (Dynamic IP mode) with Ubuntu (Migration to Azure Linux)
AKS Resource Group Creation
Create a Resource Group
AKS Network creation
Create a virtual network with a subnet for nodes and a subnet for pods and retrieve the subnetID
AKS Cluster creation
Create an AKS cluster referencing the node subnet using –vnet-subnet-id and the pod subnet using –pod-subnet-id. Make sure to use the argument –network-plugin as azure
and network-dataplane as cilium
.
Set the Subscription
Choose the subscription you want to use if you have multiple Azure subscriptions.
- Replace SubscriptionName with your subscription name.
- You can also use your subscription ID instead of your subscription name.
Set the Kubernetes Context
Log in to the Azure portal, browse to Kubernetes Services>, select the respective Kubernetes service created ( AKS Cluster), and click connect. This will help you connect to your AKS cluster and set the respective Kubernetes context.
Cluster Status Check
Check the status of the nodes and make sure they are in a ‘Ready’ state and are running ‘Ubuntu’ as the host OS.
Add nodepool with OS-type as AzureLinux.
Add an Azure Linux node pool to your existing cluster. In the case of Azure CNI (Dynamic IP allocation), you need to add a new subnet for pods and nodes in addition to what was created originally at the time of the AKS cluster creation.
Note- When adding a new Azure Linux node pool, you need to add at least one as --mode System
. Otherwise, AKS will not allow you to delete your existing node pool.
Cluster Status Check
Check the status of the newly added nodes and make sure they are in a ‘Ready’ state and are running ‘CBL-Mariner/Linux’ as the host OS.
Migrate the default nodes to Azure Linux.
You can migrate the default nodes created while creating the AKS cluster and run Ubuntu as the host OS. This is optional; if not required, you can skip this step. Migration is a 3-part process:
Cordon the existing Nodes (Default)
Cordoning marks specified nodes as unschedulable and prevents any more pods from being added to the nodes.
First, obtain the names of the nodes you’d like to cordon with kubectl get nodes
:
Next, using kubectl cordon <node-names>
, specify the desired nodes in a space-separated list:
Check the status of the nodes that are being cordoned:
Drain the existing nodes (Default)
To successfully drain nodes and evict running pods, ensure that any PodDisruptionBudgets (PDBs) allow for at least 1 pod replica to be moved at a time; otherwise, the drain/evict operation will fail. To check this, you can run kubectl get pdb -A
, and make sure ALLOWED DISRUPTIONS is at least 1 or higher.
Draining nodes will cause pods running on them to be evicted and recreated on the other schedulable nodes.
To drain nodes, use kubectl drain <node-names> --ignore-daemonsets --delete-emptydir-data
, again, a space-separated list of node names:
Note- Using --delete-emptydir-data
is required to evict the AKS-created coredns and metrics-server pods. If this flag isn’t used, an error is expected.
Remove the existing nodes (Default)
To remove the existing nodes, use the az aks delete
command. The final result is the AKS cluster having a single Azure Linux node pool with the desired SKU size and all the applications and pods properly running.
Check the status of the nodes to ensure that the default node has been deleted and the additional node running AzureLinux is in a ‘Ready’ state:
Scenario 5: In-place OS SKU migration (preview)
You can now migrate your existing Ubuntu node pools to Azure Linux by changing the OS SKU of the node pool, which rolls the cluster through the standard node image upgrade process. This new feature doesn’t require the creation of new node pools.
Install the aks-preview
extension.
Install the aks-preview
extension using the az extension add
command.
Register the OSSKUMigrationPreview
feature flag
- Register the
OSSKUMigrationPreview
feature flag on your subscription using theaz feature register
command.
- Check the registration status using the
az feature list
command.
- Refresh the registration of the
OSSKUMigrationPreview
feature flag using theaz provider register
command.
Set the Subscription
Choose the subscription you want to use if you have multiple Azure subscriptions.
- Replace SubscriptionName with your subscription name.
- You can also use your subscription ID instead of your subscription name.
Set the Kubernetes Context
Log in to the Azure portal, browse Kubernetes Services>, select the respective Kubernetes service created ( AKS Cluster), and click connect. This will help you connect to your AKS cluster and set the respective Kubernetes context.
Note- You can use an existing scenario to see how OS SKU migration occurs on an existing AKS cluster running Ubuntu as the host OS.
Migrate the OS SKU of your Ubuntu node pool.
- Migrate the OS SKU of your node pool to Azure Linux using the
az aks nodepool update
command. This command updates the OS SKU for your node pool from Ubuntu to Azure Linux. The OS SKU change triggers an immediate upgrade operation, which takes several minutes to complete.
Truncated O/P:
Verify the OS SKU migration
Once the migration is complete on your test clusters, you should verify the following to ensure a successful migration:
- If your migration target is Azure Linux, run the
kubectl get nodes -o wide
command. The output should showCBL-Mariner/Linux
as your OS image and.cm2
at the end of your kernel version. - Run the
kubectl get pods -o wide -A
command to verify that your pods and daemonsets are running on the new node pool. - Run the
kubectl get nodes --show-labels
command to verify that all of the node labels in your upgraded node pool are what you expect.
Blue-Green tests during upgrade (Optional)
Blue-green deployments allow you to run multiple versions of your application side by side and do a clean cutover from v1 to v2. You would bring v2 of your application live, next to production, and have the ability to test it out without impacting production. Once you’re happy with v2, you will switch production from blue to green.
The easiest way to set up blue-green deployments is by using the Service object in Kubernetes. In the example below, you can deploy a blue and green service and observe how the traffic is switched from green to blue while the upgrade continues. In between, the traffic is also manually switched back to green to see if there are no traffic drops.
Deployment
- Create a deployment
myapp-blue
- Create a deployment
myapp-green
- Deploy the deployments
myapp-green
&myapp-blue
- Check the status of the deployments.
- Creation of pods- the deployments will lead to creation of two distinct pods with prefixes myapp-blue-* and myapp-green-*. They should be up and running.
- Create a service object (
svc.yaml
) and apply theservice
object.
- Ensure that the service has a Public IP (Optional).
Traffic generation and upgrade
- Before the upgrade, you can send traffic toward the public IP of the myapp Service (Load Balancer), which has previously created pods as its backend.
- Initiate the upgrade on the nodepool
nodepool-name
for the cluster. (nodepool1 in this case)
- Notice that the
green
app is running on node 1.
- Edit the service object and change the selector type to
blue
. Notice that theblue
app is running on node 2
- Notice that nodes are upgraded from Ubuntu to Azure Linux without traffic disruption.
- As the upgrade proceeds, the apps are moved to nodes node0 and node1, respectively, but the user experience is not compromised.
Scenario 6: AKS cluster on Isovalent Enterprise for Cilium with Azure Linux
Note- You can upgrade your existing clusters as described in Scenarios 1 to 4 to Isovalent Enterprise for Cilium through Azure Marketplace. We have chosen one of those options to highlight the upgrade process. The steps for upgrading all the 4 scenarios are the same.
You can follow this blog and the steps to upgrade an existing AKS cluster to Isovalent Enterprise for Cilium. Make sure you take care of the prerequisites.
- In the Azure portal, search for Marketplace on the top search bar. In the results, under Services, select Marketplace.
- Type ‘Isovalent’ In the search window and select the offer.
- On the Plans + Pricing tab, select an option. Ensure that the terms are acceptable, and then select Create.
- Select the resource group in which the cluster exists that we will be upgraded.
- Click Create New Dev Cluster, select ‘No,’ and click Next: Cluster Details.
- As ‘No’ was selected, this will upgrade an already existing cluster in that region.
- The name for the AKS cluster will be auto-populated by clicking on the drop-down selection.
- Click ‘Next: Review + Create’ Details.
- Once Final validation is complete, click ‘Create.’
- When the application is deployed, the portal will show ‘Your deployment is complete,’ along with deployment details.
- Verify that the nodes are running Azure Linux. Click > Resource Groups> Kubernetes Services> Select the AKS cluster> Nodepools
How do you upgrade Azure Linux Container Host Nodes?
The Azure Linux Container Host ships updates through Updated Azure Linux node images.
Note- Ensure you have an AKS cluster running Azure Linux or migrated to Azure Linux by following the steps outlined in the previous sections.
Manually upgrade your cluster.
To manually upgrade the node-image on a cluster:
Validation- Isovalent Enterprise for Cilium
Validate the version of Isovalent Enterprise for Cilium
Check the version of Isovalent Enterprise for Cilium with cilium version
:
Cilium Health Check
cilium-health is a tool available in Cilium that provides visibility into the overall health of the cluster’s networking connectivity. You can check node-to-node health with cilium-health status
:
Cilium Connectivity Test
The Cilium connectivity test deploys a series of services and deployments, and CiliumNetworkPolicy will use various connectivity paths to connect. Connectivity paths include with and without service load-balancing and various network policy combinations.
The cilium connectivity test was run for all of the above scenarios, and the tests were passed successfully. A truncated output for one such test result is added.
Caveats/ Troubleshooting
- If you add a nodepool with network plugins Azure CNI Dynamic IP or Azure CNI powered by Cilium and a different/new subnet for both pods and nodes has not been added, you will observe this error.
- If you are deleting a nodepool in any of the above scenarios that have been explained, ensure that there is one nodepool that was created with
--mode System
else you will observe this error.
Conclusion
Hopefully, this post gave you a good overview of installing or migrating your existing or new AKS clusters running Azure CNI powered by Cilium with Azure Linux and upgrading to Isovalent Enterprise for Cilium. If you have any feedback on the solution, please share it with us. You’ll find us on the Cilium Slack channel.
Try it out
- Azure CNI powered by Cilium.
- Isovalent Enterprise for Cilium on the Azure marketplace.
Further Reading
- Tutorial on installing Isovalent Enterprise for Cilium in Azure
- Enabling Enterprise features on Isovalent Enterprise from Cilium in Azure
- Tutorial on installing an AKS cluster running Azure CNI powered by Cilium
- Upgrade to cilium in Azure
- Azure and Isovalent main partner page
Amit Gupta is a Senior Technical Marketing Engineer at Isovalent that is powering eBPF cloud-native networking and security. Amit has 20+ years of experience in Networking, Telecommunications, Cloud, Security, and Open-Source and has worked in the past with companies like Motorola, Juniper, Avi Networks (acquired by VMware), and Prosimo. He is keen to learn and try out new technologies that aid in solving day-to-day problems for operators and customers.
He has worked in the Indian start-up ecosystem for a long time and helps new folks in that area in his time outside of work. Amit is an avid runner and cyclist and also spends a considerable amount of time helping kids in orphanages.