Testing the new AKS Edge Essentials

Microsoft recently released the GA of "Azure Kubernetes Service Edge Essentials" after some years in preview. From the official documentation [emphasis mine]:

Azure Kubernetes Service Edge Essentials is an on-premises Kubernetes implementation of Azure Kubernetes Service (AKS) that automates running containerized applications at scale. AKS Edge Essentials includes a Microsoft-supported Kubernetes platform that includes a lightweight Kubernetes distribution with a small footprint and simple installation experience, making it easy for you to deploy Kubernetes on PC-class or "light" edge hardware. AKS Edge Essentials makes it easier to get started with your containerized application, bringing cloud-native best practices to your edge application.

Sounds very attractive for two specific use cases that if often see in my daily work:

Let's see how AKS EE can help with this!

The details

Minimum hardware requirements per machine *:

* This is the bare minimum for Linux node only machines, in case of Windows containers you will need more CPU/Memory/disk ;-)

When you create an AKS EE deployment, it creates a VM for each of the nodes on the machine. A single machine can have 1 Linux node (pure worker or control pane) and optionally 1 Windows node. You can have a maximum of 15 machines in total.


This also means that AKS EE runs side-by-side with Docker Desktop both in Linux and Windows mode!

You can deploy on your workstation, dedicated server(s) or for testing purpose; virtual machines(s).

TIP: Remember to enable nested virtualization and MAC address spoofing if you are testing with Hyper-V based virtual machines!

There are two deployment options:

If you choose "single" then there is no option to add more machines later but if you choose "full" then you can (but don't have to). Choosing full you have to use an external VM switch and with single you can only use a private VM switch.

BTW licensing is free for development and testing and pricing for production is around $2.50/device/month when connected to an Azure subscription.

Getting started

Let's try to install AKS-EE, deploy a mixed node cluster and then deploy a minimal Sitecore XM workload!

Install AKS EE and deploy mixed Linux/Windows cluster on a single machine

Get and prepare downloads:

Start-BitsTransfer "https://aka.ms/aks-edge/k8s-msi" -Destination "~\Downloads\k8s.msi"
Start-BitsTransfer "https://aka.ms/aks-edge/windows-node-zip" -Destination "~\Downloads\windows-node.zip"
Expand-Archive -Path "~\Downloads\windows-node.zip" -DestinationPath "~\Downloads"

Then start installation with:

msiexec.exe /i (Get-Item "~\Downloads\k8s.msi").FullName /passive ADDLOCAL=CoreFeature,WindowsNodeFeature

...or if you want the VM disks on another drive (see docs for all arguments):

msiexec.exe /i (Get-Item "~\Downloads\k8s.msi").FullName /passive ADDLOCAL=CoreFeature,WindowsNodeFeature VHDXDIR=D:\Data\AksEdge

Now verify you the installation was successful:

Import-Module AksEdge
Get-Command -Module AKSEdge | Format-Table Name, Version

IMPORTANT! Before starting the cluster deployment, please notice that the following command will overwrite your current kubeconfig at $HOME/.kube/config, so make sure to backup if needed! See https://github.com/Azure/AKS-Edge/issues/125 for details.

Now you can deploy a new cluster (takes about ~6 minutes):

New-AksEdgeDeployment -JsonConfigString (curl.exe "https://raw.githubusercontent.com/pbering/aksee-test/main/single-mixed.json" | Out-String)

You now have a running Kuberneters cluster with a Linux node and Windows node on a single machine, you can verify this with kubectl get node.

Post deployment steps

  1. Taint the Windows node: kubectl taint node "$(Get-AksEdgeNodeName -NodeType Windows)" os=windows:NoSchedule
  2. Install nginx ingress:
    • Install: kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.8.0/deploy/static/provider/baremetal/deploy.yaml
    • Configure nginx with: kubectl edit svc ingress-nginx-controller -n ingress-nginx, change spec.type from NodePort to LoadBalancer.
    • Verify that nginx got an external IP address: kubectl get services -n ingress-nginx ingress-nginx-controller
  3. Install local path provisioner for Linux container persistence:
    • Run: kubectl apply -f https://raw.githubusercontent.com/Azure/AKS-Edge/main/samples/storage/local-path-provisioner/local-path-storage.yaml
    • For details, see docs...

Deploy Sitecore XM

  1. Create a new project directory: mkdir .\sitecore-xm;cd .\sitecore-xm

  2. Download workload: curl.exe https://raw.githubusercontent.com/pbering/aksee-test/main/test/xm/kustomization.yaml -o .\kustomization.yaml

  3. Generate certificates (download mkcert https://github.com/FiloSottile/mkcert/releases):

    mkcert -install
    mkcert -cert-file ".\cm-tls.crt" -key-file ".\cm-tls.key" "cm.aksee.local"
    mkcert -cert-file ".\id-tls.crt" -key-file ".\id-tls.key" "id.aksee.local"
  4. Prepare HOST file, run kubectl get services -n ingress-nginx ingress-nginx-controller and use the external IP to add two new entries:

    • id.aksee.local
    • cm.aksee.local
  5. Prepare the Sitecore license, assuming you have the SitecoreDockerTools PowerShell module installed you can then run:

    Import-Module SitecoreDockerTools
    ConvertTo-CompressedBase64String -Path "C:\License\license.xml" | Out-File -Encoding ascii -NoNewline -FilePath .\sitecore-license.txt
  6. Then deploy the workload:

    kubectl create namespace xm
    kubectl apply -n xm -k .
  7. You can observe the progress with kubectl -n xm get pods or https://k9scli.io/ or just wait with kubectl wait --for=condition=complete jobs/mssql-init jobs/solr-init -n xm --timeout 15m

  8. Done! Now you can visit https://cm.aksee.local.


There you have it, finally a easy and production friendly way to install Kubernetes on-prem / locally with Windows container support. While single machine deployments with Linux nodes are in GA, multi machine and Windows nodes are still in preview but looks like they will hit GA soon.