Nutanix Cluster Design : Live Migration and Cross Hypervisor Mobility

Live migration is the feature you lean on every time you patch a host, every time a workload spikes, and every time you retire old hardware. On AHV it is part of the platform, not an add on. This article covers how AHV live migration actually runs, what triggers it, how the Acropolis Dynamic Scheduler decides what to move, and how you get a workload onto AHV from another hypervisor in the first place. Baseline is AOS 7.3 with AHV 10.3.

Sources: nutanixbible.com (Book of AHV Architecture), portal.nutanix.com (AHV Administration Guide, Nutanix Move documentation), next.nutanix.com. Verify any version specific maximums against the AOS 7.3 Configuration Maximums page before you design to them.

The KVM foundation under AHV

AHV is Linux plus QEMU plus KVM. Every guest runs as a qemu-kvm process using hardware assisted virtualization. AOS talks to the hypervisor through libvirtd, and the Controller VM runs as a VM with its storage devices presented over PCI passthrough. That is the whole stack: a KVM kernel module, libvirt as the management layer, and one QEMU process per running VM.

This matters for migration because a live migration is a QEMU and libvirt operation that AOS orchestrates. The same machinery gives you processor generation leveling and the iterative memory copy that makes a migration safe. Nothing here is a bolt on appliance. It is the hypervisor doing what KVM does, coordinated cluster wide by AOS.

How an AHV live migration runs

Live migration moves a powered on VM from one host to another with no workload interruption, whether an administrator or an automated process starts it. The VM keeps serving traffic on the source host through almost the entire operation. There are seven stages.

1. 1A VM placeholder is created on the destination host.
2. 2The source VM memory is copied to the destination, iteratively, while the VM keeps running.
3. 3The VM is paused on the source host.
4. 4The final VM state is copied to the destination placeholder.
5. 5The network switches are updated so traffic follows the VM.
6. 6The VM resumes on the destination host.
7. 7The paused copy on the source host is deleted.

The iterative memory copy

Stage 2 is where migrations succeed or fail. AHV copies the VM memory over the network in passes, up to 50 of them as documented. Because the VM is still running, it keeps dirtying pages during each pass. After every pass AHV looks at how much memory still needs to go across and the transfer rate it is achieving, then decides whether another pass is needed or whether it can move on. The rate at which the guest dirties memory matters more than how much memory the VM has. AHV will deliberately slow the guest down to let the copy catch up.

The stun window

The source VM is only paused once AHV calculates that the remaining memory can transfer in 300ms or less. That is the maximum stun window, and it is what keeps the VM responsive on the destination after a very short pause. If the remaining memory cannot move inside that window after the final pass, the migration can abort with a failure to converge error. ADS, a host evacuation, or you can retry it.

When the VM comes up on the destination, AHV sends a RARP broadcast onto the subnet so the physical switches relearn where that MAC address now lives. That is what keeps TCP connections from dropping across the move.

What triggers a live migration

Migrations happen for four reasons, and most of them are not manual.

• Maintenance operations. Putting a host into maintenance mode evacuates its VMs first by live migrating them elsewhere. Every rolling AOS and AHV upgrade does this under the covers. Cluster updates are their own topic, covered in NCD-06.
• ADS workload balancing. The scheduler moves VMs to clear host hotspots. Covered in the next section.
• Node expansion. Adding nodes gives ADS room to redistribute placement across the larger cluster.
• Administrator requests. A manual migration from Prism or from acli, usually to stage hardware work or test a path.

Maintenance mode is the one to understand cold. Enter it on a host, AHV live migrates every VM off, you do the work, you exit, and the host rejoins the placement pool. The trap is a VM that cannot be migrated off, which brings us to affinity.

The Acropolis Dynamic Scheduler

The Acropolis Dynamic Scheduler, ADS, is the placement and migration brain of the cluster. It runs in the background at all times. Its job is broader than load balancing:

• Initial VM placement. Choosing the host for a new VM, and defragmenting placement across the cluster if needed to make room.
• Hotspot mitigation. Watching every host and resolving a hotspot when one appears.
• Policy enforcement. Moving VMs in the background to respect affinity and anti affinity rules.
• High availability guarantees. When HA is enabled, keeping placement such that every host failure can be absorbed.
• GPU management. Placing VMs against NVIDIA vGPU profiles.

The service that implements ADS on AHV is Lazan. A Lazan manager gathers statistics from the components it watches, a Lazan solver checks those statistics for anomalies and works out whether and how to resolve them, and the manager then runs the resulting tasks, such as VM migrations.

What it watches, and when it acts

Lazan monitors guest vCPU usage and Stargate storage controller CPU usage per VM or iSCSI target. It does not load balance on memory usage, but it keeps every migration plan inside memory limits, including HA reservations. A host that stays above 85 percent for the 15 minute analysis window triggers a migration plan, and Lazan runs on roughly that 15 minute cadence.

ADS puts a penalty on movement and prefers the lowest cost move that resolves the problem. The result is far fewer migrations than a constant rebalancer would make. It also will not move a VM blindly. If there is an obvious hotspot but no move resolves it without creating a new one, for example a single large VM saturating its host while no other host can absorb it, ADS leaves placement alone and raises an alert instead.

Disabling it

ADS is enabled by default and Nutanix recommends leaving it on. You can disable it from a CVM with acli ads.update enable=false, but the contention checks keep running in the background and still raise alerts. Nothing gets moved automatically once it is off, so you own remediation by hand.

Affinity and anti affinity

AHV gives you two placement controls, and both shape every migration and HA decision the scheduler makes.

VM host affinity

VM host affinity ties a VM to a host or a group of hosts so it only runs there. The common reasons are socket locked software licensing, where you must limit how many hosts an application can run on, and appliances bound to specific hardware.

VM anti affinity

VM anti affinity declares that a named set of VMs should not run on the same host. Use it to keep the members of a clustered or replicated tier apart, so one host failure does not take out database replicas, domain controllers, or load balanced front ends all at once.

There is one behavior you must design around. When the cluster becomes constrained, AHV overrides anti affinity to keep the VMs running. Availability of the VM wins over separation. Anti affinity is therefore best effort under pressure, not a hard guarantee. The way to keep it honest is to size the cluster so that, in your worst tolerated failure case, there is always somewhere for the separated VMs to go without breaking the rule.

Processor compatibility and migration between clusters

Mixed CPU generations inside one cluster are supported, and migration still works across them. AHV finds the lowest processor generation in the cluster and constrains every QEMU domain to that feature level, the same idea as VMware Enhanced vMotion Compatibility. Every VM then sees a feature set that exists on all hosts, so it can land anywhere.

Advanced Processor Compatibility, APC, improves on that. It lets a cluster present a consistent CPU model so you are not permanently held to the lowest common denominator across hardware refreshes, and it lets you set a baseline CPU model per VM for on demand live migration to a different generation of hardware, including to another cluster.

Live migration between clusters

You can live migrate a running VM to another physical cluster, in the same location or a different one, to rebalance workloads or to drain a cluster for maintenance. Nutanix recommends a network with 5ms of latency between the clusters and supports up to 40ms.

Getting workloads onto AHV from another hypervisor

The supported tool for moving VMs onto AHV from another hypervisor is Nutanix Move, a downloadable appliance you run close to the destination. It is separate from AOS and you upgrade it on its own track. Move is on its own release train. Use the current Move guide and release notes for the migration source, target, guest OS, VirtIO, permissions, and upgrade path you are designing around. For this article, the architecture point is simple: Move is a seed plus incremental sync plus cutover tool, not AHV live migration.

Source	Target
VMware ESXi	AHV
Microsoft Hyper-V	AHV
Microsoft Azure	AHV (and ESXi on Nutanix)
AWS EC2	AHV (and ESXi on Nutanix)
VMware Cloud on AWS	AHV and NC2 on AWS
AHV	AWS EC2

How Move works

Move is largely agentless. It seeds a full copy of each source VM disk onto AHV, then keeps that copy current using incremental change tracking while the source VM keeps running. The only downtime is the cutover: the source is quiesced, the last delta is shipped, and the VM boots on AHV. Move injects the VirtIO drivers the guest needs to run on AHV as part of the process.

Move details worth checking before you deploy it:

• Direct in place upgrade from Move 5.6.0. Older 5.4.x and 5.5.x come forward through migration.
• It can carry VMware NSX security policies into Nutanix Flow Network Security.
• It reserves 172.17.0.0/16 and 172.18.0.0/16 for internal services, so do not overlap those ranges.
• ESXi sources need the Cryptographic Operations: Direct Access permission granted to the Move service account.
• The supported VirtIO version is 1.2.4.

Running ESXi on Nutanix without migrating

If you are running ESXi on Nutanix hardware, AOS lets you perform common VM operations from Prism, so you are not forced into vCenter for everyday work. That is hypervisor coexistence on the same platform, not a migration, but it is worth knowing when you are planning a phased move rather than a single cutover.

Common design mistakes