HA, Metro Availability and Multi Site Design on AOS 7.3

NCD-12 closes the Nutanix Cluster Design series. This is the availability layer: how a cluster survives a host failure, how two clusters survive a site failure, and how you stitch many clusters together for disaster recovery. We cover AHV VM high availability, the replication spectrum from async to zero RPO, synchronous AHV Metro Availability, the move from Protection Domains to entity centric Nutanix DR, and the multi site topologies that hold it all together.

Sources: Verified against the Nutanix AHV Administration Guide v10.3, the AHV Best Practices solution document (BP-2029), the Nutanix Disaster Recovery and Data Protection guides on portal.nutanix.com, and Nutanix Tech Center and community material on AHV Metro Availability and Protection Policies. The technical claims and version sensitive values below were checked against those sources. Distance, latency, and RPO floors should be confirmed against portal.nutanix.com for your exact AOS build before you commit a multi site design.

Two different problems: HA and DR

Availability has two layers and they solve different failures. High availability handles the loss of a host inside one cluster: a node dies, its VMs restart on surviving nodes, and the cluster keeps running. Disaster recovery handles the loss of a whole cluster or a whole site: you bring workloads up somewhere else entirely. Conflating them is how people end up with a beautifully configured HA cluster and no answer when the building loses power. Design both, separately, and know which one each control belongs to.

AHV VM high availability

VM high availability on AHV restarts the VMs from a failed host on the surviving hosts. It comes in two modes and the difference is whether restart is guaranteed.

Guarantee mode is enabled in Prism Element under Manage VM High Availability by selecting Enable HA Reservation. The system then shows the memory it has reserved and how many host failures the cluster can absorb. VMHA sizes the reservation against VM memory and respects host affinity and anti affinity rules when it computes where VMs can restart.

Failures tolerated, and the storage tie in

How many host failures the reservation covers tracks your replication factor. With every container at RF2, the reservation protects against one host failure. With any container at RF3, it can protect against two. You set the tolerance explicitly from the CVM:

nutanix@CVM$ acli ha.update num_host_failures_to_tolerate=1

The trap is setting compute tolerance higher than storage can back. If VM data has only two copies, losing two hosts at once can make the data unavailable even when the cluster reserved enough CPU and RAM to restart the VMs. Do not set num_host_failures_to_tolerate higher than your storage availability supports. Compute reservation and data resilience have to agree.

One more control worth knowing: a VM set to ha_priority=-1 is excluded from HA restart. It comes back only when its original host returns. That is sometimes correct for scratch or test VMs you would rather not have competing for restart capacity, and wrong for anything that matters.

ADS and HA together

The Acropolis Dynamic Scheduler covered in NCD-11 is what enforces these guarantees at runtime. When HA reservation is on, ADS keeps the cluster in a state where the surviving hosts can actually absorb a failure, moving VMs as needed so the reserved capacity stays usable. HA decides that VMs must restart; ADS makes sure there is somewhere to restart them.

Failure domains revisited for availability

NCD-01 covered failure domains as a cluster construction choice. For availability the point is what each level lets you survive. Replication factor decides how many data copies exist. Awareness decides where those copies are placed so a single failure cannot take all of them.

• Node awareness places copies on different nodes. That is the baseline. A node loss should not take all copies of the data.
• Block awareness places copies in different blocks (chassis), so losing a whole block, including the nodes and the shared power and cooling inside it, does not take all copies.
• Rack awareness extends the same idea to racks, so a rack level failure, a top of rack switch or a PDU, does not take all copies.

HA restarts the VM; awareness keeps the data underneath it available so there is something to restart. The two have to line up, which is the same lesson as the reservation and RF tie in above: compute survivability is worthless if the data copy it needs is gone.

The replication spectrum

Cross cluster protection is a spectrum, and the axis is recovery point objective traded against bandwidth and distance. Nutanix DR offers three replication types and the design job is matching each workload tier to the right one.

The mistake to avoid is putting everything on the tightest tier. Synchronous replication writes to both sites before the guest sees the write complete, so it adds latency to every write and consumes the most bandwidth. Reserve it for the workloads that justify it and let the long tail ride async.

The replication spectrum: RPO traded against bandwidth and distance, from async hours to synchronous zero.

Synchronous replication and AHV Metro Availability

Synchronous replication is natively supported on AHV. Nutanix delivers synchronous replication through Protection Policies in Prism Central for supported entities, and the result is what the platform calls AHV Metro Availability: zero RPO for protected entities, with writes replicated to the recovery cluster before completion.

The pieces of a synchronous design:

• Protection Policy with RPO zero. You assign supported entities, often through categories rather than by hand, name the recovery cluster in a different fault domain, and set the RPO to zero.
• Recovery cluster. The paired cluster that holds the synchronous copy and runs the VMs after a failover.
• Witness. A lightweight service in a third fault domain that monitors both sites and arbitrates failover so the surviving site can come up automatically without a split brain.
• Recovery Plan. Orchestrates the failover and the cross cluster live migration, including boot order and network mapping, so recovery is a runbook rather than a scramble. Cross cluster live migration can move VMs between the two clusters with zero downtime.

The constraints are physics and architecture, and they are firm:

• Latency. Synchronous replication needs the two sites under roughly 5 ms round trip. Past that, the per write penalty makes it impractical.
• Distance. The real requirement is latency, not a map distance. Roughly 100 km can be a useful metro planning shorthand, but fiber route, carrier path, encryption, and switching design decide whether the sites actually stay under the latency budget. Synchronous is a measured metro latency design, not a continental one.
• No cross hypervisor synchronous. Do not design synchronous replication as a cross hypervisor feature. Cross Hypervisor Disaster Recovery is an async or NearSync style DR workflow, not a synchronous zero RPO Metro design. For synchronous protection, keep the protected stack compatible on both sides and validate the exact release limitations.

Protection Domains versus Nutanix DR

There are two generations of the DR machinery and you should know which one you are building on.

For anything new, design on Nutanix DR with category based Protection Policies and Recovery Plans. The category model is what makes protection survive day two: a VM created next quarter lands in a category and is protected the moment it exists, with no human remembering to add it. Protection Domains still work and still appear in older designs, but they do not scale and they leave gaps when someone forgets a step.

Multi site topologies

With Protection Policies you are not limited to thinking in only one primary and one recovery site, but the exact number of recovery AZs, policies, and synchronous schedules is release and topology dependent. Validate the current Nutanix DR limits before designing one to many or many to one layouts. The common arrangements:

• One to one. A primary and a recovery site. The simplest design and the right starting point for most.
• One to many. One production site protecting different workload groups to different recovery targets, where the release and policy limits support it. A common arrangement is a metro design for the tier one set plus a separate async copy for regional disaster recovery.
• Many to one. Several production clusters, often branch or regional sites, replicating into one central recovery cluster. Common for distributed organizations consolidating DR.
• Cloud target. NC2 lets a public cloud region act as a recovery target, which removes the need to own and run a second physical site purely for DR.

A realistic enterprise design layers these. Tier one workloads get synchronous replication to a metro partner under 5 ms away for zero RPO and automated failover. The same workloads, plus everything else, get an async copy to a geographically distant site or an NC2 region so a regional event that takes both metro sites still has a recovery point. That layering is the whole reason the replication spectrum exists: one tier rarely fits the entire estate.

The design mistakes that cost you later