Your Next Recovery Test Should Break Something on Purpose

A clean restore proves something useful. The backup data can be read, the recovery target can be reached, and the workflow completes while the dependencies the runbook expects are available. You need that baseline before you start changing variables.

What it does not prove is that recovery will still work when the environment is already damaged. After the baseline is established, I want the next exercise to remove one dependency on purpose and make the team recover around it.

That dependency might be a proxy or gateway, a network path, a credential, DNS, a repository, a storage path, a recovery target, or the management plane used to orchestrate the process. The point is not to create random outages. The point is to find out whether the recovery design still works when one realistic assumption is false.

The Happy Path Is a Baseline

Most recovery tests are designed to pass. The normal management server is online. DNS works. The preferred repository is reachable. The recovery cluster is already prepared. The same engineer who built the runbook executes it with the same credentials used during the last test.

The baseline establishes the known good path. It gives you a timing reference, confirms that the restore point can be consumed, and exposes obvious procedural problems before you add failure conditions.

Documented NIST SP 800-53 CP-4 calls for contingency plan testing to determine plan effectiveness and readiness, followed by review of test results and corrective action when needed. CP-4(5), Self-Challenge, goes further by describing tests that deliberately disrupt a system or component. Its discussion includes disabling critical components, changing configuration, restricting network bandwidth, and altering privileges as examples of ways to expose unexpected dependencies.[1]

A second clean restore does not test a new assumption. If the architecture claims an alternate path exists, the next useful question is whether recovery can use it when the preferred path is gone.

Controlled Recovery Validation Is Not Random Chaos

Deliberately removing one recovery dependency is different from creating a broad, unpredictable outage. I would not start by killing several services in production and watching what happens. That may create an incident, but it makes a poor first recovery experiment because too many variables change at once.

A controlled validation starts with a named dependency, an expected result, a bounded blast radius, a rollback method, and a point where the exercise stops if conditions leave the approved test envelope. The recovery process itself stays real. The injected failure is the controlled variable.

If the test will touch production, I would also name who approved the injection, who owns the change, and who has authority to stop it.

Documented AWS Fault Injection Service guidance recommends reviewing architecture, dependencies, and recovery procedures before an experiment; defining steady state behavior and a hypothesis; starting with small, simple experiments; and beginning in a test or preproduction environment. AWS also documents stop conditions that end an experiment when a defined CloudWatch alarm threshold is reached.[2][3]

The same discipline works for recovery testing even when no fault injection product is involved. Disable one test gateway. Block one route. Revoke one test credential. Remove one resolver from the recovery segment. The mechanism can be simple as long as the condition is deliberate, reversible, and measurable.

Break the Dependency the Runbook Assumes

A recovery runbook is full of dependencies even when it does not list them as dependencies. It assumes something can resolve a name, something can authenticate an operator, something can read the backup, something can move data, and something can provide compute, network, and storage at the destination.

The fastest way to expose those assumptions is to choose one and remove it under controlled conditions.

Not every row needs an automatic fallback. Some architectures intentionally accept a manual replacement step or a documented single dependency because the cost of duplicating it is not justified. The test should prove that the actual behavior matches that decision instead of assuming every component will fail over by itself.

Expected Behavior Matters More Than Expected Success

Before the fault is introduced, decide what a passing result looks like. There are at least three legitimate outcomes: the process continues automatically, the operator follows a documented alternate path, or the workflow stops cleanly at a known dependency and gives the team enough information to execute a planned recovery action.

A test fails when the observed behavior does not match the design, when the fallback cannot be used, when the team has to invent undocumented steps, or when the recovery misses the objective the exercise was meant to validate. A safe stop can be a correct result if the architecture never claimed that component was redundant.

I would write the expected behavior directly into the test plan. “Disable gateway A” is only the injection. “Recovery should select gateway B without operator intervention and remain inside the recovery objective” is the test.

Test One Failure at a Time

The progression should be boring on purpose. Start with a clean restore, then change one condition and keep everything else as stable as practical. If the team cannot explain why the outcome changed, adding another fault only makes the evidence harder to interpret.

I would normally progress in this order:

1. Baseline recovery. Recover the workload through the normal path and record the restore point, target, elapsed time, validation steps, and any manual actions.
2. Single component failure. Remove one dependency such as a proxy, gateway, credential, resolver, or storage path.
3. Path or target change. Force recovery to use a different network path, repository, cluster, site, or other prepared target.
4. Management dependency loss. Remove access to a console, API, management server, or control plane the team normally expects to use.
5. Alternate infrastructure. Run the recovery using the infrastructure intended for a real site or platform loss instead of the comfortable lab path.

Each stage should produce a result you can compare with the baseline. Did recovery still complete? Did the fallback activate? Did a manual decision appear that was never documented? Did the elapsed time change enough to invalidate the recovery objective?

The Management Plane Deserves Its Own Test

Management systems are easy to trust because they are usually present during testing. They are also exactly the kind of dependency that can disappear during a larger outage, a security event, or a platform failure.

AWS makes this point directly in its Well-Architected reliability guidance. It recommends minimizing control plane operations during recovery and evaluating failover runbooks for actions that depend on the control plane, with resources created ahead of time or data plane alternatives where appropriate.[4] AWS is the example here, but the design question is vendor-neutral: which recovery steps require the system you use to create, configure, or manage the recovery infrastructure?

Architecture point If the answer is “most of them,” I would test management plane loss before signing off on the design. The exercise may show that some resources need to be created ahead of time, that a secondary management path is required, or simply that the current recovery plan has a dependency the organization has consciously accepted.

Evidence Is Part of the Test Design

A recovery exercise should leave behind enough evidence to explain what happened without relying on memory. Screenshots can help, but I want the sequence as well: when the failure was introduced, what the system did, which path it selected, what the operator changed, when the workload became usable, and what application level validation passed.

At minimum, I would capture:

• the injected failure and exact start time;
• the expected behavior and stop condition;
• the restore point, source copy, recovery target, and data path actually used;
• authentication, DNS, network, storage, and management events relevant to the failure;
• every manual decision or command that was not already in the runbook;
• recovery timing at the infrastructure and application level;
• the final service validation, not just a successful restore task;
• the runbook or design change created by the result.

The evidence is what lets you distinguish a real fallback from an engineer quietly fixing the environment until the restore works. Both may end with a running VM or application. Only one proves the recovery path you intended to test.

The Runbook Is One of the Things Under Test

AWS's disaster recovery guidance recommends regularly exercising recovery paths and using the runbook during recovery to document problems and find solutions before the next test.[5] I would take that literally. If the engineer has to leave the runbook and improvise a DNS change, find an undocumented credential, remap a network from memory, or rebuild a target that was supposed to exist already, the test found something useful.

The workload may still recover, but the procedure did not pass cleanly. Update the runbook while the evidence is fresh, decide whether the fix belongs in documentation or in the architecture, and rerun the same failure until the result is repeatable.

A corrected document is not proof that the correction works.

A Practical Recovery Test Cycle

The cycle does not need a chaos platform or a large game day. It needs discipline.

1. Prove the normal recovery path.
2. Pick one dependency whose loss is realistic for the failure you are designing around.
3. Define the expected behavior, recovery objective, evidence to capture, rollback method, and stop condition.
4. Introduce the failure in a controlled scope and execute the same recovery runbook.
5. Validate the recovered service, not only the restore job.
6. Compare the result with the baseline, update the runbook or architecture, and repeat the same test until the behavior is understood.

After those single failure cases are repeatable, more complicated exercises can combine conditions when there is a reason to do it. I would not use a multiple failure scenario to compensate for weak single failure testing. Complexity does not make a recovery exercise more realistic if nobody can tell which dependency actually broke the process.

The Bottom Line

A clean restore tells you the normal recovery path works. Keep doing that. Just do not confuse repeating the same healthy path exercise with increasing confidence in the recovery architecture.

Once the baseline is proven, remove one dependency on purpose. Take away a gateway, a route, a credential, DNS, a storage path, a target, or the management plane and see whether the recovery process behaves the way the design says it should. If the alternate path works, capture the evidence. If it does not exist, document that decision. If the team has to invent one during the test, change the design or the runbook and test it again.

Recovery gets more credible when the test stops protecting the assumptions the architecture is supposed to survive.

Source Notes

Product behavior and guidance change. These sources were checked against public primary documentation available on September 18, 2026.