Automic DR Is a Dependency Sequence, Not a Service Restart

Stopping one set of Windows services and starting another set at the recovery site is not an Automic disaster recovery plan. It is one action inside a much larger dependency sequence.

Broadcom Automic Automation sits on a database, a distributed set of Automation Engine processes, a web tier, communication processes, agents, DNS, load balancing, and the business workloads that ultimately prove the scheduler is doing useful work. A DR runbook has to move those pieces in an order that prevents two active sides, keeps jobs from firing at the wrong time, and gives the team a hard gate before production processing resumes.

The workflow I want is simple to describe even if the implementation is not: stop new work, withdraw the source, move the database role, start the target Automation Engine, prove the server-process set, publish the web path, prove agent communications, run real validation jobs, then reverse the process deliberately when it is time to fail back.

This article is about the controlled recovery path: a DR exercise, planned site transition, or deliberate failback where the source is reachable long enough to hold work, withdraw traffic, stop AE, and hand the database across cleanly. A real site loss can remove some of those steps. Microsoft treats forced failover as a different database case when planned-failover conditions are unavailable, so an outage runbook should start at the first surviving dependency and use the recovery method that matches the remaining state and acceptable RPO.[1]

Hold Work Before You Touch the Platform

The first recovery action should control workload execution, not infrastructure.

If scheduled jobs can continue to start while the application, database, network path, or agent population is moving between sites, the recovery team has already lost control of state. Job queues should be held or stopped according to the application design, and the team should confirm that currently running work has reached an acceptable point before the source platform is withdrawn.

This is separate from stopping the Automation Engine. Holding queues answers a business-processing question: should new work start? Stopping the source AE answers an infrastructure question: which site owns the scheduler right now?

I also want the web path withdrawn before source-side AE shutdown. Users should not keep landing on an application tier that is about to lose its engine or database. A load balancer drain, health-state change, or explicit pool disable gives the source site a clean exit from user traffic before the control plane moves.

Architecture point Treat workload state, user traffic, application processes, and database ownership as separate controls. The runbook can then prove each one before it advances.

The Database Move Has to Finish Before AE Starts

Documented For a planned SQL Server Always On availability-group failover without data loss, Microsoft requires the current primary and target secondary to use synchronous-commit mode and the target secondary to be synchronized before the role change. Microsoft also notes that a failover command can return before database recovery has fully completed because recovery continues asynchronously after the availability group accepts the transition.[1][2]

That behavior is why I do not want "SQL failover" represented as one opaque line in an application runbook.

The database section should have its own gates:

1. Confirm the expected current primary and secondary roles.
2. Confirm the availability group is healthy.
3. Move the participating replicas to synchronous commit when the design requires it.
4. Confirm synchronization and failover readiness.
5. Perform the planned role transition.
6. Confirm the new primary and the health of the group.
7. Return to the normal replication mode if the steady-state design uses asynchronous commit.
8. Confirm the application connection name resolves to the intended listener before AE starts.

The exact SQL implementation belongs to the DBA team when that is how operational ownership is divided. The application runbook still needs a hard dependency on the DBA result. "DBA says complete" should map to something concrete: expected primary role, healthy replicas, the right listener, and name resolution from the application side.

Operational view A database role change that takes longer on failback than it did on failover is not automatically an application failure. Measure failback separately and make room for database recovery time instead of copying the forward-path estimate into the return plan.

Starting the Automation Engine Means Proving the Process Topology

Documented Broadcom documents multiple server-process types inside the Automation Engine, including PWP, WP, DWP, JWP, JCP, CP, and REST processes. Work processes perform the core server work, while the primary work process handles central tasks that are not distributed. The Processes and Utilization page is the administration view used to inspect the running AE process set.[3][4]

That is a much better recovery gate than "the Windows service is running."

A service wrapper can be started while the expected AE process population is incomplete, still initializing, or unstable. A target site should not be considered ready because one executable launched successfully.

I want the validation step to prove the expected markers across the AE cluster. The exact marker set depends on the installation, but the idea is consistent:

• one healthy PWP role is established,
• the expected WP and DWP population is present,
• required JWP processes are running,
• JCP, CP, and REST processes are present where the design expects them,
• the process set remains stable through a short observation window,
• the processes are on the intended recovery nodes rather than accidentally split across old and new sites.

Documented Broadcom also supports distributed server environments across multiple AE nodes and recommends structurally identical server nodes for that design.[5] Cluster-wide validation matters because a process that is intentionally present on only one node should not be treated as a per-host requirement.

Operational clue Validate the aggregate AE process set first, then look at per-node placement where the design cares about it. A per-host check can create false failures when a role is supposed to exist once across the cluster.

AWI Health and Traffic Eligibility Are Two Different Tests

Documented Broadcom documents an AWI health check at /AWI/status. A healthy deployment returns HTTP 200 and an OK response, and Broadcom explicitly notes that the endpoint can be used by internal monitoring systems.[6]

That endpoint tells you AWI is running. It does not, by itself, answer every traffic-management question in a DR design.

A load balancer normally needs to answer at least two things:

• Is the web application alive?
• Is this site supposed to accept user traffic right now?

Those are not the same condition.

One practical design is to combine the application health check with a site-eligibility canary. The canary can be a small static file or another deterministic signal that exists only when the site is intentionally allowed to receive traffic. The load balancer should require both conditions before it considers the node eligible.

That canary is a design control, not an Automic product requirement. It is useful because a healthy Tomcat or AWI instance can still be on the wrong side of a recovery event.

The health check also has to test the resource you think it is testing. If a missing canary path is redirected to the AWI login page and the monitor accepts the redirect or the resulting 200 response, the canary is not actually controlling anything. Test the path with the file present and absent, verify redirect handling, and make the negative test part of the runbook.

Architecture point Application health says the component works. Site eligibility says the component is allowed to serve production. Keep both signals.

Agent Recovery Depends on JCP Reachability, Not Just AE Health

Documented Current Automic Agents use TLS/SSL through secure WebSockets to connect to a Java communication process. Broadcom documents that when the agent's JCP list is empty, the agent uses the connection= value in its INI file for the initial connection. After a successful connection, the JCP supplies endpoint information and the agent populates its JCP list for later use.[7][9]

Documented Broadcom's current Windows Agent documentation is explicit about what happens after bootstrap: once JCPLIST is populated, later connection attempts select a random JCP from that list.[7] Entry order is not a production-first, DR-second preference mechanism. JCP_ENDPOINT can publish one or more load-balancer endpoints, and Net Areas can constrain agents to communication processes in the network zone assigned to that area.[7][8]

That behavior matters in a multi-site recovery design. Control which JCP endpoints agents learn and which ones their networks can reach. Listing primary endpoints first and recovery endpoints second does not create deterministic site priority.

The AE cluster can be healthy while a remote agent still cannot reach the recovery-side JCP path. Firewall rules, DNS, certificates, endpoint lists, old bootstrap values, or a load-balancer address can all strand agents after the core engine is already green.

Do not make "AE up" the same gate as "agents recovered."

I would validate agents separately after the target AE process set is stable and after the intended JCP path is reachable. The checks should answer:

• Are the required JCP endpoints reachable from representative agent networks?
• Do the agents reconnect to the recovered AE system?
• Are authentication and TLS healthy?
• Are the expected agent objects active in Automic?
• Can a representative job execute through an agent at the recovery site?

If an agent emits a startup error but eventually appears active, inspect the agent log and its configured connection/JCP list before treating the message as a failed recovery. The question is whether the agent established the expected production connection and can execute work, not whether startup was perfectly quiet.

Automation Should Enforce the Gates, Not Hide the Owners

Documented Ansible is a good fit for the repeatable parts of this workflow. The Windows service module provides idempotent started and stopped states, and assert can fail a run when a prerequisite is false. For HTTP gates, use ansible.windows.win_uri when the request runs on a Windows target and ansible.builtin.uri from a non-Windows or delegated controller-side host. Either check can be polled with until, retries, and delay until the endpoint reaches the expected state.[10][11][12][13][14]

The automation should not turn a multi-team recovery into one giant button that silently crosses ownership boundaries.

I prefer a framework that can run the same sections in isolation and fail closed when required information is missing. Typical controls include:

• dry-run mode that exercises sequencing without mutation,
• an explicit execution section allowlist,
• expected active-site assertions,
• unresolved-value checks before a real run,
• approval gates around DBA-owned or network-owned actions,
• cluster-wide process-marker polling with a defined timeout,
• canary control separated from web-service control,
• result-derived status instead of unconditional success messages.

Those controls make the automation easier to trust because the code states what it is allowed to do and what still belongs to another team.

A database failover can remain a DBA-controlled task while Ansible blocks target AE startup until the DBA gate is satisfied. A network team can own the load balancer while the automation validates that the expected web state is present before workload release. That is still automation. It is just automation with operational ownership left intact.

Define Exactly Where RTO Stops

Recovery time is easy to report badly.

If the clock starts when the recovery team begins and stops when agents reconnect, the result measures technical service recovery. If the clock stops only after sample jobs and business workflows pass, the result measures something closer to usable business recovery.

Both can be valid metrics, but they are not the same metric.

I like to record at least two timestamps:

• Technical recovery: database, AE, AWI, traffic path, and agent connectivity are restored.
• Functional recovery: representative jobs complete and the required business validation has passed.

If the contractual RTO ends at the technical milestone, say so directly. Then keep the functional-validation duration beside it instead of hiding that work outside the recovery record.

The sample-job stage matters because a green dashboard does not prove the scheduler can process a workload. The job should touch the same kinds of dependencies production uses: an agent, credentials, database connectivity, file paths, network access, and whatever external system the workflow needs.

Failback Is Another Recovery Event

The return path deserves its own runbook.

Do not treat failback as "undo failover." The system has been running from the recovery site, database roles have changed, JCP and agent state may have changed, users have been routed elsewhere, and new transactions have been created since the original move.

The sequence should protect the recovery site the same way the forward path protected production:

1. Hold job queues at the recovery site.
2. Withdraw recovery-site web traffic.
3. Stop the recovery-side AE processes.
4. Re-establish the database failback prerequisites and move the primary role.
5. Confirm the primary-side listener and application name resolution.
6. Start the primary AE and prove the expected process set.
7. Restore primary web traffic.
8. Validate AWI, agents, and representative jobs again.
9. Explicitly restart normal production scheduling.

That last step should be a named task. If the forward runbook has an explicit "stop production queues" action, the return runbook should have an explicit "start production queues" action. Relying on someone to remember it after a long change window is unnecessary risk.

What a Good Test Should Change

A successful DR exercise should still produce corrections.

The useful findings are often small enough to look harmless on paper: duplicate DNS aliases, a validation owner who was not available during the test window, an embedded SQL snippet that drifted from the DBA's approved procedure, a missing queue-restart task, or a failback step that took much longer than the estimate.

Those are not reasons to dismiss the test as messy. They are exactly why the runbook is exercised before an outage.

The important part is classification. Separate technical failures from documentation defects, timing observations, ownership gaps, and test-coordination problems. Fix each one in the right place, then test the changed procedure again.

That is also where this article differs from failure-injection testing. Breaking dependencies on purpose is useful after the normal path is repeatable. First prove the basic recovery sequence can move the service cleanly between sites. Then start removing assumptions one at a time.

The Bottom Line

Automic disaster recovery is a dependency problem.

The database has to be ready before AE starts. The expected AE process set has to be stable before the web tier is published. AWI health has to be separated from site traffic eligibility. JCP reachability and agent reconnection have to be proven before production work is released. Sample jobs have to prove the scheduler can actually do useful work. Failback has to repeat the same discipline in the other direction.

If the runbook only says "stop services here, start services there," it is documenting commands, not recovery.

The design is ready when each dependency has an owner, each handoff has a gate, the team knows what evidence closes that gate, and the same process can bring the service back without improvising the return path.

Source Notes

Product behavior changes. These sources were checked against public vendor documentation available on September 20, 2026.