Disaster Recovery That Works on the Day

Executive summary

Most organisations have a disaster recovery plan. Far fewer have disaster recovery that works when it is needed. The gap is rarely the technology. It is that recovery targets were never agreed with the business, every application was protected the same way, backups shared the same failure as production, and the plan was never rehearsed end to end.

This paper sets out a practical way to close that gap. It is written for the people who have to make DR decisions and defend them: CIOs and IT heads, infrastructure architects, and the risk and compliance teams who sign off.

Why DR fails in practice

Power failures, storage faults, network outages, fire, flooding, ransomware and plain human error all take data centres down. None of these are rare across a five-year horizon. The cost shows up as lost revenue, penalties, customer dissatisfaction, compliance findings and damage to reputation, and it grows with every hour of downtime.

When we review DR set-ups, the same failure modes appear again and again:

RPO and RTO, set properly

Recovery Point Objective (RPO) is the maximum amount of data, measured in time, the business can afford to lose. If data is replicated every five minutes, up to five minutes of work can be lost. Recovery Time Objective (RTO) is how long a service can be unavailable before the impact becomes unacceptable. An RPO of 15 minutes and an RTO of 2 hours means: lose at most the last 15 minutes of work, and be running again within 2 hours.

Two rules of thumb follow. A lower RPO generally requires more replication and more bandwidth. A lower RTO generally requires more infrastructure running at the DR site. Both cost money, so the targets must come from business impact, not from preference.

A short business impact analysis

For each application, ask its business owner four questions and write the answers down:

• What does one hour of downtime cost? Revenue, penalties, patient or citizen impact, regulatory exposure.
• How much recent data can be re-entered by hand from paper, email or another system?
• Does a regulator, contract or customer SLA set a number?
• What else depends on this system, and what does it depend on?

Then group applications into tiers. Three tiers are enough for most organisations:

The five DR patterns

There are five common ways to set up disaster recovery. They differ in what is already running at the DR site before a disaster, and therefore in recovery time, data loss and cost.

Backup and restore

Scheduled backups or volume snapshots are copied to storage at another site, ideally object storage with immutability. After a disaster, infrastructure is rebuilt and data restored. Cheap to run, slow to recover, and the RPO is the time since the last good backup. Right for tier 3 and for long-term retention behind every other pattern.

Pilot light

The data layer is kept live at the DR site: database replication, for example asynchronous log shipping or a Galera or Always On replica, and storage mirroring on a schedule. Application and web servers exist only as templates and are started on failover. Recovery takes hours because servers must boot, attach data and be checked, but running cost stays low.

Warm standby

A scaled-down copy of the whole stack runs continuously at DR, with continuous block or database replication. On failover it is scaled up and traffic is redirected. Because everything is already running, it can be tested without disruption, which is the pattern’s biggest practical advantage.

Active-passive

A full-size copy stands ready. With synchronous replication, every write is confirmed at both sites, which gives near-zero data loss but limits the distance between sites to roughly 100 km and adds write latency. Network gateways and load balancers are pre-configured so failover is a decision, not a build.

Active-active

Both sites serve users at the same time behind a global load balancer, and the loss of one is close to invisible. It is the most expensive pattern and usually needs application changes, because two sites writing to the same data must avoid conflicts and split-brain. Worth it only for services designed for it from day one.

Choosing a strategy: a four-question framework

The choice of pattern follows from four questions, asked per tier:

1. What does an hour of downtime cost? If nobody can put a number or a consequence on it, the system probably does not need the expensive patterns.
2. How much recent data can be re-entered? If staff can re-key the last 30 minutes, asynchronous replication is usually enough and synchronous replication is not needed.
3. Does a regulator or contract set the number? Banks, insurers, market intermediaries and many government systems have written continuity requirements. Start from those.
4. Can the application be changed? If not, active-active is off the table, and active-passive or warm standby is the realistic ceiling.

Most organisations land on the same shape: active-passive or warm standby for a handful of critical systems, pilot light for important business applications, and backup and restore for everything else, with immutable backups behind all three.

Reference architecture

The diagram shows the tiered design most organisations end up with. Users reach applications through a global DNS or load balancer with short time-to-live values, so traffic can be moved in minutes. Tier 1 runs warm at the DR site, tier 2 is a pilot light, and tier 3 relies on backups.

Numbered flows: (1) database changes replicate asynchronously to the DR replica, (2) storage is mirrored on a schedule, with a shorter interval for tier 1 volumes, (3) immutable backups are written nightly to a vault that production credentials cannot reach. Restores from the vault are tested regularly in an isolated area.

What must exist at the DR site besides data

• Identity: a directory controller and the same roles and groups, so people can log in.
• Network: firewall rules, VPN and partner links, IP plan or DNS changes, all kept as code.
• Certificates and secrets: renewed at both sites on the same schedule.
• Licences: licence servers and any standby licence entitlements, checked against contracts.
• Monitoring and logging: so the team can see what is happening after failover.
• Runbooks and contact lists: stored somewhere reachable when the primary site is not.

Sizing and cost

For example, 400 GB of changes a day, most of it in a 10-hour business window, needs about 400 × 8,000 ÷ 36,000 × 1.3, or roughly 115 Mbps of sustained replication bandwidth. Measure change rates from storage or database logs over at least two weeks, including month end, before buying links. Provision two links from different providers.

Storage

• DR replica: at least one full copy of protected data, plus room for snapshots at the DR site.
• Backups: retention multiplied by daily change, after deduplication. A 30-day retention commonly needs 1.5 to 3 times the protected data.
• Seeding: tens of terabytes can take weeks over a WAN link. Shipping encrypted disks for the first copy is often faster.

A worked example

Production has 40 virtual machines and 30 TB of data. After business impact analysis, 8 VMs are tier 1, 20 are tier 2 and 12 are tier 3.

Where the DR site is on-premises or in colocation, hardware for tiers 1 and 2 must still be ready, because servers cannot be bought on the day; the saving is in power, licences and tier 3 capacity. In cloud DR, full capacity is created only on drill or disaster day, which is where tiering saves the most.

Costs that are often missed

• Failback: moving data back after a cloud failover can carry large egress charges.
• Licences: some vendors charge in full for a standby database or hypervisor. Read the contract.
• Drills: a full failover test takes several people most of a day, every year.
• Keeping DR current: patching templates, renewing certificates and updating firewall rules at DR is ongoing work.

Ransomware changes the design

Classic DR assumes the primary site is lost but the data is good. Ransomware breaks that assumption: the data itself is the problem, and attackers deliberately go after backups and DR systems first. Replication makes it worse, because encrypted files replicate to the DR site within minutes.

A design that survives ransomware adds four things:

Testing: turning a plan into evidence

DR that has never been tested is a guess. Most problems appear the first time someone actually fails over: a missing firewall rule, an expired certificate, a licence server that only exists in production. Testing finds these while there is no real disaster.

On drill day

• Before: freeze changes, confirm replication lag is within RPO, tell users and the helpdesk, and write down what success means.
• During: time every step, record every manual fix however small, have the application owner test, and agree a hard stop for rolling back.
• After: report measured RPO and RTO against targets, turn every issue into a fix with an owner and a date, update the runbook that week, and book the next drill.

Keeping DR working between drills

• Alert on replication lag. Lag that quietly grows past the RPO is the most common silent failure.
• Add “and at DR?” to every change: new applications, firewall changes, certificate renewals.
• Patch DR on the same schedule as production, and upgrade both sites together.
• Review tiers once a year. Last year’s tier 3 system may now be critical.

A practical roadmap

For a mid-sized environment, a DR programme typically runs in four stages. Procurement of links, hardware and DR site space usually sets the pace.

Ten common mistakes

1. One RPO and RTO for the whole company. Either too expensive or not protective enough.
2. Choosing a product before agreeing targets. The product then defines the targets.
3. Treating replication as backup. It copies ransomware and mistakes just as fast.
4. DR site sharing risks with production. Same building, power feed, flood zone or admin accounts.
5. Copying data but not configuration. Networks, certificates and licences rebuilt by hand under pressure.
6. No named decision-maker. Hours lost before anyone declares the disaster.
7. Long DNS time-to-live values. Failover is done but users keep going to the dead site for hours.
8. Never testing failback. Getting back is often the harder half.
9. Not monitoring replication lag. The RPO is silently missed for weeks.
10. Letting the two sites drift. Different software versions cause surprises on failover day.

DR readiness checklist (40 points)

Use this list to score your current position. Anything you cannot tick with evidence is a gap worth closing.

Glossary