Disaster recovery on OpenStack
OpenStack has no single “DR button”. Site-level recovery is built from three things: storage replication to a second OpenStack deployment, the project's setup kept as code, and a runbook that brings instances up in the right order. This page explains how those pieces fit.
HA is not DR. Features such as live migration or Masakari instance HA protect you when a server fails inside one site. They do not help if you lose the whole site. For that you need a second OpenStack deployment in another location.
What each OpenStack service needs for DR
| Layer | Service | What we set up for DR |
|---|---|---|
| Volumes | Cinder on Ceph RBD | Ceph RBD mirroring to the DR Ceph cluster using the rbd-mirror daemon. Snapshot-based mirroring on a schedule for most volumes. Journal-based mirroring for the few volumes that need a lower RPO. |
| Images | Glance | The same images published at both sites from one image pipeline, so instances boot from identical templates. |
| Object data | Ceph RGW or Swift | Multi-site replication for application object data and backup targets. |
| Compute | Nova | Flavors, quotas and key pairs created at DR in advance. Instances are recreated from mirrored volumes, which is why we prefer boot-from-volume for protected workloads. |
| Networking | Neutron | Networks, subnets, routers and security groups recreated at DR from code. Decide early whether applications keep their IP addresses (which needs a stretched network or route changes) or get new ones with DNS updates. We usually recommend new IPs with DNS. |
| Identity | Keystone | A Keystone at each site with the same projects and roles, with users coming from a shared directory (AD or LDAP) so people log in the same way at both sites. |
| Automation | Heat or Terraform, plus Ansible | Templates that recreate each protected project, and playbooks that attach the mirrored volumes and start instances in dependency order. |
| Backup | Trilio for OpenStack or agent-based backup | Point-in-time copies kept separately from replication. Replication copies corruption and ransomware just as quickly as good data, so you need versions to go back to. |
A typical failover, step by step
- Declare the disaster. One named person decides, based on criteria agreed in advance. Most delays happen here, not in the technology.
- Promote the mirrored volumes at DR. If the primary site is unreachable, the DR copies have to be forced to primary:
rbd mirror image promote --force <pool>/<image>
In practice this runs from a playbook over every protected image, not by hand. - Recreate or verify the project at DR. Networks, security groups and ports come from the same code used to build them originally.
- Start the tiers in order. Database first, check it is consistent, then application servers, then web servers.
- Move users. Update DNS or the global load balancer to point at DR. Keep DNS TTLs short beforehand so this takes minutes, not hours.
- Check with the application owners. A real user logs in and completes a real transaction before anyone announces that the service is back.
Failback is the reverse, done as a planned switchover in a maintenance window: resync data back to the original site, demote at DR, promote at the primary, and move users back.
What trips teams up
- Volumes are replicated but nothing else is. Flavors, ports, security groups and metadata are not in the volume. If they are not in code, someone rebuilds them by hand during the outage.
- The two sites drift apart. Different OpenStack or Ceph versions at each site cause surprises on failover day. Upgrade both together.
- Nobody watches replication lag. When bandwidth runs short, mirroring falls behind quietly. Monitor
rbd mirror pool statusand alert when lag passes your RPO. - Failback is never tested. Teams test failing over, then discover that getting back is the harder half. Test both directions.
Not sure where your DR stands?
Send us your application list and a short note on how backups work today. We will come back with a plain gap review: which systems are exposed, what a realistic recovery time looks like, and what it would take to close the gap.