Restoring one VM through a healthy Proxmox interface is a valuable test. It does not answer the harder question: can the lab recover when the original host, local configuration, and convenient browser bookmarks are gone?
A disaster-recovery drill starts with an empty replacement host or isolated test machine and rebuilds the minimum platform needed to restore services. It must not endanger the live cluster or overwrite the only backup.
Define a safe scenario #
Write the simulated failure in one sentence: “The primary Proxmox boot device and its local configuration are unavailable; the independent backup server and network equipment survived.” This defines what can be used and what must be reconstructed.
Choose an isolated VLAN or disconnected switch for restored guests. Duplicate DHCP servers, domain controllers, databases, and IP addresses can disrupt the live environment. Confirm that the test host is not accidentally joining the production cluster and that its storage targets are disposable.
Set a recovery objective and start a timer. The goal is evidence, not speed at any cost. Record every missing credential, undocumented address, and decision that required memory.
Assemble the recovery packet #
The operator should have offline access to:
- supported Proxmox installation media and hardware boot instructions;
- management IP, gateway, VLAN, and DNS information;
- backup server address, datastore, user, and credential-retrieval process;
- encryption-key retrieval instructions;
- guest restore order and important VM IDs;
- storage layout and minimum capacity;
- application-level validation steps.
Do not store plaintext passwords in the packet. It should explain how an authorized person obtains them from the protected source even when normal identity or DNS services are unavailable.
Verify installation media and the backup access path before declaring the maintenance window open. This is preparation, not cheating; real recovery also depends on usable media.
Install only the minimum platform #
Install the supported Proxmox release on the test host. During setup, use the planned isolated management address and a hostname that cannot collide with the live environment. Apply updates from trusted repositories if the exercise allows internet access.
Capture the basic state after installation:
1pveversion --verbose
2ip -brief address
3ip route
4lsblk -o NAME,SIZE,TYPE,FSTYPE,MOUNTPOINTS
These commands create a checkpoint for the drill log. They also catch the common mistake of building on the wrong disk, interface, or network.
Do not spend the first hour recreating every optimization from the failed host. Build enough bridge, storage, and resolver configuration to reach the backup server and run restored guests safely. Performance tuning comes after recovery.
Host configuration backups can accelerate this stage, but restore them selectively. Interface names, disk identifiers, cluster membership, and storage paths may differ on replacement hardware. Treat saved files as evidence of the previous design, not a script to overwrite a clean installation blindly.
Reconnect the backup source #
Add the independent Proxmox Backup Server using a dedicated recovery credential with the required read access. Verify the server fingerprint through the documented trusted channel rather than accepting a value from an unexpected prompt.
From the Proxmox UI or supported CLI, confirm that backup groups and snapshots are visible. A datastore listing proves authentication and basic connectivity; it does not yet prove that chunks are readable or encryption keys are present.
Inspect available storage and snapshots with the supported tools for the installed release. Avoid scripts that parse internal datastore files. The drill should exercise the same interfaces expected during a real incident.
Restore infrastructure in dependency order #
Restore the smallest service that proves the platform first. A disposable Linux utility or internal resolver can validate guest creation, storage writes, virtual networking, console access, and boot.
Then follow the documented order. A typical sequence might be:
- routing, firewall, or DNS required by other guests;
- identity and time services;
- databases and shared storage clients;
- application servers;
- reverse proxy, monitoring, and non-critical tools.
The actual order depends on the lab. If the firewall itself is virtualized, the recovery network needs a bootstrap path that does not require that firewall to reach the backup server.
Restore a guest under a temporary ID when the original number could conflict. Keep its NIC disconnected or attached to the isolated bridge until its identity and address are understood.
After restore, inspect configuration before starting:
1qm config 120
For containers, use the corresponding pct config command. Check network
bridge, VLAN tag, boot order, mounted storage, memory, and attached devices.
Hardware passthrough may not exist on the replacement host and should not block
recovery of unrelated services.
Validate at the application layer #
A green VM state is an infrastructure result, not a service result. For each restored workload, verify:
- system time and name resolution;
- expected filesystem mounts and free space;
- application data counts or recent records;
- authentication with a test account;
- background workers and scheduled jobs;
- communication with required dependencies;
- backup agent or next scheduled protection path.
Keep external integrations disabled until they are safe. A restored mail job, webhook worker, or automation controller can repeat old actions from the isolated copy.
For databases, run the vendor-supported consistency checks and compare an application-level record or export. Do not infer database health from an open TCP port.
Record time and gaps #
At every milestone, record elapsed time: host installed, backup connected, first guest booted, first critical service usable, and recovery complete. Compare the result with the recovery time objective.
Also record the recovery point used for each service. The newest snapshot is not always the safest after corruption or a bad deployment. The runbook should explain how to choose an earlier point and how much data that choice discards.
Classify gaps immediately:
- missing information, such as an undocumented VLAN;
- missing access, such as a credential available only from the failed guest;
- missing data, such as an excluded mount point;
- incompatible hardware or capacity;
- unclear procedure or unsafe manual step.
Fixing the document during the drill is useful, but preserve the original gap in the report. Otherwise the next review cannot tell which improvements the exercise produced.
Test the return to protection #
Recovery is incomplete while the restored environment has no backup. Configure the intended backup target or a safe drill target, run a new backup of one restored guest, and restore a file from that new recovery point.
This closes the loop: the recovered platform can protect changes made after the incident. It also reveals credentials or firewall rules that allowed restore traffic but not new backup traffic.
Check monitoring and notification paths. During a real disaster, silent backup or storage failures are especially costly because the environment is changing quickly.
Tear down without erasing evidence #
Export the drill notes and relevant non-sensitive logs. Power off isolated restored guests before reconnecting any test network. Remove temporary recovery credentials or restrict them again, and verify that production backup retention was not altered.
Only then erase disposable test storage according to the lab's normal process. Never delete backup snapshots merely to make the test environment tidy.
Schedule the follow-up work with owners and dates. A finding such as “the key was hard to find” is not resolved until the retrieval path is changed and another authorized person can use it.
Hold a short review after the environment is safe. Separate changes that reduce recovery time from changes that reduce recovery risk, prioritize both, and test the highest-risk correction promptly. A rewritten runbook is not verified until someone follows it from the same constrained starting point.
Repeat after meaningful change #
Run a full drill periodically and a smaller restore test more often. Repeat after replacing backup infrastructure, changing encryption, redesigning the management network, or moving critical services between guests.
Vary the assumed failure. The next exercise might remove the backup server while keeping Proxmox, or simulate loss of the primary site and recovery from an off-site copy. No single scenario proves recovery from every event.
An empty-host drill turns disaster recovery from a diagram into a measured capability. The most valuable output is not the restored VM. It is the shorter, safer path created by every missing fact discovered while the real environment was still available.
Schedule that next test before closing the report; otherwise “later” becomes another undocumented dependency. Recovery confidence expires when the environment changes.