Data Recovery for Virtual Systems: VMware, Hyper-V, Proxmox and KVM

First Steps After Data Loss in a Virtual Environment

These measures improve the starting position for a later data recovery and cost little time. They complement our general first aid measures for data loss:

1. Shut down the affected VMs and take the host out of production. Every further write access to the storage can overwrite areas that are needed for the reconstruction.
2. Secure configuration files and logs on a separate medium, for example the VMX file, the libvirt XML or the checkpoint files. They contain details about the order and the mapping of the images.
3. Document the state: RAID level and order of the drives, the last change to the infrastructure, the first error message and when it appeared. These details noticeably shorten the diagnosis.
4. Check existing backups, but do not restore them onto the original. A restore overwrites exactly the storage that data is still to be recovered from. A restore makes sense only onto a different target.
5. With a degraded array, do not start a second rebuild. An aborted rebuild can usually not be repaired by another one.
6. Do not run recovery software on the original storage. Programs of that kind work at the level of individual drives and map neither the array nor the layers above it.

VM Data Recovery Process

The procedure always follows the same order, from the hardware up to the individual file in the guest system:

1. Diagnosis of every physical storage device. We check the readability and condition of each hard drive and each SSD individually. Mechanically or electronically damaged drives are repaired by our specialists in the clean-room laboratory, where read/write heads or electronics are replaced with spare parts from the same model. What to watch for per drive type is set out on our pages about hard drive data recovery and SSD data recovery.
2. Sector-accurate, write-protected images of all drives or of the presented LUN. Every further step runs on these copies only, the original stays unchanged.
3. Closing the array arithmetically. We determine level, order, block size, offsets and parity from the metadata, even without the original controller and without the original enclosure.
4. Evaluating the datastore layer: VMFS, LVM and thin pools, ZFS, NTFS or ReFS. Missing or overwritten metadata is reconstructed from its structures.
5. Reassembling the container files. VMDK, VHDX, QCOW2 and VDI are joined together with their delta chains in the correct order; missing descriptors are regenerated.
6. Opening the guest file system and extracting the data. Whether the guest runs Windows or Linux: on request we deliver individual files and folders, consistent SQL databases or a virtual machine that boots again.

Before handover you receive a list of the recovered files and folders and can check against it whether the data you need is complete. Handover is on a new storage device that we provide. The entire process from enquiry to return is described on our page about the data recovery process. If decommissioned drives leave your operation afterwards, our data erasure service handles them to DIN 66399, with an erasure certificate on request.

Costs and Prices for VM Data Recovery

The prices follow our regular scale for storage systems: standard data recovery from 449 €, express handling around the clock at an individual fixed price. The price becomes binding after the diagnosis, before you place the order, with no surprises afterwards.

Five points essentially determine the effort:

• the number of drives, as each one is diagnosed and cloned individually
• the type of damage, since purely logical cases take less effort than mechanical or electronic damage
• the amount of clean-room work required, for instance when replacing read/write heads
• the complexity of the environment, from a single VM on a mirror to a nested array with long delta chains
• the urgency, since in an emergency express handling runs around the clock and is scheduled with priority

A blanket success rate cannot be stated seriously for virtual environments. From more than 20 years of experience with failed server systems we can say what shifts the outcome: a complete set of drives, a state left unchanged since the failure and the absence of DIY recovery attempts all work in your favour. An overview of all services is available in our price overview.

Virtualisation Platforms We Work With

As a data recovery service with its own clean-room laboratory we work vendor-independently; what matters for the reconstruction is not the product but the structure underneath. These platforms and virtualisation technologies reach us most frequently:

We equally handle other virtualisation software such as oVirt, Red Hat Virtualization or Nutanix AHV. If the virtual machines sit on network storage, the evaluation of that device is added: our pages on NAS and SAN data recovery and on the systems from Synology, QNAP and Buffalo cover that.

Data Structure in Virtual Environments

A hypervisor consolidates many systems onto shared storage and distributes its resources among the individual guests. That is exactly what makes the difference in a failure: an outage rarely affects a single machine, but every VM that sits on the same datastore. And between the physical drive and the file a user is missing, there are considerably more management layers than on a classic server.

Two practical consequences follow:

• The order is fixed. Every layer is reassembled from the bottom upwards. An intact container file is of little use as long as the array beneath it has not been closed arithmetically.
• Recovery tools only ever see one layer. They recognise a single drive or a single file system, but neither the RAID array nor the VMFS datastore nor a broken delta chain.

Inside the virtual disk sits the file system of the guest operating system, depending on the VM for example NTFS and ReFS, ext4, XFS or UFS. How an array distributes data across several drives is described in detail on our page about RAID data recovery.

Typical Damage Patterns in Virtualised Environments

The symptoms are similar across every hypervisor. A symptom on its own, however, does not allow a reliable conclusion about the cause; only the diagnosis shows which layer the fault actually sits on.

In the laboratory we keep seeing the same causes: defective hard drives and SSDs in the array, overvoltage and power failure, mistakes during migration and implementation, accidentally deleted or formatted volumes, failed controllers and malicious software. How logical defects and mechanical defects differ is explained separately. Which warning signs precede the failure of an array is collected in our overview of RAID failure symptoms.

Snapshots and Delta Chains

A snapshot freezes the state of a virtual machine. From that moment on, the base image is no longer written to and every change lands in a delta file of its own. Each further snapshot creates another link. When reading, the hypervisor works its way back through this chain until it finds the block it needs.

This is where the most common damage pattern in virtual environments arises. Three triggers keep coming up:

• a supposedly old delta is deleted
• a link of the chain is moved to another volume
• a consolidation is aborted because space runs short

The current state is then spread across several files and looks lost. It can be reconstructed in many cases nonetheless, because the order and the mapping of the blocks can be determined from the metadata of the delta files. The condition is that no further writes reached the storage afterwards.

One classification matters here: a snapshot is not a backup. It sits on the same storage as the machine it is meant to protect and fails together with it. Our article RAID is not a backup describes the same misconception for the failure safety of storage arrays.

Ransomware in Virtualised Environments

Cyber attacks now aim at the hypervisor layer rather than at individual workstations. The reason is simple: encrypting the container files on the datastore hits every guest system of a company at once, including the servers that run applications and databases.

For the recovery that is not necessarily the worst starting position. Because virtual disks are very large, malicious software frequently limits the encryption to parts of the files, for example header areas or sections at fixed intervals. The areas in between remain unchanged, so files and database contents can in many cases be won back from the untouched parts. Whether and to what extent that succeeds is shown by the analysis; a blanket promise would not be serious.

What makes sense in this situation:

• disconnect the host from the network immediately
• delete nothing and leave the state unchanged
• pay no ransom
• keep the affected storage media complete and available for the analysis

If personal data is affected, a reporting obligation under the GDPR may apply. Our IT forensics service is available for the evidence preservation this requires and for the analysis of the attack path.

Frequently Asked Questions About Virtual System Data Recovery

Can you recover individual files or only the complete virtual machine?

Both are possible. After reconstructing the container file we open the file system of the guest and can extract specific folders, files or database contents from it. Alternatively we return the complete virtual disk so that the machine can be started again in your environment. Which option makes sense depends on whether you are rebuilding the infrastructure anyway or want to resume operation as quickly as possible.

Do you need the original host or the RAID controller?

No. We reassemble the array in software from write-protected images of the individual drives, so neither the original enclosure nor the original controller is required. In most cases the storage devices themselves are enough. What matters is that the set is complete, including supposedly defective disks, and that the bay positions are labelled.

Can a VM still be saved after an aborted consolidation?

Often yes. What matters is whether all links of the chain are still present and whether further writes reached the storage in the meantime. From the metadata of the delta files we determine the order and the mapping of the blocks and reassemble the chain in the correct sequence. Further consolidation attempts on the original, by contrast, usually make the starting position worse.

What happens to databases inside the virtual machine?

Database files sit in the file system of the guest and are recovered together with it. If the system was running at the moment of the failure, the data set can be inconsistent because transactions were only partially written. In that case we additionally evaluate the log files and provide a consistent state. The same applies to damaged backup files from common backup software such as Veeam that were stored on the same storage system. If the backup sits on tape, our page on tape data recovery takes it from there.

How quickly can you work during a production outage?

Purely logical cases are often completed within a few days. If several drives are physically damaged it takes longer, because each one has to be repaired and cloned before the reconstruction begins. For time-critical server systems an express service is available around the clock that keeps downtime as short as technically possible. You receive that assessment directly after the diagnosis.