My oVirt cluster's storage is pretty slow, so certain disk operations time out. As a result, I wasn't able to resize a disk, because the QCOW2 bitmaps were corrupt. Here's how to fix it!
My situation
- oVirt 4.5 cluster (should work for 4.3 and 4.4)
- Hosted Engine on the cluster (probably doesn't matter)
- NFS shared storage (may need adjusted steps for gluster, iSCSI should be fine)
- VMs with QCOW2 disks (not applicable to RAW obviously, unknown for other formats)
When I attempted to resize a VM, I would get a combination of any of the following issues:
The disk was "0" GB in the UI
Error in the Web UI log: "Failed to update VM '$VM' with the new volume size. VM restart is recommended."
- (usually after one of the other errors in the logs)
Error in the Web UI log: "VDSM $HOST command ExtendVmDiskSizeVDS failed: Wrong resize disk parameter"
Error in the Web UI log:
VDSM $HOST command ExtendVmDiskSizeVDS failed: Command ['/usr/bin/qemu-img', 'resize', '-f', 'qcow2', '/rhev/data-center/mnt/$DATASTORE/$DATASTORE_UUID/images/$DISK_UUID/$SNAPSHOT_UI', '$NEW_SIZE'] failed with rc=1 out=b'' err=b"qemu-img: Bitmap '1d1415c1-7f87-4643-8223-912030d92200' is inconsistent and cannot be used\nTry block-dirty-bitmap-remove to delete this bitmap from disk\n"
- Broken up, we see:
- /usr/bin/qemu-img resize -f qcow2 /rhev/data-center/mnt/$DATASTORE/$DATASTORE_UUID/images/$DISK_UUID/$SNAPSHOT_UUID $NEW_SIZE
- qemu-img: Bitmap '1d1415c1-7f87-4643-8223-912030d92200' is inconsistent and cannot be used
- Try block-dirty-bitmap-remove to delete this bitmap from disk
- Broken up, we see:
In my case, the above essentially happened in sequence - first the disk was 0 GB, then after trying a resize it failed with "Wrong resize disk parameter". Now the size was finally correct in the web UI, but re-trying the resize would fail with the "Bitmap .. is inconsistent and cannot be used".
My Fix
Shutdown the VM. Wait until it's shut down and "not running" before continuing.
I connected to the "SAN" via SSH, but you could do this on the hosts too.
cd into the NFS export path, and then into the particular disk's folder
- $DATASTORE_UUID/images/$DISK_UUID/
- In this folder, you should have three files with the same UUID. The file without an extension is the actual disk.
(Optionally) Reproduce the failing command:
- sudo qemu-img resize -f qcow2 15005aaa-91ae-46a9-97e2-b5dc6b5d2bb1 751619276800
- Predictably, this should fail the same way.
(Optionally) List all of the bitmaps on the disk image:
- sudo qemu-img info 15005aaa-91ae-46a9-97e2-b5dc6b5d2bb1
Remove the corrupt bitmap:
- sudo qemu-img bitmap --remove $DISK_UUID $BITMAP_UUID
- sudo qemu-img bitmap --remove 15005aaa-91ae-46a9-97e2-b5dc6b5d2bb1 1d1415c1-7f87-4643-8223-912030d92200
At this point you can try triggering the resize again, either through the UI or through qemu-img, but I found that I would need to remove all bitmaps before it'll work.
sudo qemu-img resize -f qcow2 15005aaa-91ae-46a9-97e2-b5dc6b5d2bb1 93530800-5cc6-47bf-87a0-3be3abbd6674`` sudo qemu-img bitmap --remove 15005aaa-91ae-46a9-97e2-b5dc6b5d2bb1 93530800-5cc6-47bf-87a0-3be3abbd6674`` sudo qemu-img bitmap --remove 15005aaa-91ae-46a9-97e2-b5dc6b5d2bb1 4e20da75-659c-43ef-8f57-8de65c2721d0`` sudo qemu-img bitmap --remove 15005aaa-91ae-46a9-97e2-b5dc6b5d2bb1 738dec0b-e020-4889-ba5e-bbae743a1d90`` sudo qemu-img bitmap --remove 15005aaa-91ae-46a9-97e2-b5dc6b5d2bb1 5d693475-8d18-4063-9d23-b7e6d4844d52`` sudo qemu-img bitmap --remove 15005aaa-91ae-46a9-97e2-b5dc6b5d2bb1 f509099d-e180-4376-805f-726d07ca545f`` sudo qemu-img bitmap --remove 15005aaa-91ae-46a9-97e2-b5dc6b5d2bb1 e05c894b-a80d-459a-881c-0c2a2102e61f`` sudo qemu-img bitmap --remove 15005aaa-91ae-46a9-97e2-b5dc6b5d2bb1 710852d7-5db4-4cf5-9b12-91664a8d8768``
Resize the disk - you can do this from qemu-img but the oVirt UI will probably be confused until it's restarted or something triggers a rescan of the pool, so do it through the UI.
Resize the partition / filesystem etc. in the VM as usual.