My oVirt cluster's storage is pretty slow, so certain disk operations time out. As a result, I wasn't able to resize a disk, because the QCOW2 bitmaps were corrupt. Here's how to fix it!

My situation

  • oVirt 4.5 cluster (should work for 4.3 and 4.4)
  • Hosted Engine on the cluster (probably doesn't matter)
  • NFS shared storage (may need adjusted steps for gluster, iSCSI should be fine)
  • VMs with QCOW2 disks (not applicable to RAW obviously, unknown for other formats)

When I attempted to resize a VM, I would get a combination of any of the following issues:

  • The disk was "0" GB in the UI

  • Error in the Web UI log: "Failed to update VM '$VM' with the new volume size. VM restart is recommended."

    • (usually after one of the other errors in the logs)
  • Error in the Web UI log: "VDSM $HOST command ExtendVmDiskSizeVDS failed: Wrong resize disk parameter"

  • Error in the Web UI log:

    VDSM $HOST command ExtendVmDiskSizeVDS failed: Command ['/usr/bin/qemu-img', 'resize', '-f', 'qcow2', '/rhev/data-center/mnt/$DATASTORE/$DATASTORE_UUID/images/$DISK_UUID/$SNAPSHOT_UI', '$NEW_SIZE'] failed with rc=1 out=b'' err=b"qemu-img: Bitmap '1d1415c1-7f87-4643-8223-912030d92200' is inconsistent and cannot be used\nTry block-dirty-bitmap-remove to delete this bitmap from disk\n"
    
    • Broken up, we see:
      • /usr/bin/qemu-img resize -f qcow2 /rhev/data-center/mnt/$DATASTORE/$DATASTORE_UUID/images/$DISK_UUID/$SNAPSHOT_UUID $NEW_SIZE
      • qemu-img: Bitmap '1d1415c1-7f87-4643-8223-912030d92200' is inconsistent and cannot be used
      • Try block-dirty-bitmap-remove to delete this bitmap from disk

In my case, the above essentially happened in sequence - first the disk was 0 GB, then after trying a resize it failed with "Wrong resize disk parameter". Now the size was finally correct in the web UI, but re-trying the resize would fail with the "Bitmap .. is inconsistent and cannot be used".

My Fix

  1. Shutdown the VM. Wait until it's shut down and "not running" before continuing.

  2. I connected to the "SAN" via SSH, but you could do this on the hosts too.

  3. cd into the NFS export path, and then into the particular disk's folder

    • $DATASTORE_UUID/images/$DISK_UUID/
    • In this folder, you should have three files with the same UUID. The file without an extension is the actual disk.
  4. (Optionally) Reproduce the failing command:

    • sudo qemu-img resize -f qcow2 15005aaa-91ae-46a9-97e2-b5dc6b5d2bb1 751619276800
    • Predictably, this should fail the same way.
  5. (Optionally) List all of the bitmaps on the disk image:

    • sudo qemu-img info 15005aaa-91ae-46a9-97e2-b5dc6b5d2bb1
  6. Remove the corrupt bitmap:

    • sudo qemu-img bitmap --remove $DISK_UUID $BITMAP_UUID
    • sudo qemu-img bitmap --remove 15005aaa-91ae-46a9-97e2-b5dc6b5d2bb1 1d1415c1-7f87-4643-8223-912030d92200
  7. At this point you can try triggering the resize again, either through the UI or through qemu-img, but I found that I would need to remove all bitmaps before it'll work.

    sudo qemu-img resize -f qcow2 15005aaa-91ae-46a9-97e2-b5dc6b5d2bb1 93530800-5cc6-47bf-87a0-3be3abbd6674``
    sudo qemu-img bitmap --remove 15005aaa-91ae-46a9-97e2-b5dc6b5d2bb1 93530800-5cc6-47bf-87a0-3be3abbd6674``
    sudo qemu-img bitmap --remove 15005aaa-91ae-46a9-97e2-b5dc6b5d2bb1 4e20da75-659c-43ef-8f57-8de65c2721d0``
    sudo qemu-img bitmap --remove 15005aaa-91ae-46a9-97e2-b5dc6b5d2bb1 738dec0b-e020-4889-ba5e-bbae743a1d90``
    sudo qemu-img bitmap --remove 15005aaa-91ae-46a9-97e2-b5dc6b5d2bb1 5d693475-8d18-4063-9d23-b7e6d4844d52``
    sudo qemu-img bitmap --remove 15005aaa-91ae-46a9-97e2-b5dc6b5d2bb1 f509099d-e180-4376-805f-726d07ca545f``
    sudo qemu-img bitmap --remove 15005aaa-91ae-46a9-97e2-b5dc6b5d2bb1 e05c894b-a80d-459a-881c-0c2a2102e61f``
    sudo qemu-img bitmap --remove 15005aaa-91ae-46a9-97e2-b5dc6b5d2bb1 710852d7-5db4-4cf5-9b12-91664a8d8768``
    
  8. Resize the disk - you can do this from qemu-img but the oVirt UI will probably be confused until it's restarted or something triggers a rescan of the pool, so do it through the UI.

  9. Resize the partition / filesystem etc. in the VM as usual.