Corruption of File Data/Filename w/ Virtual VFAT (VVFAT)

I forgot to mention this corruption was observed with both QEMU and KVM.

Thanks for the report.

I'm marking this 'critical' since the reporter claims data corruption. I haven't confirmed it yet.

A couple of questions.
 1) Can you reproduce this on an Ubuntu Jaunty host?
 2) Does the problem only happen when copying data guest->host? What about host->guest?
 3) What protocol are you using to do the copy? scp? ftp? rsync? samba? nfs? Something else?

:-Dustin

Also, what type of guest disk? qcow? qcow2?

:-Dustin

And what's the command line you're using to launch this?

Hi Dustin,

    Excellent questions. I should have included that information.

1) Can you reproduce this on an Ubuntu Jaunty host?
     I have not tried Jaunty 9.04 beta. Sorry. What version of KVM and QEMU are provided? I checked the main KVM changelog and I don't see anything about it. I thought I read something about a rewrite of the VFAT code but I cannot find that in the Changelog for the life of me. Off topic, is there a way to peek at other APT repositories to, say, get the Changelog of a package in an upcoming release?
    If you have good reason to believe this is likely to be resolved in Jaunty, let me know. I'll try to move up my upgrade to 9.04 and test for you. My test machine doesn't have the horsepower/RAM for VMs.

2) Does the problem only happen when copying data guest->host? What about host->guest?
    Yes. I have only detected corruption when exporting data from the guest system to the host (i.e. writes). I have read many small files without any trouble and more convenient than Samba...
    Well, you'll be interested in this new behavior. I thought I would just test a big file for you really quick. So, I put a new 150MB ISO image in the VFAT directory and started KVM. But, this time, I received this error:

    kvm: block-vvfat.c:96: array_get: Assertion `index < array->next' failed.

I created a new directory, moved the ISO image there, and updated the -drive parameter to point to the new directory. KVM started correctly. I performed the test copy, importing the ISO image file without corruption. I believe this problem is isolated to large file exports (i.e. VFAT writes).
    Now, I'm lost regarding that error. That sounds like the directory file on the host is getting corrupted, in addition to filenames and file data. But, I don't see any bogus data in the directory file, although I don't really know what or how to look for that. The other option is that there is some restriction on the Virtual FAT device that I don't understand. For example, the last time I used this particular VM, WindowsXP was constantly annoying me about the VFAT drive being full and that I should run Disk Cleanup on it. WindowsXP sees the VFAT drive as having a total capacity of 512MB and there was, at that time, more than 512MB used by files in the VFAT directory and its subdirectories on the host. However, the drive was nearly empty when I first noticed this problem and my tests were completed before the drive was "full." It filled up as I kept different versions of the same ISO file for comparison.

3) What protocol are you using to do the copy? scp? ftp? rsync? samba? nfs? Something else?
    The copying that produced the corruption was Windows Explorer copy/paste or drag/drop. I'm sorry I didn't try copy. SFTP network-based export worked perfectly. Mounting the NTFS filesystem within the VM image (after converting to raw with qemu-img convert) and using cp from the host also worked. I assume all TCP/IP or Network based transfers are safe.

4) Also, what type of guest disk? qcow? qcow2?
    Originally, I was using qcow2 format. But, before I thought of SFTP (doh!) I decided to try to loopback mount the VM disk image, w...

I think this might be fixed by:
 * http://git.kernel.org/?p=virt/kvm/kvm-userspace.git;a=commit;h=385aae33c08e17ec63671b9fa1c503c21283bc1f

:-Dustin

Soren disagrees in IRC though...

Soren indicates that VVFAT is a very non-standard way of doing disk allocation for VM's.

Is there some reason that you're using this rather than any of the more standard disk image formats?

:-Dustin

Granted. I'm sure that's true, but yes, I am using VVFAT for two reasons: (1) it is direct, read/write, host-to-guest file access, and (2) it is the easiest method to transfer files between host and guest.

When I first encountered the need to transfer files I had on my host to my VM, I looked for direct methods because it seemed more strait forward. VVFAT worked for small files, which is all I had to move for a long time. Also, I didn't test writes to the VVFAT because I didn't do that much, until recently. After using this feature for a while, I found it to be extremely convenient and useful.

Secondly, I found VVFAT much easier to use than the network-based transfer methods. I didn't want to use QEMU's CIFS/SMB/Samba option because it does not allow the use of an alternate Samba configuration file. I didn't want to have to configure /etc/smb.conf just run my VM in user space. (At least, I couldn't figure out how to do it, otherwise.) Similar inconveniences occur using FTP and SSH (which I'm using now, but wouldn't want to at work).

Generally speaking, I'm not really using VVFAT as a disk image. I'm using it as a file transfer method. Using an actual disk image would require mounting the disk image in on the host, either via root access, FUSE, or using filesystem editing tools, which are inconvenient. I have created ISO9660 images to import files, which works, but requires creating a new image every time I need to import more files and doesn't support writes, of course. (I don't know how UDF would work on a disk image file. Would that work?)

I suppose the next thing I should try is an NTFS-3g filesystem which I could mount with FUSE in my Linux host and natively in my Windows VM. That's still less convenient than VVFAT, however, since I would have to mount/umount the image on the host to get access and the VM would still have to be off. I would still prefer VVFAT to all of these options.

I really appreciate your help. Thank you for taking my report seriously. If VVFAT is buggy and cannot be fixed easily, I defer to the judgment of others and ask only that this limitation be well documented in the primary references. But, I'm convinced that a direct, host-to-guest file access scheme is valuable. If there is another solution, I apologize for my ignorance and I would appreciate being pointed in the right direction.

vvfat is not commonly used/tested in upstream KVM or QEMU. Patches are certainly welcome but it's definitely considered an experimental feature.

Although I haven't personally confirmed this, I'm going to mark it 'confirmed', since vvfat support is known to be 'experimental' in upstream QEMU. Furthermore, I'm going to reduce the importance from critical to wishlist. As the upstream maintainer has stated, this feature is known to be experiment, it's not under active development, though patches are welcome.

:-Dustin