ubuntu-vm-builder builds images with no post-MBR gap

I'm doing "ubuntu-vm-builder kvm lucid"

I may be reading this patch wrong:
  lp:~brightbox/vmbuilder/exclude-first-cylinder

but that would appear to create the first partition 63 sectors in. If so, this is incorrect (note I haven't tried it).

Technically you should get the geometry from the disk, but most disks (not all) have 63 sectors per track and 255 heads. 63 sectors is one track in, not one cylinder in. Partitions are normally cylinder aligned, and you want a full cylinder to allow grub to have room for the MBR. Therefore, the first partition should be 63*255 sectors in, i.e. 16065 sectors, and 16065s is the offset that should be passed to parted, not 63s.

That's just me getting my tracks and cylinders mixed up then. It
should be 'ignore first track'.

The MSDOS convention is to avoid allocating the first 63 sectors (ie
0x10000 lots of 512 bytes about 32K) and that is where grub hides
itself.

Gory details here:
http://people.apache.org/~skitching/MineOfInformation/linux/Booting_Linux_on_x86_with_Grub2.html

For some reason the current parted implementation doesn't follow that
convention with virtual disks - probably because there is no detected
geometry.

Arguably there is a bug in parted as well, although as it is just a
convention not a specification that might not hold.

On 10 December 2010 11:14, Alex Bligh <email address hidden> wrote:
> I may be reading this patch wrong:
>  lp:~brightbox/vmbuilder/exclude-first-cylinder
>
> but that would appear to create the first partition 63 sectors in. If
> so, this is incorrect (note I haven't tried it).
>
> Technically you should get the geometry from the disk, but most disks
> (not all) have 63 sectors per track and 255 heads. 63 sectors is one
> track in, not one cylinder in. Partitions are normally cylinder aligned,
> and you want a full cylinder to allow grub to have room for the MBR.
> Therefore, the first partition should be 63*255 sectors in, i.e. 16065
> sectors, and 16065s is the offset that should be passed to parted, not
> 63s.
>
> --
> You received this bug notification because you are a direct subscriber
> of the bug.
> https://bugs.launchpad.net/bugs/578199
>
> Title:
>  ubuntu-vm-builder builds images with no post-MBR gap
>

--
Neil Wilson

I'm not sure that's right. Firstly, partitions are meant to be cylinder aligned (per the original DOS specs). sfdisk etc. will refuse to write partition tables unless they are, unless --force is specified. Secondly, fdisk and other utilities are now (when they can read the geometry) creating 1Mb minimum MBR gaps - See (e.g.)
  ftp://ftp.kernel.org/pub/linux/utils/util-linux-ng/v2.18/v2.18-ReleaseNotes (under fdisk)
  http://www.spinics.net/lists/util-linux-ng/msg03405.html

You have only created a 62 sector gap (63 sectors less one for the MBR), i.e. 31kb, which isn't going to be enough for grub2 by your own article (though I am not sure where the 32kb number comes from here). Last time I checked, default Ubuntu was putting in a whole cylinder. I accept that most of the reason for cylinder aligning partitions is so that CHS loaders can work, and because whacky MS copy protection schemes write data to the first cylinder, and on virtual machines these are not going to be an issue most of the time, but it seems we are at least holding ourselves a hostage to fortune with grub2 having less than 32kb.

Does parted not use a 1 cyl default when it is told the geometry?

There appears to be lots of different interpretations that's the
problem. That article was the best description I've found of the
process.

The 63s definition comes from 'partition2disk' as used in the UEC
Eucalyptus implementation. That's the main reason I went with that.
There may be an off by one error as admittedly I didn't examine the
partition addition code to see if it started at 0 or 1. (vmbuilder
starts at 0).

I'm entirely comfortable with any higher number that gives maximum
compatibility with msdos partition tools and also the GPT conversion
and GPT partitioning tools.

On 10 December 2010 13:02, Alex Bligh <email address hidden> wrote:
> I'm not sure that's right. Firstly, partitions are meant to be cylinder aligned (per the original DOS specs). sfdisk etc. will refuse to write partition tables unless they are, unless --force is specified. Secondly, fdisk and other utilities are now (when they can read the geometry) creating 1Mb minimum MBR gaps - See (e.g.)
>  ftp://ftp.kernel.org/pub/linux/utils/util-linux-ng/v2.18/v2.18-ReleaseNotes    (under fdisk)
>  http://www.spinics.net/lists/util-linux-ng/msg03405.html
>
> You have only created a 62 sector gap (63 sectors less one for the MBR),
> i.e. 31kb, which isn't going to be enough for grub2 by your own article
> (though I am not sure where the 32kb number comes from here). Last time
> I checked, default Ubuntu was putting in a whole cylinder. I accept that
> most of the reason for cylinder aligning partitions is so that CHS
> loaders can work, and because whacky MS copy protection schemes write
> data to the first cylinder, and on virtual machines these are not going
> to be an issue most of the time, but it seems we are at least holding
> ourselves a hostage to fortune with grub2 having less than 32kb.
>
> Does parted not use a 1 cyl default when it is told the geometry?
>
> --
> You received this bug notification because you are a direct subscriber
> of the bug.
> https://bugs.launchpad.net/bugs/578199
>
> Title:
>  ubuntu-vm-builder builds images with no post-MBR gap
>

--
Neil Wilson

Just checked partition2disk from Eucalyptus and it does use '63s' as the start specification for the first partition, so the patch is consistent with UEC Eucalyptus.

That's not to say that partition2disk is right either - just another thing that needs to be made consistent.

Confirmed that without this patch, "grub-setup (hd0)" (with grub-pc's grub-setup)
complains as described above, and, with this patch, it's just fine. So I will merge
in the patch. Thanks again for the patch, Neil, and for the bug report, Alex.

I've just had a look at this patch again and I see a problem.

You always offset by 63 sectors, despite the out of ubuntu-vmbuilder saying (see attached output) that the CHS geometry used was */4/32. You are offsetting by 63 sectors, so that's not a a cyl boundary.

More seriously, if this image goes via iSCSI, you ENSURE no read is 4k aligned, which is very bad performance wise.

I suggest offsetting by 64 cylinders, not 63.

Alex

sfdisk -l -uS ./ubuntu-lucid-amd64-httpmetadata.img
Disk ./ubuntu-lucid-amd64-httpmetadata.img: cannot get geometry

Disk ./ubuntu-lucid-amd64-httpmetadata.img: 2610 cylinders, 255 heads, 63 sectors/track
Warning: The partition table looks like it was made
  for C/H/S=*/4/32 (instead of 2610/255/63).
For this listing I'll assume that geometry.
Units = sectors of 512 bytes, counting from 0

   Device Boot Start End #sectors Id System
./ubuntu-lucid-amd64-httpmetadata.img1 1 39996093 39996093 83 Linux
./ubuntu-lucid-amd64-httpmetadata.img2 0 - 0 0 Empty
./ubuntu-lucid-amd64-httpmetadata.img3 0 - 0 0 Empty
./ubuntu-lucid-amd64-httpmetadata.img4 0 - 0 0 Empty

I can confirm that simply change '63s' to '64s' in the attached patch worked fine. I only applied the portion of the patch related to cylinder offsets.

If I knew Python, I'd make the partitions file take a start position in Parted format. But I don't.

Since Lucid, partitions start at 1MiB (2048s) to ensure good performance on advanced format drives (https://wiki.ubuntu.com/LucidLynx/ReleaseNotes#Partition_alignment_changes_may_break_some_systems).

Maybe ubuntu-vm-builder could do the same ?

Actually the first partition's start value of "63s" is hardcoded into the script.
Therefore it's easy to change:

Edit /usr/share/pyshared/VMBuilder/disk.py

In line 299, change

partition_start = "63s"

to

partition_start = "2048s"