provisioning onto 4K native disks fails

so, we're dealing here with 4K native (or 4Kn) disks [1]

https://en.wikipedia.org/wiki/Advanced_Format#4K_native [1]

Related fix proposed to branch: master
Review: https://review.openstack.org/279485

Reviewed: https://review.openstack.org/279485
Committed: https://git.openstack.org/cgit/openstack/fuel-agent/commit/?id=a07e21c252053482bf79e245abdbec9eec231d97
Submitter: Jenkins
Branch: master

commit a07e21c252053482bf79e245abdbec9eec231d97
Author: Alexander Gordeev <email address hidden>
Date: Thu Feb 11 17:35:51 2016 +0300

    Always add configdrive partition to OS disk

    configdrive partition was put onto the first disk from the list of
    all available disks.
    When dealing with large number of nodes, sometimes disk names many
    vary from one node to another, so it's hard to predict onto which
    disk configdrive partition will land.

    Since now, configdive partition is always tied to OS disk.

    Change-Id: I6550b7c541050083991b4874f7230fc1606487df
    Related-Bug: #1544818

(This check performed automatically)
Please, make sure that bug description contains the following sections filled in with the appropriate data related to the bug you are describing:

actual result

expected result

steps to reproduce

For more detailed information on the contents of each of the listed sections see https://wiki.openstack.org/wiki/Fuel/How_to_contribute#Here_is_how_you_file_a_bug

> We need to change an image format from ISO to ext2 with 4k blocks and switch to data source ConfigDrive with disk label == 'config-2' (see [1])

that's it. We're using NoCloud data source right now it will not allow us to use any other FS with 4k blocks. (vfat isn't an option)

so, in order to fix, we need to modify do_configdrive action in manager.py https://github.com/openstack/fuel-agent/blob/57539aed3192969e884acb7cbba9952a291052c3/fuel_agent/manager.py#L343-L365

so, it will be producing ext2 image instead of iso. http://cloudinit.readthedocs.org/en/latest/topics/datasources.html#version-2

(This check performed automatically)
Please, make sure that bug description contains the following sections filled in with the appropriate data related to the bug you are describing:

actual result

expected result

steps to reproduce

For more detailed information on the contents of each of the listed sections see https://wiki.openstack.org/wiki/Fuel/How_to_contribute#Here_is_how_you_file_a_bug

Fix proposed to branch: master
Review: https://review.openstack.org/302397

Reviewed: https://review.openstack.org/302397
Committed: https://git.openstack.org/cgit/openstack/fuel-agent/commit/?id=8f5c03dcacfd20d278c4c1389762638fb14e1a02
Submitter: Jenkins
Branch: master

commit 8f5c03dcacfd20d278c4c1389762638fb14e1a02
Author: Dmitry Guryanov <email address hidden>
Date: Wed Apr 6 21:01:16 2016 +0300

    change configdrive format to ConfigDrive version 2

    We put configdrive with is9660 filesystem to a
    partition on a hard disk. New hard disks may have
    4K sectors, but blocksize of iso9660 fs is 2K so
    it will not work.

    To fix this bug we should use another filesystem (ext2)
    and another config drive format (files, directory
    structure), because NoCloud format, which is currently
    used support only vfat and iso9660 filesystems.

    Change-Id: Ia0f244f19bab3dfaceef8a092ad03667675a5557
    Closes-Bug: #1544818

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/318127

Reviewed: https://review.openstack.org/318127
Committed: https://git.openstack.org/cgit/openstack/fuel-agent/commit/?id=e74e783128daca56c05b7b6568e91cd6fc908f96
Submitter: Jenkins
Branch: stable/mitaka

commit e74e783128daca56c05b7b6568e91cd6fc908f96
Author: Dmitry Guryanov <email address hidden>
Date: Thu May 19 19:02:23 2016 +0300

    change configdrive format to ConfigDrive version 2

    We put configdrive with is9660 filesystem to a
    partition on a hard disk. New hard disks may have
    4K sectors, but blocksize of iso9660 fs is 2K so
    it will not work.

    To fix this bug we should use another filesystem (ext2)
    and another config drive format (files, directory
    structure), because NoCloud format, which is currently
    used support only vfat and iso9660 filesystems.

    Change-Id: Ia0f244f19bab3dfaceef8a092ad03667675a5557
    Closes-Bug: #1544818
    (cherry picked from commit 8f5c03dcacfd20d278c4c1389762638fb14e1a02)

I don't have hardware with 4K disk, so I've just checked, that configdrive formatted as ext2 with blocksize 4096 bytes. There should be no problems since ext2 works on 4K drives well.

Related fix proposed to branch: master
Review: https://review.openstack.org/332768

Reviewed: https://review.openstack.org/332768
Committed: https://git.openstack.org/cgit/openstack/fuel-agent/commit/?id=91680ad7e6fbf452f8a4a4a0115521a384ffba32
Submitter: Jenkins
Branch: master

commit 91680ad7e6fbf452f8a4a4a0115521a384ffba32
Author: Alexander Gordeev <email address hidden>
Date: Wed Jun 22 14:34:13 2016 +0300

    Remove genisoimage from deps

    genisoimage isn't needed anymore since fuel-agent started to produce
    non ISO9660 config drive image.

    Change-Id: I19beac5a253aa2db5eeb39c018c5bd90ebcefcd6
    Related-Bug: #1544818

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/350108

Reviewed: https://review.openstack.org/350108
Committed: https://git.openstack.org/cgit/openstack/fuel-agent/commit/?id=d58de668f6b86aac2ff526a6e1eb1486ccb41bc5
Submitter: Jenkins
Branch: stable/8.0

commit d58de668f6b86aac2ff526a6e1eb1486ccb41bc5
Author: Dmitry Guryanov <email address hidden>
Date: Thu May 19 19:02:23 2016 +0300

    change configdrive format to ConfigDrive version 2

    We put configdrive with is9660 filesystem to a
    partition on a hard disk. New hard disks may have
    4K sectors, but blocksize of iso9660 fs is 2K so
    it will not work.

    To fix this bug we should use another filesystem (ext2)
    and another config drive format (files, directory
    structure), because NoCloud format, which is currently
    used support only vfat and iso9660 filesystems.

    Change-Id: Ia0f244f19bab3dfaceef8a092ad03667675a5557
    Closes-Bug: #1544818
    (cherry picked from commit e74e783128daca56c05b7b6568e91cd6fc908f96)

Note for QA:
It's very difficult to find hardware with 4K disk, that's why we can verify the patch just by checking that after provisioning cloud-init drive has ext2 format (the last partition):

root@node-2:~# parted /dev/vda print list
Model: Virtio Block Device (virtblk)
Disk /dev/vda: 107GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number Start End Size File system Name Flags
 1 1049kB 26.2MB 25.2MB primary bios_grub
 2 26.2MB 236MB 210MB primary
 3 236MB 446MB 210MB ext2 primary
 4 446MB 26.3GB 25.8GB primary
 5 26.3GB 107GB 81.0GB primary
 6 107GB 107GB 21.0MB ext2 primary

@Sergii,

For VM that shouldn't be a problem to verify. Please check the comments here https://review.openstack.org/#/c/302397/9/fuel_agent/manager.py

Fix proposed to branch: stable/7.0
Review: https://review.openstack.org/359881

Verified on MOS 8.0 + MU3 updates.

Before updates:
After provisioning the last partition doesn't have ext2 format:
root@node-1:~# parted /dev/vda print list
Model: Virtio Block Device (virtblk)
Disk /dev/vda: 53.7GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number Start End Size File system Name Flags
 1 1049kB 26.2MB 25.2MB primary bios_grub
 2 26.2MB 236MB 210MB primary
 3 236MB 446MB 210MB ext2 primary
 4 446MB 19.8GB 19.3GB primary
 5 19.8GB 30.6GB 10.8GB primary
 6 30.6GB 52.1GB 21.5GB primary
 7 52.1GB 53.5GB 1335MB primary
 8 53.5GB 53.5GB 21.0MB primary

After updates:
After provisioning the last partition has ext2 format:
root@node-3:~# parted /dev/vda print list
Model: Virtio Block Device (virtblk)
Disk /dev/vda: 53.7GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number Start End Size File system Name Flags
 1 1049kB 26.2MB 25.2MB primary bios_grub
 2 27.3MB 237MB 210MB primary
 3 238MB 448MB 210MB ext2 primary
 4 449MB 19.8GB 19.4GB primary
 5 19.8GB 30.6GB 10.8GB primary
 6 30.6GB 52.2GB 21.5GB primary
 7 52.2GB 53.5GB 1355MB primary
 8 53.5GB 53.6GB 21.0MB ext2 primary

Reviewed: https://review.openstack.org/359881
Committed: https://git.openstack.org/cgit/openstack/fuel-agent/commit/?id=0cc992ba32b1608fcbfe2bc9db4e15491e9a6cb3
Submitter: Jenkins
Branch: stable/7.0

commit 0cc992ba32b1608fcbfe2bc9db4e15491e9a6cb3
Author: Dmitry Guryanov <email address hidden>
Date: Thu May 19 19:02:23 2016 +0300

    change configdrive format to ConfigDrive version 2

    We put configdrive with is9660 filesystem to a
    partition on a hard disk. New hard disks may have
    4K sectors, but blocksize of iso9660 fs is 2K so
    it will not work.

    To fix this bug we should use another filesystem (ext2)
    and another config drive format (files, directory
    structure), because NoCloud format, which is currently
    used support only vfat and iso9660 filesystems.

    Change-Id: Ia0f244f19bab3dfaceef8a092ad03667675a5557
    Closes-Bug: #1544818
    (cherry picked from commit d58de668f6b86aac2ff526a6e1eb1486ccb41bc5)

Verified on 7.0+mu6
I have a problem with verifying on vm that is probably related to libvirt but after some research with committer we decided to close the bug with justification of ext2 partition as well