Cinder LVM over iSCSI driver under qemu 2.0 and kernel <3.11

Using a novel kernel (3.13.0-36-generic) does not help to solve this problem.

The only other solution is to flush a device step by step. For this to be implemented `qemu` and `tgtd` must be updated.

Uploaded is the enhanced log file with backtraces of all the CPUs on the machine.

New logs uploaded. These are results of invocation of the `show_state` kernel function on hung-up action.

According to investigation results it was a `tgt` daemon who was responsible for the over-caching. This led to a `fsync(2)` call taking too long to flush cache, since the cache can be as big as few percents of memory (controlled by /proc/sys/vm/dirty_*).

The solution is one of the following:

1) Use 'write-cache off' option for the `tgt` volume configuration. This generates a lot of `fsync(2)`s being issued to the iSCSI LUN. Because of these synchronizations performance degrades drastically (around 50%).

Implementing this only requires updating the python-cinder OpenStack package for Fuel 5.1.1.

2) Use 'bsoflags direct' option for `tgt` volume configuration. This instructs `tgt` to open a backing-store device using O_DIRECT. All of the operations then miss the page cache and go directly to underlying device (a LVM device served by the 'dm' Linux module). This has almost no impact on the device performance.

Implementing this, however, requires updating both `tgt`(i.e. the `tgt-admin` script) and python-cinder packages for Fuel 5.1.1.

Currently both of the solutions are being benchmarked. More results to follow.

According to benchmark results only former (i.e. `bsoflags direct`) gives true results for the `ext` FS benchmark via `fio` utility with `direct=true` option.

In my opinion, the guest VM must be responsible for all the caching so it happens transparently for the user application. All other layers of this monster (qemu <-> iscsi <-> tgt <-> lvm <-> disks) must use O_DIRECT or its analogs so no false feeling of safety is experienced by the user application (and fewer abstractions to break).

I'm going to incorporate changes into the Fuel's cinder and update `tgt-admin` script as appropriately.

Package cinder has been built from changeset: https://review.fuel-infra.org/364
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1.1-stable-364/centos

Package cinder has been built from changeset: https://review.fuel-infra.org/364
DEB Repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.1.1-stable-364/ubuntu

Package tgt has been built from changeset: https://review.fuel-infra.org/379
DEB Repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.1.1-stable-379/ubuntu

Package cinder has been built from changeset: https://review.fuel-infra.org/364
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1.1-stable-364/centos

Package cinder has been built from changeset: https://review.fuel-infra.org/364
DEB Repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.1.1-stable-364/ubuntu

If there commits ready to be merged the status should be In Progress. Where's a fix for 6.0? And why is this targeted for maintenance releases if it's not High priority?

It is still unclear if there is need to apply this for the upcoming 6.0 release.

The process here is to consider the performance benchmarks and to derive a decision from these.

Status and priority are to be updated.

Package tgt has been built from changeset: https://review.fuel-infra.org/379
DEB Repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.1.1-stable-379/ubuntu

Package tgt has been built from changeset: https://review.fuel-infra.org/379
DEB Repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.1.1-stable/ubuntu

Package cinder-2014.1.1-fuel5.1.1.mira13 has been built from changeset: https://review.fuel-infra.org/364
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1.1-stable/centos

Package cinder-2014.1.1-fuel5.1.1~mira20 has been built from changeset: https://review.fuel-infra.org/364
DEB Repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.1.1-stable/ubuntu

Is this still in progress for 5.1.1?

The appropriate glitch in 6.0 should be fixed by the 'write-cache off'. Since this is not truly a bug moving to a 'Won't Fix' status.

Will merge `bsoflags` patch with the upstream version of Cinder.

That was a concern - to add this fix into 6.0 or not, because we were need to test the performance first. We have not tested it yet and considered to make this decision in 6.1 scope.

Hence, Won't fix in 6.0.

Fix proposed to branch: openstack-ci/fuel-6.1/2014.2
Change author: Pavel Boldin <email address hidden>
Review: https://review.fuel-infra.org/6295

Change abandoned by Pavel Boldin <email address hidden> on branch: openstack-ci/fuel-6.1/2014.2
Review: https://review.fuel-infra.org/5462
Reason: https://review.fuel-infra.org/#/c/6295/

Since it seems like Ceph would be a default storage backend for 6.1 I move that to 7.0 and link it with the upstream bug:

https://bugs.launchpad.net/cinder/+bug/1441935

My bad, we actually implemented a local patch for our Mirantis OpenStack. So, this is fixed in the 6.1 and I need to push the fix to the upstream.

Ubuntu Deployment:
1 Controller
1 Compute
1 Cinder with LVM over iSCSI.

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "6.1"
  openstack_version: "2014.2.2-6.1"
  api: "1.0"
  build_number: "437"
  build_id: "2015-05-19_10-05-51"
  nailgun_sha: "593c99f2b46cf52b2be6c7c6e182b6ba9f2232cd"
  python-fuelclient_sha: "e19f1b65792f84c4a18b5a9473f85ef3ba172fce"
  astute_sha: "96801c5bccb14aa3f2a0d7f27f4a4b6dd2b4a548"
  fuel-library_sha: "2814c51668f487e97e1449b078bad1942421e6b9"
  fuel-ostf_sha: "9ce1800749081780b8b2a4a7eab6586583ffaf33"
  fuelmain_sha: "68796aeaa7b669e68bc0976ffd616709c937187a"

Verification steps:

1. Create Volume
2. Run Instance.
3. Attache volume to Instance
4. Check on Cinder Node:
# tgt-admin -s
Target 1: iqn.2010-10.org.openstack:volume-bbd0fe91-0057-4f12-924b-83d08c2e8d45
    System information:
        Driver: iscsi
        State: ready
    I_T nexus information:
        I_T nexus: 1
            Initiator: iqn.1993-08.org.debian:01:b8c3f770cfbd alias: node-2.domain.tld
            Connection: 0
                IP Address: 192.168.1.2
    LUN information:
        LUN: 0
            Type: controller
            SCSI ID: IET 00010000
            SCSI SN: beaf10
            Size: 0 MB, Block size: 1
            Online: Yes
            Removable media: No
            Prevent removal: No
            Readonly: No
            SWP: No
            Thin-provisioning: No
            Backing store type: null
            Backing store path: None
            Backing store flags:
        LUN: 1
            Type: disk
            SCSI ID: IET 00010001
            SCSI SN: beaf11
            Size: 1074 MB, Block size: 512
            Online: Yes
            Removable media: No
            Prevent removal: No
            Readonly: No
            SWP: No
            Thin-provisioning: No
            Backing store type: rdwr
            Backing store path: /dev/cinder/volume-bbd0fe91-0057-4f12-924b-83d08c2e8d45
            Backing store flags:
    Account information:
        344jnGsTMZc4zFk8swxd
    ACL information:
        ALL
5.
 # cat /var/lib/cinder/volumes/*
<target iqn.2010-10.org.openstack:volume-bbd0fe91-0057-4f12-924b-83d08c2e8d45>
    backing-store /dev/cinder/volume-bbd0fe91-0057-4f12-924b-83d08c2e8d45
    lld iscsi
    bsoflags direct
    incominguser 344jnGsTMZc4zFk8swxd 4Xp2kkm9uyYtKeHB
    write-cache on

Check that "bsoflags direct" is used.

Fix proposed to branch: openstack-ci/fuel-7.0/2015.1.0
Change author: Pavel Boldin <email address hidden>
Review: https://review.fuel-infra.org/8095

Change abandoned by Anton Arefiev <email address hidden> on branch: openstack-ci/fuel-7.0/2015.1.0
Review: https://review.fuel-infra.org/8095
Reason: No needs anymore, cherry-picked from upstream version: https://review.fuel-infra.org/#/c/9421/