RabbitMQ connections lack heartbeat or TCP keepalives

related bug https://bugs.launchpad.net/oslo.messaging/+bug/1338732

Patches and requests for building packages: https://bugs.launchpad.net/mos/+bug/1340711

Fixed by https://gerrit.mirantis.com/19112

Here is the summary changes for 5.0.1:
https://gerrit.mirantis.com/19900
https://gerrit.mirantis.com/19112
https://gerrit.mirantis.com/19573
https://review.openstack.org/112681

Package oslo.messaging has been built from changeset: http://gerrit.mirantis.com/20691
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1-stable-20691/centos

Package oslo.messaging has been built from changeset: http://gerrit.mirantis.com/20691
DEB Repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.1-stable-20691/ubuntu

Porting of changes from 5.0.1 to 5.1 is done:

https://gerrit.mirantis.com/19902
https://gerrit.mirantis.com/19906
https://gerrit.mirantis.com/19905
https://review.openstack.org/112677

https://gerrit.mirantis.com/20691 - Proper heartbeat implementation is submitted and ready to be tested

Package oslo.messaging has been built from changeset: http://gerrit.mirantis.com/21233
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1-stable-21233/centos

Package oslo.messaging has been built from changeset: http://gerrit.mirantis.com/21233
DEB Repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.1-stable-21233/ubuntu

Package oslo.messaging has been built from changeset: http://gerrit.mirantis.com/20691
DEB Repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.1-stable-20691/ubuntu

Package oslo.messaging has been built from changeset: http://gerrit.mirantis.com/20691
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1-stable-20691/centos

Package oslo.messaging has been built from changeset: http://gerrit.mirantis.com/20691
DEB Repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.1-stable-20691/ubuntu

Package oslo.messaging has been built from changeset: http://gerrit.mirantis.com/20691
DEB Repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.1-stable/ubuntu

Package oslo.messaging has been built from changeset: http://gerrit.mirantis.com/20691
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1-stable/centos

What is the current status of this bug in 5.1 and 5.0.2?

We verified and merged fix to 5.1 branch, backports to 5.0.2 are pending:
https://review.openstack.org/117500
http://gerrit.mirantis.com/21389

Backport to 5.0.2 merged

{

    "build_id": "2014-09-15_00-01-46",
    "ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346",
    "build_number": "8",
    "auth_required": true,
    "api": "1.0",
    "nailgun_sha": "b8d8189cc37d6d1b26f4479be6be7313beefb1c8",
    "production": "docker",
    "fuelmain_sha": "d7ed7973034bde73d3f42c000984423b59b2312b",
    "astute_sha": "f5fbd89d1e0e1f22ef9ab2af26da5ffbfbf24b13",
    "feature_groups": [
        "experimental"
    ],
    "release": "5.1",
    "release_versions": {
        "2014.1.1-5.1": {
            "VERSION": {
                "build_id": "2014-09-15_00-01-46",
                "ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346",
                "build_number": "8",
                "api": "1.0",
                "nailgun_sha": "b8d8189cc37d6d1b26f4479be6be7313beefb1c8",
                "production": "docker",
                "fuelmain_sha": "d7ed7973034bde73d3f42c000984423b59b2312b",
                "astute_sha": "f5fbd89d1e0e1f22ef9ab2af26da5ffbfbf24b13",
                "feature_groups": [
                    "experimental"
                ],
                "release": "5.1",
                "fuellib_sha": "395fd9d20a003603cc9ad26e16cb13c1c45e24e6"
            }
        }
    },
    "fuellib_sha": "395fd9d20a003603cc9ad26e16cb13c1c45e24e6"

}

We have problems with oslo.messaging in our tests after restart nodes
Steps:
1. Create next cluster - Ubuntu, Simple, Flat nova-network, Cinder for volumes, Ceph for images, 1 controller, 1 compute, 2 cinder+ceph nodes
2. Deploy cluster
3. Reboot all nodes one by one
4. Open Health Check tab
5. Run tests

Expected - tests passed
Actual - tests on volume and instance creation failed. Lots of errors on compute node in nova logs:
2014-09-15 13:08:11 ERROR

oslo.messaging._drivers.impl_rabbit [req-c4c51ee4-f9b7-4e9e-a949-e694f3b21757 ] AMQP server on 10.108.52.4:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds.

And on cinder node in cinder logs
2014-09-15 07:40:19 ERROR

oslo.messaging._drivers.impl_rabbit [req-2fd99d10-2de0-478e-ab3e-cabc52935455 - - - - -] AMQP server on 10.108.52.4:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds.

Logs are attached

Also same errors in next configuration - Ubuntu, Ha, Flat Nova network, Cinder for volumes, 3 controllers, 2 compute
After destroy of one of controllers some OSTF tests failed with timeout
Errors on node-4 (compute)
nova.openstack.common.periodic_task [-] Error during ComputeManager.update_available_resource: Timed out waiting for a reply to message ID 2dd0eabc69984cdca76067420d9d6f12

Logs are attached

This exception has nothing to do with heartbeats, from log there is clearly issue with connection to rabbitmq. Looks like server are down.

I suggest opening new bug and returning state of this bug to commited

I agree with it

related bug related bug https://bugs.launchpad.net/mos/+bug/1371723