Mirantis OpenStack

nova-compute times out waiting for RPC reply

Bug #1427892 reported by Ryan Moe
18
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Mirantis OpenStack
Fix Released
Critical
MOS Oslo
Mirantis OpenStack 6.1
6.0.x
Fix Released
Critical
Oleksii Zamiatin
Mirantis OpenStack 6.0-mu-2

Bug Description

After some time of creating and terminating instances a nova-compute process will stop consuming replies. All calls start failing with the error "MessagingTimeout: Timed out waiting for a reply to message ID XXX". It appears that the replies make it to the correct queue but nova-compute just doesn't consume them. This issue has been confirmed on 5.0.1 and is suspected to occur on 6.0 as well.

Maybe it's related to this oslo.messaging bug: https://bugs.launchpad.net/oslo.messaging/+bug/1338732

A diagnostic snapshot will be uploaded as soon as it's available.

Ryan Moe (rmoe)
summary: - nova-compute times out waiting fro RPC reply
+ nova-compute times out waiting for RPC reply
Revision history for this message
Dmitry Borodaenko (angdraug) wrote :

Ryan, please include the reproducer script along with the diagnostic snapshot.

Changed in fuel:
importance: Undecided → Critical
assignee: nobody → MOS Oslo (mos-oslo)
milestone: none → 6.1
assignee: MOS Oslo (mos-oslo) → Ryan Moe (rmoe)
Revision history for this message
Davanum Srinivas (DIMS) (dims-v) wrote :

One possible upstream bug that looks similar according to Ilya Pekelny is:
https://bugs.launchpad.net/oslo.messaging/+bug/1338732

Revision history for this message
Davanum Srinivas (DIMS) (dims-v) wrote :

grep "MessagingTimeout: Timed out waiting for a reply" */*.log > ~/junk/untitled1.txt
grep "583c89a2-5d7b-4017-aeb2-44813f9de334" */*.log > ~/junk/untitled2.txt
cat ~/junk/untitled2.txt | cut -f 2- -d ':' | sort | uniq > ~/junk/untitled2-2.txt
grep "req-b591e322-cf7a-49d3-bb18-48ea2a71d917" */*.log > ~/junk/untitled3.txt
cat ~/junk/untitled3.txt | cut -f 2- -d ':' | sort | uniq > ~/junk/untitled3-2.txt
find . -name *rabbit* | xargs grep -H -i 2015-03-04 > ~/junk/untitled4.txt

Wading through the files, and sorting by time etc, finally end up with:
http://paste.openstack.org/show/189290/
http://paste.openstack.org/show/189291/

Which shows, clearly that between 20:20:18.402 and 20:26:04.920 there was a problem and messages were not picked up.

So checking the rabbit logs, we see:
http://paste.openstack.org/show/189306/

There do seem to be some issues around the time in question, so we should definitely look at the fix for the reconnect issue that Ilya dug up to see if that helps.

Revision history for this message
Davanum Srinivas (DIMS) (dims-v) wrote :
Revision history for this message
Davanum Srinivas (DIMS) (dims-v) wrote :

which would be this https://review.openstack.org/#/c/109373/

Dmitry Mescheryakov (dmitrymex)
affects: fuel → mos
Changed in mos:
milestone: 6.1 → none
milestone: none → 6.1
Dmitry Mescheryakov (dmitrymex)
Changed in mos:
status: New → Confirmed
Revision history for this message
OSCI Robot (oscirobot) wrote :

RPM package oslo.messaging has been built for project openstack/oslo.messaging
Package version == 1.4.1, package release == fuel6.0.mira21.git.032a4d2.ae0dd8f

Changeset: https://review.fuel-infra.org/4415
project: openstack/oslo.messaging
branch: openstack-ci/fuel-6.0-updates/2014.2
author: Oleksii Zamiatin
committer: Oleksii Zamiatin
subject: Nova-compute times out waiting for RPC reply
status: patchset-created

Files placed on repository:
python-oslo-messaging-1.4.1-fuel6.0.mira21.git.032a4d2.ae0dd8f.noarch.rpm
python-oslo-messaging-doc-1.4.1-fuel6.0.mira21.git.032a4d2.ae0dd8f.noarch.rpm

NOTE: Changeset is not merged, created temporary package repository.
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-6.0-updates-stable-4415/centos

Revision history for this message
OSCI Robot (oscirobot) wrote :

DEB package oslo.messaging has been built for project openstack/oslo.messaging
Package version == 1.4.1, package release == fuel6.0~mira19+git.032a4d2.ae0dd8f

Changeset: https://review.fuel-infra.org/4415
project: openstack/oslo.messaging
branch: openstack-ci/fuel-6.0-updates/2014.2
author: Oleksii Zamiatin
committer: Oleksii Zamiatin
subject: Nova-compute times out waiting for RPC reply
status: patchset-created

Files placed on repository:
python-oslo.messaging_1.4.1-fuel6.0~mira19+git.032a4d2.ae0dd8f_all.deb

NOTE: Changeset is not merged, created temporary package repository.
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-6.0-updates-stable-4415/ubuntu

Revision history for this message
OSCI Robot (oscirobot) wrote :

RPM package oslo.messaging has been built for project openstack/oslo.messaging
Package version == 1.4.1, package release == fuel6.0.mira21

Changeset: https://review.fuel-infra.org/4415
project: openstack/oslo.messaging
branch: openstack-ci/fuel-6.0-updates/2014.2
author: Oleksii Zamiatin
committer: Oleksii Zamiatin
subject: Nova-compute times out waiting for RPC reply
status: change-merged

Files placed on repository:
python-oslo-messaging-1.4.1-fuel6.0.mira21.noarch.rpm
python-oslo-messaging-doc-1.4.1-fuel6.0.mira21.noarch.rpm

Changeset merged. Package placed on primary repository
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-6.0-updates-stable/centos

Revision history for this message
OSCI Robot (oscirobot) wrote :

DEB package oslo.messaging has been built for project openstack/oslo.messaging
Package version == 1.4.1, package release == fuel6.0~mira19

Changeset: https://review.fuel-infra.org/4415
project: openstack/oslo.messaging
branch: openstack-ci/fuel-6.0-updates/2014.2
author: Oleksii Zamiatin
committer: Oleksii Zamiatin
subject: Nova-compute times out waiting for RPC reply
status: change-merged

Files placed on repository:
python-oslo.messaging_1.4.1-fuel6.0~mira19_all.deb

Changeset merged. Package placed on primary repository
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-6.0-updates-stable/ubuntu

Revision history for this message
OSCI Robot (oscirobot) wrote :

RPM package oslo.messaging has been built for project openstack/oslo.messaging
Package version == 1.4.1, package release == fuel6.0.mira22

Changeset: https://review.fuel-infra.org/4415
project: openstack/oslo.messaging
branch: openstack-ci/fuel-6.0-updates/2014.2
author: Oleksii Zamiatin
committer: Oleksii Zamiatin
subject: Nova-compute times out waiting for RPC reply
status: change-merged

Files placed on repository:
python-oslo-messaging-1.4.1-fuel6.0.mira22.noarch.rpm
python-oslo-messaging-doc-1.4.1-fuel6.0.mira22.noarch.rpm

Changeset merged. Package placed on primary repository
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-6.0-updates-stable/centos

Revision history for this message
OSCI Robot (oscirobot) wrote :

DEB package oslo.messaging has been built for project openstack/oslo.messaging
Package version == 1.4.1, package release == fuel6.0~mira20

Changeset: https://review.fuel-infra.org/4415
project: openstack/oslo.messaging
branch: openstack-ci/fuel-6.0-updates/2014.2
author: Oleksii Zamiatin
committer: Oleksii Zamiatin
subject: Nova-compute times out waiting for RPC reply
status: change-merged

Files placed on repository:
python-oslo.messaging_1.4.1-fuel6.0~mira20_all.deb

Changeset merged. Package placed on primary repository
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-6.0-updates-stable/ubuntu

Revision history for this message
OSCI Robot (oscirobot) wrote :

RPM package oslo.messaging has been built for project openstack/oslo.messaging
Package version == 1.4.1, package release == fuel6.0.mira23

Changeset: https://review.fuel-infra.org/4415
project: openstack/oslo.messaging
branch: openstack-ci/fuel-6.0-updates/2014.2
author: Oleksii Zamiatin
committer: Oleksii Zamiatin
subject: Nova-compute times out waiting for RPC reply
status: change-merged

Files placed on repository:
python-oslo-messaging-1.4.1-fuel6.0.mira23.noarch.rpm
python-oslo-messaging-doc-1.4.1-fuel6.0.mira23.noarch.rpm

Changeset merged. Package placed on primary repository
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-6.0-updates-stable/centos

Revision history for this message
OSCI Robot (oscirobot) wrote :

DEB package oslo.messaging has been built for project openstack/oslo.messaging
Package version == 1.4.1, package release == fuel6.0~mira21

Changeset: https://review.fuel-infra.org/4415
project: openstack/oslo.messaging
branch: openstack-ci/fuel-6.0-updates/2014.2
author: Oleksii Zamiatin
committer: Oleksii Zamiatin
subject: Nova-compute times out waiting for RPC reply
status: change-merged

Files placed on repository:
python-oslo.messaging_1.4.1-fuel6.0~mira21_all.deb

Changeset merged. Package placed on primary repository
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-6.0-updates-stable/ubuntu

Revision history for this message
OSCI Robot (oscirobot) wrote :

RPM package oslo.messaging has been built for project openstack/oslo.messaging
Package version == 1.4.1, package release == fuel6.0.1.mira24.git.9200b1e.f9e74fb

Changeset: https://review.fuel-infra.org/4447
project: openstack/oslo.messaging
branch: openstack-ci/fuel-6.0.1/2014.2
author: Oleksii Zamiatin
committer: Oleksii Zamiatin
subject: Nova-compute times out waiting for RPC reply
status: patchset-created

Files placed on repository:
python-oslo-messaging-1.4.1-fuel6.0.1.mira24.git.9200b1e.f9e74fb.noarch.rpm
python-oslo-messaging-doc-1.4.1-fuel6.0.1.mira24.git.9200b1e.f9e74fb.noarch.rpm

NOTE: Changeset is not merged, created temporary package repository.
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-6.0.1-stable-4447/centos

Revision history for this message
OSCI Robot (oscirobot) wrote :

DEB package oslo.messaging has been built for project openstack/oslo.messaging
Package version == 1.4.1, package release == fuel6.0.1~mira22+git.9200b1e.f9e74fb

Changeset: https://review.fuel-infra.org/4447
project: openstack/oslo.messaging
branch: openstack-ci/fuel-6.0.1/2014.2
author: Oleksii Zamiatin
committer: Oleksii Zamiatin
subject: Nova-compute times out waiting for RPC reply
status: patchset-created

Files placed on repository:
python-oslo.messaging_1.4.1-fuel6.0.1~mira22+git.9200b1e.f9e74fb_all.deb

NOTE: Changeset is not merged, created temporary package repository.
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-6.0.1-stable-4447/ubuntu

Revision history for this message
OSCI Robot (oscirobot) wrote :

RPM package oslo.messaging has been built for project openstack/oslo.messaging
Package version == 1.4.1, package release == fuel6.0.1.mira24.git.345064c.f9e74fb

Changeset: https://review.fuel-infra.org/4447
project: openstack/oslo.messaging
branch: openstack-ci/fuel-6.0.1/2014.2
author: Oleksii Zamiatin
committer: Oleksii Zamiatin
subject: Nova-compute times out waiting for RPC reply
status: patchset-created

Files placed on repository:
python-oslo-messaging-1.4.1-fuel6.0.1.mira24.git.345064c.f9e74fb.noarch.rpm
python-oslo-messaging-doc-1.4.1-fuel6.0.1.mira24.git.345064c.f9e74fb.noarch.rpm

NOTE: Changeset is not merged, created temporary package repository.
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-6.0.1-stable-4447/centos

Revision history for this message
OSCI Robot (oscirobot) wrote :

DEB package oslo.messaging has been built for project openstack/oslo.messaging
Package version == 1.4.1, package release == fuel6.0.1~mira22+git.345064c.f9e74fb

Changeset: https://review.fuel-infra.org/4447
project: openstack/oslo.messaging
branch: openstack-ci/fuel-6.0.1/2014.2
author: Oleksii Zamiatin
committer: Oleksii Zamiatin
subject: Nova-compute times out waiting for RPC reply
status: patchset-created

Files placed on repository:
python-oslo.messaging_1.4.1-fuel6.0.1~mira22+git.345064c.f9e74fb_all.deb

NOTE: Changeset is not merged, created temporary package repository.
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-6.0.1-stable-4447/ubuntu

Revision history for this message
OSCI Robot (oscirobot) wrote :

RPM package oslo.messaging has been built for project openstack/oslo.messaging
Package version == 1.4.1, package release == fuel6.0.1.mira24

Changeset: https://review.fuel-infra.org/4447
project: openstack/oslo.messaging
branch: openstack-ci/fuel-6.0.1/2014.2
author: Oleksii Zamiatin
committer: Oleksii Zamiatin
subject: Nova-compute times out waiting for RPC reply
status: change-merged

Files placed on repository:
python-oslo-messaging-1.4.1-fuel6.0.1.mira24.noarch.rpm
python-oslo-messaging-doc-1.4.1-fuel6.0.1.mira24.noarch.rpm

Changeset merged. Package placed on primary repository
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-6.0.1-stable/centos

Revision history for this message
OSCI Robot (oscirobot) wrote :

DEB package oslo.messaging has been built for project openstack/oslo.messaging
Package version == 1.4.1, package release == fuel6.0.1~mira22

Changeset: https://review.fuel-infra.org/4447
project: openstack/oslo.messaging
branch: openstack-ci/fuel-6.0.1/2014.2
author: Oleksii Zamiatin
committer: Oleksii Zamiatin
subject: Nova-compute times out waiting for RPC reply
status: change-merged

Files placed on repository:
python-oslo.messaging_1.4.1-fuel6.0.1~mira22_all.deb

Changeset merged. Package placed on primary repository
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-6.0.1-stable/ubuntu

Revision history for this message
Dmitry Mescheryakov (dmitrymex) wrote :

The fix for 6.0.1 should have gotten through staging by now

Revision history for this message
Nikita Konovalov (nkonovalov) wrote :

The bug was reproducing while testing Sahara @ 25 node scale lab.
The instances were failing to start/shutdown if the there were more than 60 vms trying to start/delete simultaneously.
MOS: 6.0.1_130

Revision history for this message
Vladimir Kuklin (vkuklin) wrote :

Folks, do we have this issue for 6.1 right now? Is it somehow related to FUEL project? As far as I see, there is nothing we should do in our puppet manifests or configuration files. If so, please remove Fuel developers assignees from this bug.

Vladimir Kuklin (vkuklin)
Changed in mos:
status: Confirmed → Incomplete
Revision history for this message
Andrew Woodward (xarses) wrote :

Setting this to mos-oslo,

olso team, please confirm if this patch is in oslo in 6.1

Changed in mos:
assignee: Ryan Moe (rmoe) → MOS Oslo (mos-oslo)
status: Incomplete → Triaged
Revision history for this message
OSCI Robot (oscirobot) wrote :

RPM package oslo.messaging has been built for project openstack/oslo.messaging
Package version == 1.4.1, package release == fuel6.0.mira24

Changeset: https://review.fuel-infra.org/4660
project: openstack/oslo.messaging
branch: openstack-ci/fuel-6.0-updates/2014.2
author: Eugene Nikanorov
committer: Alexandr Nevenchannyy
subject: Allow disabling client heartbeats from config file
status: change-merged

Files placed on repository:
python-oslo-messaging-1.4.1-fuel6.0.mira24.noarch.rpm
python-oslo-messaging-doc-1.4.1-fuel6.0.mira24.noarch.rpm

Changeset merged. Package placed on primary repository
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-6.0-updates-stable/centos

Revision history for this message
OSCI Robot (oscirobot) wrote :

DEB package oslo.messaging has been built for project openstack/oslo.messaging
Package version == 1.4.1, package release == fuel6.0~mira22

Changeset: https://review.fuel-infra.org/4660
project: openstack/oslo.messaging
branch: openstack-ci/fuel-6.0-updates/2014.2
author: Eugene Nikanorov
committer: Alexandr Nevenchannyy
subject: Allow disabling client heartbeats from config file
status: change-merged

Files placed on repository:
python-oslo.messaging_1.4.1-fuel6.0~mira22_all.deb

Changeset merged. Package placed on primary repository
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-6.0-updates-stable/ubuntu

Oleksii Zamiatin (ozamiatin)
Changed in mos:
status: Triaged → Fix Committed
Revision history for this message
Viktor Serhieiev (vsergeyev) wrote :

@Andrew - this was fixed in 6.1 by patch https://review.fuel-infra.org/#/c/5377/

Revision history for this message
Denis Meltsaykin (dmeltsaykin) wrote :

Previous patch introduces new issue: https://bugs.launchpad.net/mos/+bug/1457055

Kyrylo Romanenko (kromanenko)
tags: added: on-verification
Revision history for this message
Kyrylo Romanenko (kromanenko) wrote :

I could get only one appearance of this error message:
# grep "MessagingTimeout: Timed out waiting for a reply" /var/log/*/*.log
/var/log/neutron/openvswitch-agent.log:2015-06-03 12:50:56.875 12317 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent MessagingTimeout: Timed out waiting for a reply to message ID 369e8b8f3cf64fc5a2e1c1c516b510d1.

But cluster continued to work with instances further as expected/

Another issue was filed during attemts to reproduce current one:
Failed to launch instance after failure of instances batch
https://bugs.launchpad.net/mos/+bug/1461537

Revision history for this message
Kyrylo Romanenko (kromanenko) wrote :

Verified on
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "6.1"
  openstack_version: "2014.2.2-6.1"
  api: "1.0"
  build_number: "499"
  build_id: "2015-06-02_22-09-35"
  nailgun_sha: "3830bdcb28ec050eed399fe782cc3dd5fbf31bde"
  python-fuelclient_sha: "4fc55db0265bbf39c369df398b9dc7d6469ba13b"
  astute_sha: "cbae24e9904be2ff8d1d49c0c48d1bdc33574228"
  fuel-library_sha: "938f033a5da90aca0c24c89c995cf01707d746d2"
  fuel-ostf_sha: "f899e16c4ce9a60f94e7128ecde1324ea41d09d4"
  fuelmain_sha: "bcc909ffc5dd5156ba54cae348b6a07c1b607b24"

Changed in mos:
status: Fix Committed → Fix Released
Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix proposed to openstack/oslo.messaging (openstack-ci/fuel-7.0/2015.1.0)

Fix proposed to branch: openstack-ci/fuel-7.0/2015.1.0
Change author: Oleksii Zamiatin <email address hidden>
Review: https://review.fuel-infra.org/8282

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Change abandoned on openstack/oslo.messaging (openstack-ci/fuel-7.0/2015.1.0)

Change abandoned by Victor Sergeyev <email address hidden> on branch: openstack-ci/fuel-7.0/2015.1.0
Review: https://review.fuel-infra.org/8282
Reason: merged in stable/kilo

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Bug attachments

Remote bug watches

Bug watches keep track of this bug in other bug trackers.