Constant exceptions "NotFound: Basic.consume: (404) NOT_FOUND - no queue abc in vhost '/'" in log
Affects | Status | Importance | Assigned to | Milestone | ||
---|---|---|---|---|---|---|
Mirantis OpenStack | Status tracked in 10.0.x | |||||
10.0.x |
Fix Committed
|
High
|
Kirill Bespalov | |||
7.0.x |
Fix Released
|
High
|
Kirill Bespalov | |||
8.0.x |
Fix Released
|
High
|
Kirill Bespalov | |||
9.x |
Fix Released
|
High
|
Kirill Bespalov |
Bug Description
Version: 9.0
Steps to reproduce:
1. Deploy environment MOS environment.
2. Run some tests on it (exact cause is unknown yet)
Expected results:
All logs are clean
Actual results:
In one of OpenStack components log you find a lot of exceptions like
NotFound: Basic.consume: (404) NOT_FOUND - no queue 'reply_
(full stack trace from neutron-server logs - http://
This indicates that process lost a queue it was listening on and the situation does not end by itself. Loosing a queue has an impact that server stops processing messages from it, which might be crucial to its work (depends on the queue).
In rabbit logs on node-61 with grep one can find the following entries (only several earliest are shown):
http://
Note the pattern - first two queue.declare operations timed out and then basic.consume fail in endless loop.
It seems that RabbitMQ failed to create the queue due to overload or something and oslo.messaging did not notice that. Unfortunately the relevant neutron-server logs were already rotated, so it is not clear what happened in oslo.messaging at the time of the queue declaration.
description: | updated |
description: | updated |
tags: | added: on-verification |
tags: | removed: on-verification |
has been reproduced on node-119 http:// paste.openstack .org/show/ 495774/ mos-scale- share.mirantis. com/fuel- snapshot- 2016-04- 29_08-53- 26.tar. gz
from this diagnostic snapshot: http://