Comment 43 for bug 1806156

when we hit this problem, I also observed RPC message timeout from the logfile.

2019-07-25 13:19:20.607 24665 ERROR cinder.cmd.manage [req-ed246967-33af-4f36-950b-a3fb935dee99 - - - - -] Error attempting to run shared_targets_online_data_migration: MessagingTimeout: Timed out waiting for a reply to message ID fcd7e35f688c497691a4f7f1333ee732
2019-07-25 13:19:20.607 24665 ERROR cinder.cmd.manage Traceback (most recent call last):
2019-07-25 13:19:20.607 24665 ERROR cinder.cmd.manage File "/openstack/venvs/cinder-18.1.8/lib/python2.7/site-packages/cinder/cmd/manage.py", line 351, in _run_migration
2019-07-25 13:19:20.607 24665 ERROR cinder.cmd.manage found, done = migration_meth(ctxt, count)
2019-07-25 13:19:20.607 24665 ERROR cinder.cmd.manage File "/openstack/venvs/cinder-18.1.8/lib/python2.7/site-packages/cinder/cmd/manage.py", line 128, in shared_targets_online_data_migration
2019-07-25 13:19:20.607 24665 ERROR cinder.cmd.manage non_shared_hosts, total_vols_to_update = _get_non_shared_target_hosts(ctxt)
2019-07-25 13:19:20.607 24665 ERROR cinder.cmd.manage File "/openstack/venvs/cinder-18.1.8/lib/python2.7/site-packages/cinder/cmd/manage.py", line 111, in _get_non_shared_target_hosts
2019-07-25 13:19:20.607 24665 ERROR cinder.cmd.manage capabilities = rpcapi.get_capabilities(ctxt, service.host, True)
2019-07-25 13:19:20.607 24665 ERROR cinder.cmd.manage File "/openstack/venvs/cinder-18.1.8/lib/python2.7/site-packages/cinder/volume/rpcapi.py", line 341, in get_capabilities
2019-07-25 13:19:20.607 24665 ERROR cinder.cmd.manage return cctxt.call(ctxt, 'get_capabilities', discover=discover)
2019-07-25 13:19:20.607 24665 ERROR cinder.cmd.manage File "/openstack/venvs/cinder-18.1.8/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 179, in call
2019-07-25 13:19:20.607 24665 ERROR cinder.cmd.manage retry=self.retry)
2019-07-25 13:19:20.607 24665 ERROR cinder.cmd.manage File "/openstack/venvs/cinder-18.1.8/lib/python2.7/site-packages/oslo_messaging/transport.py", line 133, in _send
2019-07-25 13:19:20.607 24665 ERROR cinder.cmd.manage retry=retry)
2019-07-25 13:19:20.607 24665 ERROR cinder.cmd.manage File "/openstack/venvs/cinder-18.1.8/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 584, in send
2019-07-25 13:19:20.607 24665 ERROR cinder.cmd.manage call_monitor_timeout, retry=retry)
2019-07-25 13:19:20.607 24665 ERROR cinder.cmd.manage File "/openstack/venvs/cinder-18.1.8/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 573, in _send
2019-07-25 13:19:20.607 24665 ERROR cinder.cmd.manage call_monitor_timeout)
2019-07-25 13:19:20.607 24665 ERROR cinder.cmd.manage File "/openstack/venvs/cinder-18.1.8/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 459, in wait
2019-07-25 13:19:20.607 24665 ERROR cinder.cmd.manage message = self.waiters.get(msg_id, timeout=timeout)
2019-07-25 13:19:20.607 24665 ERROR cinder.cmd.manage File "/openstack/venvs/cinder-18.1.8/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 336, in get
2019-07-25 13:19:20.607 24665 ERROR cinder.cmd.manage 'to message ID %s' % msg_id)
2019-07-25 13:19:20.607 24665 ERROR cinder.cmd.manage MessagingTimeout: Timed out waiting for a reply to message ID fcd7e35f688c497691a4f7f1333ee732
2019-07-25 13:19:20.607 24665 ERROR cinder.cmd.manage

And I checked cinder message queue, it was only 1 message there.

So I can't explain why it timed out.