We have a bug during the creation of a trove cluster.
The taskmanager can't get ip address of cluster instances.
Everything was tested from master(trove, trove-pythonclient)
trove --debug cluster-create cluster mariadb 10.1 --instance "flavor=101,volume=5,nic='net-id=09cd3332-146b-430f-b973-2d1cf204288d'" --instance "flavor=101,volume=5,nic='net-id=09cd3332-146b-430f-b973-2d1cf204288d'" --instance "flavor=101,volume=5,nic='net-id=09cd3332-146b-430f-b973-2d1cf204288d'"
cat /var/log/kolla/trove/trove-taskmanager.log ... 2018-01-12 18:00:42.293 7 DEBUG trove.common.notification [-] Sending event: dbaas.cluster_create.error, {'client_ip': u'10.128.150.5', 'server_type': u'taskmanager', 'tenant_id': u'd20aecdc013f42739c1d110d7d592b7a', 'exception': ['Traceback (most recent call last):\n', ' File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/trove/taskmanager/manager.py", line 395, in create_cluster\n cluster_tasks.create_cluster(context, cluster_id)\n', ' File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/trove/common/strategies/cluster/experimental/galera_common/taskmanager.py", line 141, in create_cluster\n _create_cluster()\n', ' File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/trove/common/strategies/cluster/experimental/galera_common/taskmanager.py", line 88, in _create_cluster\n cluster_ips = [self.get_ip(instance) for instance in instances]\n', ' File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/trove/taskmanager/models.py", line 204, in get_ip\n IPs = instance.get_visible_ip_addresses()\n', 'IndexError: list index out of range\n'], 'server_ip': u'10.128.150.100:8779', 'request_id': u'req-9aa49155-ab23-4fa5-92c5-b13ab03ec780', 'message': 'list index out of range'} _notify /var/lib/kolla/venv/local/lib/python2.7/site-packages/trove/common/notification.py:405 2018-01-12 18:00:42.297 7 ERROR oslo_messaging.rpc.server [-] Exception during message handling: IndexError: list index out of range 2018-01-12 18:00:42.297 7 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2018-01-12 18:00:42.297 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 160, in _process_incoming 2018-01-12 18:00:42.297 7 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 2018-01-12 18:00:42.297 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 213, in dispatch 2018-01-12 18:00:42.297 7 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) 2018-01-12 18:00:42.297 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _do_dispatch 2018-01-12 18:00:42.297 7 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) 2018-01-12 18:00:42.297 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/osprofiler/profiler.py", line 153, in wrapper 2018-01-12 18:00:42.297 7 ERROR oslo_messaging.rpc.server return f(*args, **kwargs) 2018-01-12 18:00:42.297 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/trove/taskmanager/manager.py", line 395, in create_cluster 2018-01-12 18:00:42.297 7 ERROR oslo_messaging.rpc.server cluster_tasks.create_cluster(context, cluster_id) 2018-01-12 18:00:42.297 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/trove/common/strategies/cluster/experimental/galera_common/taskmanager.py", line 141, in create_cluster 2018-01-12 18:00:42.297 7 ERROR oslo_messaging.rpc.server _create_cluster() 2018-01-12 18:00:42.297 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/trove/common/strategies/cluster/experimental/galera_common/taskmanager.py", line 88, in _create_cluster 2018-01-12 18:00:42.297 7 ERROR oslo_messaging.rpc.server cluster_ips = [self.get_ip(instance) for instance in instances] 2018-01-12 18:00:42.297 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/trove/taskmanager/models.py", line 204, in get_ip 2018-01-12 18:00:42.297 7 ERROR oslo_messaging.rpc.server IPs = instance.get_visible_ip_addresses() 2018-01-12 18:00:42.297 7 ERROR oslo_messaging.rpc.server IndexError: list index out of range 2018-01-12 18:00:42.297 7 ERROR oslo_messaging.rpc.server ...
We have a bug during the creation of a trove cluster.
The taskmanager can't get ip address of cluster instances.
Everything was tested from master(trove, trove-pythonclient)
trove --debug cluster-create cluster mariadb 10.1 --instance "flavor= 101,volume= 5,nic=' net-id= 09cd3332- 146b-430f- b973-2d1cf20428 8d'" --instance "flavor= 101,volume= 5,nic=' net-id= 09cd3332- 146b-430f- b973-2d1cf20428 8d'" --instance "flavor= 101,volume= 5,nic=' net-id= 09cd3332- 146b-430f- b973-2d1cf20428 8d'"
cat /var/log/ kolla/trove/ trove-taskmanag er.log notification [-] Sending event: dbaas.cluster_ create. error, {'client_ip': u'10.128.150.5', 'server_type': u'taskmanager', 'tenant_id': u'd20aecdc013f4 2739c1d110d7d59 2b7a', 'exception': ['Traceback (most recent call last):\n', ' File "/var/lib/ kolla/venv/ local/lib/ python2. 7/site- packages/ trove/taskmanag er/manager. py", line 395, in create_cluster\n cluster_ tasks.create_ cluster( context, cluster_id)\n', ' File "/var/lib/ kolla/venv/ local/lib/ python2. 7/site- packages/ trove/common/ strategies/ cluster/ experimental/ galera_ common/ taskmanager. py", line 141, in create_cluster\n _create_ cluster( )\n', ' File "/var/lib/ kolla/venv/ local/lib/ python2. 7/site- packages/ trove/common/ strategies/ cluster/ experimental/ galera_ common/ taskmanager. py", line 88, in _create_cluster\n cluster_ips = [self.get_ ip(instance) for instance in instances]\n', ' File "/var/lib/ kolla/venv/ local/lib/ python2. 7/site- packages/ trove/taskmanag er/models. py", line 204, in get_ip\n IPs = instance. get_visible_ ip_addresses( )\n', 'IndexError: list index out of range\n'], 'server_ip': u'10.128. 150.100: 8779', 'request_id': u'req-9aa49155- ab23-4fa5- 92c5-b13ab03ec7 80', 'message': 'list index out of range'} _notify /var/lib/ kolla/venv/ local/lib/ python2. 7/site- packages/ trove/common/ notification. py:405 rpc.server [-] Exception during message handling: IndexError: list index out of range rpc.server Traceback (most recent call last): rpc.server File "/var/lib/ kolla/venv/ local/lib/ python2. 7/site- packages/ oslo_messaging/ rpc/server. py", line 160, in _process_incoming rpc.server res = self.dispatcher .dispatch( message) rpc.server File "/var/lib/ kolla/venv/ local/lib/ python2. 7/site- packages/ oslo_messaging/ rpc/dispatcher. py", line 213, in dispatch rpc.server return self._do_ dispatch( endpoint, method, ctxt, args) rpc.server File "/var/lib/ kolla/venv/ local/lib/ python2. 7/site- packages/ oslo_messaging/ rpc/dispatcher. py", line 183, in _do_dispatch rpc.server result = func(ctxt, **new_args) rpc.server File "/var/lib/ kolla/venv/ local/lib/ python2. 7/site- packages/ osprofiler/ profiler. py", line 153, in wrapper rpc.server return f(*args, **kwargs) rpc.server File "/var/lib/ kolla/venv/ local/lib/ python2. 7/site- packages/ trove/taskmanag er/manager. py", line 395, in create_cluster rpc.server cluster_ tasks.create_ cluster( context, cluster_id) rpc.server File "/var/lib/ kolla/venv/ local/lib/ python2. 7/site- packages/ trove/common/ strategies/ cluster/ experimental/ galera_ common/ taskmanager. py", line 141, in create_cluster rpc.server _create_cluster() rpc.server File "/var/lib/ kolla/venv/ local/lib/ python2. 7/site- packages/ trove/common/ strategies/ cluster/ experimental/ galera_ common/ taskmanager. py", line 88, in _create_cluster rpc.server cluster_ips = [self.get_ ip(instance) for instance in instances] rpc.server File "/var/lib/ kolla/venv/ local/lib/ python2. 7/site- packages/ trove/taskmanag er/models. py", line 204, in get_ip rpc.server IPs = instance. get_visible_ ip_addresses( ) rpc.server IndexError: list index out of range rpc.server
...
2018-01-12 18:00:42.293 7 DEBUG trove.common.
2018-01-12 18:00:42.297 7 ERROR oslo_messaging.
2018-01-12 18:00:42.297 7 ERROR oslo_messaging.
2018-01-12 18:00:42.297 7 ERROR oslo_messaging.
2018-01-12 18:00:42.297 7 ERROR oslo_messaging.
2018-01-12 18:00:42.297 7 ERROR oslo_messaging.
2018-01-12 18:00:42.297 7 ERROR oslo_messaging.
2018-01-12 18:00:42.297 7 ERROR oslo_messaging.
2018-01-12 18:00:42.297 7 ERROR oslo_messaging.
2018-01-12 18:00:42.297 7 ERROR oslo_messaging.
2018-01-12 18:00:42.297 7 ERROR oslo_messaging.
2018-01-12 18:00:42.297 7 ERROR oslo_messaging.
2018-01-12 18:00:42.297 7 ERROR oslo_messaging.
2018-01-12 18:00:42.297 7 ERROR oslo_messaging.
2018-01-12 18:00:42.297 7 ERROR oslo_messaging.
2018-01-12 18:00:42.297 7 ERROR oslo_messaging.
2018-01-12 18:00:42.297 7 ERROR oslo_messaging.
2018-01-12 18:00:42.297 7 ERROR oslo_messaging.
2018-01-12 18:00:42.297 7 ERROR oslo_messaging.
2018-01-12 18:00:42.297 7 ERROR oslo_messaging.
2018-01-12 18:00:42.297 7 ERROR oslo_messaging.
...