Juniper Openstack

Vcenter-as-compute: VM launch fails inconsistently due to glance issue

Bug #1689401 reported by Sarath
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R4.0
Invalid
High
Sarath
Juniper Openstack r4.0.1.0 "r4.0.1.0"
Trunk
Invalid
High
Sarath
Juniper Openstack r4.1.0.0-fcs "r4.1"

Bug Description

This issue seen on Sanity HA setup.
Debugged the issue with Senthil and the reason for "inconsistent" isssue is due to glance responding/working good on couple of nodes in this HA setup and one node has issue and when it gets request for VM launch, it fails.

After restarting glance registry on this node, it resumed back working fine.

Please find below the debugging details,

>>> nova-api.log

2017-05-08 12:01:17.236 24077 ERROR nova.api.openstack.extensions Traceback (most recent call last):
2017-05-08 12:01:17.236 24077 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/openstack/extensions.py", line 478, in wrapped
2017-05-08 12:01:17.236 24077 ERROR nova.api.openstack.extensions return f(*args, **kwargs)
2017-05-08 12:01:17.236 24077 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/images.py", line 145, in detail
2017-05-08 12:01:17.236 24077 ERROR nova.api.openstack.extensions **page_params)
2017-05-08 12:01:17.236 24077 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/image/api.py", line 68, in get_all
2017-05-08 12:01:17.236 24077 ERROR nova.api.openstack.extensions return session.detail(context, **kwargs)
2017-05-08 12:01:17.236 24077 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/image/glance.py", line 284, in detail
2017-05-08 12:01:17.236 24077 ERROR nova.api.openstack.extensions for image in images:
2017-05-08 12:01:17.236 24077 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/glanceclient/v1/images.py", line 254, in list
2017-05-08 12:01:17.236 24077 ERROR nova.api.openstack.extensions for image in paginate(params, return_request_id):
2017-05-08 12:01:17.236 24077 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/glanceclient/v1/images.py", line 238, in paginate
2017-05-08 12:01:17.236 24077 ERROR nova.api.openstack.extensions images, resp = self._list(url, "images")
2017-05-08 12:01:17.236 24077 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/glanceclient/v1/images.py", line 63, in _list
2017-05-08 12:01:17.236 24077 ERROR nova.api.openstack.extensions resp, body = self.client.get(url)
2017-05-08 12:01:17.236 24077 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 280, in get
2017-05-08 12:01:17.236 24077 ERROR nova.api.openstack.extensions return self._request('GET', url, **kwargs)
2017-05-08 12:01:17.236 24077 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 272, in _request
2017-05-08 12:01:17.236 24077 ERROR nova.api.openstack.extensions resp, body_iter = self._handle_response(resp)
2017-05-08 12:01:17.236 24077 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 93, in _handle_response
2017-05-08 12:01:17.236 24077 ERROR nova.api.openstack.extensions raise exc.from_response(resp, resp.content)
2017-05-08 12:01:17.236 24077 ERROR nova.api.openstack.extensions HTTPInternalServerError: 500 Internal Server Error: The server has either erred or is incapable of performing the requested operation. (H

>>> on issue node, /var/log/glance/api.log

2017-05-08 13:39:11.559 27651 ERROR glance.common.wsgi [req-d6f08d2a-42db-44fc-acc6-acb9cd003797 42d5a4304bce4d588c771cbf3ad3a8f8 8ab7a0d1ee6149cc8ccf7479de314e76 - - -] Caught error: The request returned 500 Internal Server Error.
2017-05-08 13:39:11.559 27651 ERROR glance.common.wsgi Traceback (most recent call last):
2017-05-08 13:39:11.559 27651 ERROR glance.common.wsgi File "/usr/lib/python2.7/dist-packages/glance/common/wsgi.py", line 879, in __call__
2017-05-08 13:39:11.559 27651 ERROR glance.common.wsgi request, **action_args)
2017-05-08 13:39:11.559 27651 ERROR glance.common.wsgi File "/usr/lib/python2.7/dist-packages/glance/common/wsgi.py", line 907, in dispatch
2017-05-08 13:39:11.559 27651 ERROR glance.common.wsgi return method(*args, **kwargs)
2017-05-08 13:39:11.559 27651 ERROR glance.common.wsgi File "/usr/lib/python2.7/dist-packages/glance/api/v1/images.py", line 367, in detail
2017-05-08 13:39:11.559 27651 ERROR glance.common.wsgi images = registry.get_images_detail(req.context, **params)
2017-05-08 13:39:11.559 27651 ERROR glance.common.wsgi File "/usr/lib/python2.7/dist-packages/glance/registry/client/v1/api.py", line 155, in get_images_detail
2017-05-08 13:39:11.559 27651 ERROR glance.common.wsgi return c.get_images_detailed(**kwargs)
2017-05-08 13:39:11.559 27651 ERROR glance.common.wsgi File "/usr/lib/python2.7/dist-packages/glance/registry/client/v1/client.py", line 150, in get_images_detailed
2017-05-08 13:39:11.559 27651 ERROR glance.common.wsgi res = self.do_request("GET", "/images/detail", params=params)
2017-05-08 13:39:11.559 27651 ERROR glance.common.wsgi File "/usr/lib/python2.7/dist-packages/glance/registry/client/v1/client.py", line 136, in do_request
2017-05-08 13:39:11.559 27651 ERROR glance.common.wsgi 'exc_name': exc_name})
2017-05-08 13:39:11.559 27651 ERROR glance.common.wsgi File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__
2017-05-08 13:39:11.559 27651 ERROR glance.common.wsgi six.reraise(self.type_, self.value, self.tb)
2017-05-08 13:39:11.559 27651 ERROR glance.common.wsgi File "/usr/lib/python2.7/dist-packages/glance/registry/client/v1/client.py", line 121, in do_request
2017-05-08 13:39:11.559 27651 ERROR glance.common.wsgi **kwargs)
2017-05-08 13:39:11.559 27651 ERROR glance.common.wsgi File "/usr/lib/python2.7/dist-packages/glance/common/client.py", line 71, in wrapped
2017-05-08 13:39:11.559 27651 ERROR glance.common.wsgi return func(self, *args, **kwargs)
2017-05-08 13:39:11.559 27651 ERROR glance.common.wsgi File "/usr/lib/python2.7/dist-packages/glance/common/client.py", line 375, in do_request
2017-05-08 13:39:11.559 27651 ERROR glance.common.wsgi headers=copy.deepcopy(headers))
2017-05-08 13:39:11.559 27651 ERROR glance.common.wsgi File "/usr/lib/python2.7/dist-packages/glance/common/client.py", line 88, in wrapped
2017-05-08 13:39:11.559 27651 ERROR glance.common.wsgi return func(self, method, url, body, headers)
2017-05-08 13:39:11.559 27651 ERROR glance.common.wsgi File "/usr/lib/python2.7/dist-packages/glance/common/client.py", line 532, in _do_request
2017-05-08 13:39:11.559 27651 ERROR glance.common.wsgi raise exception.ServerError()
2017-05-08 13:39:11.559 27651 ERROR glance.common.wsgi ServerError: The request returned 500 Internal Server Error.
2017-05-08 13:39:11.559 27651 ERROR glance.common.wsgi
2017-05-08 13:39:11.565 27651 INFO eventlet.wsgi.server [req-d6f08d2a-42db-44fc-acc6-acb9cd003797 42d5a4304bce4d588c771cbf3ad3a8f8 8ab7a0d1ee6149cc8ccf7479de314e76 - - -] 127.0.0.1 - - [08/May/2017 13:39:11] "GET //v1/images/detail?is_public=none&limit=20 HTTP/1.1" 500 430 0.278721
2017-05-08 13:39:28.896 27647 ERROR glance.registry.client.v1.client [req-a0dbd9a6-e28f-458b-a8fa-28219918d3f1 42d5a4304bce4d588c771cbf3ad3a8f8 8ab7a0d1ee6149cc8ccf7479de314e76 - - -] Registry client request GET /images/detail raised ServerError

>>> verifying this node always fails for glance request,

root@5a10s29:~# curl "http://127.0.0.1:9393//v1/images/detail?is_public=none&limit=20" -H 'X-AUTH-TOKEN: 7ac2492a445d4c0e8ebde0067ff1731c'

==> /var/log/glance/api.log <==
2017-05-08 13:39:28.896 27647 ERROR glance.registry.client.v1.client [req-a0dbd9a6-e28f-458b-a8fa-28219918d3f1 42d5a4304bce4d588c771cbf3ad3a8f8 8ab7a0d1ee6149cc8ccf7479de314e76 - - -] Registry client request GET /images/detail raised ServerError
2017-05-08 13:39:28.896 27647 ERROR glance.registry.client.v1.client Traceback (most recent call last):
2017-05-08 13:39:28.896 27647 ERROR glance.registry.client.v1.client File "/usr/lib/python2.7/dist-packages/glance/registry/client/v1/client.py", line 121, in do_request
2017-05-08 13:39:28.896 27647 ERROR glance.registry.client.v1.client **kwargs)
2017-05-08 13:39:28.896 27647 ERROR glance.registry.client.v1.client File "/usr/lib/python2.7/dist-packages/glance/common/client.py", line 71, in wrapped
2017-05-08 13:39:28.896 27647 ERROR glance.registry.client.v1.client return func(self, *args, **kwargs)
2017-05-08 13:39:28.896 27647 ERROR glance.registry.client.v1.client File "/usr/lib/python2.7/dist-packages/glance/common/client.py", line 375, in do_request
2017-05-08 13:39:28.896 27647 ERROR glance.registry.client.v1.client headers=copy.deepcopy(headers))
2017-05-08 13:39:28.896 27647 ERROR glance.registry.client.v1.client File "/usr/lib/python2.7/dist-packages/glance/common/client.py", line 88, in wrapped
2017-05-08 13:39:28.896 27647 ERROR glance.registry.client.v1.client return func(self, method, url, body, headers)
2017-05-08 13:39:28.896 27647 ERROR glance.registry.client.v1.client File "/usr/lib/python2.7/dist-packages/glance/common/client.py", line 532, in _do_request
2017-05-08 13:39:28.896 27647 ERROR glance.registry.client.v1.client raise exception.ServerError()
2017-05-08 13:39:28.896 27647 ERROR glance.registry.client.v1.client ServerError: The request returned 500 Internal Server Error.
2017-05-08 13:39:28.896 27647 ERROR glance.registry.client.v1.client
2017-05-08 13:39:28.899 27647 ERROR glance.common.wsgi [req-a0dbd9a6-e28f-458b-a8fa-28219918d3f1 42d5a4304bce4d588c771cbf3ad3a8f8 8ab7a0d1ee6149cc8ccf7479de314e76 - - -] Caught error: The request returned 500 Internal Server Error.
2017-05-08 13:39:28.899 27647 ERROR glance.common.wsgi Traceback (most recent call last):
2017-05-08 13:39:28.899 27647 ERROR glance.common.wsgi File "/usr/lib/python2.7/dist-packages/glance/common/wsgi.py", line 879, in __call__
2017-05-08 13:39:28.899 27647 ERROR glance.common.wsgi request, **action_args)
2017-05-08 13:39:28.899 27647 ERROR glance.common.wsgi File "/usr/lib/python2.7/dist-packages/glance/common/wsgi.py", line 907, in dispatch
2017-05-08 13:39:28.899 27647 ERROR glance.common.wsgi return method(*args, **kwargs)
2017-05-08 13:39:28.899 27647 ERROR glance.common.wsgi File "/usr/lib/python2.7/dist-packages/glance/api/v1/images.py", line 367, in detail
2017-05-08 13:39:28.899 27647 ERROR glance.common.wsgi images = registry.get_images_detail(req.context, **params)
2017-05-08 13:39:28.899 27647 ERROR glance.common.wsgi File "/usr/lib/python2.7/dist-packages/glance/registry/client/v1/api.py", line 155, in get_images_detail
2017-05-08 13:39:28.899 27647 ERROR glance.common.wsgi return c.get_images_detailed(**kwargs)
2017-05-08 13:39:28.899 27647 ERROR glance.common.wsgi File "/usr/lib/python2.7/dist-packages/glance/registry/client/v1/client.py", line 150, in get_images_detailed
2017-05-08 13:39:28.899 27647 ERROR glance.common.wsgi res = self.do_request("GET", "/images/detail", params=params)
2017-05-08 13:39:28.899 27647 ERROR glance.common.wsgi File "/usr/lib/python2.7/dist-packages/glance/registry/client/v1/client.py", line 136, in do_request
2017-05-08 13:39:28.899 27647 ERROR glance.common.wsgi 'exc_name': exc_name})
2017-05-08 13:39:28.899 27647 ERROR glance.common.wsgi File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__
2017-05-08 13:39:28.899 27647 ERROR glance.common.wsgi six.reraise(self.type_, self.value, self.tb)
2017-05-08 13:39:28.899 27647 ERROR glance.common.wsgi File "/usr/lib/python2.7/dist-packages/glance/registry/client/v1/client.py", line 121, in do_request
2017-05-08 13:39:28.899 27647 ERROR glance.common.wsgi **kwargs)
2017-05-08 13:39:28.899 27647 ERROR glance.common.wsgi File "/usr/lib/python2.7/dist-packages/glance/common/client.py", line 71, in wrapped
2017-05-08 13:39:28.899 27647 ERROR glance.common.wsgi return func(self, *args, **kwargs)
2017-05-08 13:39:28.899 27647 ERROR glance.common.wsgi File "/usr/lib/python2.7/dist-packages/glance/common/client.py", line 375, in do_request
2017-05-08 13:39:28.899 27647 ERROR glance.common.wsgi headers=copy.deepcopy(headers))
2017-05-08 13:39:28.899 27647 ERROR glance.common.wsgi File "/usr/lib/python2.7/dist-packages/glance/common/client.py", line 88, in wrapped
2017-05-08 13:39:28.899 27647 ERROR glance.common.wsgi return func(self, method, url, body, headers)
2017-05-08 13:39:28.899 27647 ERROR glance.common.wsgi File "/usr/lib/python2.7/dist-packages/glance/common/client.py", line 532, in _do_request
2017-05-08 13:39:28.899 27647 ERROR glance.common.wsgi raise exception.ServerError()
2017-05-08 13:39:28.899 27647 ERROR glance.common.wsgi ServerError: The request returned 500 Internal Server Error.
2017-05-08 13:39:28.899 27647 ERROR glance.common.wsgi
<html>
 <head>
  <title>500 Internal Server Error</title>
 </head>
 <body>
  <h1>500 Internal Server Error</h1>
  The server has either erred or is incapable of performing the requested operation.<br /><br />

root@5a10s29:~# ps ax | grep glance
18934 ? S 0:01 /usr/bin/python /usr/bin/glance-registry
20498 pts/0 S+ 0:00 grep --color=auto glance
27119 ? S 18:53 /usr/bin/python /usr/bin/glance-api
27647 ? S 0:27 /usr/bin/python /usr/bin/glance-api
27648 ? S 0:27 /usr/bin/python /usr/bin/glance-api
27650 ? S 0:27 /usr/bin/python /usr/bin/glance-api
27651 ? S 0:28 /usr/bin/python /usr/bin/glance-api
31480 ? S 0:00 /usr/bin/python /usr/bin/glance-registry
31815 ? S 0:00 /usr/bin/python /usr/bin/glance-registry
31816 ? S 0:00 /usr/bin/python /usr/bin/glance-registry
31817 ? S 0:00 /usr/bin/python /usr/bin/glance-registry
31819 ? S 0:00 /usr/bin/python /usr/bin/glance-registry
root@5a10s29:~# netstat -antp | grep 9191
tcp 0 0 0.0.0.0:9191 0.0.0.0:* LISTEN 31480/python
root@5a10s29:~# strace -p 31815 -f
Process 31815 attached
restart_syscall(<... resuming interrupted call ...>
) = 1
accept(3, 0x7fff7648cd40, [16]) = -1 EAGAIN (Resource temporarily unavailable)
poll([{fd=3, events=POLLIN|POLLPRI|POLLERR|POLLHUP}], 1, 60000

>>> After restart of glance registry, this works fine.

Revision history for this message
Sarath (nsarath) wrote :

nsarath@ubuntu-build04:/auto/cores/1689401$ ls -ltrd *
-rwxrwxrwx 1 nsarath test 79185920 May 8 15:10 Kvm-1-log.tar
-rwxrwxrwx 1 nsarath test 74997760 May 8 15:11 Esxi-1-log.tar
-rwxrwxrwx 1 nsarath test 74393600 May 8 15:11 Esxi-2-log.tar
-rwxrwxrwx 1 nsarath test 6983680 May 8 15:11 Ctrl-A-Utils.tar
-rwxrwxrwx 1 nsarath test 1424988160 May 8 15:12 Ctrl-C-log.tar
-rwxrwxrwx 1 nsarath test 1462333440 May 8 15:12 Ctrl-B-log.tar
-rwxrwxrwx 1 nsarath test 1488209920 May 8 15:12 Ctrl-A-log.tar

Revision history for this message
amudhar (amudha) wrote :

Not reproducible manually. Waiting on sanity testbed to debug.

Revision history for this message
Sarath (nsarath) wrote :

Though Inconsistently reproducible, but it is seen more than once impacing VM launch
It is also regression passing on earlier releases so raising priority

Revision history for this message
amudhar (amudha) wrote :

Issue: glance image-show <> and nova boot fail with R4.0 kilo build.

Analysis:
-> 14.04.5 base image and contrail-install-packages_4.0.0.0-6~kilo_all.deb, provisioned using fab – works fine and issue not seen
root@a6s14:/opt/contrail/utils# dpkg -l | grep python-requests
ii python-requests 2.2.1-1ubuntu0.3 all elegant and simple HTTP library for Python, built for human beings
ii python-requests-whl 2.4.3-6~cloud0 all elegant and simple HTTP library for Python, built for human beings
root@a6s14:/opt/contrail/utils# dpkg -l | grep python-urllib3
ii python-urllib3 1.9.1-3~cloud0 all HTTP library with thread-safe connection pooling for Python
ii python-urllib3-whl 1.9.1-3~cloud0 all HTTP library with thread-safe connection pooling
root@a6s14:/opt/contrail/utils#

-> 14.04.5 base image and install using SM repo, issue is seen
root@a6s14:/opt/contrail/utils# dpkg -l | grep python-requests
ii python-requests 2.2.1-1ubuntu0.3 all elegant and simple HTTP library for Python, built for human beings
ii python-requests-whl 2.4.3-6~cloud0 all elegant and simple HTTP library for Python, built for human beings
root@a6s14:/opt/contrail/utils# dpkg -l | grep python-urllib3
ii python-urllib3 1.13.1-2 all HTTP library with thread-safe connection pooling for Python
ii python-urllib3-whl 1.9.1-3~cloud0 all HTTP library with thread-safe connection pooling
root@a6s14:/opt/contrail/utils#

The issue is because of python-urllib3 1.13.1-2 version not being compatible with python-requests 2.2.1.
python-requests 2.2.1. Comes installed in the 14.04.5 base image.
Note, python-urllib3 1.13.1-2 that is being added in server-manager deb is from xenial repo (and not trusty).

Revision history for this message
amudhar (amudha) wrote :

For vcenter-as-compute, the installation recommended is using fab and contrail-install-packages.
Sanity seems to run fab install_contrail:<contrail-cloud.deb> and this sets up server_manager_installer repo.
Please run install_contrail instead, and the issue would not occur.

Update if the issue is still seen.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.