Vcenter-as-compute: VM launch fails inconsistently due to glance issue
Affects | Status | Importance | Assigned to | Milestone | ||
---|---|---|---|---|---|---|
Juniper Openstack | Status tracked in Trunk | |||||
R4.0 |
Invalid
|
High
|
Sarath | |||
Trunk |
Invalid
|
High
|
Sarath |
Bug Description
This issue seen on Sanity HA setup.
Debugged the issue with Senthil and the reason for "inconsistent" isssue is due to glance responding/working good on couple of nodes in this HA setup and one node has issue and when it gets request for VM launch, it fails.
After restarting glance registry on this node, it resumed back working fine.
Please find below the debugging details,
>>> nova-api.log
2017-05-08 12:01:17.236 24077 ERROR nova.api.
2017-05-08 12:01:17.236 24077 ERROR nova.api.
2017-05-08 12:01:17.236 24077 ERROR nova.api.
2017-05-08 12:01:17.236 24077 ERROR nova.api.
2017-05-08 12:01:17.236 24077 ERROR nova.api.
2017-05-08 12:01:17.236 24077 ERROR nova.api.
2017-05-08 12:01:17.236 24077 ERROR nova.api.
2017-05-08 12:01:17.236 24077 ERROR nova.api.
2017-05-08 12:01:17.236 24077 ERROR nova.api.
2017-05-08 12:01:17.236 24077 ERROR nova.api.
2017-05-08 12:01:17.236 24077 ERROR nova.api.
2017-05-08 12:01:17.236 24077 ERROR nova.api.
2017-05-08 12:01:17.236 24077 ERROR nova.api.
2017-05-08 12:01:17.236 24077 ERROR nova.api.
2017-05-08 12:01:17.236 24077 ERROR nova.api.
2017-05-08 12:01:17.236 24077 ERROR nova.api.
2017-05-08 12:01:17.236 24077 ERROR nova.api.
2017-05-08 12:01:17.236 24077 ERROR nova.api.
2017-05-08 12:01:17.236 24077 ERROR nova.api.
2017-05-08 12:01:17.236 24077 ERROR nova.api.
2017-05-08 12:01:17.236 24077 ERROR nova.api.
2017-05-08 12:01:17.236 24077 ERROR nova.api.
>>> on issue node, /var/log/
2017-05-08 13:39:11.559 27651 ERROR glance.common.wsgi [req-d6f08d2a-
2017-05-08 13:39:11.559 27651 ERROR glance.common.wsgi Traceback (most recent call last):
2017-05-08 13:39:11.559 27651 ERROR glance.common.wsgi File "/usr/lib/
2017-05-08 13:39:11.559 27651 ERROR glance.common.wsgi request, **action_args)
2017-05-08 13:39:11.559 27651 ERROR glance.common.wsgi File "/usr/lib/
2017-05-08 13:39:11.559 27651 ERROR glance.common.wsgi return method(*args, **kwargs)
2017-05-08 13:39:11.559 27651 ERROR glance.common.wsgi File "/usr/lib/
2017-05-08 13:39:11.559 27651 ERROR glance.common.wsgi images = registry.
2017-05-08 13:39:11.559 27651 ERROR glance.common.wsgi File "/usr/lib/
2017-05-08 13:39:11.559 27651 ERROR glance.common.wsgi return c.get_images_
2017-05-08 13:39:11.559 27651 ERROR glance.common.wsgi File "/usr/lib/
2017-05-08 13:39:11.559 27651 ERROR glance.common.wsgi res = self.do_
2017-05-08 13:39:11.559 27651 ERROR glance.common.wsgi File "/usr/lib/
2017-05-08 13:39:11.559 27651 ERROR glance.common.wsgi 'exc_name': exc_name})
2017-05-08 13:39:11.559 27651 ERROR glance.common.wsgi File "/usr/lib/
2017-05-08 13:39:11.559 27651 ERROR glance.common.wsgi six.reraise(
2017-05-08 13:39:11.559 27651 ERROR glance.common.wsgi File "/usr/lib/
2017-05-08 13:39:11.559 27651 ERROR glance.common.wsgi **kwargs)
2017-05-08 13:39:11.559 27651 ERROR glance.common.wsgi File "/usr/lib/
2017-05-08 13:39:11.559 27651 ERROR glance.common.wsgi return func(self, *args, **kwargs)
2017-05-08 13:39:11.559 27651 ERROR glance.common.wsgi File "/usr/lib/
2017-05-08 13:39:11.559 27651 ERROR glance.common.wsgi headers=
2017-05-08 13:39:11.559 27651 ERROR glance.common.wsgi File "/usr/lib/
2017-05-08 13:39:11.559 27651 ERROR glance.common.wsgi return func(self, method, url, body, headers)
2017-05-08 13:39:11.559 27651 ERROR glance.common.wsgi File "/usr/lib/
2017-05-08 13:39:11.559 27651 ERROR glance.common.wsgi raise exception.
2017-05-08 13:39:11.559 27651 ERROR glance.common.wsgi ServerError: The request returned 500 Internal Server Error.
2017-05-08 13:39:11.559 27651 ERROR glance.common.wsgi
2017-05-08 13:39:11.565 27651 INFO eventlet.
2017-05-08 13:39:28.896 27647 ERROR glance.
>>> verifying this node always fails for glance request,
root@5a10s29:~# curl "http://
==> /var/log/
2017-05-08 13:39:28.896 27647 ERROR glance.
2017-05-08 13:39:28.896 27647 ERROR glance.
2017-05-08 13:39:28.896 27647 ERROR glance.
2017-05-08 13:39:28.896 27647 ERROR glance.
2017-05-08 13:39:28.896 27647 ERROR glance.
2017-05-08 13:39:28.896 27647 ERROR glance.
2017-05-08 13:39:28.896 27647 ERROR glance.
2017-05-08 13:39:28.896 27647 ERROR glance.
2017-05-08 13:39:28.896 27647 ERROR glance.
2017-05-08 13:39:28.896 27647 ERROR glance.
2017-05-08 13:39:28.896 27647 ERROR glance.
2017-05-08 13:39:28.896 27647 ERROR glance.
2017-05-08 13:39:28.896 27647 ERROR glance.
2017-05-08 13:39:28.896 27647 ERROR glance.
2017-05-08 13:39:28.899 27647 ERROR glance.common.wsgi [req-a0dbd9a6-
2017-05-08 13:39:28.899 27647 ERROR glance.common.wsgi Traceback (most recent call last):
2017-05-08 13:39:28.899 27647 ERROR glance.common.wsgi File "/usr/lib/
2017-05-08 13:39:28.899 27647 ERROR glance.common.wsgi request, **action_args)
2017-05-08 13:39:28.899 27647 ERROR glance.common.wsgi File "/usr/lib/
2017-05-08 13:39:28.899 27647 ERROR glance.common.wsgi return method(*args, **kwargs)
2017-05-08 13:39:28.899 27647 ERROR glance.common.wsgi File "/usr/lib/
2017-05-08 13:39:28.899 27647 ERROR glance.common.wsgi images = registry.
2017-05-08 13:39:28.899 27647 ERROR glance.common.wsgi File "/usr/lib/
2017-05-08 13:39:28.899 27647 ERROR glance.common.wsgi return c.get_images_
2017-05-08 13:39:28.899 27647 ERROR glance.common.wsgi File "/usr/lib/
2017-05-08 13:39:28.899 27647 ERROR glance.common.wsgi res = self.do_
2017-05-08 13:39:28.899 27647 ERROR glance.common.wsgi File "/usr/lib/
2017-05-08 13:39:28.899 27647 ERROR glance.common.wsgi 'exc_name': exc_name})
2017-05-08 13:39:28.899 27647 ERROR glance.common.wsgi File "/usr/lib/
2017-05-08 13:39:28.899 27647 ERROR glance.common.wsgi six.reraise(
2017-05-08 13:39:28.899 27647 ERROR glance.common.wsgi File "/usr/lib/
2017-05-08 13:39:28.899 27647 ERROR glance.common.wsgi **kwargs)
2017-05-08 13:39:28.899 27647 ERROR glance.common.wsgi File "/usr/lib/
2017-05-08 13:39:28.899 27647 ERROR glance.common.wsgi return func(self, *args, **kwargs)
2017-05-08 13:39:28.899 27647 ERROR glance.common.wsgi File "/usr/lib/
2017-05-08 13:39:28.899 27647 ERROR glance.common.wsgi headers=
2017-05-08 13:39:28.899 27647 ERROR glance.common.wsgi File "/usr/lib/
2017-05-08 13:39:28.899 27647 ERROR glance.common.wsgi return func(self, method, url, body, headers)
2017-05-08 13:39:28.899 27647 ERROR glance.common.wsgi File "/usr/lib/
2017-05-08 13:39:28.899 27647 ERROR glance.common.wsgi raise exception.
2017-05-08 13:39:28.899 27647 ERROR glance.common.wsgi ServerError: The request returned 500 Internal Server Error.
2017-05-08 13:39:28.899 27647 ERROR glance.common.wsgi
<html>
<head>
<title>500 Internal Server Error</title>
</head>
<body>
<h1>500 Internal Server Error</h1>
The server has either erred or is incapable of performing the requested operation.<br /><br />
root@5a10s29:~# ps ax | grep glance
18934 ? S 0:01 /usr/bin/python /usr/bin/
20498 pts/0 S+ 0:00 grep --color=auto glance
27119 ? S 18:53 /usr/bin/python /usr/bin/glance-api
27647 ? S 0:27 /usr/bin/python /usr/bin/glance-api
27648 ? S 0:27 /usr/bin/python /usr/bin/glance-api
27650 ? S 0:27 /usr/bin/python /usr/bin/glance-api
27651 ? S 0:28 /usr/bin/python /usr/bin/glance-api
31480 ? S 0:00 /usr/bin/python /usr/bin/
31815 ? S 0:00 /usr/bin/python /usr/bin/
31816 ? S 0:00 /usr/bin/python /usr/bin/
31817 ? S 0:00 /usr/bin/python /usr/bin/
31819 ? S 0:00 /usr/bin/python /usr/bin/
root@5a10s29:~# netstat -antp | grep 9191
tcp 0 0 0.0.0.0:9191 0.0.0.0:* LISTEN 31480/python
root@5a10s29:~# strace -p 31815 -f
Process 31815 attached
restart_
) = 1
accept(3, 0x7fff7648cd40, [16]) = -1 EAGAIN (Resource temporarily unavailable)
poll([{fd=3, events=
>>> After restart of glance registry, this works fine.
nsarath@ ubuntu- build04: /auto/cores/ 1689401$ ls -ltrd *
-rwxrwxrwx 1 nsarath test 79185920 May 8 15:10 Kvm-1-log.tar
-rwxrwxrwx 1 nsarath test 74997760 May 8 15:11 Esxi-1-log.tar
-rwxrwxrwx 1 nsarath test 74393600 May 8 15:11 Esxi-2-log.tar
-rwxrwxrwx 1 nsarath test 6983680 May 8 15:11 Ctrl-A-Utils.tar
-rwxrwxrwx 1 nsarath test 1424988160 May 8 15:12 Ctrl-C-log.tar
-rwxrwxrwx 1 nsarath test 1462333440 May 8 15:12 Ctrl-B-log.tar
-rwxrwxrwx 1 nsarath test 1488209920 May 8 15:12 Ctrl-A-log.tar