We faced with an issue during boot_and_list rally scenario: http://paste.openstack.org/show/490154/
We run rally on dedicated harware node with 48 CPUs and 256 RAM
rally@6bb2d5d3882c:~$ rally --version
0.3.2~dev84
We got 500 HTTP error:
from rally log: http://paste.openstack.org/show/490147/
The error was from nova-api:
find 500 HTTP code at the time from haproxy log: http://paste.openstack.org/show/490146/
look at nova log on the node which we know from haproxy: http://paste.openstack.org/show/490145/
can't find a keystone node from haproxy log : http://paste.openstack.org/show/490150/
There is node error in keystone logs at the same time. But I've noticed that there is only 6 processes for keystone wsgi workers: http://paste.openstack.org/show/490153/
And it seems all of the workers was overloaded:
atop from node-1 controller: http://paste.openstack.org/show/490156/
atop from node-2 controller: http://paste.openstack.org/show/490158/
atop from node-3 controller: http://paste.openstack.org/show/490159/
Environment description:
3controllers,20computes+ceph,176computes,Ceph_for_all,vxlan
[root@fuel ~]# cat /etc/fuel/8.0/version.yaml
VERSION:
feature_groups:
- mirantis
production: "docker"
release: "8.0"
api: "1.0"
build_number: "570"
build_id: "570"
fuel-nailgun_sha: "558ca91a854cf29e395940c232911ffb851899c1"
python-fuelclient_sha: "4f234669cfe88a9406f4e438b1e1f74f1ef484a5"
fuel-agent_sha: "658be72c4b42d3e1436b86ac4567ab914bfb451b"
fuel-nailgun-agent_sha: "b2bb466fd5bd92da614cdbd819d6999c510ebfb1"
astute_sha: "b81577a5b7857c4be8748492bae1dec2fa89b446"
fuel-library_sha: "c2a335b5b725f1b994f78d4c78723d29fa44685a"
fuel-ostf_sha: "3bc76a63a9e7d195ff34eadc29552f4235fa6c52"
fuel-mirror_sha: "fb45b80d7bee5899d931f926e5c9512e2b442749"
fuelmenu_sha: "78ffc73065a9674b707c081d128cb7eea611474f"
shotgun_sha: "63645dea384a37dde5c01d4f8905566978e5d906"
network-checker_sha: "a43cf96cd9532f10794dce736350bf5bed350e9d"
fuel-upgrade_sha: "616a7490ec7199f69759e97e42f9b97dfc87e85b"
fuelmain_sha: "d605bcbabf315382d56d0ce8143458be67c53434"
!!! NOTE: RabbitMQ was replaced by ZeroMQ in this installation. Actually, rabbitmq wasn't removed, but zeromq was installed and oslo-messaging worked via zeromq.
Will try to generate Diagnostic Snapshot, but there is bug which can prevent to do that (https://bugs.launchpad.net/fuel/+bug/1546023)
Logs from controller nodes are here: http://mos-scale-share.mirantis.com/bug-1556102.tar.gz
According to keystone.log the average request to API takes >20 second. e.g.:
<46>Mar 11 12:48:03 node-1 keystone_
<46>Mar 11 12:48:03 node-1 keystone_
<46>Mar 11 12:48:03 node-1 keystone_
<46>Mar 11 12:48:03 node-1 keystone_
<46>Mar 11 12:48:03 node-1 keystone_
<46>Mar 11 12:48:03 node-1 keystone_
<46>Mar 11 12:48:03 node-1 keystone_
<46>Mar 11 12:48:03 node-1 keystone_
<46>Mar 11 12:48:03 node-1 keystone_
<46>Mar 11 12:48:03 node-1 keystone_
<46>Mar 11 12:48:03 node-1 keystone_
<46>Mar 11 12:48:03 node-1 keystone_
<46>Mar 11 12:48:03 node-1 keystone_
<46>Mar 11 12:48:03 node-1 keystone_
<46>Mar 11 12:48:03 node-1 keystone_
<46>Mar 11 12:48:03 node-1 keystone_
<46>Mar 11 12:48:03 node-1 keystone_