Comment 0 for bug 1590478

Revision history for this message
Volodymyr Shypyguzov (vshypyguzov) wrote :

Steps to reproduce:
1. Deploy any environment with 3 controllers and NeutronTUN or NeutronVLAN
2. Fill root filesystem on primary controller. For example, "fallocate -l 10000G /bigfile"
3. Wait for ~5 min
4. Verify networks fail (mcollective aggent fails on primary controller, network checker returns error: "Reason: systemu: Error - process interrupted!")
5. Run OSTF tests <<< FAIL

Expected result: OSTF tests passed
Actual result: All tests failed

In /var/log/apache2/keystone_wsgi_main_error.log:

[Wed Jun 08 15:22:02.028505 2016] [wsgi:error] [pid 54268:tid 140593522988800] (11)Resource temporarily unavailable: [client 10.109.3.3:46881] mod_wsgi (pid=54268): Unable to connect to WSGI daemon process 'keystone_main' on '/var/run/apache2/wsgi.46010.4.2.sock' after multiple attempts as listener backlog limit was exceeded.
[Wed Jun 08 15:22:03.192453 2016] [wsgi:error] [pid 53630:tid 140593756165888] (11)Resource temporarily unavailable: [client 10.109.3.6:52884] mod_wsgi (pid=53630): Unable to connect to WSGI daemon process 'keystone_main' on '/var/run/apache2/wsgi.46010.4.2.sock' after multiple attempts as listener backlog limit was exceeded.

root@node-2:~# haproxy-status.sh | grep DOWN
keystone-1 node-2 Status: DOWN/L7TOUT Sessions: 0 Rate: 0
keystone-1 node-3 Status: DOWN/L7TOUT Sessions: 0 Rate: 0
keystone-1 node-4 Status: DOWN/L7TOUT Sessions: 0 Rate: 0
keystone-1 BACKEND Status: DOWN Sessions: 0 Rate: 0