R3.0- HA-nova services are down after upgrade from build 39 to 40
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Juniper Openstack |
Invalid
|
Low
|
shajuvk | ||
R3.0 |
Invalid
|
Low
|
shajuvk |
Bug Description
After upgrading from R3.0.3 build 39 t0 40 some of the nova services on one of the node shows down. SQL db is in sync on all the nodes. Upgrade task was executed in parallel
Please check the node a6s32 (1st cfgm) for the testbed details
----+--
| Id | Binary | Host | Zone | Status | State | Updated_at | Disabled Reason |
+----+-
| 3 | nova-scheduler | a6s32 | internal | enabled | up | 2016-05-
| 6 | nova-console | a6s32 | internal | enabled | up | 2016-05-
| 9 | nova-consoleauth | a6s32 | internal | enabled | up | 2016-05-
| 12 | nova-conductor | a6s32 | internal | enabled | up | 2016-05-
| 15 | nova-scheduler | a6s33 | internal | enabled | up | 2016-05-
| 18 | nova-console | a6s33 | internal | enabled | up | 2016-05-
| 21 | nova-consoleauth | a6s33 | internal | enabled | up | 2016-05-
| 24 | nova-conductor | a6s33 | internal | enabled | up | 2016-05-
| 30 | nova-scheduler | a6s38 | internal | enabled | down | 2016-05-
| 33 | nova-console | a6s38 | internal | enabled | down | 2016-05-
| 36 | nova-consoleauth | a6s38 | internal | enabled | down | 2016-05-
| 39 | nova-conductor | a6s38 | internal | enabled | down | 2016-05-
| 42 | nova-compute | a6s2 | nova | enabled | up | 2016-05-
| 45 | nova-compute | a6s38 | nova | enabled | up | 2016-05-
+----+-
mysql> show status like '%wsrep%';
+------
| Variable_name | Value |
+------
| wsrep_local_
| wsrep_protocol_
| wsrep_last_
| wsrep_replicated | 82509 |
| wsrep_replicate
| wsrep_received | 8545 |
| wsrep_received_
| wsrep_local_commits | 81898 |
| wsrep_local_
| wsrep_local_replays | 0 |
| wsrep_local_
| wsrep_local_
| wsrep_local_
| wsrep_local_
| wsrep_flow_
| wsrep_flow_
| wsrep_flow_
| wsrep_cert_
| wsrep_apply_oooe | 0.000000 |
| wsrep_apply_oool | 0.000000 |
| wsrep_apply_window | 1.000000 |
| wsrep_commit_oooe | 0.000000 |
| wsrep_commit_oool | 0.000000 |
| wsrep_commit_window | 1.000000 |
| wsrep_local_state | 4 |
| wsrep_local_
| wsrep_cert_
| wsrep_causal_reads | 1663330 |
| wsrep_incoming_
| wsrep_cluster_
| wsrep_cluster_size | 3 |
| wsrep_cluster_
| wsrep_cluster_
| wsrep_connected | ON |
| wsrep_local_
| wsrep_local_index | 2 |
| wsrep_provider_name | Galera |
| wsrep_provider_
| wsrep_provider_
| wsrep_ready | ON |
+------
40 rows in set (0.00 sec)
summary: |
- R3.0- HA-nova services are down afterupgrade from build 39 to 40 + R3.0- HA-nova services are down after upgrade from build 39 to 40 |
tags: | added: sanity |
Changed in juniperopenstack: | |
importance: | Undecided → Critical |
information type: | Proprietary → Public |
tags: | added: blocker |
Changed in juniperopenstack: | |
importance: | Critical → High |
Changed in juniperopenstack: | |
assignee: | nobody → Rudra Rugge (rudrarugge) |
Changed in juniperopenstack: | |
assignee: | Rudra Rugge (rudrarugge) → Nagendra Prasath (npchandran) |
Changed in juniperopenstack: | |
status: | Incomplete → Invalid |
Nova scheduler log: common. periodic_ task Traceback (most recent call last): common. periodic_ task File "/usr/lib/ python2. 7/dist- packages/ nova/openstack/ common/ periodic_ task.py" , line 224, in run_periodic_tasks common. periodic_ task task(self, context) common. periodic_ task File "/usr/lib/ python2. 7/dist- packages/ nova/scheduler/ manager. py", line 70, in _expire_ reservations common. periodic_ task QUOTAS. expire( context) common. periodic_ task File "/usr/lib/ python2. 7/dist- packages/ nova/quota. py", line 1444, in expire common. periodic_ task self._driver. expire( context) common. periodic_ task File "/usr/lib/ python2. 7/dist- packages/ nova/quota. py", line 690, in expire common. periodic_ task db.reservation_ expire( context) common. periodic_ task File "/usr/lib/ python2. 7/dist- packages/ nova/db/ api.py" , line 1174, in reservation_expire common. periodic_ task return IMPL.reservatio n_expire( context) common. periodic_ task File "/usr/lib/ python2. 7/dist- packages/ nova/db/ sqlalchemy/ api.py" , line 215, in wrapper common. periodic_ task return f(*args, **kwargs) common. periodic_ task File "/usr/lib/ python2. 7/dist- packages/ nova/db/ sqlalchemy/ api.py" , line 271, in wrapped common. periodic_ task return f(*args, **kwargs) common. periodic_ task File "/usr/lib/ python2. 7/dist- packages/ nova/db/ sqlalchemy/ api.py" , line 3742, in reservation_expire common. periodic_ task reservation_ query.soft_ delete( synchronize_ session= False) common. periodic_ task File "/usr/lib/ python2. 7/dist- packages/ oslo_db/ sqlalchemy/ session. py", line 600, in soft_delete common. periodic_ task synchronize_ session= synchronize_ session) common. periodic_ task File "/usr/lib/ python2. 7/dist- packages/ sqlalchemy/ orm/query. py", line 2796, in update common. periodic_ task update_op.exec_() common. periodic_ task File "/usr/lib/ python2. 7/dist- packages/ sqlalchemy/ orm/persistence .py", line 897, in exec_ common. periodic_ task self._do_exec() common. periodic_ task File "/usr/lib/pyt...
===
2016-05-19 21:44:30.507 31340 TRACE nova.openstack.
2016-05-19 21:44:30.507 31340 TRACE nova.openstack.
2016-05-19 21:44:30.507 31340 TRACE nova.openstack.
2016-05-19 21:44:30.507 31340 TRACE nova.openstack.
2016-05-19 21:44:30.507 31340 TRACE nova.openstack.
2016-05-19 21:44:30.507 31340 TRACE nova.openstack.
2016-05-19 21:44:30.507 31340 TRACE nova.openstack.
2016-05-19 21:44:30.507 31340 TRACE nova.openstack.
2016-05-19 21:44:30.507 31340 TRACE nova.openstack.
2016-05-19 21:44:30.507 31340 TRACE nova.openstack.
2016-05-19 21:44:30.507 31340 TRACE nova.openstack.
2016-05-19 21:44:30.507 31340 TRACE nova.openstack.
2016-05-19 21:44:30.507 31340 TRACE nova.openstack.
2016-05-19 21:44:30.507 31340 TRACE nova.openstack.
2016-05-19 21:44:30.507 31340 TRACE nova.openstack.
2016-05-19 21:44:30.507 31340 TRACE nova.openstack.
2016-05-19 21:44:30.507 31340 TRACE nova.openstack.
2016-05-19 21:44:30.507 31340 TRACE nova.openstack.
2016-05-19 21:44:30.507 31340 TRACE nova.openstack.
2016-05-19 21:44:30.507 31340 TRACE nova.openstack.
2016-05-19 21:44:30.507 31340 TRACE nova.openstack.
2016-05-19 21:44:30.507 31340 TRACE nova.openstack.
2016-05-19 21:44:30.507 31340 TRACE nova.openstack.
2016-05-19 21:44:30.507 31340 TRACE nova.openstack.