| 2016-04-20 11:35:00 |
Alexey Stepanov |
bug |
|
|
added bug |
| 2016-04-20 11:57:11 |
Dmitry Klenov |
nominated for series |
|
fuel/mitaka |
|
| 2016-04-20 11:57:11 |
Dmitry Klenov |
bug task added |
|
fuel/mitaka |
|
| 2016-04-20 11:57:11 |
Dmitry Klenov |
nominated for series |
|
fuel/newton |
|
| 2016-04-20 11:57:11 |
Dmitry Klenov |
bug task added |
|
fuel/newton |
|
| 2016-04-20 11:57:17 |
Dmitry Klenov |
fuel/newton: milestone |
9.0 |
10.0 |
|
| 2016-04-20 11:57:19 |
Dmitry Klenov |
fuel/mitaka: milestone |
|
9.0 |
|
| 2016-04-20 11:57:31 |
Dmitry Klenov |
fuel/newton: assignee |
|
Fuel Library Team (fuel-library) |
|
| 2016-04-20 11:57:40 |
Dmitry Klenov |
fuel/mitaka: assignee |
|
Fuel Library Team (fuel-library) |
|
| 2016-04-20 11:57:42 |
Dmitry Klenov |
fuel/newton: importance |
Undecided |
High |
|
| 2016-04-20 11:57:44 |
Dmitry Klenov |
fuel/mitaka: importance |
Undecided |
High |
|
| 2016-04-20 11:58:15 |
Dmitry Klenov |
tags |
|
area-library |
|
| 2016-04-20 11:58:20 |
Dmitry Klenov |
fuel/mitaka: status |
New |
Confirmed |
|
| 2016-04-20 11:58:23 |
Dmitry Klenov |
fuel/newton: status |
New |
Confirmed |
|
| 2016-04-20 12:38:12 |
Alexey Stepanov |
attachment added |
|
Snapshot https://bugs.launchpad.net/fuel/+bug/1572521/+attachment/4640680/+files/fail_error_ceph_ha_restart-fuel-snapshot-2016-04-20_03-46-23.tar.xz |
|
| 2016-04-21 09:34:17 |
Valeriy Saharov |
fuel/mitaka: assignee |
Fuel Library Team (fuel-library) |
Valeriy Saharov (vsakharov) |
|
| 2016-04-21 09:34:22 |
Valeriy Saharov |
fuel/mitaka: status |
Confirmed |
In Progress |
|
| 2016-05-04 16:36:34 |
Valeriy Saharov |
tags |
area-library |
area-library tricky |
|
| 2016-05-04 16:36:42 |
Valeriy Saharov |
fuel/mitaka: status |
In Progress |
Won't Fix |
|
| 2016-05-06 17:05:30 |
Dmitry Pyzhov |
fuel/newton: assignee |
Fuel Library (Deprecated) (fuel-library) |
Fuel Sustaining (fuel-sustaining-team) |
|
| 2016-05-10 10:36:28 |
Dmitry Pyzhov |
fuel/newton: status |
Confirmed |
Invalid |
|
| 2016-05-10 10:36:32 |
Dmitry Pyzhov |
fuel/mitaka: status |
Won't Fix |
Invalid |
|
| 2016-05-10 10:36:35 |
Dmitry Pyzhov |
bug task deleted |
fuel/newton |
|
|
| 2016-05-10 10:37:05 |
Dmitry Pyzhov |
fuel: assignee |
Fuel Sustaining (fuel-sustaining-team) |
Valeriy Saharov (vsakharov) |
|
| 2016-05-16 14:28:48 |
Andrey Sledzinskiy |
attachment added |
|
fail_error_ceph_ha_restart-fuel-snapshot-2016-05-15_02-16-50.tar.xz https://bugs.launchpad.net/fuel/+bug/1572521/+attachment/4663871/+files/fail_error_ceph_ha_restart-fuel-snapshot-2016-05-15_02-16-50.tar.xz |
|
| 2016-05-16 14:29:03 |
Andrey Sledzinskiy |
fuel: status |
Invalid |
New |
|
| 2016-05-16 14:29:06 |
Andrey Sledzinskiy |
fuel/mitaka: status |
Invalid |
New |
|
| 2016-05-17 10:03:39 |
Matthew Mosesohn |
fuel: status |
New |
Confirmed |
|
| 2016-05-17 10:03:43 |
Matthew Mosesohn |
fuel/mitaka: status |
New |
Confirmed |
|
| 2016-05-18 13:30:31 |
Sergey Shevorakov |
tags |
area-library tricky |
area-library swarm-blocker tricky |
|
| 2016-05-19 09:37:49 |
Dmitry Klenov |
fuel: assignee |
Valeriy Saharov (vsakharov) |
Michael Polenchuk (mpolenchuk) |
|
| 2016-05-19 09:37:57 |
Dmitry Klenov |
fuel/mitaka: assignee |
Valeriy Saharov (vsakharov) |
Michael Polenchuk (mpolenchuk) |
|
| 2016-05-20 09:11:20 |
Michael Polenchuk |
bug |
|
|
added subscriber Sergey Shevorakov |
| 2016-05-20 09:11:34 |
Michael Polenchuk |
bug |
|
|
added subscriber Andrey Sledzinskiy |
| 2016-05-20 10:13:34 |
Michael Polenchuk |
fuel: assignee |
Michael Polenchuk (mpolenchuk) |
Fuel QA Team (fuel-qa) |
|
| 2016-05-20 10:13:48 |
Michael Polenchuk |
fuel/mitaka: assignee |
Michael Polenchuk (mpolenchuk) |
Fuel QA Team (fuel-qa) |
|
| 2016-05-20 10:14:54 |
Michael Polenchuk |
bug |
|
|
added subscriber Michael Polenchuk |
| 2016-05-20 11:25:37 |
Dmitry Pyzhov |
tags |
area-library swarm-blocker tricky |
area-library area-qa swarm-blocker tricky |
|
| 2016-05-23 09:47:16 |
Maksim Malchuk |
bug |
|
|
added subscriber Maksim Malchuk |
| 2016-05-23 16:23:59 |
Alexander Kurenyshev |
fuel/mitaka: assignee |
Fuel QA Team (fuel-qa) |
Alexander Kurenyshev (akurenyshev) |
|
| 2016-05-23 16:24:03 |
Alexander Kurenyshev |
fuel: assignee |
Fuel QA Team (fuel-qa) |
Alexander Kurenyshev (akurenyshev) |
|
| 2016-06-01 14:59:58 |
Alexander Kurenyshev |
fuel/mitaka: assignee |
Alexander Kurenyshev (akurenyshev) |
Fuel Sustaining (fuel-sustaining-team) |
|
| 2016-06-01 15:00:09 |
Alexander Kurenyshev |
fuel: assignee |
Alexander Kurenyshev (akurenyshev) |
Fuel Sustaining (fuel-sustaining-team) |
|
| 2016-06-02 10:01:52 |
Oleksiy Molchanov |
fuel/mitaka: milestone |
9.0 |
10.0 |
|
| 2016-06-02 10:01:54 |
Oleksiy Molchanov |
fuel: milestone |
10.0 |
9.0 |
|
| 2016-06-02 10:01:58 |
Oleksiy Molchanov |
fuel/mitaka: milestone |
10.0 |
9.0 |
|
| 2016-06-02 10:02:00 |
Oleksiy Molchanov |
fuel: milestone |
9.0 |
10.0 |
|
| 2016-06-02 10:02:06 |
Oleksiy Molchanov |
fuel/mitaka: assignee |
Fuel Sustaining (fuel-sustaining-team) |
Oleksiy Molchanov (omolchanov) |
|
| 2016-06-02 10:02:09 |
Oleksiy Molchanov |
fuel: assignee |
Fuel Sustaining (fuel-sustaining-team) |
Oleksiy Molchanov (omolchanov) |
|
| 2016-06-02 14:56:08 |
Sergey Shevorakov |
tags |
area-library area-qa swarm-blocker tricky |
area-library area-qa swarm-fail tricky |
|
| 2016-06-02 18:00:49 |
Dmitry Pyzhov |
fuel/mitaka: milestone |
9.0 |
9.0-updates |
|
| 2016-06-02 18:00:52 |
Dmitry Pyzhov |
fuel: status |
Confirmed |
Incomplete |
|
| 2016-06-02 18:00:55 |
Dmitry Pyzhov |
fuel/mitaka: status |
Confirmed |
Incomplete |
|
| 2016-06-06 12:37:07 |
Andrey Sledzinskiy |
fuel: status |
Incomplete |
Confirmed |
|
| 2016-06-06 12:37:09 |
Andrey Sledzinskiy |
fuel/mitaka: status |
Incomplete |
Confirmed |
|
| 2016-06-06 16:38:24 |
Andrey Sledzinskiy |
attachment added |
|
fail_error_ceph_partitions_repetitive_cold_restart-fuel-snapshot-2016-06-06_09-20-51.tar.gz https://bugs.launchpad.net/fuel/+bug/1572521/+attachment/4678283/+files/fail_error_ceph_partitions_repetitive_cold_restart-fuel-snapshot-2016-06-06_09-20-51.tar.gz |
|
| 2016-06-06 16:46:32 |
Dmitry Pyzhov |
tags |
area-library area-qa swarm-fail tricky |
area-library area-qa move-to-mu swarm-fail tricky |
|
| 2016-06-07 11:23:12 |
Oleksiy Molchanov |
fuel: assignee |
Oleksiy Molchanov (omolchanov) |
Fuel QA Team (fuel-qa) |
|
| 2016-06-07 11:23:16 |
Oleksiy Molchanov |
fuel/mitaka: assignee |
Oleksiy Molchanov (omolchanov) |
Fuel QA telco (fuel-qa-telco) |
|
| 2016-06-07 11:23:23 |
Oleksiy Molchanov |
fuel/mitaka: assignee |
Fuel QA telco (fuel-qa-telco) |
Fuel QA Team (fuel-qa) |
|
| 2016-06-07 11:46:19 |
Sergey Shevorakov |
fuel/mitaka: milestone |
9.0-updates |
9.0 |
|
| 2016-06-08 13:34:20 |
Alexander Kurenyshev |
fuel: assignee |
Fuel QA Team (fuel-qa) |
Alexander Kurenyshev (akurenyshev) |
|
| 2016-06-08 13:34:24 |
Alexander Kurenyshev |
fuel/mitaka: assignee |
Fuel QA Team (fuel-qa) |
Alexander Kurenyshev (akurenyshev) |
|
| 2016-06-09 09:24:50 |
Dmitry |
attachment added |
|
fail_error_ha_one_controller_backup_restore-fuel-snapshot-2016-06-08_17-34-06.tar.xz https://bugs.launchpad.net/fuel/+bug/1572521/+attachment/4680248/+files/fail_error_ha_one_controller_backup_restore-fuel-snapshot-2016-06-08_17-34-06.tar.xz |
|
| 2016-06-09 09:26:03 |
Vadim Rovachev |
nominated for series |
|
fuel/7.0.x |
|
| 2016-06-09 09:26:03 |
Vadim Rovachev |
bug task added |
|
fuel/7.0.x |
|
| 2016-06-09 09:26:10 |
Vadim Rovachev |
fuel/7.0.x: milestone |
|
7.0-updates |
|
| 2016-06-09 09:26:17 |
Vadim Rovachev |
fuel/7.0.x: importance |
Undecided |
High |
|
| 2016-06-09 17:00:20 |
Aleksey Zvyagintsev |
fuel/7.0.x: assignee |
|
Alexander Kurenyshev (akurenyshev) |
|
| 2016-06-09 17:00:25 |
Aleksey Zvyagintsev |
fuel/7.0.x: status |
New |
Confirmed |
|
| 2016-06-10 11:55:07 |
Sergey Shevorakov |
tags |
area-library area-qa move-to-mu swarm-fail tricky |
area-library area-qa move-to-mu swarm-blocker tricky |
|
| 2016-06-10 12:16:19 |
Alexander Kurenyshev |
fuel/7.0.x: status |
Confirmed |
Fix Committed |
|
| 2016-06-10 12:16:27 |
Alexander Kurenyshev |
fuel/mitaka: status |
Confirmed |
Fix Committed |
|
| 2016-06-10 12:16:31 |
Alexander Kurenyshev |
fuel: status |
Confirmed |
Fix Committed |
|
| 2016-06-13 08:52:44 |
Andrey Sledzinskiy |
fuel: status |
Fix Committed |
Confirmed |
|
| 2016-06-13 08:52:47 |
Andrey Sledzinskiy |
fuel/mitaka: status |
Fix Committed |
Confirmed |
|
| 2016-06-13 08:54:22 |
Andrey Sledzinskiy |
attachment added |
|
fail_error_ceph_ha_restart-fuel-snapshot-2016-06-13_05-40-09.tar.gz https://bugs.launchpad.net/fuel/+bug/1572521/+attachment/4682763/+files/fail_error_ceph_ha_restart-fuel-snapshot-2016-06-13_05-40-09.tar.gz |
|
| 2016-06-14 11:13:06 |
Alexander Kurenyshev |
fuel: assignee |
Alexander Kurenyshev (akurenyshev) |
Fuel Sustaining (fuel-sustaining-team) |
|
| 2016-06-14 11:13:19 |
Alexander Kurenyshev |
fuel/mitaka: assignee |
Alexander Kurenyshev (akurenyshev) |
Fuel Sustaining (fuel-sustaining-team) |
|
| 2016-06-14 12:17:47 |
Maksim Malchuk |
fuel/mitaka: assignee |
Fuel Sustaining (fuel-sustaining-team) |
Alexander Kurenyshev (akurenyshev) |
|
| 2016-06-14 12:17:55 |
Maksim Malchuk |
fuel: assignee |
Fuel Sustaining (fuel-sustaining-team) |
Alexander Kurenyshev (akurenyshev) |
|
| 2016-06-14 12:18:14 |
Maksim Malchuk |
tags |
area-library area-qa move-to-mu swarm-blocker tricky |
area-python area-qa move-to-mu swarm-blocker tricky |
|
| 2016-06-15 13:52:49 |
Nastya Urlapova |
tags |
area-python area-qa move-to-mu swarm-blocker tricky |
area-python move-to-mu tricky |
|
| 2016-06-15 13:53:00 |
Nastya Urlapova |
fuel/mitaka: milestone |
9.0 |
9.0-updates |
|
| 2016-06-30 12:15:03 |
Dmitry Pyzhov |
tags |
area-python move-to-mu tricky |
move-to-mu tricky |
|
| 2016-07-19 14:08:16 |
Sergey Shevorakov |
tags |
move-to-mu tricky |
move-to-mu swarm-fail tricky |
|
| 2016-07-20 11:30:26 |
Alexander Kurenyshev |
fuel/mitaka: assignee |
Alexander Kurenyshev (akurenyshev) |
Fuel Sustaining (fuel-sustaining-team) |
|
| 2016-07-20 11:30:48 |
Alexander Kurenyshev |
fuel: assignee |
Alexander Kurenyshev (akurenyshev) |
Fuel Sustaining (fuel-sustaining-team) |
|
| 2016-07-20 13:07:35 |
Maksim Malchuk |
fuel: assignee |
Fuel Sustaining (fuel-sustaining-team) |
Alexander Kurenyshev (akurenyshev) |
|
| 2016-07-20 13:07:44 |
Maksim Malchuk |
fuel/mitaka: assignee |
Fuel Sustaining (fuel-sustaining-team) |
Alexander Kurenyshev (akurenyshev) |
|
| 2016-07-20 13:35:35 |
Alexander Kurenyshev |
summary |
Time on nodes was not set with 'ntpdate' |
Pacemaker resources are failed to start after the nodes cold reboot |
|
| 2016-07-20 13:51:43 |
Alexander Kurenyshev |
description |
Steps to reproduce:
Scenario:
1. Revert from ceph_ha
2. Waiting up galera and cinder
3. Check ceph status
4. Run OSTF
5. Destroy and remove osd-node
6. Check ceph status
7. Run OSTF
8. Destroy and remove one compute node
9. Check ceph status
10. Run OSTF
11. Cold restart
12. Waiting up galera and cinder
13. Run single OSTF - Create volume and attach it to instance
14. Run OSTF
Actual result:
CI test failed https://product-ci.infra.mirantis.net/job/9.0.system_test.ubuntu.thread_3/83/console
2016-04-20 03:50:42,371 - ERROR decorators.py:126 -- Traceback (most recent call last):
File "/home/jenkins/workspace/9.0.system_test.ubuntu.thread_3/fuelweb_test/helpers/decorators.py", line 120, in wrapper
result = func(*args, **kwargs)
File "/home/jenkins/workspace/9.0.system_test.ubuntu.thread_3/fuelweb_test/tests/tests_strength/test_restart.py", line 165, in ceph_ha_restart
'slave-04']))
File "/home/jenkins/workspace/9.0.system_test.ubuntu.thread_3/fuelweb_test/models/fuel_web_client.py", line 1860, in cold_restart_nodes
self.environment.sync_time()
File "/home/jenkins/workspace/9.0.system_test.ubuntu.thread_3/fuelweb_test/models/environment.py", line 137, in sync_time
new_time = sync_time(self.d_env, nodes_names, skip_sync)
File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/devops/helpers/retry.py", line 27, in wrapper
return func(*args, **kwargs)
File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/devops/helpers/ntp.py", line 45, in sync_time
g_ntp.do_sync_time(g_ntp.other_ntps)
File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/devops/helpers/ntp.py", line 134, in do_sync_time
.format(self.report_not_synchronized(ntps)))
TimeoutError: Time on nodes was not set with 'ntpdate':
[(u'slave-04', ['Wed Apr 20 03:46:22 UTC 2016\n'])]
Expected result:
test pass
iso: fuel-9.0-220-2016-04-19_08-00-00.iso
Additional information:
Node startup or network start was too long. |
Steps to reproduce:
1. Create cluster
2. Add 3 nodes with controller and ceph OSD roles
3. Add 1 node with ceph OSD roles
4. Add 2 nodes with compute and ceph OSD roles
5. Deploy the cluster
6. Cold reboot all nodes in the cluster
Expected result:
The cluster is working, all pcs resources are up after nodes startup
Actual result:
The luster is broken. All resources are down.
Presumably the main problem is outage of resources on VM nodes. When we have add memory to the controller nodes mysql enabled performance mode and got all free memory again.
Such behavior doesn't appear on the baremetal lab with 64GB memory on controller nodes.
This leads to the test [1] error with ntpdate, because all resources are down including ntp one:
2016-04-20 03:50:42,371 - ERROR decorators.py:126 -- Traceback (most recent call last):
File "/home/jenkins/workspace/9.0.system_test.ubuntu.thread_3/fuelweb_test/helpers/decorators.py", line 120, in wrapper
result = func(*args, **kwargs)
File "/home/jenkins/workspace/9.0.system_test.ubuntu.thread_3/fuelweb_test/tests/tests_strength/test_restart.py", line 165, in ceph_ha_restart
'slave-04']))
File "/home/jenkins/workspace/9.0.system_test.ubuntu.thread_3/fuelweb_test/models/fuel_web_client.py", line 1860, in cold_restart_nodes
self.environment.sync_time()
File "/home/jenkins/workspace/9.0.system_test.ubuntu.thread_3/fuelweb_test/models/environment.py", line 137, in sync_time
new_time = sync_time(self.d_env, nodes_names, skip_sync)
File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/devops/helpers/retry.py", line 27, in wrapper
return func(*args, **kwargs)
File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/devops/helpers/ntp.py", line 45, in sync_time
g_ntp.do_sync_time(g_ntp.other_ntps)
File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/devops/helpers/ntp.py", line 134, in do_sync_time
.format(self.report_not_synchronized(ntps)))
TimeoutError: Time on nodes was not set with 'ntpdate':
[(u'slave-04', ['Wed Apr 20 03:46:22 UTC 2016\n'])]
This error couldn't be fixed in the test with smart waiting for ntp resource. When tried to reproduce the cluster was broken even after one day waiting.
[1] https://github.com/openstack/fuel-qa/blob/391b1219ebfa5f3d136054931076cbd03644c6eb/fuelweb_test/tests/tests_strength/test_restart.py#L90 |
|
| 2016-07-20 13:52:16 |
Alexander Kurenyshev |
fuel: assignee |
Alexander Kurenyshev (akurenyshev) |
Fuel Sustaining (fuel-sustaining-team) |
|
| 2016-07-20 13:52:31 |
Alexander Kurenyshev |
fuel/mitaka: assignee |
Alexander Kurenyshev (akurenyshev) |
Fuel Sustaining (fuel-sustaining-team) |
|
| 2016-07-20 14:48:10 |
Oleksiy Molchanov |
summary |
Pacemaker resources are failed to start after the nodes cold reboot |
ntpdate run before p_ntp pacemaker resource is up |
|
| 2016-07-20 14:49:18 |
Oleksiy Molchanov |
fuel: assignee |
Fuel Sustaining (fuel-sustaining-team) |
Fuel QA Team (fuel-qa) |
|
| 2016-07-20 14:49:23 |
Oleksiy Molchanov |
fuel/mitaka: assignee |
Fuel Sustaining (fuel-sustaining-team) |
Fuel QA Team (fuel-qa) |
|
| 2016-08-04 18:29:44 |
Nastya Urlapova |
fuel/mitaka: status |
Confirmed |
Incomplete |
|
| 2016-08-24 14:56:06 |
Alexander Kurenyshev |
fuel/mitaka: status |
Incomplete |
Invalid |
|
| 2016-08-24 14:56:21 |
Alexander Kurenyshev |
fuel: status |
Confirmed |
Invalid |
|
