| 2016-09-14 09:20:09 |
Timur Nurlygayanov |
bug |
|
|
added bug |
| 2016-09-14 09:21:07 |
Timur Nurlygayanov |
description |
The gates are broken, example:
https://review.openstack.org/#/c/369427/3/
in the logs we can see the following error:
2016-09-14T06:26:54.030000+00:00 warning: [ 2687.871970] nova-api invoked oom-killer: gfp_mask=0x24201ca, order=0, oom_score_adj=0
2016-09-14T06:26:54.030367+00:00 err: [ 2687.872346] Out of memory: Kill process 411 (mysqld) score 45 or sacrifice child
2016-09-14T06:26:54.030367+00:00 err: [ 2687.872419] Killed process 411 (mysqld) total-vm:2547180kB, anon-rss:115368kB, file-rss:0kB
The root of the issue:
We discussed the issue with Maksim Malchuk and he suggested to increase the size of RAM on controller nodes in this job to make gate more stable. |
The gates are broken, example:
https://review.openstack.org/#/c/369427/3/
in the logs we can see the following error:
2016-09-14T06:26:54.030000+00:00 warning: [ 2687.871970] nova-api invoked oom-killer: gfp_mask=0x24201ca, order=0, oom_score_adj=0
2016-09-14T06:26:54.030367+00:00 err: [ 2687.872346] Out of memory: Kill process 411 (mysqld) score 45 or sacrifice child
2016-09-14T06:26:54.030367+00:00 err: [ 2687.872419] Killed process 411 (mysqld) total-vm:2547180kB, anon-rss:115368kB, file-rss:0kB
The root of the issue:
We discussed the issue with Maksim Malchuk and he suggested to increase the size of RAM on controller nodes in this job to make gate more stable.
Note:
We need to review all gates where we run BVT/SWARM tests and make sure we increase the RAM size for all of them to avoid random false-negative fails. |
|
| 2016-09-14 09:21:13 |
Timur Nurlygayanov |
tags |
|
area-ci |
|
| 2016-09-14 09:21:21 |
Timur Nurlygayanov |
fuel: assignee |
|
Fuel CI (fuel-ci) |
|
| 2016-09-14 09:21:24 |
Timur Nurlygayanov |
fuel: importance |
Undecided |
High |
|
| 2016-09-14 09:21:28 |
Timur Nurlygayanov |
nominated for series |
|
fuel/mitaka |
|
| 2016-09-14 09:21:28 |
Timur Nurlygayanov |
bug task added |
|
fuel/mitaka |
|
| 2016-09-14 09:21:28 |
Timur Nurlygayanov |
nominated for series |
|
fuel/newton |
|
| 2016-09-14 09:21:28 |
Timur Nurlygayanov |
bug task added |
|
fuel/newton |
|
| 2016-09-14 09:21:45 |
Timur Nurlygayanov |
fuel/mitaka: importance |
Undecided |
High |
|
| 2016-09-14 09:21:52 |
Timur Nurlygayanov |
fuel/mitaka: assignee |
|
Fuel CI (fuel-ci) |
|
| 2016-09-14 09:21:55 |
Timur Nurlygayanov |
fuel/mitaka: milestone |
|
9.1 |
|
| 2016-09-14 09:21:58 |
Timur Nurlygayanov |
fuel/newton: milestone |
|
10.0 |
|
| 2016-09-14 09:42:29 |
Maksim Malchuk |
bug |
|
|
added subscriber Maksim Malchuk |
| 2016-09-14 10:03:35 |
Nastya Urlapova |
fuel/mitaka: status |
New |
Incomplete |
|
| 2016-09-14 10:03:41 |
Nastya Urlapova |
fuel/newton: status |
New |
Incomplete |
|
| 2016-09-14 10:03:45 |
Roman Vyalov |
fuel/mitaka: assignee |
Fuel CI (fuel-ci) |
Dmitry Kaigarodеsev (dkaiharodsev) |
|
| 2016-09-14 10:03:51 |
Roman Vyalov |
fuel/newton: assignee |
Fuel CI (fuel-ci) |
Dmitry Kaigarodеsev (dkaiharodsev) |
|
| 2016-09-14 10:03:58 |
Roman Vyalov |
fuel/mitaka: status |
Incomplete |
Won't Fix |
|
| 2016-09-14 10:04:03 |
Roman Vyalov |
fuel/mitaka: status |
Won't Fix |
Confirmed |
|
| 2016-09-14 10:04:11 |
Roman Vyalov |
fuel/newton: status |
Incomplete |
Invalid |
|
| 2016-09-14 10:04:38 |
Roman Vyalov |
fuel/mitaka: status |
Confirmed |
New |
|
| 2016-09-14 10:04:42 |
Roman Vyalov |
fuel/newton: status |
Invalid |
Incomplete |
|
| 2016-09-14 10:04:44 |
Roman Vyalov |
fuel/newton: status |
Incomplete |
New |
|
| 2016-09-14 10:05:32 |
Nastya Urlapova |
summary |
The gates are broken: need to increase the size of RAM, MySQL crashed |
[gates] Cluster is not deployed: some nodes are in the Error state |
|
| 2016-09-14 10:22:16 |
Roman Vyalov |
fuel/newton: assignee |
Dmitry Kaigarodеsev (dkaiharodsev) |
Fuel QA Team (fuel-qa) |
|
| 2016-09-14 10:24:20 |
Roman Vyalov |
fuel/mitaka: status |
New |
Confirmed |
|
| 2016-09-14 10:30:07 |
Nastya Urlapova |
fuel/newton: assignee |
Fuel QA Team (fuel-qa) |
|
|
| 2016-09-14 10:57:14 |
Roman Vyalov |
fuel/newton: assignee |
|
MOS Linux (mos-linux) |
|
| 2016-09-14 11:03:50 |
Maksim Malchuk |
fuel/newton: status |
New |
Confirmed |
|
| 2016-09-14 11:03:59 |
Maksim Malchuk |
fuel/newton: assignee |
MOS Linux (mos-linux) |
Dmitry Kaigarodеsev (dkaiharodsev) |
|
| 2016-09-14 11:04:06 |
Maksim Malchuk |
bug task deleted |
fuel/newton |
|
|
| 2016-09-14 11:04:32 |
Maksim Malchuk |
fuel/mitaka: status |
Confirmed |
Invalid |
|
| 2016-09-14 11:53:20 |
Roman Vyalov |
fuel: assignee |
Dmitry Kaigarodеsev (dkaiharodsev) |
|
|
| 2016-09-14 11:53:22 |
Roman Vyalov |
fuel: status |
Confirmed |
New |
|
| 2016-09-14 12:25:33 |
Dmitry Kaigarodеsev |
fuel: assignee |
|
Fuel QA Team (fuel-qa) |
|
| 2016-09-14 12:25:39 |
Dmitry Kaigarodеsev |
fuel: status |
New |
Confirmed |
|
| 2016-09-14 12:29:55 |
Nastya Urlapova |
fuel: assignee |
Fuel QA Team (fuel-qa) |
Maksim Malchuk (mmalchuk) |
|
| 2016-09-14 12:30:03 |
Nastya Urlapova |
fuel: status |
Confirmed |
Won't Fix |
|
| 2016-09-14 12:30:06 |
Nastya Urlapova |
fuel: status |
Won't Fix |
Incomplete |
|
| 2016-09-14 12:49:43 |
Maksim Malchuk |
description |
The gates are broken, example:
https://review.openstack.org/#/c/369427/3/
in the logs we can see the following error:
2016-09-14T06:26:54.030000+00:00 warning: [ 2687.871970] nova-api invoked oom-killer: gfp_mask=0x24201ca, order=0, oom_score_adj=0
2016-09-14T06:26:54.030367+00:00 err: [ 2687.872346] Out of memory: Kill process 411 (mysqld) score 45 or sacrifice child
2016-09-14T06:26:54.030367+00:00 err: [ 2687.872419] Killed process 411 (mysqld) total-vm:2547180kB, anon-rss:115368kB, file-rss:0kB
The root of the issue:
We discussed the issue with Maksim Malchuk and he suggested to increase the size of RAM on controller nodes in this job to make gate more stable.
Note:
We need to review all gates where we run BVT/SWARM tests and make sure we increase the RAM size for all of them to avoid random false-negative fails. |
The gates are broken, example:
https://review.openstack.org/#/c/369427/3/
in the logs we can see the following error:
2016-09-14T06:26:54.030000+00:00 warning: [ 2687.871970] nova-api invoked oom-killer: gfp_mask=0x24201ca, order=0, oom_score_adj=0
2016-09-14T06:26:54.030367+00:00 err: [ 2687.872346] Out of memory: Kill process 411 (mysqld) score 45 or sacrifice child
2016-09-14T06:26:54.030367+00:00 err: [ 2687.872419] Killed process 411 (mysqld) total-vm:2547180kB, anon-rss:115368kB, file-rss:0kB
The root of the issue:
We discussed the issue with Maksim Malchuk and he suggested to increase the size of RAM on controller nodes in this job to make gate more stable.
Note:
We need to review all gates where we run BVT/SWARM tests and make sure we increase the RAM size for all of them to avoid random false-negative fails.
Also failed tests due to out of memory issue:
https://ci.fuel-infra.org/job/master.puppet-openstack.fuel-library.pkgs.ubuntu.smoke_neutron/4628
https://product-ci.infra.mirantis.net/view/10.0/job/10.0.main.ubuntu.smoke_neutron/656/
https://product-ci.infra.mirantis.net/view/10.0/job/10.0.main.ubuntu.smoke_neutron/652/
https://ci.fuel-infra.org/job/master.puppet-openstack.fuel-library.pkgs.ubuntu.smoke_neutron/4678/
https://ci.fuel-infra.org/job/master.puppet-openstack.fuel-library.pkgs.ubuntu.smoke_neutron/4683/
https://ci.fuel-infra.org/job/master.puppet-openstack.fuel-library.pkgs.ubuntu.smoke_neutron/4681/
https://ci.fuel-infra.org/job/master.puppet-openstack.fuel-library.pkgs.ubuntu.smoke_neutron/4679/
https://ci.fuel-infra.org/job/master.puppet-openstack.fuel-library.pkgs.ubuntu.smoke_neutron/4695/ |
|
| 2016-09-14 12:49:49 |
Maksim Malchuk |
fuel: status |
Incomplete |
Confirmed |
|
| 2016-09-14 12:50:01 |
Maksim Malchuk |
fuel: assignee |
Maksim Malchuk (mmalchuk) |
Dmitry Kaigarodеsev (dkaiharodsev) |
|
| 2016-09-14 14:23:34 |
Roman Vyalov |
fuel: assignee |
Dmitry Kaigarodеsev (dkaiharodsev) |
Maksim Malchuk (mmalchuk) |
|
| 2016-09-14 14:23:37 |
Roman Vyalov |
fuel: status |
Confirmed |
New |
|
| 2016-09-14 14:39:56 |
OpenStack Infra |
fuel: status |
New |
In Progress |
|
| 2016-09-19 15:56:37 |
Maksim Malchuk |
fuel/mitaka: status |
Invalid |
New |
|
| 2016-09-19 15:56:48 |
Maksim Malchuk |
fuel/mitaka: assignee |
Dmitry Kaigarodеsev (dkaiharodsev) |
Fuel CI (fuel-ci) |
|
| 2016-09-20 11:10:36 |
Roman Vyalov |
fuel/mitaka: assignee |
Fuel CI (fuel-ci) |
|
|
| 2016-09-20 11:27:26 |
Maksim Malchuk |
fuel/mitaka: status |
New |
Incomplete |
|
| 2016-09-21 14:15:45 |
Michael Polenchuk |
bug |
|
|
added subscriber Michael Polenchuk |
| 2016-09-27 07:21:10 |
Nikita Karpin |
bug |
|
|
added subscriber Nikita Karpin |
| 2016-09-27 08:15:00 |
Alexander Arzhanov |
nominated for series |
|
fuel/newton |
|
| 2016-09-27 08:15:00 |
Alexander Arzhanov |
bug task added |
|
fuel/newton |
|
| 2016-09-27 08:16:48 |
Maksim Malchuk |
summary |
[gates] Cluster is not deployed: some nodes are in the Error state |
[gates] OOM failures on CI |
|
| 2016-09-27 08:35:09 |
Maksim Malchuk |
bug task deleted |
fuel/newton |
|
|
| 2016-10-18 14:43:47 |
Roman Vyalov |
fuel/mitaka: milestone |
9.1 |
9.2 |
|
| 2016-10-20 10:21:12 |
Maksim Malchuk |
fuel: assignee |
Maksim Malchuk (mmalchuk) |
Fuel Sustaining (fuel-sustaining-team) |
|
| 2016-10-20 10:21:21 |
Maksim Malchuk |
fuel: status |
In Progress |
Confirmed |
|
| 2016-11-07 13:10:40 |
Alexander Kurenyshev |
fuel/mitaka: assignee |
|
Fuel Sustaining (fuel-sustaining-team) |
|
| 2016-11-07 13:10:45 |
Alexander Kurenyshev |
fuel/mitaka: status |
Incomplete |
Confirmed |
|
| 2016-12-12 14:53:04 |
Alexey Shtokolov |
fuel: status |
Confirmed |
Fix Committed |
|
| 2016-12-12 14:53:08 |
Alexey Shtokolov |
fuel/mitaka: status |
Confirmed |
Fix Committed |
|
