Mirantis OpenStack

API services consume CPU on idle env

Series 10.0.x
Bug #1516557

Bug #1516557 reported by Ilya Shakhat on 2015-11-16

This bug affects 1 person

	Status	Importance	Assigned to	Milestone
Mirantis OpenStack	Status tracked in 10.0.x
10.0.x	Fix Committed	Medium	MOS Oslo	Mirantis OpenStack 10.0
8.0.x	Won't Fix	Medium	MOS Oslo	Mirantis OpenStack 8.0
9.x	Fix Released	Medium	MOS Oslo	Mirantis OpenStack 9.2

Bug Description

On the idle fresh env several services consume CPU, in average 1,5% per each. The issue is introduced in 8.0 and not observable in 7.0. Affected services: nova-api, cinder-api, glance-api, swift-api (neutron is not).

strace shows the following loop:
select(0, NULL, NULL, NULL, {0, 9714}) = 0 (Timeout)
gettimeofday({1447668681, 29496}, NULL) = 0
wait4(0, 0x7ffe4ca6d890, WNOHANG, NULL) = 0
gettimeofday({1447668681, 29639}, NULL) = 0
gettimeofday({1447668681, 29711}, NULL) = 0

On MOS 7 the loop was different from project to project and didn't result in visible CPU consumption. In all cases it didn't contain calls to gettimeofday.

See original description

Tags:

Revision history for this message

Ilya Shakhat (shakhat) wrote on 2015-11-16:

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  openstack_version: "2015.1.0-8.0"
  api: "1.0"
  build_number: "155"
  build_id: "155"
  fuel-nailgun_sha: "37a535e11a7939e206ffababf3ecf12504cd91c4"
  python-fuelclient_sha: "e685d68c1c0d0fa0491a250f07d9c3a8d0f9608c"
  fuel-agent_sha: "07560a9fc3ce5301ace04d2d3e5d68db6ee4f8d5"
  fuel-nailgun-agent_sha: "3e9d17211d65c80bf97c8d83979979f6c7feb687"
  astute_sha: "959b06c5ef8143125efd1727d350c050a922eb12"
  fuel-library_sha: "1e690ed95452297294c710a2f5886ef671d6b6da"
  fuel-ostf_sha: "9690a2de829d3b063ed1e64b0b10dde39f711dc0"
  fuel-createmirror_sha: "a034dcb06520df58a7338816900a431a6b61d83f"
  fuelmenu_sha: "8a32c53c1fa13b036000f589f96e876277dbd071"
  shotgun_sha: "25dd78a3118267e3616df0727ce746e7dead2d67"
  network-checker_sha: "a57e1d69acb5e765eb22cab0251c589cd76f51da"
  fuel-upgrade_sha: "1e894e26d4e1423a9b0d66abd6a79505f4175ff6"
  fuelmain_sha: "45b79c9121a08c2b467a3246dc1fa714e4c2043d"

DEPLOYMENT:
  1 controller + 1 compute on KVM
  Neutron + VLAN
  no additional services

Revision history for this message

Ilya Shakhat (shakhat) wrote on 2015-11-16:

Steps to repro:
1. Deploy new env and leave alone on several days (e.g. over weekend)
2. Check services via "top -b -n 1 -o TIME | head -n 30" Affected should be running for 1,5% of total uptime

description:	updated
Changed in mos:
milestone:	none → 8.0

Ilya Shakhat (shakhat) on 2015-11-16

Changed in mos:
assignee:	nobody → MOS Nova (mos-nova)

Roman Podoliaka (rpodolyaka) on 2015-11-17

Changed in mos:
importance:	Undecided → Medium
status:	New → Confirmed

Revision history for this message

Roman Podoliaka (rpodolyaka) wrote on 2015-11-17:

Unfortunately, this was fixed in oslo.service, but caused https://bugs.launchpad.net/oslo-incubator/+bug/1446583, so it was reverted.

Changed in mos:
assignee:	MOS Nova (mos-nova) → MOS Oslo (mos-oslo)

Revision history for this message

Roman Podoliaka (rpodolyaka) wrote on 2015-11-17:

Let's ask Oslo guys to take another look, but it's mostly annoying rather than causing troubles.

Revision history for this message

Dmitry Mescheryakov (dmitrymex) wrote on 2015-12-29:

Moving to 9.0 since we've passed SCF

Timur Nurlygayanov (tnurlygayanov) on 2016-02-10

Changed in mos:
status:	Confirmed → Won't Fix

Revision history for this message

Dmitry Mescheryakov (dmitrymex) wrote on 2016-04-04:

We will not fix this issue until 9.0 SCF, hence bumping target milestone

Dmitry Mescheryakov (dmitrymex) on 2016-07-11

tags:

added: 10.0-reviewed

Revision history for this message

Dmitry Mescheryakov (dmitrymex) wrote on 2016-12-16:

I am pretty sure that it is the same bug as https://bugs.launchpad.net/mos/+bug/1380220

Revision history for this message

Dmitry Mescheryakov (dmitrymex) wrote on 2016-12-16:

The issue should be fixed in 9.2 after sync of stable/mitaka https://review.fuel-infra.org/#/c/29072/ which got us https://review.openstack.org/#/c/386656/ into 9.2. Correspondingly the issue is fixed in 10.0 as the fix is Newton already.

Dmitry (dtsapikov) on 2017-01-25

tags:

added: on-verification

Revision history for this message

Dmitry (dtsapikov) wrote on 2017-02-03:

Verified on 9.2

tags:

removed: on-verification

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.