Ubuntu
python-eventlet package

glance-registry process spins if rsyslog restarted with syslog logging enabled

Bug #1452312 reported by James Page on 2015-05-06

This bug affects 9 people

	Status	Importance	Assigned to
glance (Ubuntu)
Trusty	Invalid	Undecided	Unassigned
Utopic	Invalid	Undecided	Unassigned
Vivid	Invalid	Undecided	Unassigned
Wily	Invalid	Undecided	Unassigned
python-eventlet (Ubuntu)	Fix Released	High	Unassigned
Trusty	Fix Released	High	Unassigned
Utopic	Fix Released	High	Unassigned
Vivid	Fix Released	High	Unassigned
Wily	Fix Released	High	Unassigned

Bug Description

[Impact]
Glance, when configured to log to syslog, will spin consuming 100% cpu if syslog is restarted, locking all function.

[Test Case]
Install OpenStack Glance as part of a OpenStack Cloud
Configure to log to syslog
Restart syslog (sudo service rsyslog restart)

[Regression Potential]
Minimal - disconnected sockets are added to the list of types to re-raise exceptions for.

[Original Bug Report]
When configured to log to syslog, if the rsyslog process is restarted, then the glance-registry process goes into a spin, consuming 100% of CPU - strace:

poll([{fd=3, events=POLLOUT|POLLERR|POLLHUP}, {fd=5, events=POLLIN|POLLPRI|POLLERR|POLLHUP}], 2, 60000) = 1 ([{fd=3, revents=POLLOUT}])
sendto(3, "<14>2015-05-06 14:58:16.186 2579"..., 112, 0, NULL, 0) = -1 ENOTCONN (Transport endpoint is not connected)
gettimeofday({1430924440, 430244}, NULL) = 0
poll([{fd=3, events=POLLOUT|POLLERR|POLLHUP}, {fd=5, events=POLLIN|POLLPRI|POLLERR|POLLHUP}], 2, 60000) = 1 ([{fd=3, revents=POLLOUT}])
sendto(3, "<14>2015-05-06 14:58:16.186 2579"..., 112, 0, NULL, 0) = -1 ENOTCONN (Transport endpoint is not connected)

This maybe related to bug 1081022 but that should have only impacted 12.04 deployments.

ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: glance-registry 1:2014.1.4-0ubuntu2
ProcVersionSignature: User Name 3.13.0-51.84-generic 3.13.11-ckt18
Uname: Linux 3.13.0-51-generic x86_64
ApportVersion: 2.14.1-0ubuntu3.10
Architecture: amd64
Date: Wed May 6 15:00:00 2015
Ec2AMI: ami-00000350
Ec2AMIManifest: FIXME
Ec2AvailabilityZone: nova
Ec2InstanceType: m1.small.osci
Ec2Kernel: aki-00000002
Ec2Ramdisk: ari-00000002
PackageArchitecture: all
SourcePackage: glance
UpgradeStatus: No upgrade log present (probably fresh install)
modified.conffile..etc.glance.glance.registry.conf: [deleted]
modified.conffile..etc.glance.glance.registry.paste.ini: [deleted]

See original description

Tags:

Related branches

lp://qastaging/ubuntu/wily-proposed/python-eventlet

lp://qastaging/ubuntu/vivid-proposed/python-eventlet

lp://qastaging/ubuntu/trusty-proposed/python-eventlet

lp://qastaging/ubuntu/utopic-proposed/python-eventlet

lp://qastaging/ubuntu/wily/python-eventlet

Revision history for this message

James Page (james-page) wrote on 2015-05-06:

Dependencies.txt Edit (5.4 KiB, text/plain; charset="utf-8")
ProcEnviron.txt Edit (315 bytes, text/plain; charset="utf-8")

Revision history for this message

James Page (james-page) wrote on 2015-05-06:

Using the reproducer on bug 1081022 would indicate that this is something different, as the installation does not exhibit quite the same issue.

Revision history for this message

James Page (james-page) wrote on 2015-05-06:

Raising a task for eventlet as this appears to be related to https://gist.github.com/temoto/8259860

Revision history for this message

Launchpad Janitor (janitor) wrote on 2015-05-06:

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in glance (Ubuntu):
status:	New → Confirmed
Changed in python-eventlet (Ubuntu):
status:	New → Confirmed

Revision history for this message

James Page (james-page) wrote on 2015-05-06:

I'm reticent to pull in the patch detail in #3 as this is not upstream - the codebase in the master branch is fundamentally the same as that found in eventlet in 14.04.

Changed in python-eventlet (Ubuntu):
importance:	Undecided → High
Changed in glance (Ubuntu):
importance:	Undecided → High
no longer affects:	glance (Ubuntu)
Changed in glance (Ubuntu Trusty):
status:	New → Invalid
Changed in glance (Ubuntu Utopic):
status:	New → Invalid
Changed in glance (Ubuntu Vivid):
status:	New → Invalid
Changed in glance (Ubuntu Wily):
status:	Confirmed → Invalid
Changed in python-eventlet (Ubuntu Trusty):
status:	New → Confirmed
importance:	Undecided → High
Changed in python-eventlet (Ubuntu Wily):
status:	Confirmed → New
Changed in python-eventlet (Ubuntu Vivid):
importance:	Undecided → High
Changed in python-eventlet (Ubuntu Utopic):
importance:	Undecided → High
Changed in glance (Ubuntu Wily):
importance:	High → Undecided

Revision history for this message

James Page (james-page) wrote on 2015-05-06:

Also confirmed with kilo - using eventlet 0.16.0

Changed in python-eventlet (Ubuntu Vivid):
status:	New → Confirmed
Changed in python-eventlet (Ubuntu Wily):
status:	New → Confirmed

Revision history for this message

James Page (james-page) wrote on 2015-05-07:

https://github.com/eventlet/eventlet/issues/192

Revision history for this message

James Page (james-page) wrote on 2015-05-26:

Potential fix:

https://github.com/eventlet/eventlet/commit/ed0655333007fe97624b745e92deb407d192022c

James Page (james-page) on 2015-05-26

Changed in python-eventlet (Ubuntu Utopic):
status:	New → Confirmed

James Page (james-page) on 2015-05-26

description:

updated

Revision history for this message

Chris J Arges (arges) wrote on 2015-05-27: Please test proposed package

#10

Hello James, or anyone else affected,

Accepted python-eventlet into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-eventlet/0.13.0-1ubuntu2.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in python-eventlet (Ubuntu Trusty):
status:	Confirmed → Fix Committed
tags:	added: verification-needed
Changed in python-eventlet (Ubuntu Utopic):
status:	Confirmed → Fix Committed

Revision history for this message

Chris J Arges (arges) wrote on 2015-05-27:

#11

Hello James, or anyone else affected,

Accepted python-eventlet into utopic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-eventlet/0.13.0-1ubuntu3.3 in a few hours, and then in the -proposed repository.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in python-eventlet (Ubuntu Vivid):
status:	Confirmed → Fix Committed

Revision history for this message

Chris J Arges (arges) wrote on 2015-05-27:

#12

Hello James, or anyone else affected,

Accepted python-eventlet into vivid-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-eventlet/0.16.1-0ubuntu1.1 in a few hours, and then in the -proposed repository.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Revision history for this message

Stuart McLaren (stuart-mclaren) wrote on 2015-05-29:

#13

Note that this behaviour was originally reported here:

https://bugs.launchpad.net/oslo.log/+bug/1076466

Revision history for this message

Launchpad Janitor (janitor) wrote on 2015-05-30:

#14

This bug was fixed in the package python-eventlet - 0.17.3-2ubuntu2

---------------
python-eventlet (0.17.3-2ubuntu2) wily; urgency=medium

  * d/p/handle-ENOTCONN.patch: Ensure that ENOTCONN socket errors are
    raised correctly, resolving issues with disconnected syslog sockets
    in OpenStack Glance (LP: #1452312).

-- James Page <email address hidden> Tue, 26 May 2015 16:45:17 +0100

Changed in python-eventlet (Ubuntu Wily):
status:	Confirmed → Fix Released

Revision history for this message

George Shuklin (george-shuklin) wrote on 2015-06-01:

#15

Hello.

I can't see update nor in trusty-updates nor in trusty-updates/juno/main. It still contains 0.13.0-1ubuntu3.2~cloud0 instead of 0.13.0-1ubuntu3.3

Revision history for this message

Edward Hope-Morley (hopem) wrote on 2015-06-09:

#16

@george-shuklin the package will remain in trusty-proposed until it has been verified. If you want to help test this package please enable the proposed pocket in your /etc/apt/sources.list, run apt-get update and install the package. Thanks!

Revision history for this message

Seyeong Kim (seyeongkim) wrote on 2015-06-15:

#17

0. set up keystone, glance
1. stop rsyslogd while glance-api, registry is running
2. run comamnd "glance image-list"
3. CPU usage is getting high

trusty
ii python-eventlet 0.13.0-1ubuntu2.2 all concurrent networking library for Python

utopic
ii python-eventlet 0.13.0-1ubuntu3.3 all concurrent networking library for Python

vivid
ii python-eventlet 0.16.1-0ubuntu1.1 all concurrent networking library - Python 2.x

tags:

added: verification-done
removed: verification-needed

Revision history for this message

George Shuklin (george-shuklin) wrote on 2015-06-15:

#18

@hopem, We've running on this (proposed) code for few weeks already, works fine.

Revision history for this message

Launchpad Janitor (janitor) wrote on 2015-06-17:

#19

This bug was fixed in the package python-eventlet - 0.13.0-1ubuntu2.2

---------------
python-eventlet (0.13.0-1ubuntu2.2) trusty; urgency=medium

  * d/p/handle-ENOTCONN.patch: Ensure that ENOTCONN socket errors are
    raised correctly, resolving issues with disconnected syslog sockets
    in OpenStack Glance (LP: #1452312).

-- James Page <email address hidden> Tue, 26 May 2015 17:22:24 +0100

Changed in python-eventlet (Ubuntu Trusty):
status:	Fix Committed → Fix Released

Revision history for this message

Chris J Arges (arges) wrote on 2015-06-17: Update Released

#20

The verification of the Stable Release Update for python-eventlet has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message

Launchpad Janitor (janitor) wrote on 2015-06-17:

#21

This bug was fixed in the package python-eventlet - 0.13.0-1ubuntu3.3

---------------
python-eventlet (0.13.0-1ubuntu3.3) utopic; urgency=medium

  * d/p/handle-ENOTCONN.patch: Ensure that ENOTCONN socket errors are
    raised correctly, resolving issues with disconnected syslog sockets
    in OpenStack Glance (LP: #1452312).

-- James Page <email address hidden> Tue, 26 May 2015 17:20:06 +0100

Changed in python-eventlet (Ubuntu Utopic):
status:	Fix Committed → Fix Released

Revision history for this message

Launchpad Janitor (janitor) wrote on 2015-06-17:

#22

This bug was fixed in the package python-eventlet - 0.16.1-0ubuntu1.1

---------------
python-eventlet (0.16.1-0ubuntu1.1) vivid; urgency=medium

  * d/p/handle-ENOTCONN.patch: Ensure that ENOTCONN socket errors are
    raised correctly, resolving issues with disconnected syslog sockets
    in OpenStack Glance (LP: #1452312).

-- James Page <email address hidden> Tue, 26 May 2015 17:28:40 +0100

Changed in python-eventlet (Ubuntu Vivid):
status:	Fix Committed → Fix Released

Report a bug

This report contains Public information

Everyone can see this information.

Duplicates of this bug

You are

Subscribing...

Edit bug mail

Other bug subscribers

Bug attachments

Add attachment

Remote bug watches

Bug watches keep track of this bug in other bug trackers.

Ubuntupython-eventlet package

glance-registry process spins if rsyslog restarted with syslog logging enabled

Bug Description

Related branches

Duplicates of this bug

Other bug subscribers

Bug attachments

Remote bug watches

Ubuntu
python-eventlet package