instances do not stop correctly
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
nova (Ubuntu) |
Fix Released
|
High
|
Unassigned |
Bug Description
The instances do not stop correctly. An error is triggered and the instances are not removed from the DB and are still shown as ACTIVE (even if the KVM processes and the network configuration are deleted correctly ).
* Package used: nova 2011.3-0ubuntu5 (but the problem is probably there since 2011.3-0ubuntu4)
* Commands: nova delete or euca-terminate-
* Error:
(nova.exception): TRACE: File "/usr/lib/
(nova.exception): TRACE: os.close(self.fd)
(nova.exception): TRACE: OSError: [Errno 9] Bad file descriptor
The error happens in:
File: /usr/lib/
Class: ConsoleLogger(
It looks like this error happens since the modifications made to have a ring/fifo console log for KVM:
nova (2011.3-0ubuntu5) oneiric; urgency=low
* debian/
- Patch updated to use correct patchset from upstream - incorrect version
was uploaded in -0ubuntu4 (LP: #832507).
nova (2011.3-0ubuntu4) oneiric; urgency=low
* debian/
Move console.log to a ringbuffer so that the console.log
keeps filling up. (LP: #832507)
If I install Nova from the OpenStack PPA trunk (ppa:nova-
Complete log:
2011-10-05 14:29:24,767 INFO nova.compute.
2011-10-05 14:29:24,768 DEBUG nova.compute.
2011-10-05 14:29:24,843 INFO nova.compute.
2011-10-05 14:29:24,843 INFO nova.compute.
2011-10-05 14:29:24,843 INFO nova.compute.
2011-10-05 14:29:24,897 AUDIT nova.compute.
2011-10-05 14:29:24,897 DEBUG nova.rpc [-] Making asynchronous call on network ... from (pid=912) multicall /usr/lib/
2011-10-05 14:29:24,897 DEBUG nova.rpc [-] MSG_ID is 5e8492b8c59e48c
2011-10-05 14:29:25,044 DEBUG nova.rpc [-] Making asynchronous cast on network... from (pid=912) cast /usr/lib/
2011-10-05 14:29:26,701 DEBUG nova.utils [-] Attempting to grab semaphore "iptables" for method "apply"... from (pid=912) inner /usr/lib/
2011-10-05 14:29:26,701 DEBUG nova.utils [-] Attempting to grab file lock "iptables" for method "apply"... from (pid=912) inner /usr/lib/
2011-10-05 14:29:26,702 DEBUG nova.utils [-] Running cmd (subprocess): sudo iptables-save -t filter from (pid=912) execute /usr/lib/
2011-10-05 14:29:27,397 INFO nova.virt.
2011-10-05 14:29:27,416 DEBUG nova.utils [-] Running cmd (subprocess): sudo iptables-restore from (pid=912) execute /usr/lib/
2011-10-05 14:29:27,442 DEBUG nova.utils [-] Running cmd (subprocess): sudo iptables-save -t nat from (pid=912) execute /usr/lib/
2011-10-05 14:29:27,472 DEBUG nova.utils [-] Running cmd (subprocess): sudo iptables-restore from (pid=912) execute /usr/lib/
2011-10-05 14:29:28,627 DEBUG nova.virt.
2011-10-05 14:29:28,631 ERROR nova.exception [-] Uncaught exception
(nova.exception): TRACE: Traceback (most recent call last):
(nova.exception): TRACE: File "/usr/lib/
(nova.exception): TRACE: return f(*args, **kw)
(nova.exception): TRACE: File "/usr/lib/
(nova.exception): TRACE: function(self, context, instance_id, *args, **kwargs)
(nova.exception): TRACE: File "/usr/lib/
(nova.exception): TRACE: self._shutdown_
(nova.exception): TRACE: File "/usr/lib/
(nova.exception): TRACE: self.driver.
(nova.exception): TRACE: File "/usr/lib/
(nova.exception): TRACE: self._cleanup(
(nova.exception): TRACE: File "/usr/lib/
(nova.exception): TRACE: self._stop_
(nova.exception): TRACE: File "/usr/lib/
(nova.exception): TRACE: self.console_
(nova.exception): TRACE: File "/usr/lib/
(nova.exception): TRACE: os.close(self.fd)
(nova.exception): TRACE: OSError: [Errno 9] Bad file descriptor
(nova.exception): TRACE:
2011-10-05 14:29:28,638 ERROR nova.rpc [-] Exception during message handling
(nova.rpc): TRACE: Traceback (most recent call last):
(nova.rpc): TRACE: File "/usr/lib/
(nova.rpc): TRACE: rval = node_func(
(nova.rpc): TRACE: File "/usr/lib/
(nova.rpc): TRACE: raise Error(str(e))
(nova.rpc): TRACE: Error: [Errno 9] Bad file descriptor
(nova.rpc): TRACE:
2011-10-05 14:30:22,457 INFO nova.compute.
2011-10-05 14:30:23,056 INFO nova.compute.
2011-10-05 14:31:23,742 INFO nova.compute.
Related branches
- James Page: Approve
-
Diff: 250 lines (+49/-46)2 files modifieddebian/changelog (+7/-0)
debian/patches/backport-libvirt-console-pipe.patch (+42/-46)
tags: | added: server-o-rs |
Changed in nova (Ubuntu): | |
importance: | Undecided → High |
Changed in nova (Ubuntu): | |
status: | Confirmed → Fix Committed |
Eric,
Thanks for reporting this.
I haven't been able to reproduce this in my test environment. I suspect a race, I think I know where it is and I've tried writing a few test cases but haven't been able to catch this.
Can you tell me if you are able to reliably reproduce this every time, or at all?