Ironic

Missing dependencies for third-party driver causes exception traceback

Bug #1298529 reported by aeva black on 2014-03-27

This bug affects 1 person

	Status	Importance	Assigned to	Milestone
Ironic	Fix Released	High	aeva black	Ironic 2014.2 "juno"
oslo-incubator	Fix Released	High	Doug Hellmann	oslo-incubator 2014.1 "icehouse"
stevedore	Fix Released	High	Doug Hellmann

Bug Description

Some operators (and the tools they employ) may consider an exception traceback in a service's log to be an indication of a serious problem.

When ironic's drivers are unable to be loaded (eg, due to a missing third-party library) they raise an exception to signal to stevedore that the plugin should not be loaded at all (loading a non-functional driver would cause the ironic-conductor service to advertise support for that driver). Here is an example of that traceback:

2014-03-27 06:56:56.132 13833 ERROR stevedore.extension [-] Could not load 'pxe_seamicro': PXEAndSeaMicroDriver
2014-03-27 06:56:56.133 13833 ERROR stevedore.extension [-] PXEAndSeaMicroDriver
2014-03-27 06:56:56.133 13833 TRACE stevedore.extension Traceback (most recent call last):
2014-03-27 06:56:56.133 13833 TRACE stevedore.extension File "/opt/stack/new/stevedore/stevedore/extension.py", line 162, in _load_plugins
2014-03-27 06:56:56.133 13833 TRACE stevedore.extension verify_requirements,
2014-03-27 06:56:56.133 13833 TRACE stevedore.extension File "/opt/stack/new/stevedore/stevedore/enabled.py", line 66, in _load_one_plugin
2014-03-27 06:56:56.133 13833 TRACE stevedore.extension verify_requirements,
2014-03-27 06:56:56.133 13833 TRACE stevedore.extension File "/opt/stack/new/stevedore/stevedore/extension.py", line 179, in _load_one_plugin
2014-03-27 06:56:56.133 13833 TRACE stevedore.extension obj = plugin(*invoke_args, **invoke_kwds)
2014-03-27 06:56:56.133 13833 TRACE stevedore.extension File "/opt/stack/new/ironic/ironic/drivers/pxe.py", line 102, in __init__
2014-03-27 06:56:56.133 13833 TRACE stevedore.extension raise exception.DriverNotFound('PXEAndSeaMicroDriver')
2014-03-27 06:56:56.133 13833 TRACE stevedore.extension DriverNotFound: PXEAndSeaMicroDriver

This should be logged as a WARNING, not an ERROR, but currently, stevedore provides no mechanism for capturing these exceptions and selectively choosing what to log.

Revision history for this message

aeva black (tenbrae) wrote on 2014-03-27:

Fixed in stevedore:
https://review.openstack.org/#/c/83509/

Changed in ironic:
status:	New → Triaged
importance:	Undecided → High
milestone:	none → icehouse-rc1

Doug Hellmann (doug-hellmann) on 2014-03-27

Changed in python-stevedore:
status:	New → In Progress
importance:	Undecided → High
assignee:	nobody → Doug Hellmann (doug-hellmann)
Changed in oslo:
status:	New → In Progress
importance:	Undecided → High
assignee:	nobody → Doug Hellmann (doug-hellmann)
milestone:	none → icehouse-rc1

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-03-27: Fix proposed to ironic (master)

Fix proposed to branch: master
Review: https://review.openstack.org/83572

Changed in ironic:
assignee:	nobody → Devananda van der Veen (devananda)
status:	Triaged → In Progress

Revision history for this message

Doug Hellmann (doug-hellmann) wrote on 2014-03-27:

stevedore 0.15

Changed in python-stevedore:
status:	In Progress → Fix Released
Changed in oslo:
status:	In Progress → Fix Released

Revision history for this message

aeva black (tenbrae) wrote on 2014-03-31:

After further discussion and getting feedback on the patch I proposed to Ironic, it's fairly clear that the right fix for this is going to need a new CONF option, which I hesitate to introduce this late in getting the RC ready.

I'm going to untarget this, but will land a more comprehensive fix as soon as Juno opens.

Changed in ironic:
milestone:	icehouse-rc1 → juno-1

Thierry Carrez (ttx) on 2014-04-17

Changed in oslo:
milestone:	icehouse-rc1 → 2014.1

Revision history for this message

Openstack Gerrit (openstack-gerrit) wrote on 2014-04-17: Fix merged to ironic (master)

Reviewed: https://review.openstack.org/83572
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=c023f3394d4b2ffd92849fa0eed8d1dfbbb1165b
Submitter: Jenkins
Branch: master

commit c023f3394d4b2ffd92849fa0eed8d1dfbbb1165b
Author: Devananda van der Veen <email address hidden>
Date: Thu Mar 27 13:13:18 2014 -0700

Better handling of missing drivers

    Instead of loading all available drivers, this patch adds add a CONF
    option for "enabled_drivers" which explicitly sets the expected drivers.
    A failure to load an enabled driver will prevent the conductor service
    from starting.

    Use a callback hook in stevedore to capture and clean up import exceptions.
    Third-party drivers are expected to use importutils to attempt to load
    their external requirements, and when unable to, they should raise a
    DriverNotFound error to tie into this hook.

    Sets the default enabled drivers to:
    - pxe_ssh
    - pxe_ipmitool

    tests/api/v1/test_ports was creating a ConductorManager instance
    instead of appropriately mocking the conductor.rpcapi, which led to test
    failures when changing how ConductorManager loads drivers. This patch
    corrects this by properly mocking the rpcapi within that test class.

    Change-Id: I2da37409cea98731a2ca18dfe7298861fde37783
    Closes-bug: #1298529
    Closes-bug: #1302562

Changed in ironic:
status:	In Progress → Fix Committed

Thierry Carrez (ttx) on 2014-06-12

Changed in ironic:
status:	Fix Committed → Fix Released

Thierry Carrez (ttx) on 2014-10-16

Changed in ironic:
milestone:	juno-1 → 2014.2

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.