DB disconnect errors can leave executions running forever

Bug #1751018 reported by Andras Kovi
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fix Released

Bug Description

If is DB disconnected during the engine.workflow_handler._check_and_complete operation, the call is tried to be rescheduled in a finally block. This overwrites the original disconnection error to something similar to the following:

  Delayed call failed, method: <function _check_and_complete at 0x62f3140>, exception: (sqlalchemy.exc.InvalidRequestError) Can't reconnect until invalid transaction is rolled back [SQL: u'INSERT INTO delayed_calls_v2

As a result, the retry mechanism is not triggered and the scheduled call is lost forever.

Changed in mistral:
milestone: none → rocky-1
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to mistral (master)

Reviewed: https://review.openstack.org/547069
Committed: https://git.openstack.org/cgit/openstack/mistral/commit/?id=a59af3f24729e3bca9ad7125ffd5090cb7d1b5ba
Submitter: Zuul
Branch: master

commit a59af3f24729e3bca9ad7125ffd5090cb7d1b5ba
Author: Andras Kovi <email address hidden>
Date: Thu Feb 22 17:42:13 2018 +0100

    Ensure workflow check is retried on DB error

    On DB error, the _check_and_complete logic is retried. DB disconnect errors
    are not shadowed by subsequent DB operations in the failed transaction. Other
    non DB disconnect or MistralExceptions are considered fatal, thus, the consistency
    check is not rescheduled.

    Change-Id: I093d5e2442a5489d406f4b8a188552f201b1a076
    Closes-Bug: 1751018

Changed in mistral:
status: New → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/mistral

This issue was fixed in the openstack/mistral development milestone.

Dougal Matthews (d0ugal)
Changed in mistral:
importance: Undecided → Medium
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.