Mistral

DB disconnect errors can leave executions running forever

Bug #1751018 reported by Andras Kovi on 2018-02-22

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	Mistral	Fix Released	Medium	Unassigned	Mistral rocky-1 "r1"

Bug Description

If is DB disconnected during the engine.workflow_handler._check_and_complete operation, the call is tried to be rescheduled in a finally block. This overwrites the original disconnection error to something similar to the following:

Delayed call failed, method: <function _check_and_complete at 0x62f3140>, exception: (sqlalchemy.exc.InvalidRequestError) Can't reconnect until invalid transaction is rolled back [SQL: u'INSERT INTO delayed_calls_v2

As a result, the retry mechanism is not triggered and the scheduled call is lost forever.

Renat Akhmerov (rakhmerov) on 2018-02-26

Changed in mistral:
milestone:	none → rocky-1

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-03-21: Fix merged to mistral (master)

Reviewed: https://review.openstack.org/547069
Committed: https://git.openstack.org/cgit/openstack/mistral/commit/?id=a59af3f24729e3bca9ad7125ffd5090cb7d1b5ba
Submitter: Zuul
Branch: master

commit a59af3f24729e3bca9ad7125ffd5090cb7d1b5ba
Author: Andras Kovi <email address hidden>
Date: Thu Feb 22 17:42:13 2018 +0100

Ensure workflow check is retried on DB error

    On DB error, the _check_and_complete logic is retried. DB disconnect errors
    are not shadowed by subsequent DB operations in the failed transaction. Other
    non DB disconnect or MistralExceptions are considered fatal, thus, the consistency
    check is not rescheduled.

Change-Id: I093d5e2442a5489d406f4b8a188552f201b1a076
Closes-Bug: 1751018