Mistral

mistral can create multiple action executions due to bad async action

Bug #1743662 reported by Matt B
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Mistral
Fix Released
High
Vitalii Solodilov
Mistral rocky-1 "r1"

Bug Description

This is really the fault of the async action on the other end, but it does seem like mistral should handle it better. If a task has a retry configuration, and you repeatedly update a single action execution's state to ERROR, the task will generate a new action execution each time. I tested this on mistral 5.2 using the provided docker image.

root@f316915139b2:~# mistral workflow-create repeated_retry.yml
+--------------------------------------+----------------+-------------------+--------+-------+---------------------+------------+
| ID | Name | Project ID | Tags | Input | Created at | Updated at |
+--------------------------------------+----------------+-------------------+--------+-------+---------------------+------------+
| 80fd8f0c-a245-4f78-9053-02a8fb9cef66 | repeated_retry | <default-project> | <none> | | 2018-01-16 20:39:35 | None |
+--------------------------------------+----------------+-------------------+--------+-------+---------------------+------------+

root@f316915139b2:~# mistral execution-create repeated_retry
+-------------------+--------------------------------------+
| Field | Value |
+-------------------+--------------------------------------+
| ID | c542cd24-e072-4c51-a00e-6bb66d8cbdf9 |
| Workflow ID | 80fd8f0c-a245-4f78-9053-02a8fb9cef66 |
| Workflow name | repeated_retry |
| Description | |
| Task Execution ID | <none> |
| State | RUNNING |
| State info | None |
| Created at | 2018-01-16 20:39:51 |
| Updated at | 2018-01-16 20:39:51 |
+-------------------+--------------------------------------+

root@f316915139b2:~# mistral task-list c542cd24-e072-4c51-a00e-6bb66d8cbdf9
+--------------------------------------+------------+----------------+--------------------------------------+---------+------------+---------------------+------------+
| ID | Name | Workflow name | Execution ID | State | State info | Created at | Updated at |
+--------------------------------------+------------+----------------+--------------------------------------+---------+------------+---------------------+------------+
| 531c799a-23cb-453a-8b0e-44558ce2980d | async_http | repeated_retry | c542cd24-e072-4c51-a00e-6bb66d8cbdf9 | RUNNING | None | 2018-01-16 20:39:51 | <none> |
+--------------------------------------+------------+----------------+--------------------------------------+---------+------------+---------------------+------------+

root@f316915139b2:~# mistral action-execution-list 531c799a-23cb-453a-8b0e-44558ce2980d
+--------------------------------------+------------------+----------------+------------+--------------------------------------+---------+----------+---------------------+------------+
| ID | Name | Workflow name | Task name | Task ID | State | Accepted | Created at | Updated at |
+--------------------------------------+------------------+----------------+------------+--------------------------------------+---------+----------+---------------------+------------+
| acc19f4f-af85-433c-964f-72cbe5e62201 | std.mistral_http | repeated_retry | async_http | 531c799a-23cb-453a-8b0e-44558ce2980d | RUNNING | False | 2018-01-16 20:39:51 | <none> |
+--------------------------------------+------------------+----------------+------------+--------------------------------------+---------+----------+---------------------+------------+

root@f316915139b2:~# mistral action-execution-update acc19f4f-af85-433c-964f-72cbe5e62201 --state=ERROR
+---------------+--------------------------------------+
| Field | Value |
+---------------+--------------------------------------+
| ID | acc19f4f-af85-433c-964f-72cbe5e62201 |
| Name | std.mistral_http |
| Workflow name | repeated_retry |
| Task name | None |
| Task ID | 531c799a-23cb-453a-8b0e-44558ce2980d |
| State | ERROR |
| State info | None |
| Accepted | False |
| Created at | 2018-01-16 20:39:51 |
| Updated at | 2018-01-16 20:41:15 |
+---------------+--------------------------------------+

root@f316915139b2:~# mistral action-execution-list 531c799a-23cb-453a-8b0e-44558ce2980d
+--------------------------------------+------------------+----------------+------------+--------------------------------------+---------+----------+---------------------+---------------------+
| ID | Name | Workflow name | Task name | Task ID | State | Accepted | Created at | Updated at |
+--------------------------------------+------------------+----------------+------------+--------------------------------------+---------+----------+---------------------+---------------------+
| acc19f4f-af85-433c-964f-72cbe5e62201 | std.mistral_http | repeated_retry | async_http | 531c799a-23cb-453a-8b0e-44558ce2980d | ERROR | False | 2018-01-16 20:39:51 | 2018-01-16 20:41:15 |
| 153c09d5-cc44-4eee-9d29-470927e992c3 | std.mistral_http | repeated_retry | async_http | 531c799a-23cb-453a-8b0e-44558ce2980d | RUNNING | False | 2018-01-16 20:41:25 | <none> |
+--------------------------------------+------------------+----------------+------------+--------------------------------------+---------+----------+---------------------+---------------------+

root@f316915139b2:~# mistral action-execution-update acc19f4f-af85-433c-964f-72cbe5e62201 --state=ERROR
+---------------+--------------------------------------+
| Field | Value |
+---------------+--------------------------------------+
| ID | acc19f4f-af85-433c-964f-72cbe5e62201 |
| Name | std.mistral_http |
| Workflow name | repeated_retry |
| Task name | None |
| Task ID | 531c799a-23cb-453a-8b0e-44558ce2980d |
| State | ERROR |
| State info | None |
| Accepted | False |
| Created at | 2018-01-16 20:39:51 |
| Updated at | 2018-01-16 20:41:15 |
+---------------+--------------------------------------+

root@f316915139b2:~# mistral action-execution-list 531c799a-23cb-453a-8b0e-44558ce2980d
+--------------------------------------+------------------+----------------+------------+--------------------------------------+---------+----------+---------------------+---------------------+
| ID | Name | Workflow name | Task name | Task ID | State | Accepted | Created at | Updated at |
+--------------------------------------+------------------+----------------+------------+--------------------------------------+---------+----------+---------------------+---------------------+
| acc19f4f-af85-433c-964f-72cbe5e62201 | std.mistral_http | repeated_retry | async_http | 531c799a-23cb-453a-8b0e-44558ce2980d | ERROR | False | 2018-01-16 20:39:51 | 2018-01-16 20:41:15 |
| 153c09d5-cc44-4eee-9d29-470927e992c3 | std.mistral_http | repeated_retry | async_http | 531c799a-23cb-453a-8b0e-44558ce2980d | RUNNING | False | 2018-01-16 20:41:25 | <none> |
| 615e6e4a-57d6-4149-994a-adb82f3ed180 | std.mistral_http | repeated_retry | async_http | 531c799a-23cb-453a-8b0e-44558ce2980d | RUNNING | False | 2018-01-16 20:45:00 | <none> |
+--------------------------------------+------------------+----------------+------------+--------------------------------------+---------+----------+---------------------+---------------------+

Revision history for this message
Matt B (matty-brennan) wrote :
Vitalii Solodilov (mcdoker18)
Changed in mistral:
status: New → Confirmed
assignee: nobody → Vitalii Solodilov (mcdoker18)
OpenStack Infra (hudson-openstack)
Changed in mistral:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to mistral (master)

Reviewed: https://review.openstack.org/539745
Committed: https://git.openstack.org/cgit/openstack/mistral/commit/?id=0369fdd29931965631f3f9b15494ba6590670205
Submitter: Zuul
Branch: master

commit 0369fdd29931965631f3f9b15494ba6590670205
Author: Vitalii Solodilov <email address hidden>
Date: Thu Mar 8 17:02:46 2018 +0400

    Prevent a action completion multiple times

    If action is completed for second time we must raise and propagate
    exception to the client to prevent task completion twice.

    Change-Id: I36c5a85da4549ecdf97f6b5996ba436f153dabe8
    Closes-Bug: 1743662
    Signed-off-by: Vitalii Solodilov <email address hidden>

Changed in mistral:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/mistral 7.0.0.0b1

This issue was fixed in the openstack/mistral 7.0.0.0b1 development milestone.

Dougal Matthews (d0ugal)
Changed in mistral:
milestone: none → rocky-1
importance: Undecided → High
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.