OpenStack Heat

Periodic watcher task not killed on stack delete

Bug #1315044 reported by Steven Hardy
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Heat
Fix Released
Undecided
Jason Dunsmore
OpenStack Heat 2014.2 "juno"
Icehouse
Fix Released
Undecided
Unassigned
OpenStack Heat 2014.1.2
oslo-incubator
Fix Released
Medium
Jason Dunsmore
oslo-incubator juno-2 "j2"

Bug Description

As reported by Qiming on IRC, and confirmed by me, there's an issue where periodic watcher tasks are not cleaned up on delete - looks like a regression caused by the rework for engine scale-out.

Steps to reproduce:

heat_template_version: 2013-05-23

description: |
    Simple template to validate watcher task.
resources:
    my_alarm:
        type: AWS::CloudWatch::Alarm
        properties:
            AlarmDescription: Restart the Server if it goes down
            MetricName: Heartbeat
            Namespace: system/linux
            Statistic: SampleCount
            Period: '10'
            EvaluationPeriods: '1'
            Threshold: '1'
            ComparisonOperator: LessThanThreshold

Create a stack with that template, then delete it, then you get these errors in the logs:

2014-05-01 16:42:42.220 DEBUG heat.engine.service [-] Periodic watcher task for stack f3fce986-9d6a-4fe2-8b25-8ea8576b56b5 from (pid=25559) check_stack_watches /opt/stack/heat/heat/engine/service.py:196
2014-05-01 16:42:42.223 ERROR heat.engine.service [req-1f94d4b3-a436-4197-8da0-803ef7fb4bdb None None] Unable to retrieve stack f3fce986-9d6a-4fe2-8b25-8ea8576b56b5 for periodic task

Note this will only happen if you have OS::Heat::CWLiteAlarm selected in default.yaml as the implementation for AWS::CloudWatch::Alarm

Jason Dunsmore (jasondunsmore)
Changed in heat:
assignee: nobody → Jason Dunsmore (jasondunsmore)
Revision history for this message
Openstack Gerrit (openstack-gerrit) wrote : Fix proposed to heat (master)

Fix proposed to branch: master
Review: https://review.openstack.org/91656

Changed in heat:
status: New → In Progress
Doug Hellmann (doug-hellmann)
Changed in oslo:
assignee: nobody → Jason Dunsmore (jasondunsmore)
importance: Undecided → Medium
status: New → In Progress
milestone: none → juno-1
Revision history for this message
Steven Hardy (shardy) wrote :

Jason, can you confirm if this will require backporting to stable/icehouse? AFAICS it will:

https://github.com/openstack/heat/blob/stable/icehouse/heat/engine/service.py#L705

tags: added: icehouse-backport-potential
Revision history for this message
Jason Dunsmore (jasondunsmore) wrote :

Looks like it is necessary. I'll go ahead and backport it now since it looks like https://review.openstack.org/#/c/91656/ is going to merge soon.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to heat (master)

Reviewed: https://review.openstack.org/91656
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=33ea3573422ac32f4bd1788603fef64bea7e85bd
Submitter: Jenkins
Branch: master

commit 33ea3573422ac32f4bd1788603fef64bea7e85bd
Author: Jason Dunsmore <email address hidden>
Date: Thu May 1 14:17:45 2014 -0500

    Stop periodic watcher tasks before deleting stack

    In the case where a stack that has a watch task associated with it is
    being deleted and the engine successfully acquires a lock, no attempt to
    stop the stack will be made since it has no action in-progress. This
    will result in orphaned watcher tasks being left behind, since the
    watcher tasks are normally stopped as part of the call to
    ThreadGroup.stop().

    To fix this, explicitly stop the periodic watcher tasks before calling
    stack.delete().

    Change-Id: Idf5b7c1100fc85743ac6dc5831731f7ecc96bfd6
    Closes-Bug: #1315044

Changed in heat:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to heat (stable/icehouse)

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/92887

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/94957

Doug Hellmann (doug-hellmann)
Changed in oslo:
milestone: juno-1 → juno-2
Thierry Carrez (ttx)
Changed in heat:
milestone: none → juno-1
status: Fix Committed → Fix Released
Jason Dunsmore (jasondunsmore)
Changed in oslo:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to heat (stable/icehouse)

Reviewed: https://review.openstack.org/94957
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=48ed4831ecb3ed6203cfa6907a72846baf6ff86d
Submitter: Jenkins
Branch: stable/icehouse

commit 48ed4831ecb3ed6203cfa6907a72846baf6ff86d
Author: Jason Dunsmore <email address hidden>
Date: Thu May 1 14:17:45 2014 -0500

    Stop periodic watcher tasks before deleting stack

    In the case where a stack that has a watch task associated with it is
    being deleted and the engine successfully acquires a lock, no attempt to
    stop the stack will be made since it has no action in-progress. This
    will result in orphaned watcher tasks being left behind, since the
    watcher tasks are normally stopped as part of the call to
    ThreadGroup.stop().

    To fix this, explicitly stop the periodic watcher tasks before calling
    stack.delete().

    Conflicts:
     heat/engine/service.py

    Change-Id: Idf5b7c1100fc85743ac6dc5831731f7ecc96bfd6
    Closes-Bug: #1315044
    (cherry picked from commit 33ea3573422ac32f4bd1788603fef64bea7e85bd)

tags: added: in-stable-icehouse
Thierry Carrez (ttx)
Changed in heat:
milestone: juno-1 → 2014.2
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.