stack-delete/stack-abandon fails with missing resources from global environment

Bug #1429141 reported by Johannes Grassler
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Heat
Fix Released
Low
Peter Razumovsky
Juno
Fix Released
Low
Peter Razumovsky
Kilo
Fix Released
Low
Peter Razumovsky

Bug Description

We created a bunch of custom resources for use in global environments. They
work just fine if they are registered from a user's environment (specified
using -e) using resource_registry, both for stack-create and stack-delete. If
the same environment is copied to /etc/heat/environment.d stack creation will
continue to work. stack-delete depends on the continued availability of the
global environment used to create the stack, though. If the global environment
is removed before stack-delete the stack will be remain stuck in state
DELETE_IN_PROGRESS indefinitely. Additionally, stack-abandon will result in an
error message:

  $ heat stack-abandon test_delete
  ERROR: Unknown resource Type : sys11::bootstrap

Recovery from the error condition is possible through restoring the missing
environment and issuing another stack-delete.

A minimum example template, environment and costum resource to reproduce the
problem we observed is attached. We ran into this problem with the stable/juno
version of heat-engine (Ubuntu Package version 2014.2.1-0ubuntu2~c).

Steps to reproduce with user environment (working):

1. tar -C /tmp -xf test_global_env.tar
2. heat stack-create -e /tmp/test_global_env/env.yaml -f /tmp/test_global_env/test.yaml -P key_name=<nova key name> test_delete
3. heat stack-delete test_delete # Works fine.
2. heat stack-create -e /tmp/test_global_env/env.yaml -f /tmp/test_global_env/test.yaml -P key_name=<nova key name> test_delete
3. heat stack-abandon test_delete # Works fine.

Steps to reproduce with global environment (breaks upon stack-delete):

1. tar -C /tmp -xf test_global_env.tar # on the machine where heat-engine is running.
2. cp /tmp/test_global_env/env.yaml /etc/heat/environment.d/
3. service heat-engine restart
4. heat stack-create -f /tmp/test_global_env/test.yaml -P key_name=<nova key name> test_delete
5. rm /etc/heat/environment.d/env.yaml
6. service heat-engine restart
7. heat stack-delete test_delete # Stack gets stuck in state DELETE_IN_PROGRESS.
8. heat stack-abandon test_delete # Results in the error message shown above.
9. cp /tmp/test_global_env/env.yaml /etc/heat/environment.d/
10. service heat-engine restart
11. heat stack-delete test_delete # Will work now.

Revision history for this message
Johannes Grassler (jgr-launchpad) wrote :
Changed in heat:
assignee: nobody → Razumovsky Peter (prazumovsky)
Angus Salkeld (asalkeld)
Changed in heat:
status: New → Triaged
importance: Undecided → Medium
milestone: none → kilo-rc1
Angus Salkeld (asalkeld)
Changed in heat:
milestone: kilo-rc1 → none
milestone: none → next
importance: Medium → Low
tags: added: kilo-rc-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to heat (master)

Fix proposed to branch: master
Review: https://review.openstack.org/168865

Changed in heat:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to heat (stable/juno)

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/168885

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on heat (stable/juno)

Change abandoned by Peter Razumovsky (<email address hidden>) on branch: stable/juno
Review: https://review.openstack.org/168885
Reason: this patch requires backporting many other patcches

Revision history for this message
Peter Razumovsky (prazumovsky) wrote :

I afraid, this bug cannot be resolved for juno, because juno db don't support environment storing, so, if our env is losing, we cannot load it from raw_template table (even we store file from this env). So this bug can be solved only for master, where db supports env storing.

Revision history for this message
Angus Salkeld (asalkeld) wrote :

Peter, In juno it's just called stack.parameters. But the contents are the same.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to heat (master)

Reviewed: https://review.openstack.org/168865
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=d8fe35eee518036eab7f73aca28cf4d770ae7dc1
Submitter: Jenkins
Branch: master

commit d8fe35eee518036eab7f73aca28cf4d770ae7dc1
Author: Peter Razumovsky <email address hidden>
Date: Mon Mar 30 14:48:32 2015 +0300

    Add env storing for loaded environments

    If env loaded from outer file, there is probability
    that this outer file will be deleted. So, if template
    uses some resource with resource registry env, we need
    to store this env in db.

    Change-Id: Ib0c191db2dccac6d467a9961062424bc1f924c24
    Closes-bug: #1429141

Changed in heat:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to heat (stable/kilo)

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/177154

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to heat (stable/juno)

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/177167

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on heat (stable/juno)

Change abandoned by Peter Razumovsky (<email address hidden>) on branch: stable/juno
Review: https://review.openstack.org/168885
Reason: Duplicate of https://review.openstack.org/#/c/177167/

Thierry Carrez (ttx)
tags: removed: kilo-rc-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to heat (stable/juno)

Reviewed: https://review.openstack.org/177167
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=6a117f3930049d37b03770e2cb1c94a9795a08e0
Submitter: Jenkins
Branch: stable/juno

commit 6a117f3930049d37b03770e2cb1c94a9795a08e0
Author: Peter Razumovsky <email address hidden>
Date: Mon Mar 30 14:48:32 2015 +0300

    Add env storing for loaded environments

    If env loaded from outer file, there is probability
    that this outer file will be deleted. So, if template
    uses some resource with resource registry env, we need
    to store this env in db.

    Change-Id: Ib0c191db2dccac6d467a9961062424bc1f924c24
    Closes-bug: #1429141
    (cherry picked from commit d8fe35eee518036eab7f73aca28cf4d770ae7dc1)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to heat (stable/kilo)

Reviewed: https://review.openstack.org/177154
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=b0be01301837a9930d2f4a097c815a083a0cac1c
Submitter: Jenkins
Branch: stable/kilo

commit b0be01301837a9930d2f4a097c815a083a0cac1c
Author: Peter Razumovsky <email address hidden>
Date: Mon Mar 30 14:48:32 2015 +0300

    Add env storing for loaded environments

    If env loaded from outer file, there is probability
    that this outer file will be deleted. So, if template
    uses some resource with resource registry env, we need
    to store this env in db.

    Change-Id: Ib0c191db2dccac6d467a9961062424bc1f924c24
    Closes-bug: #1429141
    (cherry picked from commit d8fe35eee518036eab7f73aca28cf4d770ae7dc1)

Changed in heat:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.