Murano environment deletion hangs for 1 hour

Bug #1566274 reported by Tatyana Kuterina
24
This bug affects 3 people
Affects Status Importance Assigned to Milestone
Mirantis OpenStack
Status tracked in 10.0.x
10.0.x
Fix Committed
High
Stan Lagun
9.x
Fix Released
High
Stan Lagun

Bug Description

https://bugs.launchpad.net/murano/+bug/1576703

Detailed bug description:
Murano environment deletion hangs for 1 hour.
All instances are removed almost immediately.
Murano environment stays in deleting status after page reload and after relogin in Murano dashboard.

Steps to reproduce:
        1. Create Murano environment
        2. Add Kubernetes Cluster application to the environment: Set
        initial_gateways=1, max_gateways=2, initial_nodes=1, max_nodes=2
        3. Add Kubernetes Pod to the environment
        4. Add some application to the environment
        5. Deploy environment
        6. Check deployment status and make sure that all nodes are active
        7. Execute scaleNodesUp action and wait deployment is finished
        8. Check deployment status and make sure that all nodes are active
        9. Delete Murano environment

Expected results:
  1-8 OK, deployment is finished without errors
  9. OK, Murano environment is deleted in several minutes

Actual result:
  1-8 OK, deployment is finished without errors
  9. NOK, Murano environment hangs in deleting status during 1 hour.
  In Deployment Logs (murano dashboard) appears incorrect message:
   2016-04-04 14:03:33 — Action None is scheduled
                After 1 hour, environment is removed from the environment list.
                Error message 'Unable to retrieve list of deployments' appears in Murano dashboard.

Reproducibility:
often

Description of the environment:
MOS 8.0
1 controller node
1 compute node

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "570"
  build_id: "570"
  fuel-nailgun_sha: "558ca91a854cf29e395940c232911ffb851899c1"
  python-fuelclient_sha: "4f234669cfe88a9406f4e438b1e1f74f1ef484a5"
  fuel-agent_sha: "658be72c4b42d3e1436b86ac4567ab914bfb451b"
  fuel-nailgun-agent_sha: "b2bb466fd5bd92da614cdbd819d6999c510ebfb1"
  astute_sha: "b81577a5b7857c4be8748492bae1dec2fa89b446"
  fuel-library_sha: "c2a335b5b725f1b994f78d4c78723d29fa44685a"
  fuel-ostf_sha: "3bc76a63a9e7d195ff34eadc29552f4235fa6c52"
  fuel-mirror_sha: "fb45b80d7bee5899d931f926e5c9512e2b442749"
  fuelmenu_sha: "78ffc73065a9674b707c081d128cb7eea611474f"
  shotgun_sha: "63645dea384a37dde5c01d4f8905566978e5d906"
  network-checker_sha: "a43cf96cd9532f10794dce736350bf5bed350e9d"
  fuel-upgrade_sha: "616a7490ec7199f69759e97e42f9b97dfc87e85b"

Scrinshot: https://drive.google.com/a/mirantis.com/file/d/0Bz15vbpS5ZPNUUVwQTlPZEltdkU/view?usp=sharing
Snapshot: https://drive.google.com/a/mirantis.com/file/d/0Bz15vbpS5ZPNSWJ1b2lhcmZhYjQ/view?usp=sharing
murano-api.log: https://drive.google.com/a/mirantis.com/file/d/0Bz15vbpS5ZPNV1BHUHlDS2cxTTQ/view?usp=sharing
murano-engine.log: https://drive.google.com/a/mirantis.com/file/d/0Bz15vbpS5ZPNNHY0aXpURUFtTGs/view?usp=sharing

Tags: area-murano
Dina Belova (dbelova)
tags: added: area-murano
Changed in mos:
importance: Undecided → Medium
assignee: nobody → MOS Murano (mos-murano)
milestone: none → 8.0-updates
status: New → Confirmed
Revision history for this message
Dina Belova (dbelova) wrote :

Won't be fixed in 9.0 due to medium priority after SCF

Revision history for this message
Dina Belova (dbelova) wrote :

Added move-to-10.0 tag due to the fact bug was transferred from 9.0 to 10.0

tags: added: move-to-10.0
Revision history for this message
Victor Ryzhenkin (vryzhenkin) wrote :

Priority was raised to high due team decision. Fix for this should be present in 9.0

summary: - Murano environment deletion hangs for 1 hour Edit
+ Murano environment deletion hangs for 1 hour
description: updated
tags: removed: move-to-10.0
Revision history for this message
Serg Melikyan (smelikyan) wrote :
Revision history for this message
Serg Melikyan (smelikyan) wrote :

Fixed in upstream for 9.0 by https://review.openstack.org/312508, waiting for a sync

Revision history for this message
Serg Melikyan (smelikyan) wrote :
Revision history for this message
Mykola Stolyarenko (mstolyarenko) wrote :

Still reproduce. Fix not help

Revision history for this message
Stan Lagun (slagun) wrote :

Tested on my environment. The only place in Murano that has this 1 hour timeout is wait for response from agents. But there were no agent calls during environment deletion with the fix above applied. There was no hang on my env

Revision history for this message
Stan Lagun (slagun) wrote :

Found the root cause. The packages on apps.openstack.org are missing this commit: https://review.openstack.org/#/q/Ia6e8064ae50c24ce595ed08e446ed45b96b4a330. It is available on all stable branches since liberty so either a.o.o returns kilo packages or we haven't updated the packages since that. So if a.o.o is working as expected we need to update packages there. There is nothing to do in MOS

Revision history for this message
Stan Lagun (slagun) wrote :

Fixed packages were uploaded to apps.openstack.org. Need to test again

Revision history for this message
Alex Kholkin (akholkin) wrote :

Verified on #390 ISO
9.0-mos.all

Revision history for this message
Alex Kholkin (akholkin) wrote :

Status for 10.0 have to be Fix Committed due to the reason, that it was a problem with packages on apps.openstack.org. So it is not effect MOS.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.