Deploy failed with error DeploymentAlreadyStarted

Bug #1619653 reported by ElenaRossokhina
22
This bug affects 3 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Released
Critical
Georgy Kibardin

Bug Description

Detailed bug description:
found in snapshot #209
https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.bonding_ha/46/testReport/(root)/deploy_bonding_neutron_vlan/deploy_bonding_neutron_vlan/
https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.bonding_ha_one_controller/47/testReport/(root)/deploy_bonding_one_controller_tun/deploy_bonding_one_controller_tun/
https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.bonding_ha/46/testReport/(root)/deploy_bonding_neutron_vlan/deploy_bonding_neutron_vlan/

Steps to reproduce:

Expected results:
Pass

Actual result:
Test failed with error: deploy_cluster_changes raised: BadRequest('Bad Request (HTTP 400)',)

api.log
2016-09-02 02:29:51.085 DEBUG [7fef15ac1880] (logger) Request PUT /api/clusters/1/changes/ from 10.109.10.1:38015
2016-09-02 02:29:51.180 DEBUG [7fef15ac1880] (logger) Response code '400 Bad Request' for PUT /api/clusters/1/changes/ from 10.109.10.1:38015

app.log
2016-09-02 02:29:51.177 ERROR [7fef15ac1880] (base) Error during execution of deployment task on environment '1': Cannot perform the actions because there are another running tasks.
2016-09-02 02:29:51.178 ERROR [7fef15ac1880] (base) NailgunException occured
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/nailgun/api/v1/handlers/base.py", line 293, in handle_errors
    return func(cls, *args, **kwargs)
  File "<string>", line 2, in PUT
  File "/usr/lib/python2.7/site-packages/nailgun/api/v1/handlers/base.py", line 351, in validate
    return func(cls, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/nailgun/api/v1/handlers/base.py", line 592, in PUT
    task = task_manager.execute(**options)
  File "/usr/lib/python2.7/site-packages/nailgun/task/manager.py", line 210, in execute
    'Cannot perform the actions because '
DeploymentAlreadyStarted: Cannot perform the actions because there are another running tasks.
[pid: 7167|app: 0|req: 83/332] 10.109.10.1 () {40 vars in 603 bytes} [Fri Sep 2 02:29:51 2016] PUT /api/clusters/1/changes/ => generated 67 bytes in 100 msecs (HTTP/1.1 400) 5 headers in 213 bytes (2 switches on core 0)

logs are available here: https://drive.google.com/open?id=0B2ag_Bf-ShtTQzdEM3ZYRDBEc3M

Revision history for this message
ElenaRossokhina (esolomina) wrote :
tags: added: swarm-blocker
Changed in fuel:
milestone: none → 9.1
Revision history for this message
Dmitry Pyzhov (dpyzhov) wrote :

Please add test scenario and where exactly it failed

Changed in fuel:
status: New → Incomplete
Revision history for this message
Alexey. Kalashnikov (akalashnikov) wrote :

there is a duplicate of this issue, information from it might be helpful:
https://bugs.launchpad.net/fuel/+bug/1620274

Changed in fuel:
status: Incomplete → New
Changed in fuel:
assignee: nobody → Fuel Sustaining (fuel-sustaining-team)
importance: Undecided → High
Dmitry Pyzhov (dpyzhov)
Changed in fuel:
assignee: Fuel Sustaining (fuel-sustaining-team) → Georgy Kibardin (gkibardin)
Changed in fuel:
status: New → In Progress
Revision history for this message
Georgy Kibardin (gkibardin) wrote :

The reason is that nailgun failed to process check_dhcp response and the corresponding task has hanged.
2016-09-02 02:29:24.862 ERROR [7fa2b0fef740] (receiverd) Message consume failed: <kombu.transport.pyamqp.Message object at 0x49c7938>
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/nailgun/rpc/receiverd.py", line 54, in consume_msg
    callback(**body["args"])
  File "/usr/lib/python2.7/site-packages/nailgun/rpc/receiver.py", line 1130, in check_dhcp_resp
    master_network_mac):
  File "/usr/lib/python2.7/site-packages/nailgun/network/utils.py", line 29, in is_same_mac
    raise ValueError(e)
ValueError: failed to detect EUI version: u''

Digging further

Revision history for this message
Georgy Kibardin (gkibardin) wrote :
Revision history for this message
Timur Nurlygayanov (tnurlygayanov) wrote :

You can find more information here:
https://bugs.launchpad.net/fuel/+bug/1620526

Do we really have the fix on review for the issue? Could you please share the link to the fix?

Changed in fuel:
importance: High → Critical
tags: added: blocker-for-qa
Revision history for this message
Timur Nurlygayanov (tnurlygayanov) wrote :

The priority is Critical because the issue is blocker for all MOs 9.1-related QA activities like bugs verification / acceptance testing.

Revision history for this message
Timur Nurlygayanov (tnurlygayanov) wrote :

Fix is on review for stable/mitaka:
https://review.openstack.org/#/c/366496

Revision history for this message
Georgy Kibardin (gkibardin) wrote :
Changed in fuel:
status: In Progress → Fix Committed
tags: added: on-verification
Revision history for this message
Valentyn Yakovlev (vyakovlev) wrote :

did not help

Changed in fuel:
status: Fix Committed → Confirmed
Revision history for this message
Valentyn Yakovlev (vyakovlev) wrote :
tags: removed: on-verification
Revision history for this message
Timur Nurlygayanov (tnurlygayanov) wrote :

The proposed fix doesn't help, could you please revert the fix and prepare another one?

Thank you!

Revision history for this message
Georgy Kibardin (gkibardin) wrote :

Its a different bug.

Changed in fuel:
status: Confirmed → Fix Committed
Revision history for this message
Georgy Kibardin (gkibardin) wrote :

There is no original DeploymentAlreadyStarted: Cannot perform the actions because there are another running tasks. in /var/log/nailgun/app.log

Revision history for this message
Georgy Kibardin (gkibardin) wrote :

The only thing I could get about the new problem in logs is:
2016-09-08 12:52:18.244 WARNING [7f123124d880] (base) Node '1': there is no interface 'enp0s8' found for bond 'bond0' in DB

I think we must make log level DEBUG for Nailgun in tests.

Revision history for this message
Dmitry Belyaninov (dbelyaninov) wrote :

The last test failed on step #5. Deployment is step #6.

tags: added: on-verification
Revision history for this message
Dmitry Belyaninov (dbelyaninov) wrote :
Changed in fuel:
status: Fix Committed → Fix Released
tags: removed: on-verification
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.