Openstack deployment failed with message: Deployment has failed. All nodes are finished. Failed tasks: Task[memcached/1], Task[apache/1], Task[connectivity_tests/1] Stopping the deployment process!

Bug #1566968 reported by Igor Marnat
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Committed
High
Maksim Malchuk
Mitaka
Fix Released
High
Maksim Malchuk

Bug Description

Deployment of Openstack cluster failed with message

Deployment has failed. All nodes are finished. Failed tasks: Task[memcached/1], Task[apache/1], Task[connectivity_tests/1] Stopping the deployment process!

Quote from astute logs:
2016-04-06 15:56:14 WARNING [29233] Validation of node:
{"uid"=>"virtual_sync_node",
 "status"=>"error",
 "error_type"=>"deploy",
 "error_msg"=>
  "All nodes are finished. Failed tasks: Task[memcached/1], Task[apache/1], Task[connectivity_tests/1] Stopping the deployment process!"}
 for report failed: Task status provided '' is not supported; Task name is not provided
2016-04-06 15:56:14 WARNING [29233] Validation of node:
{"uid"=>"2",
 "status"=>"error",
 "error_type"=>"deploy",
 "error_msg"=>
  "All nodes are finished. Failed tasks: Task[memcached/1], Task[apache/1], Task[connectivity_tests/1] Stopping the deployment process!"}
 for report failed: Task status provided '' is not supported; Task name is not provided
2016-04-06 15:56:14 WARNING [29233] Validation of node:
{"uid"=>"3",
 "status"=>"error",
 "error_type"=>"deploy",
 "error_msg"=>
  "All nodes are finished. Failed tasks: Task[memcached/1], Task[apache/1], Task[connectivity_tests/1] Stopping the deployment process!"}
 for report failed: Task status provided '' is not supported; Task name is not provided

1 controller, 2 compute nodes, all the settings are default (Next->Next->Next ...) except that Murano deployment is enabled.

Environment: Windows 7 Pro, cygwin, virtualbox, standard virtualbox script launch.sh, each node is executed in a VM of 3 GB RAM.
MOS 9.0, build #165

Diagnostic snapshot: https://dl.dropboxusercontent.com/u/14062034/fuel-snapshot-2016-04-06_16-07-08.tar.xz

Tags: need-info
Revision history for this message
Bug Checker Bot (bug-checker) wrote : Autochecker

(This check performed automatically)
Please, make sure that bug description contains the following sections filled in with the appropriate data related to the bug you are describing:

actual result

expected result

steps to reproduce

For more detailed information on the contents of each of the listed sections see https://wiki.openstack.org/wiki/Fuel/How_to_contribute#Here_is_how_you_file_a_bug

tags: added: need-info
Revision history for this message
Igor Marnat (imarnat) wrote :

expected result
Openstack cluster is deployed successfully

actual results
Openstack deployment failed with error message

steps to reproduce
Take build #165, configure env with 1 controller, 2 computes, all default settings + Murano and try to deply the cluster.

Igor Marnat (imarnat)
Changed in fuel:
assignee: nobody → Fuel for Openstack (fuel)
Revision history for this message
Oleksiy Molchanov (omolchanov) wrote :

2016-04-06T15:52:56.952633+00:00 err: (Package[memcached](provider=apt_fuel)) Cannot initiate the connection to archive.ubuntu.com:80 (2001:67c:1560:8001::11). - connect (101: Network is unreachable) [IP: 2001:67c:1560:8001::11 80]

Changed in fuel:
assignee: Fuel for Openstack (fuel) → Fuel Library Team (fuel-library)
Revision history for this message
Tatyanka (tatyana-leontovich) wrote :

Guys, looks like https://github.com/openstack/fuel-virtualbox/commit/3b40e0c28484b938b9958d72c936a95cbdf2e97e broke vb scripts, as result nodes do not have connectivity to repos

Changed in fuel:
status: New → Confirmed
Revision history for this message
Tatyanka (tatyana-leontovich) wrote :

review that should fix the problem
https://review.openstack.org/#/c/302164/

Changed in fuel:
assignee: Fuel Library Team (fuel-library) → Dmitry Bilunov (dbilunov)
Changed in fuel:
assignee: Dmitry Bilunov (dbilunov) → Maksim Malchuk (mmalchuk)
Changed in fuel:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-virtualbox (master)

Reviewed: https://review.openstack.org/302164
Committed: https://git.openstack.org/cgit/openstack/fuel-virtualbox/commit/?id=80e86854be082ccc81099235ff6e5f16606e80e1
Submitter: Jenkins
Branch: master

commit 80e86854be082ccc81099235ff6e5f16606e80e1
Author: Igor Kalnitsky <email address hidden>
Date: Tue Apr 5 18:43:02 2016 +0300

    Fix public network routing from slaves

    Since we assign public gateway address on master node, all public
    traffic goes through it. Unfortunately, it doesn't reach destination due
    to rejecting rule in FORWARD chain:

        ACCEPT all -- 10.20.0.0/24 anywhere
        ACCEPT all -- anywhere anywhere
        REJECT all -- anywhere anywhere
        ext-filter-forward all -- anywhere anywhere

    The commit fixes that problem by inserting (-I), not appending (-A)
    ext-filter-forward entry. In that case that rule will have higher
    priority and won't break routing.

        ACCEPT all -- 10.20.0.0/24 anywhere
        ACCEPT all -- anywhere anywhere
        ext-filter-forward all -- anywhere anywhere
        REJECT all -- anywhere anywhere

    Change-Id: I7887f08a175fa0ce06654dc1fc18ab412cb296f5
    Closes-Bug: #1566968

Changed in fuel:
status: In Progress → Fix Committed
Changed in fuel:
milestone: 9.0 → 10.0
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-virtualbox (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/303920

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-virtualbox (stable/mitaka)

Reviewed: https://review.openstack.org/303920
Committed: https://git.openstack.org/cgit/openstack/fuel-virtualbox/commit/?id=9373c6e955bd580aeff6bd62726574b9179df3bf
Submitter: Jenkins
Branch: stable/mitaka

commit 9373c6e955bd580aeff6bd62726574b9179df3bf
Author: Igor Kalnitsky <email address hidden>
Date: Tue Apr 5 18:43:02 2016 +0300

    Fix public network routing from slaves

    Since we assign public gateway address on master node, all public
    traffic goes through it. Unfortunately, it doesn't reach destination due
    to rejecting rule in FORWARD chain:

        ACCEPT all -- 10.20.0.0/24 anywhere
        ACCEPT all -- anywhere anywhere
        REJECT all -- anywhere anywhere
        ext-filter-forward all -- anywhere anywhere

    The commit fixes that problem by inserting (-I), not appending (-A)
    ext-filter-forward entry. In that case that rule will have higher
    priority and won't break routing.

        ACCEPT all -- 10.20.0.0/24 anywhere
        ACCEPT all -- anywhere anywhere
        ext-filter-forward all -- anywhere anywhere
        REJECT all -- anywhere anywhere

    Change-Id: I7887f08a175fa0ce06654dc1fc18ab412cb296f5
    Closes-Bug: #1566968
    (cherry picked from commit 80e86854be082ccc81099235ff6e5f16606e80e1)

tags: added: on-verification
Revision history for this message
Andrey Lavrentyev (alavrentyev) wrote :

Wasn't able to reproduce it on 9.0-mos #458, so closing...

[root@nailgun ~]# shotgun2 short-report
cat /etc/fuel_build_id:
 458
cat /etc/fuel_build_number:
 458
cat /etc/fuel_release:
 9.0
cat /etc/fuel_openstack_version:
 mitaka-9.0
...

tags: removed: on-verification
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.