system_test.ubuntu.plugins.thread_2_separate_services failed with 'Task[keystone-db/4] Stopping the deployment process!'

Bug #1591256 reported by Artem Hrechanychenko
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Released
Critical
Alex Schultz
Mitaka
Fix Released
Critical
Alex Schultz

Bug Description

Detailed bug description:
 2016-06-10 05:00:48,678 - ERROR decorators.py:126 -- Traceback (most recent call last):
  File "/home/jenkins/workspace/9.0.system_test.ubuntu.plugins.thread_2_separate_services/fuelweb_test/helpers/decorators.py", line 120, in wrapper
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/9.0.system_test.ubuntu.plugins.thread_2_separate_services/fuelweb_test/tests/tests_separate_services/test_separate_multiroles.py", line 147, in separate_all_service
    self.fuel_web.deploy_cluster_wait(cluster_id)
  File "/home/jenkins/workspace/9.0.system_test.ubuntu.plugins.thread_2_separate_services/fuelweb_test/helpers/decorators.py", line 455, in wrapper
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/9.0.system_test.ubuntu.plugins.thread_2_separate_services/fuelweb_test/helpers/decorators.py", line 440, in wrapper
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/9.0.system_test.ubuntu.plugins.thread_2_separate_services/fuelweb_test/helpers/decorators.py", line 491, in wrapper
    return func(*args, **kwargs)
  File "/home/jenkins/workspace/9.0.system_test.ubuntu.plugins.thread_2_separate_services/fuelweb_test/helpers/decorators.py", line 498, in wrapper
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/9.0.system_test.ubuntu.plugins.thread_2_separate_services/fuelweb_test/helpers/decorators.py", line 382, in wrapper
    return func(*args, **kwargs)
  File "/home/jenkins/workspace/9.0.system_test.ubuntu.plugins.thread_2_separate_services/fuelweb_test/models/fuel_web_client.py", line 897, in deploy_cluster_wait
    self.check_cluster_status(cluster_id, allow_partially_deploy)
  File "/home/jenkins/workspace/9.0.system_test.ubuntu.plugins.thread_2_separate_services/fuelweb_test/models/fuel_web_client.py", line 925, in check_cluster_status
    assert_true(check, error_msg)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/asserts.py", line 163, in assert_true
    raise ASSERTION_ERROR(message)
AssertionError: Cluster is not deployed: some nodes are in the Error state

Steps to reproduce:

            1. Create cluster
            2. Add 3 nodes with controller role
            3. Add 3 nodes with database, keystone, rabbit,
               horizon
            4. Add 1 compute and cinder
            5. Verify networks
            6. Deploy the cluster <<<failed here
            7. Verify networks
            8. Run OSTF

Expected results:
 Deployment finished successfully

Actual result:
 Cluster is not deployed: some nodes are in the Error state

Reproducibility:
 https://product-ci.infra.mirantis.net/job/9.0.system_test.ubuntu.plugins.thread_2_separate_services/138/console

Workaround:
 -
Impact:
 swarm
Description of the environment:
  cat /etc/fuel_build_id:
 465
cat /etc/fuel_build_number:
 465
cat /etc/fuel_release:
 9.0
cat /etc/fuel_openstack_version:
 mitaka-9.0
rpm -qa | egrep 'fuel|astute|network-checker|nailgun|packetary|shotgun':
 fuel-release-9.0.0-1.mos6349.noarch
 fuel-misc-9.0.0-1.mos8454.noarch
 python-packetary-9.0.0-1.mos140.noarch
 fuel-bootstrap-cli-9.0.0-1.mos285.noarch
 fuel-migrate-9.0.0-1.mos8454.noarch
 rubygem-astute-9.0.0-1.mos750.noarch
 fuel-mirror-9.0.0-1.mos140.noarch
 shotgun-9.0.0-1.mos90.noarch
 fuel-openstack-metadata-9.0.0-1.mos8742.noarch
 fuel-notify-9.0.0-1.mos8454.noarch
 nailgun-mcagents-9.0.0-1.mos750.noarch
 python-fuelclient-9.0.0-1.mos325.noarch
 fuel-9.0.0-1.mos6349.noarch
 fuel-utils-9.0.0-1.mos8454.noarch
 fuel-setup-9.0.0-1.mos6349.noarch
 fuel-provisioning-scripts-9.0.0-1.mos8742.noarch
 fuel-library9.0-9.0.0-1.mos8454.noarch
 network-checker-9.0.0-1.mos74.x86_64
 fuel-agent-9.0.0-1.mos285.noarch
 fuel-ui-9.0.0-1.mos2717.noarch
 fuel-ostf-9.0.0-1.mos935.noarch
 fuelmenu-9.0.0-1.mos274.noarch
 fuel-nailgun-9.0.0-1.mos8742.noarch

Revision history for this message
Artem Hrechanychenko (agrechanichenko) wrote :
summary: - luster is not deployed: some nodes are in the Error state
+ Cluster is not deployed: some nodes are in the Error state
Changed in fuel:
status: New → Confirmed
Revision history for this message
Maksim Malchuk (mmalchuk) wrote :
Download full text (3.9 KiB)

node-4 failed with:

2016-06-10 04:55:36 +0000 Scope(Class[Openstack_tasks::Keystone::Db]) (notice): MODULAR: keystone/db.pp
2016-06-10 04:55:37 +0000 Scope(Class[main]) (warning): osnailyfacter/modular/./keystone/db.pp is deprecated in mitaka and will be removed in newton. Please use openstack_tasks/examples/./keystone/db.pp
2016-06-10 04:55:37 +0000 Puppet (notice): Compiled catalog for node-4.test.domain.local in environment production in 1.06 seconds
2016-06-10 04:55:41 +0000 /Stage[main]/Mysql::Client::Install/Package[mysql_client]/ensure (notice): ensure changed 'purged' to 'present'
2016-06-10 04:55:41 +0000 /Stage[main]/Keystone::Deps/Anchor[keystone::service::end] (notice): Triggered 'refresh' from 1 events
2016-06-10 04:55:41 +0000 /Stage[main]/Osnailyfacter::Mysql_access/File[10.109.1.13-mysql-access]/ensure (notice): created
2016-06-10 04:55:41 +0000 /Stage[main]/Osnailyfacter::Mysql_access/File[/root/.my.cnf]/ensure (notice): created
2016-06-10 04:55:41 +0000 Puppet (err): Could not prefetch mysql_database provider 'mysql': Execution of '/usr/bin/mysql --defaults-extra-file=/root/.my.cnf -NBe show databases' returned 1: ERROR 2013 (HY000): Lost connection to MySQL server at 'reading initial communication packet', system error: 2
/usr/lib/ruby/vendor_ruby/puppet/util/execution.rb:219:in `execute'
/usr/lib/ruby/vendor_ruby/puppet/provider/command.rb:23:in `execute'
/usr/lib/ruby/vendor_ruby/puppet/provider.rb:237:in `block in has_command'
/etc/puppet/modules/mysql/lib/puppet/provider/mysql_database/mysql.rb:8:in `instances'
/etc/puppet/modules/mysql/lib/puppet/provider/mysql_database/mysql.rb:25:in `prefetch'
/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:307:in `prefetch'
/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:197:in `prefetch_if_necessary'
/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:96:in `block in evaluate'
/usr/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:116:in `call'
/usr/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:116:in `traverse'
/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:138:in `evaluate'
/usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:169:in `block in apply'
/usr/lib/ruby/vendor_ruby/puppet/util/log.rb:149:in `with_destination'
/usr/lib/ruby/vendor_ruby/puppet/transaction/report.rb:112:in `as_logging_destination'
/usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:168:in `apply'
/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:120:in `block in apply_catalog'
/usr/lib/ruby/vendor_ruby/puppet/util.rb:161:in `block in benchmark'
/usr/lib/ruby/1.9.1/benchmark.rb:295:in `realtime'
/usr/lib/ruby/vendor_ruby/puppet/util.rb:160:in `benchmark'
/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:119:in `apply_catalog'
/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:227:in `run_internal'
/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:134:in `block in run'
/usr/lib/ruby/vendor_ruby/puppet/context.rb:64:in `override'
/usr/lib/ruby/vendor_ruby/puppet.rb:246:in `override'
/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:133:in `run'
/usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:302:in `apply_catalog'
/usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:236:in `block in main'
/usr/lib/ruby/...

Read more...

summary: - Cluster is not deployed: some nodes are in the Error state
+ system_test.ubuntu.plugins.thread_2_separate_services failed with
+ 'Task[keystone-db/4] Stopping the deployment process!'
Revision history for this message
Maksim Malchuk (mmalchuk) wrote :

this issue should be fixed already https://review.openstack.org/#/c/327235/

Changed in fuel:
status: Confirmed → Incomplete
tags: added: swarm-blocker
Changed in fuel:
importance: High → Critical
Revision history for this message
Alex Schultz (alex-schultz) wrote :

It already contains that patch, so something else needs to be fixed.

Changed in fuel:
status: Incomplete → Confirmed
no longer affects: fuel/newton
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/328466

Changed in fuel:
assignee: Fuel Sustaining (fuel-sustaining-team) → Alex Schultz (alex-schultz)
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/328468

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/328466
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=bedd29639a500038a4d9f864f1d3d5a6e50345cc
Submitter: Jenkins
Branch: master

commit bedd29639a500038a4d9f864f1d3d5a6e50345cc
Author: Alex Schultz <email address hidden>
Date: Fri Jun 10 13:32:11 2016 -0600

    Fix database tasks when detached database is used

    This patch essentially reverts change sets
    Ia1d45f45cbbd91b7c6cacd4aa46dae0a92e4156e and
    I38ad9d5fd63dda374274858632710c121d721451 which were created because
    of the regression introduced by
    Icc27b6701726afb98df71050c344c6cd9525301e. But we are also reducing
    the cross-depends to only match the primary-database and database
    tasks to prevent possible loops.

    We can only wait on the mysql backend if it's configured and we don't
    configure the mysql haproxy backend if mysql is not enabled on a node.
    Additionally we need to wait until the database is actually setup before
    running any of the db tasks.

    Change-Id: I4b9bc29b4134872908af1900f6a3e5ccbdb9e392
    Closes-Bug: #1591256

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/mitaka)

Reviewed: https://review.openstack.org/328468
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=5a6ac37d88d61e6e727d221d8d742b6ed091066a
Submitter: Jenkins
Branch: stable/mitaka

commit 5a6ac37d88d61e6e727d221d8d742b6ed091066a
Author: Alex Schultz <email address hidden>
Date: Fri Jun 10 13:32:11 2016 -0600

    Fix database tasks when detached database is used

    This patch essentially reverts change sets
    Ia1d45f45cbbd91b7c6cacd4aa46dae0a92e4156e and
    I38ad9d5fd63dda374274858632710c121d721451 which were created because
    of the regression introduced by
    Icc27b6701726afb98df71050c344c6cd9525301e. But we are also reducing
    the cross-depends to only match the primary-database and database
    tasks to prevent possible loops.

    We can only wait on the mysql backend if it's configured and we don't
    configure the mysql haproxy backend if mysql is not enabled on a node.
    Additionally we need to wait until the database is actually setup before
    running any of the db tasks.

    Change-Id: I4b9bc29b4134872908af1900f6a3e5ccbdb9e392
    Closes-Bug: #1591256

Revision history for this message
Nastya Urlapova (aurlapova) wrote :
Changed in fuel:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.