Fuel for OpenStack

Deployment failed due inability to connect to mongodb server (after partition preservation)

Bug #1572104 reported by Vasily Gorin on 2016-04-19

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	Fuel for OpenStack	Fix Committed	High	Maksim Malchuk	Fuel for OpenStack 10.0
	Mitaka	Fix Released	High	Maksim Malchuk	Fuel for OpenStack 9.0

Bug Description

CI: https://product-ci.infra.mirantis.net/job/9.0.system_test.ubuntu.partition_preservation/82/testReport/(root)/mongo_mysql_partition_preservation/mongo_mysql_partition_preservation/

Filtered Log:
node-3.test.domain.local 2016-04-19T01:16:02.917967 err: Unable to connect to mongodb server! (127.0.0.1:27017)

node-3.test.domain.local 2016-04-19T01:16:02.928107 err: (/Stage[main]/Mongodb::Server::Service/Mongodb_conn_validator[mongodb]/ensure) change from absent to present failed: Unable to connect to mongodb server! (127.0.0.1:27017)
node-3.test.domain.local 2016-04-19T01:16:15.213183 err: Could not prefetch mongodb_replset provider 'mongo': Could not evalute MongoDB shell command: load('/root/.mongorc.js'); printjson(rs.conf())
node-3.test.domain.local 2016-04-19T01:16:46.318352 err: Could not prefetch mongodb_database provider 'mongodb': Could not evalute MongoDB shell command: load('/root/.mongorc.js'); printjson(db.getMongo().getDBs())
node-3.test.domain.local 2016-04-19T01:16:46.376416 err: Could not prefetch mongodb_user provider 'mongodb': Execution of '/usr/bin/mongo admin --quiet --host 127.0.0.1:27017 --eval load('/root/.mongorc.js'); printjson(db.isMaster())' returned 1: 2016-04-19T01:16:46.379+0000 warning: Failed to connect to 127.0.0.1:27017, reason: errno:111 Connection refused
node-3.test.domain.local 2016-04-19T01:16:46.376526 err: 2016-04-19T01:16:46.380+0000 Error: couldn't connect to server 127.0.0.1:27017 (127.0.0.1), connection attempt failed at src/mongo/shell/mongo.js:148
node-3.test.domain.local 2016-04-19T01:16:46.376526 err: exception: connect failed

Tags:

Revision history for this message

Bug Checker Bot (bug-checker) wrote on 2016-04-19: Autochecker

(This check performed automatically)
Please, make sure that bug description contains the following sections filled in with the appropriate data related to the bug you are describing:

actual result

version

expected result

steps to reproduce

For more detailed information on the contents of each of the listed sections see https://wiki.openstack.org/wiki/Fuel/How_to_contribute#Here_is_how_you_file_a_bug

tags:

added: need-info

Maksim Malchuk (mmalchuk) on 2016-04-19

Changed in fuel:
status:	New → Confirmed
assignee:	Fuel Library Team (fuel-library) → Maksim Malchuk (mmalchuk)

Revision history for this message

Maksim Malchuk (mmalchuk) wrote on 2016-04-19: Re: Deployment failed due inability to connect to mongodb server

actually deploy failed due:

2016-04-19 00:54:42 +0000 Puppet (err): Could not run: Could not find file /etc/puppet/modules/openstack_tasks/examples/swift/proxy.pp

Changed in fuel:
status:	Confirmed → In Progress

Revision history for this message

Maksim Malchuk (mmalchuk) wrote on 2016-04-19:

looks like we have a regression added by: https://review.openstack.org/#/c/305329/
investigating

Revision history for this message

Sergey Novikov (snovikov) wrote on 2016-04-19:

These tests
https://product-ci.infra.mirantis.net/job/9.0.system_test.ubuntu.full_cluster_reinstallation/82/testReport/(root)/full_cluster_reinstallation/full_cluster_reinstallation/

https://product-ci.infra.mirantis.net/job/9.0.system_test.ubuntu.ready_node_reinstallation/82/testReport/(root)/reinstall_single_regular_controller_node/

are affected by this bug too

Maksim Malchuk (mmalchuk) on 2016-04-20

tags:

added: area-library team-bugfix

Sergey Shevorakov (sshevorakov) on 2016-04-20

tags:

added: swarm-blocker

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-04-20: Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/308368

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-04-20: Fix proposed to fuel-library (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/308373

Revision history for this message

Maksim Malchuk (mmalchuk) wrote on 2016-04-22: Re: Deployment failed due inability to connect to mongodb server

fix proposed and passed CI, but temporarily blocked by another issue: https://bugs.launchpad.net/fuel/+bug/1573105

Maksim Malchuk (mmalchuk) on 2016-04-25

summary:

- Deployment failed due inability to connect to mongodb server
+ Deployment failed due inability to connect to mongodb server (after
+ partition preservation)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-04-25: Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/308368
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=f1f838b6effddad69dbb993b92a628ceec630599
Submitter: Jenkins
Branch: master

commit f1f838b6effddad69dbb993b92a628ceec630599
Author: Maksim Malchuk <email address hidden>
Date: Wed Apr 20 17:32:26 2016 +0300

Fix swift proxy task dependencies

    Change-Id: I9de36798dddb404c3a40474211281a416217cb9c
    Closes-Bug: #1572104
    Signed-off-by: Maksim Malchuk <email address hidden>

Changed in fuel:
status:	In Progress → Fix Committed

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-04-25: Fix merged to fuel-library (stable/mitaka)

Reviewed: https://review.openstack.org/308373
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=b65fbd5753a17b4d11880298ebaa0766ed9d4683
Submitter: Jenkins
Branch: stable/mitaka

commit b65fbd5753a17b4d11880298ebaa0766ed9d4683
Author: Maksim Malchuk <email address hidden>
Date: Wed Apr 20 17:32:26 2016 +0300

Fix swift proxy task dependencies

    Change-Id: I9de36798dddb404c3a40474211281a416217cb9c
    Closes-Bug: #1572104
    Signed-off-by: Maksim Malchuk <email address hidden>
    (cherry picked from commit f1f838b6effddad69dbb993b92a628ceec630599)

Revision history for this message

Maksim Malchuk (mmalchuk) wrote on 2016-04-27:

#10

RCA: a regression was added by https://review.openstack.org/#/c/305329/1/deployment/puppet/openstack_tasks/examples/swift/tasks.yaml because custom-ci scenario didn't used in upstream ci.

Revision history for this message

Dmitriy Kruglov (dkruglov) wrote on 2016-06-01:

#11

There was an occurrence of the issue on the latest swarm ran against MOS 9.0, build 424 -
https://product-ci.infra.mirantis.net/job/9.0.system_test.ubuntu.partition_preservation/126/testReport/(root)/mongo_mysql_partition_preservation/mongo_mysql_partition_preservation/

Traceback from puppet.log http://paste.openstack.org/show/506856/

Revision history for this message

Dmitriy Kruglov (dkruglov) wrote on 2016-06-01:

#12

fail_error_mongo_mysql_partition_preservation-fuel-snapshot-2016-06-01_05-25-46.tar.gz Edit (76.1 MiB, application/x-tar)

Dmitry Pyzhov (dpyzhov) on 2016-06-01

no longer affects:

fuel/newton

Revision history for this message

Michael Polenchuk (mpolenchuk) wrote on 2016-06-01:

#13

in logs: mongodb main process terminated with status 100

EXIT_UNCAUGHT = 100 , // top level exception that wasn't caught

possible reasons:
* permission issue on the data directory (/var/lib/mongo)
* lock file exists

Revision history for this message

Michael Polenchuk (mpolenchuk) wrote on 2016-06-01:

#14

Mongodb::Server::Config/File[/var/lib/mongo/mongodb]/owner (notice): owner changed 'aodh' to 'mongodb'
Mongodb::Server::Config/File[/var/lib/mongo/mongodb]/group (notice): group changed 'aodh' to 'mongodb'

Revision history for this message

Maksim Malchuk (mmalchuk) wrote on 2016-06-01:

#15

9.0 ETA - 3 Jun 2016

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-06-01: Fix proposed to fuel-library (stable/mitaka)

#16

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/323966

Sergey Shevorakov (sshevorakov) on 2016-06-02

tags:

added: swarm-fail
removed: swarm-blocker

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-06-03: Fix merged to fuel-library (master)

#17

Reviewed: https://review.openstack.org/323952
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=4ed71cd20e2f38d7c1184492d889e705f29175b6
Submitter: Jenkins
Branch: master

commit 4ed71cd20e2f38d7c1184492d889e705f29175b6
Author: Maksim Malchuk <email address hidden>
Date: Wed Jun 1 17:36:07 2016 +0300

Fix mongodb permissions after partition preservation

This would solve the problem with incorrect permisions of mongodb
files after the partition preservation feature.

    Change-Id: Id874889f0af4a97ca6a098ca9a7a0052ec9695a9
    Closes-Bug: #1572104
    Co-Authored-By: Michael Polenchuk <email address hidden>
    Signed-off-by: Maksim Malchuk <email address hidden>

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-06-03: Fix merged to fuel-library (stable/mitaka)

#18

Reviewed: https://review.openstack.org/323966
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=265b3de11aeb8215073e9a24da12bd3971056fa0
Submitter: Jenkins
Branch: stable/mitaka

commit 265b3de11aeb8215073e9a24da12bd3971056fa0
Author: Maksim Malchuk <email address hidden>
Date: Wed Jun 1 17:36:07 2016 +0300

Fix mongodb permissions after partition preservation

This would solve the problem with incorrect permisions of mongodb
files after the partition preservation feature.

    Change-Id: Id874889f0af4a97ca6a098ca9a7a0052ec9695a9
    Closes-Bug: #1572104
    Co-Authored-By: Michael Polenchuk <email address hidden>
    Signed-off-by: Maksim Malchuk <email address hidden>
    (cherry picked from commit 4ed71cd20e2f38d7c1184492d889e705f29175b6)

Revision history for this message

Maksim Malchuk (mmalchuk) wrote on 2016-06-03:

#19

Fix in upstream also merged: https://github.com/puppetlabs/puppetlabs-mongodb/pull/310

Revision history for this message

Dmitriy Kruglov (dkruglov) wrote on 2016-06-08:

#20

The issue is reproduced on the latest swarm run (MOS 9.0 ISO build 453), although the committed patch is present in the ISO.

The puppet.log trace http://paste.openstack.org/show/509008/

Revision history for this message

Dmitriy Kruglov (dkruglov) wrote on 2016-06-08:

#21

fail_error_mongo_mysql_partition_preservation-fuel-snapshot-2016-06-08_01-39-26.tar.gz Edit (56.3 MiB, application/x-tar)

Revision history for this message

Michael Polenchuk (mpolenchuk) wrote on 2016-06-09:

#22

LOGS:
Could not look up qualified variable '::mongodb::server::user'; class ::mongodb::server has not been evaluated

FIX:
include ::mongodb::params
$user = $::mongodb::params::user
$group = $::mongodb::params::group

DESC:
mongodb module has really confused params getter:
server <-> globals <-> config <-> params

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-06-09: Related fix proposed to fuel-library (master)

#23

Related fix proposed to branch: master
Review: https://review.openstack.org/327548

Dmitry Pyzhov (dpyzhov) on 2016-06-09

Changed in fuel:
status:	Fix Committed → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-06-09: Fix proposed to fuel-library (stable/mitaka)

#24

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/327954

Changed in fuel:
status:	In Progress → Fix Committed

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-06-09: Fix merged to fuel-library (master)

#25

Reviewed: https://review.openstack.org/327548
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=c53136f947001ded0509a8c6f21d24f75f6e40ca
Submitter: Jenkins
Branch: master

commit c53136f947001ded0509a8c6f21d24f75f6e40ca
Author: Maksim Malchuk <email address hidden>
Date: Thu Jun 9 12:49:08 2016 +0300

Fix for look up qualified variables '::mongodb::server::...'

    Change-Id: Ib056ada75ddf140f600dfaeec81726c7687497f3
    Closes-Bug: #1572104
    Signed-off-by: Maksim Malchuk <email address hidden>

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-06-10: Fix merged to fuel-library (stable/mitaka)

#26

Reviewed: https://review.openstack.org/327954
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=925133b90cf400ecd230f9ab2e7c4d3f8d003671
Submitter: Jenkins
Branch: stable/mitaka

commit 925133b90cf400ecd230f9ab2e7c4d3f8d003671
Author: Maksim Malchuk <email address hidden>
Date: Thu Jun 9 12:49:08 2016 +0300

Fix for look up qualified variables '::mongodb::server::...'

    Change-Id: Ib056ada75ddf140f600dfaeec81726c7687497f3
    Closes-Bug: #1572104
    Signed-off-by: Maksim Malchuk <email address hidden>
    (cherry picked from commit c53136f947001ded0509a8c6f21d24f75f6e40ca)

Kyrylo Romanenko (kromanenko) on 2016-06-15

tags:

added: on-verification

Revision history for this message

Kyrylo Romanenko (kromanenko) wrote on 2016-06-15:

#27

Job https://product-ci.infra.mirantis.net/job/9.0.system_test.ubuntu.partition_preservation/ is still failing. I`d like to reopen issue for clarification.
Vasily Gorin, could you look at that and check: do we have regression or some another bug causes test job to fail?

Revision history for this message

Maksim Malchuk (mmalchuk) wrote on 2016-06-15:

#28

the partition preservation test for mongodb and mysql is broken, the related bugs were created:
1. https://bugs.launchpad.net/fuel/mitaka/+bug/1591212
2. https://bugs.launchpad.net/fuel/+bug/1591271

Revision history for this message

Maksim Malchuk (mmalchuk) wrote on 2016-06-15:

#29

tested on mos 9.0 iso #465. both fixes exists.

tags:

removed: on-verification

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Bug attachments

Add attachment

Remote bug watches

Bug watches keep track of this bug in other bug trackers.