ERR: ceph-deploy --overwrite-conf config pull node-3 returned 1 instead of one of [0]

Bug #1430407 reported by Anastasia Palkina
18
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Invalid
High
Oleksiy Molchanov
6.0.x
Invalid
High
Oleksiy Molchanov

Bug Description

"build_id": "2015-03-09_22-54-44",
"ostf_sha": "8df5f2fcdae3bc9ea7d700ffd64db820baf51914",
"build_number": "178",
"release_versions": {"2014.2-6.1": {"VERSION": {"build_id": "2015-03-09_22-54-44", "ostf_sha": "8df5f2fcdae3bc9ea7d700ffd64db820baf51914", "build_number": "178", "api": "1.0", "nailgun_sha": "a9a6578a649a2a006c4810b3d0aa6876ac6e8b83", "production": "docker", "python-fuelclient_sha": "4eb787f1ad969bd23c93d192865543dbd45a8626", "astute_sha": "2d61ee42ec6dae3181d292c7769d32e40d463893", "feature_groups": ["mirantis"], "release": "6.1", "fuelmain_sha": "c73b87f7cbc371307a21c368a45a65aa3f4b7a5d", "fuellib_sha": "62e68af896887ebe18944e6a0a9721e269119ad4"}}},
"auth_required": true,
"api": "1.0",
"nailgun_sha": "a9a6578a649a2a006c4810b3d0aa6876ac6e8b83",
"production": "docker",
"python-fuelclient_sha": "4eb787f1ad969bd23c93d192865543dbd45a8626",
"astute_sha": "2d61ee42ec6dae3181d292c7769d32e40d463893",
"feature_groups": ["mirantis"],
"release": "6.1",
"fuelmain_sha": "c73b87f7cbc371307a21c368a45a65aa3f4b7a5d",
"fuellib_sha": "62e68af896887ebe18944e6a0a9721e269119ad4"

1. Create new environment (CentOS)
2. Choose Neutron, VLAN
3. Choose Ceph for volumes and Ceph for images
4. Add 1 controller, 1 compute, 2 ceph
5. Start deployment. It has failed

Compute (node-4) and both ceph nodes (node-5,6) in 'error' state.

There is only error in puppet.log on ceph node (node-5):

2015-03-10 15:33:12 ERR

 (/Stage[main]/Ceph::Conf/Exec[ceph-deploy config pull]/returns) change from notrun to 0 failed: ceph-deploy --overwrite-conf config pull node-3 returned 1 instead of one of [0]

Logs are here: https://drive.google.com/a/mirantis.com/file/d/0B6SjzarTGFxaRFVwSDhpLS1LOUU/view?usp=sharing

Tags: qa-approve
Changed in fuel:
status: New → Confirmed
Revision history for this message
Kyrylo Romanenko (kromanenko) wrote :

Cluster on CentOS with:
1 controller
1 compute
1 Ceph OSD node.
Ceph for Cinder and Glance. Neutron with VLAN.

Deployment failed on Compute and Storage nodes.

Logs.

Astute:
2015-03-12 16:22:42
ERR
[422] No more tasks will be executed on the node 2
2015-03-12 16:22:42
ERR
[422] Task '{"priority"=>1100, "type"=>"puppet", "uids"=>["2"], "parameters"=>{"puppet_modules"=>"/etc/puppet/modules", "puppet_manifest"=>"/etc/puppet/modules/osnailyfacter/modular/roles/compute.pp", "timeout"=>3600, "cwd"=>"/"}}' on node 2 valid, but failed
2015-03-12 16:21:00
ERR
[422] No more tasks will be executed on the node 3
2015-03-12 16:21:00
ERR
[422] Task '{"priority"=>1100, "type"=>"puppet", "uids"=>["3"], "parameters"=>{"puppet_modules"=>"/etc/puppet/modules", "puppet_manifest"=>"/etc/puppet/modules/osnailyfacter/modular/roles/ceph-osd.pp", "timeout"=>3600, "cwd"=>"/"}}' on node 3 valid, but failed

Storage node puppet log:
2015-03-12 16:20:59
ERR
 ceph-deploy --overwrite-conf config pull node-1 returned 1 instead of one of [0]
2015-03-12 16:20:20
ERR
 (/Stage[main]/Ceph::Conf/Exec[ceph-deploy config pull]/returns) change from notrun to 0 failed: ceph-deploy --overwrite-conf config pull node-1 returned 1 instead of one of [0]

Compute node puppet log:
2015-03-12 16:22:39
ERR
 ceph-deploy --overwrite-conf config pull node-1 returned 1 instead of one of [0]
2015-03-12 16:21:54
ERR
 (/Stage[main]/Ceph::Conf/Exec[ceph-deploy config pull]/returns) change from notrun to 0 failed: ceph-deploy --overwrite-conf config pull node-1 returned 1 instead of one of [0]

Diagnostic snapshot
https://drive.google.com/file/d/0B6E70aHvCcRQWHREYVlvVG04dVE/view?usp=sharing

Revision history for this message
Oleksiy Butenko (obutenko) wrote :

{"build_id": "2015-03-11_21-47-59", "ostf_sha": "ecb8e294b0acbdc5b0300d5e39028fb26ecc9088", "build_number": "187", "release_versions": {"2014.2-6.1": {"VERSION": {"build_id": "2015-03-11_21-47-59", "ostf_sha": "ecb8e294b0acbdc5b0300d5e39028fb26ecc9088", "build_number": "187", "api": "1.0", "nailgun_sha": "a720a2da99690eb2d2c19ddc5d739384312a8ac2", "production": "docker", "python-fuelclient_sha": "0f4ca9c2798da34797dd082130d22cac04c998a9", "astute_sha": "5cdd4ae4037aa29f4c876d441af15cad82f5a6cb", "feature_groups": ["mirantis"], "release": "6.1", "fuelmain_sha": "0791400dd8224647ff9a5cb8051ce82b2c8863b1", "fuellib_sha": "cfdfcbdb0197f606b4c93e6dd4011525df9a3ff8"}}}, "auth_required": true, "api": "1.0", "nailgun_sha": "a720a2da99690eb2d2c19ddc5d739384312a8ac2", "production": "docker", "python-fuelclient_sha": "0f4ca9c2798da34797dd082130d22cac04c998a9", "astute_sha": "5cdd4ae4037aa29f4c876d441af15cad82f5a6cb", "feature_groups": ["mirantis"], "release": "6.1", "fuelmain_sha": "0791400dd8224647ff9a5cb8051ce82b2c8863b1", "fuellib_sha": "cfdfcbdb0197f606b4c93e6dd4011525df9a3ff8"}

Steps to reproduce:
Deploy cluster on CentOS with 1 controller + Ceph and 1 compute + Ceph. Ceph for Cinder and Glance. Neutron with vlan.
Deployment has failed with errors on computer node:

http://paste.openstack.org/show/192052/

Diagnostic snapshot:
http://goo.gl/IUK5Dw

Changed in fuel:
importance: Critical → High
Changed in fuel:
assignee: Fuel Library Team (fuel-library) → Oleksiy Molchanov (omolchanov)
Revision history for this message
Oleksiy Molchanov (omolchanov) wrote :

Cannot implement this on new ISO.

The issue was that ceph mon node was not reachable by osd. I suppose it was fixed somehow during intensive feature code merge before feature freeze.

Revision history for this message
Oleksiy Molchanov (omolchanov) wrote :

I am marking this as incomplete and if it happens again please provide me with the failed env.

Changed in fuel:
status: Confirmed → Incomplete
Revision history for this message
Anastasia Palkina (apalkina) wrote :

Verified on ISO #300

"build_id": "2015-04-09_22-54-31", "ostf_sha": "4bda5bbf9ea033189f16518032c063d43e4d0e5c", "build_number": "300", "release_versions": {"2014.2-6.1": {"VERSION": {"build_id": "2015-04-09_22-54-31", "ostf_sha": "4bda5bbf9ea033189f16518032c063d43e4d0e5c", "build_number": "300", "api": "1.0", "nailgun_sha": "d6e351189666e8afa01003e643e63216ef7abd26", "openstack_version": "2014.2-6.1", "production": "docker", "python-fuelclient_sha": "9208ff4a08dcb674ce2df132399a5aa3ddfac21c", "astute_sha": "5041b2fb508e6860c3cb96474ca31ec97e549e8b", "feature_groups": ["mirantis"], "release": "6.1", "fuelmain_sha": "2ca546b86e651d5638dbb1be9bae44b86c84a893", "fuellib_sha": "e9c3ba332b05120c967b20260c7b223afc1b4f1a"}}}, "auth_required": true, "api": "1.0", "nailgun_sha": "d6e351189666e8afa01003e643e63216ef7abd26", "openstack_version": "2014.2-6.1", "production": "docker", "python-fuelclient_sha": "9208ff4a08dcb674ce2df132399a5aa3ddfac21c", "astute_sha": "5041b2fb508e6860c3cb96474ca31ec97e549e8b", "feature_groups": ["mirantis"], "release": "6.1", "fuelmain_sha": "2ca546b86e651d5638dbb1be9bae44b86c84a893", "fuellib_sha": "e9c3ba332b05120c967b20260c7b223afc1b4f1a"

I deployed for CentOS and Ubuntu with separate Ceph nodes and multinodes. Deployment were successful, no errors for ceph.

tags: added: qa-approve
Changed in fuel:
status: Incomplete → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.