Ceph deployment failed with ceph auth error auth add client.compute --in-file=/etc/ceph/ceph.client.compute.keyring returned 1 instead of one of [0]

> 2. Add 3 nodes with controller and ceph OSD roles

First of all deploying ceph OSD and monitor onto the same node is not supported [1]
(although it's possible to setup ceph that way manually for testing/development purposes).

2016-05-05 01:29:36 +0000 /Stage[main]/Ceph::Osds/Ceph::Osd[/dev/vdc3]/Exec[ceph-osd-prepare-/dev/vdc3]/returns (notice): Error: Partition(s) 1 on /dev/vdc3 have been written, but we have been unable to inform the kernel of the change, probably because it/they are in use. As a result, the old partition(s) will remain in use. You should reboot now before making further changes.
2016-05-05 01:29:36 +0000 /Stage[main]/Ceph::Osds/Ceph::Osd[/dev/vdc3]/Exec[ceph-osd-prepare-/dev/vdc3]/returns (notice): + udevadm settle

Secondly it looks like Fuel tries to alter auth rules before fetching the client.admin keyring:

$ ls -1 node-2/etc/ceph/
ceph.client.compute.keyring
ceph.conf
rbdmap

There's no ceph.client.admin.keyring so any ceph command is deemed to fail:

2016-05-05T01:30:26.898433+00:00 notice: (/Stage[main]/Osnailyfacter::Ceph::Ceph_compute/Ceph::Key[client.compute]/Exec[ceph-injectkey-client.compute]/returns) + ceph auth add client.compute --in-file=/etc/ceph/ceph.client.compute.keyring
2016-05-05T01:30:26.898433+00:00 notice: (/Stage[main]/Osnailyfacter::Ceph::Ceph_compute/Ceph::Key[client.compute]/Exec[ceph-injectkey-client.compute]/returns) 2016-05-05 01:30:26.873878 7f5d717d5700 -1 monclient(hunting): ERROR: missing keyring, cannot use cephx for authentication
2016-05-05T01:30:26.898433+00:00 notice: (/Stage[main]/Osnailyfacter::Ceph::Ceph_compute/Ceph::Key[client.compute]/Exec[ceph-injectkey-client.compute]/returns) 2016-05-05 01:30:26.873957 7f5d717d5700 0 librados: client.admin initialization error (2) No such file or directory
2016-05-05T01:30:26.898433+00:00 notice: (/Stage[main]/Osnailyfacter::Ceph::Ceph_compute/Ceph::Key[client.compute]/Exec[ceph-injectkey-client.compute]/returns) Error connecting to cluster: ObjectNotFound

[1] http://docs.ceph.com/docs/hammer/start/hardware-recommendations/#hard-disk-drives

> Fuel tries to alter auth rules before fetching the client.admin keyring:

> $ ls -1 node-2/etc/ceph/
> ceph.client.compute.keyring
> ceph.conf
> rbdmap

> There's no ceph.client.admin.keyring so any ceph command is deemed to fail

Reassigning to MOS puppet team

> There's no ceph.client.admin.keyring so any ceph command is deemed to fail

There's a nice tool called ceph-deploy which handles a lot of such tiny details, please use that instead of reinventing the wheel

Fix proposed to branch: master
Review: https://review.openstack.org/315486

fix is merged, status changed to Fix Committed.
We are going to verify the fix.

Reviewed: https://review.openstack.org/315486
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=4790ef88dedd2d1d1f90477997224eda70ae8f39
Submitter: Jenkins
Branch: master

commit 4790ef88dedd2d1d1f90477997224eda70ae8f39
Author: Oleksiy Molchanov <email address hidden>
Date: Thu May 12 14:07:53 2016 +0300

    Add admin key to compute nodes

    * Add admin-key file to compute nodes in order to inject compute keys
    * Remove keys/fsid defaults and add validation

    Change-Id: I25774dbcc3beb00e6795c3cc29ab4c346a7af221
    Closes-Bug: 1578566