No any error message in Fuel UI if CEPH puppet 'Create compute pool' failed

Bug #1660962 reported by Sergey Galkin
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Invalid
High
Fuel Sustaining
Mitaka
Invalid
High
Fuel Sustaining
Newton
Invalid
High
Fuel Sustaining

Bug Description

Steps for reproduce:
1. Install 9.0
2. Upgrade to 9.2
3. Deploy big cluster (~400 nodes in my case) with CEPH and incorrect configured vlan for storage network on the some nodes

In my case this node is node-134

This node has a lot of messages in /var/log/astute/astute.log like

2017-02-01 12:27:06 DEBUG [8419] Node[134]: Node 134: task ceph-compute, task status running
2017-02-01 12:27:07 DEBUG [8419] Node 134, task ceph-compute, manifest /etc/puppet/modules/osnailyfacter/modular/ceph/ceph_compute.pp, status: running
2017-02-01 12:27:07 DEBUG [8419] Reset undefined retries to original value: 3

In the puppet.log on the node-134 a lot of
2017-02-01 12:24:37 +0000 /Stage[main]/Osnailyfacter::Ceph::Ceph_compute/Ceph::Pool[compute]/Exec[Create compute pool]/unless (err): Check "rados lspools | grep -q '^compute$'" exceeded timeout

Output from 'rados lspools' on node-134 is
root@node-134:~# rados lspools
2017-02-01 12:32:41.132668 7f479466e700 0 -- :/1913953563 >> 10.31.0.49:6789/0 pipe(0x7f479764f270 sd=3 :0 s=1 pgs=0 cs=0 l=1 c=0x7f479764b360).fault
2017-02-01 12:32:44.132496 7f479426d700 0 -- :/1913953563 >> 10.31.0.22:6789/0 pipe(0x7f4780000c00 sd=5 :0 s=1 pgs=0 cs=0 l=1 c=0x7f4780004ef0).fault
2017-02-01 12:32:47.132551 7f479466e700 0 -- :/1913953563 >> 10.31.0.49:6789/0 pipe(0x7f47800081b0 sd=5 :0 s=1 pgs=0 cs=0 l=1 c=0x7f478000c450).fault

After this node-134 added successfully added to cluster

end of /var/log/astute/astute.log for node-134

2017-02-01 12:44:42 DEBUG [8419] Node 134, task ceph-compute, manifest /etc/puppet/modules/osnailyfacter/modular/ceph/ceph_compute.pp, status: running
2017-02-01 12:44:42 DEBUG [8419] Cluster[]: Process node: Node[134]
2017-02-01 12:44:42 DEBUG [8419] Node[134]: Node 134: task ceph-compute, task status running
2017-02-01 12:44:42 DEBUG [8419] Node 134, task ceph-compute, manifest /etc/puppet/modules/osnailyfacter/modular/ceph/ceph_compute.pp, status: succeed
2017-02-01 12:44:42 INFO [8419] Node 134 has succeed to deploy. Node 134, task ceph-compute, manifest /etc/puppet/modules/osnailyfacter/modular/ceph/ceph_compute.pp
2017-02-01 12:44:42 INFO [8419] Node[134]: Finished task: Task[ceph-compute/134] with status: busy
2017-02-01 12:44:42 DEBUG [8419] Node[134]: Decreasing node concurrency to: 0

End of puppet.log has error
http://paste.openstack.org/show/597167/

Tags: scale
Sergey Galkin (sgalkin)
summary: - Nodes without storage network successfully added to cluster
+ No any error message in Fuel UI if CEPH puppet 'Create compute pool'
+ failed
Changed in fuel:
assignee: nobody → Fuel Sustaining (fuel-sustaining-team)
milestone: none → 9.x-updates
Dmitry Pyzhov (dpyzhov)
Changed in fuel:
milestone: 9.x-updates → 11.0
importance: Undecided → High
Revision history for this message
Oleksiy Molchanov (omolchanov) wrote :

By design, timeout of puppet exec unless command doesn't lead to failure.

Changed in fuel:
status: New → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.