crm_attribute could not establish cib_rw connection

Bug #1281468 reported by Nikolay Fedotov
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Invalid
Medium
Fuel Library (Deprecated)

Bug Description

ISO: {"build_id": "2014-02-17_01-17-30", "mirantis": "no", "build_number": "128", "nailgun_sha": "7585c9538fc7cae779aee2389cb6f03b3f611857", "ostf_sha": "f86abe5544b5ffcf621e0c450bca15737c92361f", "fuelmain_sha": "439577ed2795d86d57351d6e7cec6ed0049101c9", "astute_sha": "7eed50fc30cec675fff7787c37fcf6da6dd518ee", "release": "4.1", "fuellib_sha": "8e0b1ae5b1c4c137c1dd2a0be06d0d68e99d75bf"}

Steps:
- Create environment: CentOS, Multi-node with HA, Nova network
- Add nodes: 3 controller, 2 computes
- deploy changes

Result:
errors in puppet on a controller node
<27>Feb 17 23:48:05 node-4 puppet_err /usr/lib/ruby/1.8/timeout.rb:64:in `block_until_ready': execution expired (Timeout::Error)
<27>Feb 17 23:48:05 node-4 puppet_err from /etc/puppet/modules/corosync/lib/puppet/provider/corosync.rb:48:in `block_until_ready'
<27>Feb 17 23:48:05 node-4 puppet_err from /etc/puppet/modules/corosync/lib/puppet/provider/cs_property/crm.rb:19:in `instances'
<27>Feb 17 23:48:05 node-4 puppet_err from /etc/puppet/modules/corosync/lib/puppet/provider/corosync.rb:61:in `prefetch'
<27>Feb 17 23:48:05 node-4 puppet_err from /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:286:in `prefetch'
<27>Feb 17 23:48:05 node-4 puppet_err from /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:274:in `prefetch_if_necessary'
<27>Feb 17 23:48:05 node-4 puppet_err from /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:379:in `traverse'
<27>Feb 17 23:48:05 node-4 puppet_err from /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:99:in `evaluate'
<27>Feb 17 23:48:05 node-4 puppet_err from /usr/lib/ruby/site_ruby/1.8/puppet/resource/catalog.rb:141:in `apply'
<27>Feb 17 23:48:05 node-4 puppet_err from /usr/lib/ruby/site_ruby/1.8/puppet/configurer.rb:122:in `retrieve_and_apply_catalog'
<27>Feb 17 23:48:05 node-4 puppet_err from /usr/lib/ruby/site_ruby/1.8/puppet/util.rb:161:in `benchmark'
<27>Feb 17 23:48:05 node-4 puppet_err from /usr/lib/ruby/1.8/benchmark.rb:308:in `realtime'
<27>Feb 17 23:48:05 node-4 puppet_err from /usr/lib/ruby/site_ruby/1.8/puppet/util.rb:160:in `benchmark'
<27>Feb 17 23:48:05 node-4 puppet_err from /usr/lib/ruby/site_ruby/1.8/puppet/configurer.rb:121:in `retrieve_and_apply_catalog'
<27>Feb 17 23:48:05 node-4 puppet_err from /usr/lib/ruby/site_ruby/1.8/puppet/configurer.rb:152:in `run'
<27>Feb 17 23:48:05 node-4 puppet_err from /usr/lib/ruby/site_ruby/1.8/puppet/application/apply.rb:229:in `main'
<27>Feb 17 23:48:05 node-4 puppet_err from /usr/lib/ruby/site_ruby/1.8/puppet/application/apply.rb:149:in `run_command'
<27>Feb 17 23:48:05 node-4 puppet_err from /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:309:in `run'
<27>Feb 17 23:48:05 node-4 puppet_err from /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:416:in `hook'
<27>Feb 17 23:48:05 node-4 puppet_err from /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:309:in `run'
<27>Feb 17 23:48:05 node-4 puppet_err from /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:407:in `exit_on_fail'
<27>Feb 17 23:48:05 node-4 puppet_err from /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:309:in `run'
<27>Feb 17 23:48:05 node-4 puppet_err from /usr/lib/ruby/site_ruby/1.8/puppet/util/command_line.rb:69:in `execute'
<27>Feb 17 23:48:05 node-4 puppet_err from /usr/bin/puppet:4

corosync.rb:48 executes "crm_attribute --type crm_config --query --name dc-version" that returns:
Could not establish cib_rw connection: Connection refused (111)
Error signing on to the CIB service: Transport endpoint is not connected

Tags: library
Revision history for this message
Nikolay Fedotov (nfedotov) wrote :
Mike Scherbakov (mihgen)
Changed in fuel:
assignee: nobody → Fuel Library Team (fuel-library)
Changed in fuel:
status: New → In Progress
Revision history for this message
Bogdan Dobrelya (bogdando) wrote :

Looks like one of the controller nodes is missing iptables rules for corosync, thus dropping all connections to the CIB if selected for queries, see http://pastebin.com/mahQbLqm

Mike Scherbakov (mihgen)
Changed in fuel:
importance: High → Critical
Mike Scherbakov (mihgen)
Changed in fuel:
status: In Progress → New
Changed in fuel:
importance: Critical → Medium
status: New → Incomplete
Revision history for this message
Mike Scherbakov (mihgen) wrote :

Why incomplete? Please provide your observations.

Revision history for this message
Vladimir Kuklin (vkuklin) wrote :

Noone was able to reproduce and diagnostic snapshot lacks puppet logs.

Changed in fuel:
status: Incomplete → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.