Fuel 4.1 HA CentOS. Post-deployment. Upload cirros image failed.

Bug #1306525 reported by Valeriy Saharov
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Invalid
Medium
Fuel Library (Deprecated)

Bug Description

Description of the environment:
{"ostf_sha": "de0222fed646525d248dc6892eeceab139d5c469", "fuelmain_sha": "", "astute_sha": "55df06b2e84fa5d71a1cc0e78dbccab5db29d968", "release": "4.1", "nailgun_sha": "7a05e365240ab27c492b20585ef8ac8557102cc0", "fuellib_sha": "52e7f57695f33bafa5d84d524d77f1bc3a2289b2", "mirantis": "no"}
CentOS 6.4, HA, 3 nodes (only controllers), 2 NIC on each node (eth0 - PXE, eth1 - Public, Storage (vlan tag), Management (vlan tag)), no RAID, no Storage nodes.

Steps to reproduce:
1. Create a new cluster
2. Select HA mode
3. Run cluster installation

Result:
In orchestrator's log error:
2014-04-10 15:59:03 ERR
[9282] Error running RPC method deploy: Upload cirros image failed, trace: ["/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/astute-0.0.2/lib/astute/orchestrator.rb:381:in `upload_cirros_image'", "/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/astute-0.0.2/lib/astute/orchestrator.rb:48:in `deploy'", "/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/naily-0.1.0/lib/naily/dispatcher.rb:101:in `deploy'", "/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/naily-0.1.0/lib/naily/server.rb:124:in `dispatch_message'", "/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/naily-0.1.0/lib/naily/server.rb:87:in `block in dispatch'", "/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/naily-0.1.0/lib/naily/task_queue.rb:59:in `call'", "/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/naily-0.1.0/lib/naily/task_queue.rb:59:in `block in each'", "/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/naily-0.1.0/lib/naily/task_queue.rb:52:in `each'", "/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/naily-0.1.0/lib/naily/task_queue.rb:52:in `each'", "/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/naily-0.1.0/lib/naily/server.rb:85:in `each_with_index'", "/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/naily-0.1.0/lib/naily/server.rb:85:in `dispatch'", "/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/naily-0.1.0/lib/naily/server.rb:70:in `block in perform_main_job'"]
2014-04-10 15:59:03 INFO
[9282] Casting message to fuel: {"method"=>"deploy_resp", "args"=>{"task_uuid"=>"0988fe03-000e-482f-9b4c-9fb702408d8c", "nodes"=>[{"uid"=>"13", "status"=>"error", "error_type"=>"deploy", "role"=>"primary-controller"}]}}
2014-04-10 15:59:03 DEBUG
[9282] Data send by DeploymentProxyReporter to report it up: {"nodes"=>[{"uid"=>"13", "status"=>"error", "error_type"=>"deploy", "role"=>"primary-controller"}]}
2014-04-10 15:59:03 DEBUG
[9282] Data received by DeploymentProxyReporter to report it up: {"nodes"=>[{"uid"=>"13", "status"=>"error", "error_type"=>"deploy", "role"=>"primary-controller"}]}
2014-04-10 15:59:03 ERR
[9282] 0988fe03-000e-482f-9b4c-9fb702408d8c: Upload cirros image failed
2014-04-10 15:59:03 DEBUG
[9282] 0988fe03-000e-482f-9b4c-9fb702408d8c: cmd: /usr/bin/glance -N http://192.168.0.2:5000/v2.0/ -T admin -I admin -K admin image-create --name 'TestVM' --is-public true --container-format='bare' --disk-format='qcow2' --property murano_image_info='{"title": "Murano Demo", "type": "cirros.demo"}' --file '/opt/vm/cirros-x86_64-disk.img'
                                               stdout:
                                               stderr: Request returned failure status. 400 Bad Request
Client disconnected before sending all data to backend
    (HTTP 400) exit code: 1

Impact:
Br-ex on primary controller doesn't get public IP address or drop it.

Revision history for this message
Valeriy Saharov (vsakharov) wrote :
Mike Scherbakov (mihgen)
Changed in fuel:
milestone: none → 5.0
assignee: nobody → Fuel Library Team (fuel-library)
Revision history for this message
Vladimir Kuklin (vkuklin) wrote :

Valeriy. It seems that assignment of br-ex IP addresses failed on all the controllers:

2014-04-10T15:10:59.823912+00:00 notice: (L3_if_downup[br-ex](provider=ruby)) Interface 'br-ex' down.
2014-04-10T15:10:59.823912+00:00 notice: (L3_if_downup[br-ex](provider=ruby)) Interface 'br-ex' flush.
2014-04-10T15:10:59.823912+00:00 notice: (L3_if_downup[br-ex](provider=ruby)) Can't put interface 'br-ex' to UP state.

This is very weird. We'll try to investigate the issue, but I am not sure that it is easy to reproduce.

Changed in fuel:
status: New → Confirmed
importance: Undecided → Medium
Revision history for this message
Sergey Vasilenko (xenolog) wrote :

it looks as https://bugs.launchpad.net/fuel/+bug/1308483

please ssh to one of failed node, execute 'ifdown br-ex ; echo $?; sleep 10 ; ifup br-ex ; echo $?'
and attach all output to this ticket.

Do not use ip address from br-ex for ssh in this case.

Revision history for this message
Valeriy Saharov (vsakharov) wrote :

Problem was solved.
After careful analysis was detected a conflict in the network address with other labs.

Changed in fuel:
status: Confirmed → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.