Fuel for OpenStack

Paramiko error <type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'error' occured after executing tests with fuel-qa

Bug #1587554 reported by Sergey Kreys on 2016-05-31

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	Fuel for OpenStack	Won't Fix	High	Sergey Kreys	Fuel for OpenStack 9.0
	Mitaka	In Progress	High	Sergey Kreys	Fuel for OpenStack 9.x-updates

Bug Description

Sometimes it's just a bunch of errors, but sometimes it fails with segfault which leads to test failure.
Note: more SSH connections used in the test -> more probability to reproduce the issue.
Thus, more nodes in the environment - more chance to hit this bug as well.

Please find logs from 'failed' test attached.

This bug is related to known issues:
paramiko: https://github.com/paramiko/paramiko/issues/17
python: http://bugs.python.org/issue1856

We believe it's possible to avoid this bug if we will close SSH connections opened by fuel-qa & fuel-devops.
There is method clean_all_connections() in fuel-qa, but it is called in resume_environment() only.
And in fuel-devops used SSH connections are not being closed at some places, e.g. in sync_time() function.

Tags:

Revision history for this message

Sergey Kreys (skreys) wrote on 2016-05-31:

Logs from failed test Edit (24.8 KiB, text/plain)

Revision history for this message

Sergey Kreys (skreys) wrote on 2016-05-31:

It's the same issue as described in https://bugs.launchpad.net/fuel/+bug/1528579, but it is NOT FIXED yet.

OpenStack Infra (hudson-openstack) on 2016-05-31

Changed in fuel:
assignee:	nobody → Sergey Kreys (skreys)
status:	New → In Progress

Ilya Kutukov (ikutukov) on 2016-05-31

Changed in fuel:
milestone:	none → 9.0
importance:	Undecided → High
tags:	added: non-release

Revision history for this message

Alexey Stepanov (astepanov-m) wrote on 2016-06-10:

1. Python crush is resolver in 2014 (update python on Ubuntu 14.0, it's old and buggy)
2.
On Fuel-Devops 3.0 memorization for SSHClient is arrived, currently we seriously reduced amount of ssh sessions and introduced session close for nodes on shutdown/destroy/suspend/revert call.
On Fuel-Devops 2.9 this changes in adoption state (simple cherry-pick is impossible due to significant changes in driver model).
3. Close all session on process close in investigation state (we could close process without shutdown/suspend and snapshot node states, probably fix will be in fuel-qa, but only after requirements bump).

Revision history for this message

Sergey Kreys (skreys) wrote on 2016-06-10:

Thanks for the information. BTW, we use Fuel-Devops 2.9.x, so the back port will be highly appreciated.
As a temporary solution for fuel-qa, now we close all ssh connections stored in SSHManager after all tests are done.

Regarding python crash - it's not resolved in python2.7, fix for http://bugs.python.org/issue1856 was reverted because of regression in cinder. And I'm pretty sure that we're not gonna use specific python version that contains the commit for bug not contains reverting commit.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-07-12: Change abandoned on fuel-devops (master)

Change abandoned by Fuel DevOps Robot (<email address hidden>) on branch: master
Review: https://review.openstack.org/323502
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

Roman Vyalov (r0mikiam) on 2017-02-03