Improve exceptions of decorator @log_snapshot_on_error

Yegor, do you already have ideas how we can fix it?

It could be failed on any step of admin_node installation, so we should extend the method 'create_diagnostic_snapshot' or make an alternative method, getting an archive of necessary logs thru ssh instead of nailgun.

We could save:

/var/log/anaconda.* #about master node provisioning

/var/log/puppet/bootstrap_admin_node.log #about bootstrapping process

/var/log/supervisor/supervisord.log #about containers
and
/var/log/docker*.log

So we may need to do next here:
1. If test destroy nailgun container - we receive exception and it is expected, so you need to implement some kind of exception - then catch it and copy and tar /var/log directory (or may be invoke shotgun to generate snapshot)
2. for case when admin node installation fail - we should use libguestfs to extract the log data (we have separate issue for this case)

For second item we have https://bugs.launchpad.net/fuel/+bug/1318059

Fix proposed to branch: master
Review: https://review.openstack.org/136368

Fix proposed to branch: stable/5.1
Review: https://review.openstack.org/136378

Reviewed: https://review.openstack.org/136368
Committed: https://git.openstack.org/cgit/stackforge/fuel-main/commit/?id=5d31addc064234e64f66822dcf2012d9c357a7f1
Submitter: Jenkins
Branch: master

commit 5d31addc064234e64f66822dcf2012d9c357a7f1
Author: Artem Panchenko <email address hidden>
Date: Fri Nov 21 15:42:51 2014 +0200

    Improve tests exceptions handling

    If test fails and then creation of diagnostic
    snapshot using nailgun also fails, try to get
    logs stored on master node via SSH.

    Change-Id: I5456616cf7088210860a4d4a786531333070f3e5
    Closes-bug: #1389287

Reviewed: https://review.openstack.org/136378
Committed: https://git.openstack.org/cgit/stackforge/fuel-main/commit/?id=dc83cfb238e5a55feac3fdaa04ea66805edb5f6f
Submitter: Jenkins
Branch: stable/5.1

commit dc83cfb238e5a55feac3fdaa04ea66805edb5f6f
Author: Artem Panchenko <email address hidden>
Date: Fri Nov 21 15:42:51 2014 +0200

    Improve tests exceptions handling

    If test fails and then creation of diagnostic
    snapshot using nailgun also fails, try to get
    logs stored on master node via SSH.

    Change-Id: I5456616cf7088210860a4d4a786531333070f3e5
    Closes-bug: #1389287

Related fix proposed to branch: master
Review: https://review.openstack.org/136755

We still need one more improvement: if an exception is caught in @log_snapshot_on_error decorator, we perform additional actions (downloading logs, creating snapshots) and then raise an exception again, but it has modified traceback. For example:

Traceback (most recent call last):
  File "/usr/lib/python2.7/unittest/case.py", line 331, in run
    testMethod()
  File "/usr/lib/python2.7/unittest/case.py", line 1043, in runTest
    self._testFunc()
  File "/usr/lib/python2.7/dist-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
    compatability.capture_type_error(s_func)
  File "/usr/lib/python2.7/dist-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
    func()
  File "/usr/lib/python2.7/dist-packages/proboscis/case.py", line 350, in func
    func(test_case.state.get_state())
  File "/home/jenkins/workspace/5.1_fuelmain.system_test.ubuntu.thread_3/fuelweb_test/helpers/decorators.py", line 78, in wrapper
    raise test_exception
TimeoutError: Waiting timed out

So, printing of original traceback is required.

Fix proposed to branch: master
Review: https://review.openstack.org/137081

Fix proposed to branch: stable/5.1
Review: https://review.openstack.org/137082

Reviewed: https://review.openstack.org/137081
Committed: https://git.openstack.org/cgit/stackforge/fuel-main/commit/?id=e3f1d2cd35ff8479d7ad0c2811880764be96bb6f
Submitter: Jenkins
Branch: master

commit e3f1d2cd35ff8479d7ad0c2811880764be96bb6f
Author: Artem Panchenko <email address hidden>
Date: Tue Nov 25 15:51:03 2014 +0200

    Log traceback of test exception that's caught

    Print traceback of original test exception. This
    will help to understand on what step test fails.

    Change-Id: Ia9a2f396d267141cfa164d11e4ebe1fecfee524b
    Partial-Bug: #1389287

Reviewed: https://review.openstack.org/137082
Committed: https://git.openstack.org/cgit/stackforge/fuel-main/commit/?id=7a7bdfc1051c270c4d5e3acb6cd3edc3d2f234e8
Submitter: Jenkins
Branch: stable/5.1

commit 7a7bdfc1051c270c4d5e3acb6cd3edc3d2f234e8
Author: Artem Panchenko <email address hidden>
Date: Tue Nov 25 15:51:03 2014 +0200

    Log traceback of test exception that's caught

    Print traceback of original test exception. This
    will help to understand on what step test fails.

    Change-Id: Ia9a2f396d267141cfa164d11e4ebe1fecfee524b
    Partial-Bug: #1389287

Reviewed: https://review.openstack.org/136755
Committed: https://git.openstack.org/cgit/stackforge/fuel-devops/commit/?id=db7e0d574ee777780e99a6f3644efe30208c1f47
Submitter: Jenkins
Branch: master

commit db7e0d574ee777780e99a6f3644efe30208c1f47
Author: Artem Panchenko <email address hidden>
Date: Mon Nov 24 15:36:07 2014 +0200

    Add download method to SSHClient

    Add possibility to download files via SCP
    from remote host.

    Change-Id: I1601236f6b265c3aa785bc27b512389d4e361cd4
    Related-bug: #1389287