[IT][Vanilla2] The check for active "nodemanager" nodes does not work

Bug #1300248 reported by Yaroslav Lobankov
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Sahara
Fix Released
Medium
Sergey Reshetnyak

Bug Description

Now integration tests execute command 'sudo -u hadoop bash -lc "hadoop job -list-active-trackers"' to check how many "tasktracker" nodes are active.

This command does not work with any Vanilla2 images created with elements from sahara-image-elements repo.

Trace of tests is
======================================================================
FAIL: tests.gating.test_vanilla_two_gating.VanillaTwoGatingTest.test_vanilla_two_plugin_gating[vanilla2]
tags: worker-0
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/skolekonov/sahara/sahara/tests/integration/tests/gating/test_vanilla_two_gating.py", line 227, in test_vanilla_two_plugin_gating
    self._create_cluster()
  File "sahara/tests/integration/tests/base.py", line 46, in wrapper
    ITestCase.print_error_log(message, e)
  File "sahara/openstack/common/excutils.py", line 68, in __exit__
    six.reraise(self.type_, self.value, self.tb)
  File "sahara/tests/integration/tests/base.py", line 43, in wrapper
    fct(*args, **kwargs)
  File "/home/skolekonov/sahara/sahara/tests/integration/tests/gating/test_vanilla_two_gating.py", line 157, in _create_cluster
    self.vanilla_two_config)
  File "sahara/tests/integration/tests/base.py", line 338, in await_active_workers_for_namenode
    % self.common_config.HDFS_INITIALIZATION_TIMEOUT
  File "/home/skolekonov/sahara/.tox/integration/local/lib/python2.7/site-packages/unittest2/case.py", line 415, in fail
    raise self.failureException(msg)
AssertionError: Tasktracker or datanode cannot be started within 5 minute(s) for namenode.

Output of command 'sudo -u hadoop bash -lc "hadoop job -list-active-trackers"' may be as follows
"Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library /opt/hadoop-2.3.0/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.\nIt's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.\ntracker_sk-test-vanilla-v2-worker-node-nm-dn-002.novalocal:35697\ntracker_sk-test-vanilla-v2-worker-node-nm-001.novalocal:46901\ntracker_sk-test-vanilla-v2-worker-node-nm-dn-001.novalocal:59270\n"

A possible solution is to use the following command 'sudo -u hadoop bash -lc "hadoop job -list-active-trackers" | grep tracker_'.
Output of command 'sudo -u hadoop bash -lc "hadoop job -list-active-trackers" | grep tracker_' is
"tracker_sk-test-vanilla-v2-worker-node-nm-dn-002.novalocal:35697\ntracker_sk-test-vanilla-v2-worker-node-nm-001.novalocal:46901\ntracker_sk-test-vanilla-v2-worker-node-nm-dn-001.novalocal:59270\n"

Changed in sahara:
assignee: nobody → Sergey Reshetnyak (sreshetniak)
summary: - [IT][Vanilla2] The check for active data node does not work
+ [IT][Vanilla2] The check for active "tasktracker" nodes does not work
description: updated
summary: - [IT][Vanilla2] The check for active "tasktracker" nodes does not work
+ [IT][Vanilla2] The check for active "nodemanager" nodes does not work
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to sahara (master)

Fix proposed to branch: master
Review: https://review.openstack.org/84228

Changed in sahara:
status: New → In Progress
Changed in sahara:
importance: Undecided → Low
milestone: none → icehouse-rc1
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to sahara (master)

Reviewed: https://review.openstack.org/84228
Committed: https://git.openstack.org/cgit/openstack/sahara/commit/?id=ecffd9fb5c6024ba6d9a56ca03f85a120884f17a
Submitter: Jenkins
Branch: master

commit ecffd9fb5c6024ba6d9a56ca03f85a120884f17a
Author: Sergey Reshetnyak <email address hidden>
Date: Mon Mar 31 23:51:49 2014 +0400

    Fix check active nodemanagers for vanilla 2 plugin

    Fixes bug: #1300248

    Change-Id: I9afdded0ef47643aaf2ade8a8921ee37e3040fc8

Changed in sahara:
status: In Progress → Fix Committed
Changed in sahara:
importance: Low → Medium
Thierry Carrez (ttx)
Changed in sahara:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in sahara:
milestone: icehouse-rc1 → 2014.1
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.