lava-test fails to complete for beagle board hwpack

Bug #845720 reported by Deepti B. Kalakeri
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
LAVA Dispatcher
Fix Released
Medium
Paul Larson

Bug Description

lava-test failed with the following error when trying to install and test the beagle board hwpack.

root@master:~# umount /mnt/rootLava failed at action lava_test_install with error: lava-test deployment failed
Traceback (most recent call last):
  File "/usr/lib/pymodules/python2.6/lava_dispatcher/__init__.py", line 70, in run
    action.run(**params)
  File "/usr/lib/pymodules/python2.6/lava_dispatcher/actions/lava-test.py", line 122, in run
    _install_lava_test(client)
  File "/usr/lib/pymodules/python2.6/lava_dispatcher/actions/lava-test.py", line 97, in _install_lava_test
    raise OperationFailed("lava-test deployment failed")
OperationFailed: lava-test deployment failed

here is the reference for the test job where the problem occured.
http://validation.linaro.org/lava-server/scheduler/job/1450

Revision history for this message
Spring Zhang (qzhang) wrote :

Yes, sometimes it may fail, and we don't try to run the following case by lava-run, see bug 844299

Revision history for this message
Spring Zhang (qzhang) wrote :

The handler process is duplicated with #844299, but if for why lava-test deployment failed, need to run with some debug info opened.

Revision history for this message
Deepti B. Kalakeri (deeptik) wrote :

How to enable the debug option ? Do we need to enable it while running the lava-tool submit-job ?

Thanks!!!
Deepti.

Revision history for this message
Le Chi Thu (le-chi-thu) wrote : Re: [Bug 845720] Re: lava-test fails to complete for beagle board hwpack

Lava-test help command can take longer than 10 seconds. Please increase it
to 2-3 minutes.

#Test if lava-test installed
    try:
        client.run_shell_command(
            'chroot /mnt/root lava-test help',
            response="list-test", timeout=10)
BR

/Chi Thu

Revision history for this message
Paul Larson (pwlars) wrote :

I already patched this on the server to 1 min. to see if that will be sufficient, but asked deepti to open the bug so that we track it and make sure the fix goes into the version we release this month. Deepti, please let us know if you see any more problems with it.

Changed in lava-dispatcher:
assignee: nobody → Paul Larson (pwlars)
importance: Undecided → Medium
status: New → In Progress
milestone: none → 2011.09
Revision history for this message
Deepti B. Kalakeri (deeptik) wrote :

I tried the tests again and this time it failed with

root@master:/tmp#

Traceback (most recent call last):
  File "/usr/bin/lava-dispatch", line 49, in <module>
    job.run()
  File "/usr/lib/pymodules/python2.6/lava_dispatcher/__init__.py", line 103, in run
    action.run(**params)
  File "/usr/lib/pymodules/python2.6/lava_dispatcher/actions/launch_control.py", line 137, in run
    srv.put_ex(json_bundle, 'lava-dispatcher.bundle', stream)
  File "/usr/lib/python2.6/xmlrpclib.py", line 1199, in __call__
    return self.__send(self.__name, args)
  File "/usr/lib/python2.6/xmlrpclib.py", line 1489, in __request
    verbose=self.__verbose
  File "/usr/lib/python2.6/xmlrpclib.py", line 1235, in request
    self.send_content(h, request_body)
  File "/usr/lib/python2.6/xmlrpclib.py", line 1349, in send_content
    connection.endheaders()
  File "/usr/lib/python2.6/httplib.py", line 908, in endheaders
    self._send_output()
  File "/usr/lib/python2.6/httplib.py", line 780, in _send_output
    self.send(msg)
  File "/usr/lib/python2.6/httplib.py", line 739, in send
    self.connect()
  File "/usr/lib/python2.6/httplib.py", line 720, in connect
    self.timeout)
  File "/usr/lib/python2.6/socket.py", line 561, in create_connection
    raise error, msg
socket.error: [Errno 110] Connection timed out

Thanks!!!
Deepti.

Revision history for this message
Paul Larson (pwlars) wrote :

Deepti, could you please open a new bug on that and point at the job where you got that? This is something completely different, and indicates that it failed to connect to the dashboard to upload the results. Go ahead and open it against the dispatcher for now, as it certainly needs to retry that connection if it fails. However we should also investigate what was going on at that time. Do you know about when this happened?

Revision history for this message
Deepti B. Kalakeri (deeptik) wrote :

I am able to get the test run successfully completed.
But there are times when the tests fail intermittently with the error like the below:

root@master:/tmp# ifconfig eth0 | grep 'inet addr' | awk -F: '{print $2}' |awk '{print $1}'
ifconfig eth0 | grep 'inet addr' | awk -F: '{print $2}' |awk '
�{print $1}'

192.168.1.121

root@master:/tmp#
id=0
python -m SimpleHTTPServer 80 &> /dev/null &
Traceback (most recent call last):
  File "/usr/bin/lava-dispatch", line 49, in <module>
    job.run()
  File "/usr/lib/pymodules/python2.6/lava_dispatcher/__init__.py", line 103, in run
    action.run(**params)
  File "/usr/lib/pymodules/python2.6/lava_dispatcher/actions/launch_control.py", line 109, in run
    result_path = download(result_tarball, tarball_dir)
  File "/usr/lib/pymodules/python2.6/lava_dispatcher/utils.py", line 41, in download
    raise RuntimeError("Could not retrieve %s" % url)
RuntimeError: Could not retrieve http://192.168.1.121/lava_results.tgz

The above error can be seen in the job http://validation.linaro.org/lava-server/scheduler/job/1569
I will open a new bug if the problem persists and the frequency is more often, but for now we can close this bug.
Do I need to open a new bug for the above error ?

Thanks!!!
Deepti.

Revision history for this message
Paul Larson (pwlars) wrote :

I increased the timeout to 60 seconds and it seems to be a good fit.

Changed in lava-dispatcher:
status: In Progress → Fix Committed
Fathi Boudra (fboudra)
Changed in lava-dispatcher:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.