toci_quickstart.sh: recording the size of logs directory can fail with "sort: write error"

Bug #1731289 reported by Attila Darazs
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
Medium
Unassigned

Bug Description

Example:

http://logs.openstack.org/48/518548/1/gate/legacy-tripleo-ci-centos-7-undercloud-oooq/da2d936/job-output.txt.gz#_2017-11-09_00_40_50_055344

2017-11-09 00:40:49.970367 | primary | +(./toci_quickstart.sh:109): du -L -ch /home/zuul/workspace/logs/ara_oooq /home/zuul/workspace/logs/delorean_logs /home/zuul/workspace/logs/devstack-gate-setup-host.txt /home/zuul/workspace/logs/devstack-gate-setup-workspace-new.txt /home/zuul/workspace/logs/full_logs.html /home/zuul/workspace/logs/postci.txt.gz /home/zuul/workspace/logs/quickstart_collect_logs.log /home/zuul/workspace/logs/quickstart_install.log /home/zuul/workspace/logs/README-logs.html /home/zuul/workspace/logs/reproduce.sh /home/zuul/workspace/logs/testrepository.subunit.gz /home/zuul/workspace/logs/undercloud
2017-11-09 00:40:49.970509 | primary | +(./toci_quickstart.sh:109): sort -rh
2017-11-09 00:40:49.970575 | primary | +(./toci_quickstart.sh:109): head -n 200
2017-11-09 00:40:50.055214 | primary | sort: write failed: standard output: Broken pipe
2017-11-09 00:40:50.055344 | primary | sort: write error

This is the problematic line: https://github.com/openstack-infra/tripleo-ci/blob/701ff27cfb11897a0c998ea6d3556fa94bfd5634/toci_quickstart.sh#L109

My suspicion is that the sorting happens in the /tmp directory which could get full (or has some other problem?) and the sort operation fails.

Possible solutions are:

* using `sort -T <dir>` and point it somewhere where there is more space.
* reduce the amount of files that we check by using `du -d 1` for example

Tags: ci
Attila Darazs (adarazs)
Changed in tripleo:
importance: High → Medium
description: updated
Revision history for this message
wes hayutin (weshayutin) wrote :

This error does not impact the success of the job FYI

wes hayutin (weshayutin)
Changed in tripleo:
milestone: none → queens-3
Revision history for this message
wes hayutin (weshayutin) wrote :

need to look at this again, but this was an attempt to resolve the error
https://review.openstack.org/#/c/491884/

Changed in tripleo:
milestone: queens-3 → queens-rc1
Changed in tripleo:
milestone: queens-rc1 → rocky-1
Changed in tripleo:
milestone: rocky-1 → rocky-2
Changed in tripleo:
milestone: rocky-2 → rocky-3
Changed in tripleo:
milestone: rocky-3 → rocky-rc1
Changed in tripleo:
milestone: rocky-rc1 → stein-1
Changed in tripleo:
milestone: stein-1 → stein-2
Changed in tripleo:
milestone: stein-2 → stein-3
Changed in tripleo:
status: Triaged → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.