Buildd-manager should deal with transient communication failures with builders
Bug #369109 reported by
Celso Providelo
This bug affects 2 people
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| Launchpad itself |
Fix Released
|
High
|
Julian Edwards | ||
Bug Description
As described in bug #343683 and bug #31546, builders are subjected to transient communication failures and shouldn't be immediately marked as NOT_OK when it happens.
Instead we should allow an acceptable period of unavailability before excluding the builder from the poll, this way builders victims of network hiccups, which have continue with their jobs won't mistakenly be reset.
On the flip-side, builders excluded from the poll should be re-probed periodically, so they are automatically made available again once whatever was preventing them to build was fixed.
Related branches
lp://qastaging/~julian-edwards/launchpad/buildd-failure-counting
- Jonathan Lange (community): Approve
- Stuart Bishop (community): Approve (db)
- Robert Collins: Pending (db) requested
- Diff: 0 lines
| tags: | added: buildd-manager |
Revision history for this message
| Julian Edwards (julian-edwards) wrote | #1 |
| Changed in soyuz: | |
| assignee: | Celso Providelo (cprov) → nobody |
| Changed in soyuz: | |
| status: | Triaged → In Progress |
| assignee: | nobody → Julian Edwards (julian-edwards) |
| milestone: | pending → none |
Revision history for this message
| Launchpad QA Bot (lpqabot) wrote Bug fixed by a commit | #2 |
| Changed in soyuz: | |
| milestone: | none → 10.09 |
| tags: | added: qa-needstesting |
| Changed in soyuz: | |
| status: | In Progress → Fix Committed |
Revision history for this message
| Julian Edwards (julian-edwards) wrote | #3 |
| tags: |
added: qa-untestable removed: qa-needstesting |
| Changed in soyuz: | |
| status: | Fix Committed → Fix Released |
To post a comment you must log in.
Today we had this in the log:
2010-04-15 23:53:23+0100 [-] Starting scanning cycle.
2010-04-15 23:56:46+0100 [-] Disabling builder: http://
2010-04-15 23:56:46+0100 [-] Traceback (most recent call last):
2010-04-15 23:56:46+0100 [-] File "/srv/launchpad
2010-04-15 23:56:46+0100 [-] builder.
2010-04-15 23:56:46+0100 [-] File "/srv/launchpad
2010-04-15 23:56:46+0100 [-] if self.slave.
2010-04-15 23:56:46+0100 [-] File "/usr/lib/
2010-04-15 23:56:46+0100 [-] return self.__
2010-04-15 23:56:46+0100 [-] File "/usr/lib/
2010-04-15 23:56:46+0100 [-] verbose=
2010-04-15 23:56:46+0100 [-] File "/usr/lib/
2010-04-15 23:56:46+0100 [-] errcode, errmsg, headers = h.getreply()
2010-04-15 23:56:46+0100 [-] File "/usr/lib/
2010-04-15 23:56:46+0100 [-] response = self._conn.
2010-04-15 23:56:46+0100 [-] File "/usr/lib/
2010-04-15 23:56:46+0100 [-] response.begin()
2010-04-15 23:56:46+0100 [-] File "/usr/lib/
2010-04-15 23:56:46+0100 [-] version, status, reason = self._read_status()
2010-04-15 23:56:46+0100 [-] File "/usr/lib/
2010-04-15 23:56:46+0100 [-] line = self.fp.readline()
2010-04-15 23:56:46+0100 [-] File "/usr/lib/
2010-04-15 23:56:46+0100 [-] data = recv(1)
2010-04-15 23:56:46+0100 [-] timeout: timed out
It seems as though there are two competing ways of timing stuff out.
1. the code in lib/lp/
2. lib/lp/
Different actions seem to cause timeouts in each of these. This is crap.
It also seems as though the updateBuilderSt