fwts jobs run out of order

Hi Jeff,

I assume you were using server-full-14.04.whitelist? It has this (which looks correct):

__miscellanea__
miscellanea/fwts_test
miscellanea/fwts_results.log
miscellanea/ipmi_test

I also looked at the code and it *is* using select_jobs which should honor whitelist ordering.

However when doing the actual run, the tests seem to be reordered, in the selection list I see them as:

miscellanea/fwts_results.log
miscellanea/fwts_test

indeed, when they run, they do so in the order I showed, which is incorrect and not what the whitelist specifies.

My package versions are:

Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-===============================-====================-====================-====================================================================
un checkbox <none> <none> (no description available)
ii checkbox-ng 0.3-2 all PlainBox based test runner
ii plainbox 0.6~dev+bzr2922+pkg2 all toolkit for software and hardware integration testing
un plainbox-insecure-policy <none> <none> (no description available)
ii plainbox-provider-certification 0.1~dev+bzr2922+pkg8 all Server Certification
ii plainbox-provider-checkbox 0.5~dev+bzr2922+pkg2 amd64 CheckBox provider for PlainBox
ii plainbox-provider-resource-gene 0.4~dev+bzr2922+pkg5 amd64 CheckBox generic resource jobs provider
ii plainbox-secure-policy 0.6~dev+bzr2922+pkg2 all policykit policy required to use plainbox (secure version)
ii python3-checkbox-ng 0.3-2 all PlainBox based test runner (Python 3 library)
ii python3-checkbox-support 0.2-1 all collection of Python modules used by PlainBox providers
ii python3-plainbox 0.6~dev+bzr2922+pkg2 all toolkit for software and hardware testing (python3 module)

Prior to running the selection tree, the manager.state.desired_job_list contains the correct order. This is just the relevant fragment as the djl is originally a bit long:

 <JobDefinition id:'2013.com.canonical.certification::__miscellanea__' plugin:'local'>,
 <JobDefinition id:'2013.com.canonical.certification::miscellanea/fwts_test' plugin:'shell'>,
 <JobDefinition id:'2013.com.canonical.certification::miscellanea/fwts_results.log' plugin:'attachment'>,
 <JobDefinition id:'2013.com.canonical.certification::miscellanea/ipmi_test' plugin:'shell'>,

I selected only the miscellaneous tests in the tree selection, and the tree.selection now has an incorrect order:

djl from tree
[<JobDefinition id:'2013.com.canonical.certification::optical_drive_sr0' plugin:'resource'>,
 <JobDefinition id:'2013.com.canonical.certification::miscellanea/fwts_results.log' plugin:'attachment'>,
 <JobDefinition id:'2013.com.canonical.certification::miscellanea/submission-resources' plugin:'shell'>,
 <JobDefinition id:'2013.com.canonical.certification::miscellanea/fwts_test' plugin:'shell'>,
 <JobDefinition id:'2013.com.canonical.certification::miscellanea/ipmi_test' plugin:'shell'>,
 <JobDefinition id:'2013.com.canonical.certification::package' plugin:'resource'>,
 <JobDefinition id:'2013.com.canonical.certification::device' plugin:'resource'>,
 <JobDefinition id:'2013.com.canonical.certification::cpuinfo' plugin:'resource'>,
 <JobDefinition id:'2013.com.canonical.certification::uname' plugin:'resource'>,
 <JobDefinition id:'2013.com.canonical.certification::requirements' plugin:'resource'>,
 <JobDefinition id:'2013.com.canonical.certification::dpkg' plugin:'resource'>,
 <JobDefinition id:'2013.com.canonical.certification::lsb' plugin:'resource'>,
 <JobDefinition id:'2013.com.canonical.certification::dmi' plugin:'resource'>,
 <JobDefinition id:'2013.com.canonical.certification::block_device' plugin:'resource'>,
 <JobDefinition id:'2013.com.canonical.certification::rtc' plugin:'resource'>,
 <JobDefinition id:'2013.com.canonical.certification::usb' plugin:'resource'>,
 <JobDefinition id:'2013.com.canonical.certification::virtualization' plugin:'resource'>]

Then apparently when updating the desired job list, the wrong order seen above is considered:

self._update_desired_job_list(manager, tree.selection)

after this, the manager.state.desired_job_list contains the bad order:

[<JobDefinition id:'2013.com.canonical.certification::optical_drive_sr0' plugin:'resource'>,
 <JobDefinition id:'2013.com.canonical.certification::miscellanea/fwts_results.log' plugin:'attachment'>,
 <JobDefinition id:'2013.com.canonical.certification::miscellanea/submission-resources' plugin:'shell'>,
 <JobDefinition id:'2013.com.canonical.certification::miscellanea/fwts_test' plugin:'shell'>,
 <JobDefinition id:'2013.com.canonical.certification::miscellanea/ipmi_test' plugin:'shell'>,
 <JobDefinition id:'2013.com.canonical.certification::package' plugin:'resource'>,
 <JobDefinition id:'2013.com.canonical.certification::device' plugin:'resource'>,
 <JobDefinition id:'2013.com.canonical.certification::cpuinfo' plugin:'resource'>,
 <JobDefinition id:'2013.com.canonical.certification::uname' plugin:...

OK, in cli, SelectableJobTreeNode has a selection method that returns the selection for the current node and children. I'm sure I'm misunderstanding how it works, but the line where it adds selected jobs to the final selection_list it will returns is pretty clear:

        for job in self.job_selection:
            if self.job_selection[job]:
                self._selection_list.append(job)

A walk of jobs in self.job_selection and then they're appended to a list (self._selection_list is a [] so it will conserve order). What is self.job_selection? It's initialized in __init__:

        self.job_selection = {}

So it's a dictionary, whose key ordering is not guaranteed, this is why we see inconsistent ordering in this case.

I very simplistically changed self.job_selection to an OrderedDict and this seems to make things more stable, I see the fwts and fwts_results.log in the desired_job_list in the correct order all the time (and also, submission_resources correctly appears *before* those two). I'd prefer if someone with more knowledge of this code implemented a solution (which may or may not be OrderedDict), since my simplistic approach may break other things, but the fact that we're using a dict to keep a list of selected tests *seems* to be the problem here.