Comment 3 for bug 1308106

OK, in cli, SelectableJobTreeNode has a selection method that returns the selection for the current node and children. I'm sure I'm misunderstanding how it works, but the line where it adds selected jobs to the final selection_list it will returns is pretty clear:

        for job in self.job_selection:
            if self.job_selection[job]:
                self._selection_list.append(job)

A walk of jobs in self.job_selection and then they're appended to a list (self._selection_list is a [] so it will conserve order). What is self.job_selection? It's initialized in __init__:

        self.job_selection = {}

So it's a dictionary, whose key ordering is not guaranteed, this is why we see inconsistent ordering in this case.

I very simplistically changed self.job_selection to an OrderedDict and this seems to make things more stable, I see the fwts and fwts_results.log in the desired_job_list in the correct order all the time (and also, submission_resources correctly appears *before* those two). I'd prefer if someone with more knowledge of this code implemented a solution (which may or may not be OrderedDict), since my simplistic approach may break other things, but the fact that we're using a dict to keep a list of selected tests *seems* to be the problem here.