test_cache_layer_*_crud depend on test execution order

The tests test_cache_layer_domain_crud, test_cache_layer_project_crud, and test_cache_layer_role_crud in keystone.tests.test_backend.IdentityTests may fail depending on which tests have run previously. In particular, if no tests have run previously, then they all fail. Here's an example running each test separately using the KVS backend:

% tox -e py27 -- keystone.tests.test_backend_sql.SqlIdentity.test_cache_layer_role_crud
py27 develop-inst-nodeps: /opt/stack/keystone
py27 runtests: commands[0] | python setup.py testr --testr-args=keystone.tests.test_backend_sql.SqlIdentity.test_cache_layer_role_crud
running testr
running=${PYTHON:-python} -m subunit.run discover \
-t ./ ./keystone/tests \
--list
running=${PYTHON:-python} -m subunit.run discover \
-t ./ ./keystone/tests \
 --load-list /tmp/tmpCi8sRv
======================================================================
FAIL: keystone.tests.test_backend_sql.SqlIdentity.test_cache_layer_role_crud
tags: worker-0
----------------------------------------------------------------------
Traceback (most recent call last):
  File "keystone/tests/test_backend.py", line 2682, in test_cache_layer_role_crud
    self.assertDictEqual(role_ref, self.assignment_api.get_role(role_id))
  File "keystone/tests/core.py", line 423, in assertDictEqual
    self.assertEqual(d1, d2, msg)
  File "/opt/stack/keystone/.tox/py27/local/lib/python2.7/site-packages/testtools/testcase.py", line 322, in assertEqual
    self.assertThat(observed, matcher, message)
  File "/opt/stack/keystone/.tox/py27/local/lib/python2.7/site-packages/testtools/testcase.py", line 417, in assertThat
    raise MismatchError(matchee, matcher, mismatch, verbose)
MismatchError: !=:
reference = {'id': u'bf17c51445794329987eda59f9ffd64f',
 'name': u'ea760f7d48aa4c22a6dcaa54098fd2a5'}
actual = {'id': u'bf17c51445794329987eda59f9ffd64f',
 'name': u'815c02c26b3048308b1fb4be22412882'}
======================================================================
FAIL: process-returncode
tags: worker-0
----------------------------------------------------------------------
Binary content:
  traceback (test/plain; charset="utf8")
Ran 3 (+1) tests in 0.143s (+0.004s)
FAILED (id=151, failures=2)
error: testr failed (1)
ERROR: InvocationError: '/opt/stack/keystone/.tox/py27/bin/python setup.py testr --testr-args=keystone.tests.test_backend_sql.SqlIdentity.test_cache_layer_role_crud'
____________________________________________________________________________________ summary _____________________________________________________________________________________
ERROR: py27: commands failed
% tox -e py27 -- keystone.tests.test_backend_sql.SqlIdentity.test_cache_layer_project_crud
py27 develop-inst-nodeps: /opt/stack/keystone
py27 runtests: commands[0] | python setup.py testr --testr-args=keystone.tests.test_backend_sql.SqlIdentity.test_cache_layer_project_crud
running testr
running=${PYTHON:-python} -m subunit.run discover \
-t ./ ./keystone/tests \
--list
running=${PYTHON:-python} -m subunit.run discover \
-t ./ ./keystone/tests \
 --load-list /tmp/tmp4QfzG9
======================================================================
FAIL: keystone.tests.test_backend_sql.SqlIdentity.test_cache_layer_project_crud
tags: worker-0
----------------------------------------------------------------------
Traceback (most recent call last):
  File "keystone/tests/test_backend.py", line 2636, in test_cache_layer_project_crud
    project, self.assignment_api.get_project(project_id))
  File "keystone/tests/core.py", line 480, in assertDictContainsSubset
    self.fail(self._formatMessage(msg, standardMsg))
  File "/usr/lib/python2.7/unittest/case.py", line 412, in fail
    raise self.failureException(msg)
AssertionError: Mismatched values: 'name', expected: '4c6b94aaf6324a2c8d3da20fc19ced63', actual: u'aebeb924708f4cc7a326043d240be3af'
======================================================================
FAIL: process-returncode
tags: worker-0
----------------------------------------------------------------------
Binary content:
  traceback (test/plain; charset="utf8")
Ran 3 (+1) tests in 0.180s (+0.040s)
FAILED (id=152, failures=2)
error: testr failed (1)
ERROR: InvocationError: '/opt/stack/keystone/.tox/py27/bin/python setup.py testr --testr-args=keystone.tests.test_backend_sql.SqlIdentity.test_cache_layer_project_crud'
____________________________________________________________________________________ summary _____________________________________________________________________________________
ERROR: py27: commands failed
% tox -e py27 -- keystone.tests.test_backend_sql.SqlIdentity.test_cache_layer_domain_crud
py27 develop-inst-nodeps: /opt/stack/keystone
py27 runtests: commands[0] | python setup.py testr --testr-args=keystone.tests.test_backend_sql.SqlIdentity.test_cache_layer_domain_crud
running testr
running=${PYTHON:-python} -m subunit.run discover \
-t ./ ./keystone/tests \
--list
running=${PYTHON:-python} -m subunit.run discover \
-t ./ ./keystone/tests \
 --load-list /tmp/tmpY27vIC
======================================================================
FAIL: keystone.tests.test_backend_sql.SqlIdentity.test_cache_layer_domain_crud
tags: worker-0
----------------------------------------------------------------------
Traceback (most recent call last):
  File "keystone/tests/test_backend.py", line 2586, in test_cache_layer_domain_crud
    domain_ref, self.assignment_api.get_domain(domain_id))
  File "keystone/tests/core.py", line 480, in assertDictContainsSubset
    self.fail(self._formatMessage(msg, standardMsg))
  File "/usr/lib/python2.7/unittest/case.py", line 412, in fail
    raise self.failureException(msg)
AssertionError: Mismatched values: 'name', expected: u'2414056b220a4e44ae11c9f4fb00376e', actual: u'a9e3a4ff47844a0f9f4911c9d7dbfa0e'
======================================================================
FAIL: process-returncode
tags: worker-0
----------------------------------------------------------------------
Binary content:
  traceback (test/plain; charset="utf8")
Ran 3 (+1) tests in 0.143s (-0.034s)
FAILED (id=153, failures=2)
error: testr failed (1)
ERROR: InvocationError: '/opt/stack/keystone/.tox/py27/bin/python setup.py testr --testr-args=keystone.tests.test_backend_sql.SqlIdentity.test_cache_layer_domain_crud'
____________________________________________________________________________________ summary _____________________________________________________________________________________
ERROR: py27: commands failed

The results are similar for the SQL backend. I haven't tried the other backends.

This test order dependency is particularly when running the tests in parallel since the test execution order is nondeterministic.