[tempest] test_update_account_metadata_with_create_and_delete_metadata fails with 'matches Contains' error

Important note: sometimes this test passed, the root of the issue is unknown.

Timur, yes, I haven't reproduced it yet, all my tests passed successfully. I'm trying to find some condition, which force it to fail.

This only failed in 5% of runs for the last 30 days. Moving this to 8.0-updates, as we can't fix it in time for 8.0. We'll continue the investigation of the root cause.

Is it possible to see something like snapshot of system (or at least swift logs), where it'll fail next time? I'm trying to reproduce this, but it'll take too long time to get in this 5%.

Alyona, yes, we will provide the env when we reproduce it next time.

Any news here? I still have no env with this bug.

Snapshot:
https://drive.google.com/file/d/0B8hkiEm94sEtYWljVVZjY1owZG8/view?usp=sharing

Trace: http://paste.openstack.org/show/495526/

Configuration:
    ISO: http://paste.openstack.org/show/495500/
    Compute - QEMU.
    Network - Neutron with VLAN segmentation, DVR.
    Storage Backends - Ceph RBD for volumes (Cinder), Ceph RBD for ephemeral volumes (Nova), Ceph RBD for images (Glance)
    RADOS
    Additional services - Install Ironic, Sahara

In tab Settings->Compute check Nova quotas

Nodes: controller, compute, ironic, ceph-osd

Steps to reproduce:
    1. Deploy ISO in configuration see (Detailed bug description)
    2. Navigate to controller node
    3. Install git (use apt-get install git)
    4. Clone script to deploy rally + tempest
       # git clone https://github.com/obutenko/mos-rally-verify.git
    5. Navigate to the https://github.com/obutenko/mos-rally-verify
    6. Execute necessary steps to deploy tempest
    7. Tun test in debug mode
        #rally --debug verify start --regex tempest.api.object_storage.test_account_services.AccountTest.test_update_account_metadata_with_create_and_delete_metadata

Sergey, please take a look at the traceback you've attached: http://paste.openstack.org/show/495526/

Captured pythonlogging:
~~~~~~~~~~~~~~~~~~~~~~~
    2016-04-27 14:17:14,304 2190 INFO [tempest.lib.common.rest_client] Request (AccountTest:test_update_account_metadata_with_create_and_delete_metadata): 405 POST https://public.fuel.local:8080/swift/v1 0.016s
    2016-04-27 14:17:14,305 2190 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'X-Account-Meta-test-account-meta1': 'Meta1', 'X-Auth-Token': '<omitted>'}
            Body: None
        Response - Headers: {'status': '405', 'content-length': '16', 'content-location': 'https://public.fuel.local:8080/swift/v1', 'accept-ranges': 'bytes', 'server': 'Apache', 'connection': 'close', 'x-trans-id': 'tx000000000000000028f3a-005720c9ea-8597-default', 'date': 'Wed, 27 Apr 2016 14:17:14 GMT', 'content-type': 'text/plain; charset=utf-8'}
            Body: MethodNotAllowed

it clearly says that POST is not allowed to this particular URL: /swift/v1

and that's *correct* according to Swift docs (http://developer.openstack.org/api-ref-objectstorage-v1.html) - there is no such endpoint, thus Swift returns 405 for a reason.

Please check the test and its configuration: maybe you have broken string interpolation somewhere and the URL is produced incorrectly.

It seems that someone is trying to reach an account through URL typical for RadosGW (https://public.fuel.local:8080/swift/v1) instead of a Swift-compatible one (https://public.fuel.local:8080/swift/v1/AUTH_$(tenant_id)s). At the moment RadosGW extracts the tenant_id from a token provided by a HTTP client. That's the reason why those URLs are different.

The question is: how has the URL been provided to Tempest? AFAIK Tempest should go to Keystone for endpoint locations.

Change to Invalid
Can't reproduce this bug manual and on ci