Mirantis OpenStack

[Tempest] [SSL] 3 tests for object storage failed with ssl configuration

Bug #1537077 reported by Timur Nurlygayanov on 2016-01-22

This bug affects 1 person

	Status	Importance	Assigned to	Milestone
Mirantis OpenStack	Status tracked in 10.0.x
10.0.x	Invalid	High	Kostiantyn Danylov	Mirantis OpenStack 10.0
7.0.x	Invalid	High	Kostiantyn Danylov	Mirantis OpenStack 7.0-updates
8.0.x	Invalid	High	Kostiantyn Danylov	Mirantis OpenStack 8.0-updates
9.x	Invalid	High	Kostiantyn Danylov	Mirantis OpenStack 9.0

Bug Description

Steps To Reproduce:
1. Deploy environment with TLS
2. Run all Tempest tests for object storage using mos-tempest-runner scripts [1]

Expected Result:
All tests will pass

Observed Result:
3 tests will fail, if disable SSL, these tests pass.

The following tests failed:
test_create_object_with_transfer_encoding
test_create_object_with_expect_continue
test_basic_scenario

Tests failed with the following error:

traceback
Traceback (most recent call last):
  File "/home/developer/mos-tempest-runner/tempest/tempest/api/object_storage/test_object_services.py", line 214, in test_create_object_with_transfer_encoding
    chunk_size=512)
  File "/home/developer/mos-tempest-runner/tempest/tempest/services/object_storage/object_client.py", line 163, in put_object_with_chunk
    chunk_size, headers)
  File "/home/developer/mos-tempest-runner/tempest/tempest/services/object_storage/object_client.py", line 241, in put_object_connection
    conn.endheaders()
  File "/usr/local/lib/python2.7/httplib.py", line 997, in endheaders
    self._send_output(message_body)
  File "/usr/local/lib/python2.7/httplib.py", line 850, in _send_output
    self.send(msg)
  File "/usr/local/lib/python2.7/httplib.py", line 812, in send
    self.connect()
  File "/usr/local/lib/python2.7/httplib.py", line 1212, in connect
    server_hostname=server_hostname)
  File "/usr/local/lib/python2.7/ssl.py", line 350, in wrap_socket
    _context=self)
  File "/usr/local/lib/python2.7/ssl.py", line 566, in __init__
    self.do_handshake()
  File "/usr/local/lib/python2.7/ssl.py", line 788, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)

Tags:

Timur Nurlygayanov (tnurlygayanov) on 2016-01-22

tags:	added: ceph ssl swift tempest
tags:	added: customer-found

Revision history for this message

Timur Nurlygayanov (tnurlygayanov) wrote on 2016-01-22:

Priority changed to High because 3 Tempest tests failed for any configurations with SSL and it looks like we don't have workaround for the issue.

Revision history for this message

Roman Podoliaka (rpodolyaka) wrote on 2016-01-22:

Timur, are you sure you pass the correct SSL cacert to python-*client's / Tempest?

Revision history for this message

Timur Nurlygayanov (tnurlygayanov) wrote on 2016-01-22:

Roman, it is good question, we are going to check it.

Thank you!

Revision history for this message

Timur Nurlygayanov (tnurlygayanov) wrote on 2016-01-27:

Roman, yes, we checked it and it looks like we passed certificates correctly (all tests failed if we not pass the certificate).

Revision history for this message

Ivan Berezovskiy (iberezovskiy) wrote on 2016-01-28:

Change assignee to Oleksiy for researching of root cause.

Revision history for this message

Timur Nurlygayanov (tnurlygayanov) wrote on 2016-01-29:

Assigned to MOS Ceph team because we reproduced the issue but we don't know the root of the issue yet. It looks like we need someone with experience in object storage API and Swift.

Roman Podoliaka (rpodolyaka) on 2016-02-02

tags:

added: move-to-mu

Roman Podoliaka (rpodolyaka) on 2016-02-02

tags:

added: area-ceph
removed: ceph

Revision history for this message

Roman Podoliaka (rpodolyaka) wrote on 2016-02-03:

Need more time to understand the root cause. This may even be a problem of Tempest configuration. Moving to MU.

Roman Podoliaka (rpodolyaka) on 2016-02-04

tags:

removed: customer-found

Alexey Stupnikov (astupnikov) on 2016-05-23

tags:

added: non-release

Revision history for this message

Kostiantyn Danylov (kdanylov) wrote on 2016-05-23:

For 7.0 error caused by wrong TLS configuration - either self-signed certificates or by wrong tempest settings.

For 8.0+ a lot of storage test were failed due to check headers like 'content-length', which is not provided by ceph (and NOT required by object storage API) in some cases.

Revision history for this message

Kostiantyn Danylov (kdanylov) wrote on 2016-05-23:

BTW: instruction from https://github.com/Mirantis/mos-tempest-runner#installation-and-usage

leads to:

${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover} --list
--- import errors ---
Failed to import test module: tempest.test_discover.test_discover
Traceback (most recent call last):
  File "/home/developer/mos-tempest-runner/.venv/lib/python2.7/site-packages/unittest2/loader.py", line 456, in _find_test_path
    module = self._get_module_from_name(name)
  File "/home/developer/mos-tempest-runner/.venv/lib/python2.7/site-packages/unittest2/loader.py", line 395, in _get_module_from_name
    __import__(name)
  File "/home/developer/mos-tempest-runner/tempest/tempest/test_discover/test_discover.py", line 18, in <module>
    from tempest.test_discover import plugins
  File "/home/developer/mos-tempest-runner/tempest/tempest/test_discover/plugins.py", line 20, in <module>
    from tempest_lib.common.utils import misc
  File "/home/developer/mos-tempest-runner/.venv/lib/python2.7/site-packages/tempest_lib/common/utils/misc.py", line 19, in <module>
    from oslo_log import log as logging
  File "/home/developer/mos-tempest-runner/.venv/lib/python2.7/site-packages/oslo_log/log.py", line 49, in <module>
    from oslo_log import _options
  File "/home/developer/mos-tempest-runner/.venv/lib/python2.7/site-packages/oslo_log/_options.py", line 38, in <module>
    help='If set to true, the logging level will be set to '
  File "/home/developer/mos-tempest-runner/.venv/lib/python2.7/site-packages/oslo_config/cfg.py", line 965, in __init__
    super(BoolOpt, self).__init__(name, type=types.Boolean(), **kwargs)
TypeError: __init__() got an unexpected keyword argument 'mutable'

Ran 0 tests in 0.425s

OK
Error: Subunit stream 0 is not found
cat: /home/developer/mos-tempest-runner/tempest-reports/tempest-report.xml: No such file or directory
/home/developer/mos-tempest-runner/.venv/bin/run_tests: line 88: [: : integer expression expected

BTW: instruction from https://github.com/Mirantis/mos-tempest-runner#installation-and-usage

leads to:

${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover} --list 
--- import errors ---
Failed to import test module: tempest.test_discover.test_discover
Traceback (most recent call last):
  File "/home/developer/mos-tempest-runner/.venv/lib/python2.7/site-packages/unittest2/loader.py", line 456, in _find_test_path
    module = self._get_module_from_name(name)
  File "/home/developer/mos-tempest-runner/.venv/lib/python2.7/site-packages/unittest2/loader.py", line 395, in _get_module_from_name
    __import__(name)
  File "/home/developer/mos-tempest-runner/tempest/tempest/test_discover/test_discover.py", line 18, in <module>
    from tempest.test_discover import plugins
  File "/home/developer/mos-tempest-runner/tempest/tempest/test_discover/plugins.py", line 20, in <module>
    from tempest_lib.common.utils import misc
  File "/home/developer/mos-tempest-runner/.venv/lib/python2.7/site-packages/tempest_lib/common/utils/misc.py", line 19, in <module>
    from oslo_log import log as logging
  File "/home/developer/mos-tempest-runner/.venv/lib/python2.7/site-packages/oslo_log/log.py", line 49, in <module>
    from oslo_log import _options
  File "/home/developer/mos-tempest-runner/.venv/lib/python2.7/site-packages/oslo_log/_options.py", line 38, in <module>
    help='If set to true, the logging level will be set to '
  File "/home/developer/mos-tempest-runner/.venv/lib/python2.7/site-packages/oslo_config/cfg.py", line 965, in __init__
    super(BoolOpt, self).__init__(name, type=types.Boolean(), **kwargs)
TypeError: __init__() got an unexpected keyword argument 'mutable'

Ran 0 tests in 0.425s

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.