Fuel for OpenStack

Deployment fails on adding Cinder types if TLS is enabled for Fuel environment

Bug #1585562 reported by Serg Lystopad
12
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Invalid
High
Stanislaw Bogatkin
Fuel for OpenStack 10.0
8.0.x
Fix Released
High
Stanislaw Bogatkin
Fuel for OpenStack 8.0-mu-2
Mitaka
Invalid
High
Stanislaw Bogatkin
Fuel for OpenStack 9.0

Bug Description

Detailed bug description:
Deplyment fails on primary controller on adding 'cinder types'
2016-05-24 17:15:22 +0000 /Package[python-cinderclient] (info): Starting to evaluate the resource
2016-05-24 17:15:22 +0000 /Package[python-cinderclient] (info): Evaluated in 0.00 seconds
2016-05-24 17:15:22 +0000 /Stage[main]/Main/Create_cinder_types[netapp]/Cinder::Type[netapp]/Exec[cinder type-create netapp] (info): Starting to evaluate the resource
2016-05-24 17:15:22 +0000 Exec[cinder type-create netapp](provider=posix) (debug): Executing check 'cinder type-list | grep -qP '\bnetapp\b''
2016-05-24 17:15:22 +0000 Puppet (debug): Executing 'cinder type-list | grep -qP '\bnetapp\b''
2016-05-24 17:15:22 +0000 /Stage[main]/Main/Create_cinder_types[netapp]/Cinder::Type[netapp]/Exec[cinder type-create netapp]/unless (debug): /usr/lib/python2.7/dist-pack
ages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause
certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
2016-05-24 17:15:22 +0000 /Stage[main]/Main/Create_cinder_types[netapp]/Cinder::Type[netapp]/Exec[cinder type-create netapp]/unless (debug): InsecurePlatformWarning
2016-05-24 17:15:22 +0000 /Stage[main]/Main/Create_cinder_types[netapp]/Cinder::Type[netapp]/Exec[cinder type-create netapp]/unless (debug): ERROR: SSL exception connect
ing to https://horizon.jdc.stag.stsc:5000/v2.0/tokens: [Errno 1] _ssl.c:510: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
2016-05-24 17:15:22 +0000 /Stage[main]/Main/Create_cinder_types[netapp]/Cinder::Type[netapp]/Exec[cinder type-create netapp]/returns (debug): Exec try 1/2
2016-05-24 17:15:22 +0000 Exec[cinder type-create netapp](provider=posix) (debug): Executing 'cinder type-create netapp'
2016-05-24 17:15:22 +0000 Puppet (debug): Executing 'cinder type-create netapp'
2016-05-24 17:15:23 +0000 /Stage[main]/Main/Create_cinder_types[netapp]/Cinder::Type[netapp]/Exec[cinder type-create netapp]/returns (debug): Sleeping for 5.0 seconds be
tween tries
2016-05-24 17:15:28 +0000 /Stage[main]/Main/Create_cinder_types[netapp]/Cinder::Type[netapp]/Exec[cinder type-create netapp]/returns (debug): Exec try 2/2
2016-05-24 17:15:28 +0000 Exec[cinder type-create netapp](provider=posix) (debug): Executing 'cinder type-create netapp'
2016-05-24 17:15:28 +0000 Puppet (debug): Executing 'cinder type-create netapp'
2016-05-24 17:15:28 +0000 /Stage[main]/Main/Create_cinder_types[netapp]/Cinder::Type[netapp]/Exec[cinder type-create netapp]/returns (debug): Sleeping for 5.0 seconds be
tween tries
2016-05-24 17:15:33 +0000 /Stage[main]/Main/Create_cinder_types[netapp]/Cinder::Type[netapp]/Exec[cinder type-create netapp]/returns (notice): /usr/lib/python2.7/dist-pa
ckages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may caus
e certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
2016-05-24 17:15:33 +0000 /Stage[main]/Main/Create_cinder_types[netapp]/Cinder::Type[netapp]/Exec[cinder type-create netapp]/returns (notice): InsecurePlatformWarning
2016-05-24 17:15:33 +0000 /Stage[main]/Main/Create_cinder_types[netapp]/Cinder::Type[netapp]/Exec[cinder type-create netapp]/returns (notice): ERROR: SSL exception conne
cting to https://horizon.jdc.stag.stsc:5000/v2.0/tokens: [Errno 1] _ssl.c:510: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
2016-05-24 17:15:33 +0000 Puppet (err): cinder type-create netapp returned 1 instead of one of [0]
/usr/lib/ruby/vendor_ruby/puppet/util/errors.rb:106:in `fail'

Steps to reproduce:
Deploy Fuel 8.0, apply MU1
Install plugins listed in 'Related projects installed' section below
Create Fuel environment with Contrail networking
Enable and configure cinder_netapp plugin: 'Clustered Data ONTAP', 'NetApp Storage Protocol' - NFS
Modify settins in 'Public TLS' section on Security tab for Fuel environment:
* enable checkbox 'TLS for OpenStack public endpoints'
* enable checkbox 'HTTPS for Horizon'
* Source for certificate - 'self-signed'
* set 'DNS hostname for public TLS endpoints' to 'horizon.dc11.stag.smth'
Deploy-changes

Expected results:
environment successfully deployed
Actual result:
deployment fails

Impact:
 affects environment deployment in customer environment

Description of the environment:
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "570"
  build_id: "570"
  fuel-nailgun_sha: "558ca91a854cf29e395940c232911ffb851899c1"
  python-fuelclient_sha: "4f234669cfe88a9406f4e438b1e1f74f1ef484a5"
  fuel-agent_sha: "658be72c4b42d3e1436b86ac4567ab914bfb451b"
  fuel-nailgun-agent_sha: "b2bb466fd5bd92da614cdbd819d6999c510ebfb1"
  astute_sha: "b81577a5b7857c4be8748492bae1dec2fa89b446"
  fuel-library_sha: "c2a335b5b725f1b994f78d4c78723d29fa44685a"
  fuel-ostf_sha: "3bc76a63a9e7d195ff34eadc29552f4235fa6c52"
  fuel-mirror_sha: "fb45b80d7bee5899d931f926e5c9512e2b442749"
  fuelmenu_sha: "78ffc73065a9674b707c081d128cb7eea611474f"
  shotgun_sha: "63645dea384a37dde5c01d4f8905566978e5d906"
  network-checker_sha: "a43cf96cd9532f10794dce736350bf5bed350e9d"
  fuel-upgrade_sha: "616a7490ec7199f69759e97e42f9b97dfc87e85b"
  fuelmain_sha: "d605bcbabf315382d56d0ce8143458be67c53434"

Operation system: Ubuntu 14.04.4 LTS
Reference architecture: HA
Network model: Juniper Contrail SDN
Related projects installed:
id | name | version | package_version
---|-----------------------------|---------|----------------
3 | influxdb_grafana | 0.9.0 | 4.0.0
2 | elasticsearch_kibana | 0.9.0 | 4.0.0
4 | lma_collector | 0.9.0 | 4.0.0
5 | lma_infrastructure_alerting | 0.9.0 | 4.0.0
6 | nova_nfs | 3.2.1 | 3.0.0
9 | ldap | 2.0.0 | 3.0.0
11 | cinder_netapp | 4.1.1 | 4.0.0
12 | contrail | 4.0.1 | 4.0.0

Additional information:
my env contains
  id roles online
 --- ------------------------------------------------ --------
   1 cinder, controller True
   4 contrail-config, contrail-control, contrail-db True
   9 compute True
   8 elasticsearch_kibana True
  14 mongo True
   7 influxdb_grafana, infrastructure_alerting True
   6 contrail-config, contrail-control, contrail-db True
   2 cinder, controller True
  11 compute True
   5 contrail-config, contrail-control, contrail-db True
   3 cinder, controller True
  12 mongo True
  13 mongo True
  10 compute True

Revision history for this message
Serg Lystopad (slystopad) wrote :

`dpkg -l` output from primary-controller (node on which puppet run failed)
https://drive.google.com/file/d/0BzqvkqZNKRGddTN3LWRuNGNzOHc/view?usp=sharing

Dmitry Pyzhov (dpyzhov)
Changed in fuel:
assignee: nobody → Fuel Plugin NetApp (fuel-plugin-cinder-netapp)
Changed in fuel-plugin-cinder-netapp:
status: New → Invalid
status: Invalid → New
Changed in fuel:
milestone: none → 10.0
status: New → Incomplete
Revision history for this message
Stanislaw Bogatkin (sbogatkin) wrote :

I believe that cinder_netapp plugin creator should handle situation when ssl with unknown crt is used. Anyway, to understand more - at least deployment snapshot is needed.

Revision history for this message
Serg Lystopad (slystopad) wrote :

Deployment fail on non-plugin specific code
2016-05-25 12:31:55 INFO [323] Casting message to Nailgun:
{"method"=>"deploy_resp",
 "args"=>
  {"task_uuid"=>"36c21cd5-8fe9-4f54-a038-af3d1276ab13",
   "nodes"=>
    [{"uid"=>"1",
      "status"=>"error",
      "error_type"=>"deploy",
      "role"=>"primary-controller",
      "task"=>
       {"priority"=>7600,
        "type"=>"puppet",
        "id"=>"create-cinder-types",
        "parameters"=>
         {"puppet_modules"=>"/etc/puppet/modules",
          "puppet_manifest"=>
           "/etc/puppet/modules/osnailyfacter/modular/openstack-cinder/create_cinder_types.pp",
          "timeout"=>1200,
          "cwd"=>"/"},
        "uids"=>["1"]},
      "progress"=>100}]}}

Revision history for this message
Serg Lystopad (slystopad) wrote :

Stanislav, I've shared snapshot for you

Revision history for this message
Stanislaw Bogatkin (sbogatkin) wrote :

This task works normally without netapp plugin, I believe? If so, plugin developer must change logic in his own plugin.

Revision history for this message
Serg Lystopad (slystopad) wrote :

Deployment of the environment without cinder-netapp plugin also fails
Below is respective puppet log
https://drive.google.com/file/d/0BzqvkqZNKRGdNGxtRWxLazFxQWM/view?usp=sharing

Revision history for this message
Serg Lystopad (slystopad) wrote :

cinder-create-types specifies public keystone endpoint for cinder client (and uses is HTTPS scheme in case of enabled TLS)
https://github.com/openstack/fuel-library/blob/8.0/deployment/puppet/osnailyfacter/modular/openstack-cinder/create_cinder_types.pp#L44

Cinder client also by default uses public endpoint for cinder API (and it is also HTTPS in case of enabled TLS).

Workaround might be specify internal endpoint for cinder client
https://review.openstack.org/#/c/321101/

and use internal keystone endpoint for authentication.

What is reason of using public endpoints? We have isolated management network and endpoint for all services in it.

Revision history for this message
Stanislaw Bogatkin (sbogatkin) wrote :

real questions here is why cinder client doesn't trust to certificate all other services trust. Upstream bug?

Stanislaw Bogatkin (sbogatkin)
Changed in fuel:
status: Incomplete → Confirmed
importance: Undecided → Medium
Revision history for this message
Stanislaw Bogatkin (sbogatkin) wrote :

After speak with Serhii L. it seems to be a bug in cinder client itself:

root@node-1:~# cinder --debug list
DEBUG:keystoneclient.session:REQ: curl -g -i -X GET https://horizon.dc11.stag.smth:5000/v2.0/ -H "Accept: application/json" -H "User-Agent: python-keystoneclient"
/usr/lib/python2.7/dist-packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
DEBUG:keystoneclient.auth.identity.v2:Making authentication request to https://horizon.dc11.stag.smth:5000/v2.0/tokens
ERROR: SSL exception connecting to https://horizon.dc11.stag.smth:5000/v2.0/tokens: [Errno 1] _ssl.c:510: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

but:

root@node-1:~# curl -g -i -X GET https://horizon.dc11.stag.smth:5000/v2.0/ -H "Accept: application/json" -H "User-Agent: python-keystoneclient"
HTTP/1.1 200 OK
Date: Thu, 26 May 2016 09:56:31 GMT
Server: Apache
Content-Length: 349
Vary: X-Auth-Token
x-openstack-request-id: req-c22d283b-3592-46cf-a89f-05cf6d390988
Connection: close
Content-Type: application/json

{"version": {"status": "stable", "updated": "2014-04-17T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v2.0+json"}], "id": "v2.0", "links": [{"href": "https://horizon.dc11.stag.smth:5000/v2.0/", "rel": "self"}, {"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}]}}

So, certificate with which url is hosted, is trusted, but cinder client still doesn't trust it. As a solution I can propose this [0] for case if selective ssl is not used here. Other way is to fix upstream cinder client or fix logic and point to non-ssl url when create cinder types.

[0] https://review.openstack.org/#/c/243107/

Stanislaw Bogatkin (sbogatkin)
Changed in fuel:
importance: Medium → High
status: Confirmed → Invalid
assignee: Fuel Plugin NetApp (fuel-plugin-cinder-netapp) → Stanislaw Bogatkin (sbogatkin)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/8.0)

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/321464

Ilya Kutukov (ikutukov)
Changed in fuel-plugin-cinder-netapp:
status: New → Confirmed
Andrey Volochay (wallavv)
no longer affects: fuel-plugin-cinder-netapp
Ilya Kutukov (ikutukov)
tags: added: area-python
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/8.0)

Reviewed: https://review.openstack.org/321464
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=85cc4bbde1410ef2af63400e2d4c581d75c8ace6
Submitter: Jenkins
Branch: stable/8.0

commit 85cc4bbde1410ef2af63400e2d4c581d75c8ace6
Author: Bartłomiej Piotrowski <email address hidden>
Date: Mon Nov 9 14:36:00 2015 +0100

    Add OS_CACERT to openrc if SSL is enabled

    Change-Id: Ie38dc225d1aa2104ef7959644c4e16a2923fa15e
    Closes-bug: 1585562
    (cherry picked from commit ce2ac8ed8e41c3174f21e34aa446a65f4737e67e)

TatyanaGladysheva (tgladysheva)
tags: added: on-verification
Revision history for this message
TatyanaGladysheva (tgladysheva) wrote :

Bug is not reproducible on MOS 8.0 mu1.
Environment with Contrail networking: 1 controller+cinder node, 1 compute node, 1 contrail-config, contrail-control, contrail-db node; cinder_netapp plugin.

Also I tried to reproduce on MOS 8.0 with/without cinder_netapp, without contrail plugin. Every time deployment was completed successfully.

tags: removed: on-verification
Revision history for this message
TatyanaGladysheva (tgladysheva) wrote :

Verified on MOS 8.0 + MU2 updates.

Environment:
1 controller+cinder node, 1 compute node, 1 contrail-config, contrail-control, contrail-db node; cinder_netapp, contrail plugins.

Actual results:
Deployment is completed successfully.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.