Fuel for OpenStack

Duplicated keystone endpoints

Bug #1375254 reported by Dmitry Ukov
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Released
Medium
Aleksandr Didenko
Fuel for OpenStack 6.0
5.1.x
Fix Released
Medium
Aleksandr Didenko
Fuel for OpenStack 5.1.1

Bug Description

{"build_id": "2014-09-17_21-40-34",
"ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346",
"build_number": "11",
"auth_required": true,
"api": "1.0",
"nailgun_sha": "eb8f2b358ea4bb7eb0b2a0075e7ad3d3a905db0d",
"production": "docker",
"fuelmain_sha": "8ef433e939425eabd1034c0b70e90bdf888b69fd",
"astute_sha": "f5fbd89d1e0e1f22ef9ab2af26da5ffbfbf24b13",
"feature_groups": ["mirantis"],
"release": "5.1",
"release_versions": {
    "2014.1.1-5.1": {
        "VERSION": {
            "build_id": "2014-09-17_21-40-34",
            "ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346",
            "build_number": "11",
            "api": "1.0",
            "nailgun_sha": "eb8f2b358ea4bb7eb0b2a0075e7ad3d3a905db0d",
            "production": "docker",
            "fuelmain_sha": "8ef433e939425eabd1034c0b70e90bdf888b69fd",
            "astute_sha": "f5fbd89d1e0e1f22ef9ab2af26da5ffbfbf24b13",
            "feature_groups": ["mirantis"],
            "release": "5.1",
            "fuellib_sha": "d9b16846e54f76c8ebe7764d2b5b8231d6b25079"
        }
    }
},
"fuellib_sha": "d9b16846e54f76c8ebe7764d2b5b8231d6b25079"}

Steps to reproduce:
1. Create environment
2. Use HA mode foe deployment
3. Leave default values for all parameters
4. Add 3 controllers and 1 compute
5. Deploy
6. Execute keystone endpoint-list

Expected result: 10 endpoints

Actual result: 24 endpoints

Root cause of the issue is wrong usage of puppet resource keystone_endpoint for nova, cinder, glance, heat and ceilometer.
It looks like puppet modules for those components were updated from puppetlabs upstream but keystone was not.
Hence there are to solutons here:
1. Fix this in each module which was mentioned above.
2. Update keystone module (can bring up unpredictable consequences).

Dmitry Ukov (dukov)
tags: added: customer-found
Ihor Kalnytskyi (ikalnytskyi)
Changed in fuel:
milestone: none → 6.0
importance: Undecided → Medium
assignee: nobody → MOS Keystone (mos-keystone)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/124744

Changed in fuel:
assignee: MOS Keystone (mos-keystone) → Dmitry Ukov (dukov)
status: New → In Progress
Revision history for this message
Aleksandr Didenko (adidenko) wrote :

> Root cause of the issue is wrong usage of puppet resource keystone_endpoint for nova, cinder, glance, heat and ceilometer.

This is not the root cause. The root cause is adaptation patch for keystone which makes keystone_endpoint support both old and new puppet modules. I'm working on the fix.

Changed in fuel:
assignee: Dmitry Ukov (dukov) → Aleksandr Didenko (adidenko)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: master
Review: https://review.openstack.org/128384

Revision history for this message
Aleksandr Didenko (adidenko) wrote :

After https://review.openstack.org/128384 on CentOS HA env, fuel 6.0 master iso:

[root@node-1 ~]# keystone endpoint-list | grep RegionOne | wc -l
10

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-library (master)

Change abandoned by Dmitry Ukov (<email address hidden>) on branch: master
Review: https://review.openstack.org/124744

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/5.1)

Fix proposed to branch: stable/5.1
Review: https://review.openstack.org/128672

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/128384
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=6a0631be7195c31c1beeb761e12f5b58a760a747
Submitter: Jenkins
Branch: master

commit 6a0631be7195c31c1beeb761e12f5b58a760a747
Author: Aleksandr Didenko <email address hidden>
Date: Tue Oct 14 21:03:31 2014 +0300

    Fix keystone_endpoint provider

    Provider logic was broken for the new style keystone_endpoints
    (the ones with "region/service" resource names). Due to this
    puppet created new keystone endpoint even for existing ones on the
    2nd+ runs.

    We need to use common hash key for endpoint_hash in the provider
    for both resource types ("region/service" and "service":region =>)
    So we just force endpoint_hash to use "region/service" as hash key
    for any "keystone_endpoint" resource type.

    This patch and the problem will go away after we sync keystone
    module with upstream.

    Change-Id: I9eaaaf0f63db36a9837cc2141c293efd8adfd68e
    Closes-bug: 1375254

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/5.1)

Reviewed: https://review.openstack.org/128672
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=288170085b24ee797b27aec739bf5e11fcedf2e9
Submitter: Jenkins
Branch: stable/5.1

commit 288170085b24ee797b27aec739bf5e11fcedf2e9
Author: Aleksandr Didenko <email address hidden>
Date: Tue Oct 14 21:03:31 2014 +0300

    Fix keystone_endpoint provider

    Provider logic was broken for the new style keystone_endpoints
    (the ones with "region/service" resource names). Due to this
    puppet created new keystone endpoint even for existing ones on the
    2nd+ runs.

    We need to use common hash key for endpoint_hash in the provider
    for both resource types ("region/service" and "service":region =>)
    So we just force endpoint_hash to use "region/service" as hash key
    for any "keystone_endpoint" resource type.

    This patch and the problem will go away after we sync keystone
    module with upstream.

    Cherry-picked from: https://review.openstack.org/#/c/128384/
    Change-Id: I9eaaaf0f63db36a9837cc2141c293efd8adfd68e
    Closes-bug: 1375254

Revision history for this message
Mike Scherbakov (mihgen) wrote :

Looks like fix was merged to stable/5.1. Are we safe to close this bug for 5.1.1 milestone?

Revision history for this message
Aleksandr Didenko (adidenko) wrote :

> Are we safe to close this bug for 5.1.1 milestone?

We need to check how puppet handles duplicate endpoints with this fix, that are arleady created by 5.1 (i.e. run 5.1 -> 5.1.1 upgrade). We may need to run a pre-upgrade hook which will remove duplicate keystone endpoints.

Revision history for this message
Anastasia Palkina (apalkina) wrote :

Verified on ISO #79

"build_id": "2014-11-05_21-28-15", "ostf_sha": "9c6fadca272427bb933bc459e14bb1bad7f614aa", "build_number": "79", "auth_required": true, "api": "1.0", "nailgun_sha": "01579a53c2be841dc4cafb0be00213243dde7667", "production": "docker", "fuelmain_sha": "35eed79fd6dcfc86a9206ffcd90d385ba5dcca56", "astute_sha": "c72dac7b31646fbedbfc56a2a87676c6d5713fcf", "feature_groups": ["mirantis"], "release": "6.0", "release_versions": {"2014.2-6.0": {"VERSION": {"build_id": "2014-11-05_21-28-15", "ostf_sha": "9c6fadca272427bb933bc459e14bb1bad7f614aa", "build_number": "79", "api": "1.0", "nailgun_sha": "01579a53c2be841dc4cafb0be00213243dde7667", "production": "docker", "fuelmain_sha": "35eed79fd6dcfc86a9206ffcd90d385ba5dcca56", "astute_sha": "c72dac7b31646fbedbfc56a2a87676c6d5713fcf", "feature_groups": ["mirantis"], "release": "6.0", "fuellib_sha": "03c63d4233f6e1c182cc58213cd4542aba5b706d"}}}, "fuellib_sha": "03c63d4233f6e1c182cc58213cd4542aba5b706d"

[root@node-1 ~]# keystone endpoint-list | grep RegionOne | wc -l
12

No duplicated endpoints

Changed in fuel:
status: Fix Committed → Fix Released
Revision history for this message
Anastasia Palkina (apalkina) wrote :

Verified on ISO #17

"build_id": "2014-11-16_21-00-23", "ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346", "build_number": "17", "auth_required": true, "api": "1.0", "nailgun_sha": "2fc6fc4261092a591779a8fb7e3fb1623c6abb85", "production": "docker", "fuelmain_sha": "b118fa4475833ce031ef189ce280772c676fa1c9", "astute_sha": "702af3db6f5bca92525bc8322d7d5d7675ec857e", "feature_groups": ["mirantis"], "release": "5.1.1", "release_versions": {"2014.1.3-5.1.1": {"VERSION": {"build_id": "2014-11-16_21-00-23", "ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346", "build_number": "17", "api": "1.0", "nailgun_sha": "2fc6fc4261092a591779a8fb7e3fb1623c6abb85", "production": "docker", "fuelmain_sha": "b118fa4475833ce031ef189ce280772c676fa1c9", "astute_sha": "702af3db6f5bca92525bc8322d7d5d7675ec857e", "feature_groups": ["mirantis"], "release": "5.1.1", "fuellib_sha": "0d3909b9a291880af28dbe48b9c7d25215aa98ea"}}}, "fuellib_sha": "0d3909b9a291880af28dbe48b9c7d25215aa98ea"

[root@node-3 ~]# keystone endpoint-list | grep RegionOne | wc -l
11

No duplicated endpoints

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.