The service reconfiguration doesn't work for nova and keystone

Bug #1644656 reported by Sergey Novikov
26
This bug affects 3 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Committed
High
Dmitry Ukov
Nominated for Ocata by Oleksiy Molchanov
Mitaka
Fix Released
High
Dmitry Ukov
Newton
Fix Released
High
Dmitry Ukov

Bug Description

Detailed bug description:
The issue was found by
https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.services_reconfiguration_thread_2/133/testReport/(root)/reconfigure_with_new_fields/

https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.services_reconfiguration_thread_1/132/testReport/(root)/reconfigure_keystone_to_use_ldap/

https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.services_reconfiguration_thread_1/132/testReport/(root)/reconfigure_nova_ephemeral_disk/

The serialization of new config data is executed correctly. The corresponding data is passed to target node in astute.yaml, in section "configuration", some manifests are triggered, but a new config is not applied. Probably, this change https://review.openstack.org/#/c/396199/ affects a correct behaviour of openstack service reconfiguration.
Steps to reproduce:
 1. deploy cluster compute+controller
 2. upload a new configuration for nova(compute) and apply that for compute node
 3. check corresponding nova config file on compute node

Description of the environment:
snapshot #549

Changed in fuel:
assignee: nobody → Fuel Sustaining (fuel-sustaining-team)
importance: Undecided → Medium
status: New → Confirmed
Changed in fuel:
assignee: Fuel Sustaining (fuel-sustaining-team) → Oleksiy Molchanov (omolchanov)
Dmitry Ukov (dukov)
Changed in fuel:
assignee: Oleksiy Molchanov (omolchanov) → Dmitry Ukov (dukov)
Changed in fuel:
milestone: 9.2 → 11.0
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/403778

Changed in fuel:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/newton)

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/404093

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/404094

Revision history for this message
Timur Nurlygayanov (tnurlygayanov) wrote :

Priority changed to High because it affects BVT pass rate.

Changed in fuel:
importance: Medium → High
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/403778
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=4b0630f05739665e0c77099f9c03f88760702fea
Submitter: Jenkins
Branch: master

commit 4b0630f05739665e0c77099f9c03f88760702fea
Author: Dmitry Ukov <email address hidden>
Date: Mon Nov 28 18:56:55 2016 +0300

    Changed default behaviour for override_resources

    Puppet type override_resources should create new resource in case this
    resource is not in catalog

    Change-Id: Idbf48dd8148dc46890792847420e5050fad9e3d0
    Closes-Bug: 1644656

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/newton)

Reviewed: https://review.openstack.org/404093
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=741ce9e47a5cf6ddf07f95409f58ff17a3788ab4
Submitter: Jenkins
Branch: stable/newton

commit 741ce9e47a5cf6ddf07f95409f58ff17a3788ab4
Author: Dmitry Ukov <email address hidden>
Date: Mon Nov 28 18:56:55 2016 +0300

    Changed default behaviour for override_resources

    Puppet type override_resources should create new resource in case this
    resource is not in catalog

    Change-Id: Idbf48dd8148dc46890792847420e5050fad9e3d0
    Closes-Bug: 1644656

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/mitaka)

Reviewed: https://review.openstack.org/404094
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=ac5912b0c886f746fe632aa12c54596325ceb5f1
Submitter: Jenkins
Branch: stable/mitaka

commit ac5912b0c886f746fe632aa12c54596325ceb5f1
Author: Dmitry Ukov <email address hidden>
Date: Mon Nov 28 18:56:55 2016 +0300

    Changed default behaviour for override_resources

    Puppet type override_resources should create new resource in case this
    resource is not in catalog

    Change-Id: Idbf48dd8148dc46890792847420e5050fad9e3d0
    Closes-Bug: 1644656

tags: added: on-verification
Revision history for this message
Sergey Novikov (snovikov) wrote :
Revision history for this message
Roman Podoliaka (rpodolyaka) wrote :

I wonder if it's a duplicate of https://bugs.launchpad.net/fuel/+bug/1646833 or just a similar problem.

Revision history for this message
Dmitry Ukov (dukov) wrote :

This bug is related to slightly changed configuration format for Advanced Configuration feature
and is not causing issues with OSTF described here https://bugs.launchpad.net/fuel/+bug/1646833

This bug should be fixed by https://review.openstack.org/#/c/409893/ and one more commit to fuel-qa

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/411659

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-qa (master)

Fix proposed to branch: master
Review: https://review.openstack.org/411724

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-qa (stable/newton)

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/411726

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-qa (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/411727

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/newton)

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/411751

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-qa (stable/mitaka)

Reviewed: https://review.openstack.org/411727
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=c8f829a8beb0c159952a4bcb2788f9616dab5171
Submitter: Jenkins
Branch: stable/mitaka

commit c8f829a8beb0c159952a4bcb2788f9616dab5171
Author: Dmitry Ukov <email address hidden>
Date: Fri Dec 16 13:18:24 2016 +0400

    Configuration templates updated

    We need to update configuration templates with services definition and
    appropriate "notify" parameters since new OpenStack configuration format
    has been introduced.

    Change-Id: Id195ee1bc9ecd4443ad3eae6e3bc5fae65b232e3
    Closes-Bug: 1644656

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-qa (master)

Reviewed: https://review.openstack.org/411724
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=5a7c0baec52fee64a4e095c45cf778b24eae3ecd
Submitter: Jenkins
Branch: master

commit 5a7c0baec52fee64a4e095c45cf778b24eae3ecd
Author: Dmitry Ukov <email address hidden>
Date: Fri Dec 16 13:18:24 2016 +0400

    Configuration templates updated

    We need to update configuration templates with services definition and
    appropriate "notify" parameters since new OpenStack configuration format
    has been introduced.

    Change-Id: Id195ee1bc9ecd4443ad3eae6e3bc5fae65b232e3
    Closes-Bug: 1644656

Revision history for this message
Dmitry Belyaninov (dbelyaninov) wrote :

failed again, waiting tomorrows run

Revision history for this message
Dmitry Ukov (dukov) wrote :

Commits to fuel-libry have not been merged
We need to merge all these [1] commits to fix the issue

[1] https://review.openstack.org/#/q/topic:bug/1644656

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/409893
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=0e29cd247d46a8886be0b84580011925b1bddf84
Submitter: Jenkins
Branch: master

commit 0e29cd247d46a8886be0b84580011925b1bddf84
Author: Dmitry Ukov <email address hidden>
Date: Mon Dec 12 23:15:48 2016 +0400

    Added support for Notify/Subscribe

    Since eval_generate executed after relationship_graph has been formed we
    need to insert dependencies for new resources

    Change-Id: Ic9ad10cc4eb5ddbcd9ecf673b0bff3728b1d51e3
    Closes-Bug: 1649380
    Closes-Bug: 1644656

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/newton)

Reviewed: https://review.openstack.org/411751
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=75682bbdd3bafe5f5485b6b4b87d8d62c37c3190
Submitter: Jenkins
Branch: stable/newton

commit 75682bbdd3bafe5f5485b6b4b87d8d62c37c3190
Author: Dmitry Ukov <email address hidden>
Date: Mon Dec 12 23:15:48 2016 +0400

    Added support for Notify/Subscribe

    Since eval_generate executed after relationship_graph has been formed we
    need to insert dependencies for new resources

    Change-Id: Ic9ad10cc4eb5ddbcd9ecf673b0bff3728b1d51e3
    Closes-Bug: 1649380
    Closes-Bug: 1644656

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/mitaka)

Reviewed: https://review.openstack.org/411659
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=e3bd10c2365dc52f617013a8360c3a48e797ecff
Submitter: Jenkins
Branch: stable/mitaka

commit e3bd10c2365dc52f617013a8360c3a48e797ecff
Author: Dmitry Ukov <email address hidden>
Date: Mon Dec 12 23:15:48 2016 +0400

    Added support for Notify/Subscribe

    Since eval_generate executed after relationship_graph has been formed we
    need to insert dependencies for new resources

    Change-Id: Ic9ad10cc4eb5ddbcd9ecf673b0bff3728b1d51e3
    Closes-Bug: 1649380
    Closes-Bug: 1644656

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-qa (master)

Fix proposed to branch: master
Review: https://review.openstack.org/414545

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-qa (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/414546

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-qa (stable/newton)

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/414547

Revision history for this message
Alexander Kurenyshev (akurenyshev) wrote :

Moved to In progress due to fixes are still on review

Changed in fuel:
status: Fix Committed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-qa (stable/newton)

Reviewed: https://review.openstack.org/411726
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=3a0841a5e7d49259e4c53be0a04f63b54772bfe0
Submitter: Jenkins
Branch: stable/newton

commit 3a0841a5e7d49259e4c53be0a04f63b54772bfe0
Author: Dmitry Ukov <email address hidden>
Date: Fri Dec 16 13:18:24 2016 +0400

    Configuration templates updated

    We need to update configuration templates with services definition and
    appropriate "notify" parameters since new OpenStack configuration format
    has been introduced.

    Change-Id: Id195ee1bc9ecd4443ad3eae6e3bc5fae65b232e3
    Closes-Bug: 1644656

tags: added: swarm-blocker
removed: swarm-fail
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-qa (stable/mitaka)

Reviewed: https://review.openstack.org/414546
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=94a7196300d98c0325b0e1ea8d06275eecc94e08
Submitter: Jenkins
Branch: stable/mitaka

commit 94a7196300d98c0325b0e1ea8d06275eecc94e08
Author: Dmitry Ukov <email address hidden>
Date: Fri Dec 23 15:28:36 2016 +0400

    Configuration should be applied for specific nodes

    We need to apply configuration to the node with particular service
    running or we need to extent configuration templates processing to
    support additional parameters of override_resources Puppet type

    Change-Id: Iae6c2e5f2b3c43fa4f750802a98e927dca95c251
    Closes-Bug: 1644656

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-qa (master)

Change abandoned by Dmitry Ukov (<email address hidden>) on branch: master
Review: https://review.openstack.org/419489

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-qa (master)

Fix proposed to branch: master
Review: https://review.openstack.org/419875

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-qa (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/419876

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-qa (stable/newton)

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/419877

Revision history for this message
Alexander Kurenyshev (akurenyshev) wrote :

Moved to In progress due to fixes are still on review

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/421887

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/newton)

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/422148

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/422151

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/421887
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=63805ab80a362d04795952110424c44d1eeae0cf
Submitter: Jenkins
Branch: master

commit 63805ab80a362d04795952110424c44d1eeae0cf
Author: Dmitry Ukov <email address hidden>
Date: Wed Jan 18 17:03:38 2017 +0400

    Resources shoud be created in a separate task

    We shoud not create resources in all tasks. Only resouce parameters
    update should be available for user. Separate task will be used
    in order to create new resources (e.g. to introduce new
    configuration file parameters).

    Change-Id: I89ff7d9b6434519107f88d26fceb564a3bda2edc
    Closes-Bug: 1644656
    Closes-Bug: 1655928

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/mitaka)

Reviewed: https://review.openstack.org/422151
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=24318bea019fd2a8312b280853f1e7fa34f2edff
Submitter: Jenkins
Branch: stable/mitaka

commit 24318bea019fd2a8312b280853f1e7fa34f2edff
Author: Dmitry Ukov <email address hidden>
Date: Wed Jan 18 17:03:38 2017 +0400

    Resources shoud be created in a separate task

    We shoud not create resources in all tasks. Only resouce parameters
    update should be available for user. Separate task will be used
    in order to create new resources (e.g. to introduce new
    configuration file parameters).

    Change-Id: I89ff7d9b6434519107f88d26fceb564a3bda2edc
    Closes-Bug: 1644656
    Closes-Bug: 1655928

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/newton)

Reviewed: https://review.openstack.org/422148
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=62e6754e7f30ded4d57357ab576f046edc37fe8b
Submitter: Jenkins
Branch: stable/newton

commit 62e6754e7f30ded4d57357ab576f046edc37fe8b
Author: Dmitry Ukov <email address hidden>
Date: Wed Jan 18 17:03:38 2017 +0400

    Resources shoud be created in a separate task

    We shoud not create resources in all tasks. Only resouce parameters
    update should be available for user. Separate task will be used
    in order to create new resources (e.g. to introduce new
    configuration file parameters).

    Change-Id: I89ff7d9b6434519107f88d26fceb564a3bda2edc
    Closes-Bug: 1644656
    Closes-Bug: 1655928

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-qa (stable/newton)

Change abandoned by Dmitry Ukov (<email address hidden>) on branch: stable/newton
Review: https://review.openstack.org/419877

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-qa (master)

Change abandoned by Dmitry Ukov (<email address hidden>) on branch: master
Review: https://review.openstack.org/419875

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-qa (stable/mitaka)

Change abandoned by Dmitry Ukov (<email address hidden>) on branch: stable/mitaka
Review: https://review.openstack.org/419876

Revision history for this message
Dmitry Belyaninov (dbelyaninov) wrote :
Revision history for this message
Dmitry Ukov (dukov) wrote :

We need to remove
    identity/driver:
      value: keystone.identity.backends.ldap.Identity
from config template in fuel-qa.
We can not switch keystone identity driver on ldap with fake ldap server
    ldap/url:
      value: ldap://dc.example.com

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-qa (master)

Reviewed: https://review.openstack.org/414545
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=7cc13fd54c0b6c6805e2a7b8dff7d1923d6e20ef
Submitter: Jenkins
Branch: master

commit 7cc13fd54c0b6c6805e2a7b8dff7d1923d6e20ef
Author: Dmitry Ukov <email address hidden>
Date: Fri Dec 23 15:28:36 2016 +0400

    Configuration should be applied for specific nodes

    We need to apply configuration to the node with particular service
    running or we need to extent configuration templates processing to
    support additional parameters of override_resources Puppet type

    Change-Id: Iae6c2e5f2b3c43fa4f750802a98e927dca95c251
    Closes-Bug: 1644656

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-qa (stable/newton)

Reviewed: https://review.openstack.org/414547
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=c0f52686259c46fd7c7dbf5963948d07ba4384c9
Submitter: Jenkins
Branch: stable/newton

commit c0f52686259c46fd7c7dbf5963948d07ba4384c9
Author: Dmitry Ukov <email address hidden>
Date: Fri Dec 23 15:28:36 2016 +0400

    Configuration should be applied for specific nodes

    We need to apply configuration to the node with particular service
    running or we need to extent configuration templates processing to
    support additional parameters of override_resources Puppet type

    Change-Id: Iae6c2e5f2b3c43fa4f750802a98e927dca95c251
    Closes-Bug: 1644656

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.openstack.org/423106
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=c490e5af21e737b349263d63243f46d27e075649
Submitter: Jenkins
Branch: stable/newton

commit c490e5af21e737b349263d63243f46d27e075649
Author: Dmitry Ukov <email address hidden>
Date: Fri Jan 20 11:30:39 2017 +0400

    Updated config upload procedure and params

    * Configuration should be changed on the appropriate
      nodes (e.g. keystone should be configured only on
      controller roles)
    * We should not change identity driver on LDAP with
      fake LDAP server since this breaks keystone and whole
      cloud as a result

    Change-Id: I3218bcf13588cbecef7678b7bebae0d713e1ad7b
    Closes-Bug: 1644656

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-qa (stable/mitaka)

Reviewed: https://review.openstack.org/423107
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=903500d63b09fec236c02fa01301a37ebf6eca0f
Submitter: Jenkins
Branch: stable/mitaka

commit 903500d63b09fec236c02fa01301a37ebf6eca0f
Author: Dmitry Ukov <email address hidden>
Date: Fri Jan 20 11:30:39 2017 +0400

    Updated config upload procedure and params

    * Configuration should be changed on the appropriate
      nodes (e.g. keystone should be configured only on
      controller roles)
    * We should not change identity driver on LDAP with
      fake LDAP server since this breaks keystone and whole
      cloud as a result

    Change-Id: I3218bcf13588cbecef7678b7bebae0d713e1ad7b
    Closes-Bug: 1644656

tags: removed: on-verification
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-qa (master)

Reviewed: https://review.openstack.org/423101
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=33a6f979e310fc600de81de1613e0381bb45e49d
Submitter: Jenkins
Branch: master

commit 33a6f979e310fc600de81de1613e0381bb45e49d
Author: Dmitry Ukov <email address hidden>
Date: Fri Jan 20 11:30:39 2017 +0400

    Updated config upload procedure and params

    * Configuration should be changed on the appropriate
      nodes (e.g. keystone should be configured only on
      controller roles)
    * We should not change identity driver on LDAP with
      fake LDAP server since this breaks keystone and whole
      cloud as a result

    Change-Id: I3218bcf13588cbecef7678b7bebae0d713e1ad7b
    Closes-Bug: 1644656

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/fuel-qa 11.0.0.0rc1

This issue was fixed in the openstack/fuel-qa 11.0.0.0rc1 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/fuel-library 11.0.0.0rc1

This issue was fixed in the openstack/fuel-library 11.0.0.0rc1 release candidate.

Revision history for this message
Ilya Bumarskov (ibumarskov) wrote :
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.