Fuel for OpenStack

'reconfigure_with_new_fields' test fails by timeout on applying configuration

Bug #1597321 reported by Dmitriy Kruglov
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Committed
High
Michael Polenchuk
Fuel for OpenStack 10.0

Bug Description

Reconfiguration of some neutron configs fails on executing the corresponding deployment task.
The config is successfully applied (whether the corresponding puppet is executed manually or '/api/openstack-config/execute/' Nailgun API is called). But in case of API call the deployment process is stuck on a particular task on primary controller node and is failed on expiry of 30 minutes timeout.
The following tasks were noticed to manifest the issue:
  primary-openstack-network-agents-l3
  primary-openstack-network-agents-metadata
  primary-openstack-network-dhcp_agent

Scenario:
1. Create cluster
2. Add 1 node with compute role and 3 nodes with controller role
3. Deploy the cluster
4. Verify network and run OSTF
5. Update config of, e.g. neutron l3 agent, on primary controller - add a new dummy option and value
6. Apply the configuration for a node by calling '/api/openstack-config/execute/' Nailgun API call

Expected result: the deployment is finished and the config changes are successfully applied.
Actual result: the deployment process is stuck, the puppet.log shows that the task is continuously running. On expiry of 30 minutes timeout the deployment is failed, the corresponding warning can observed in astute.log - http://paste.openstack.org/show/525501/.

Description of the environment:
MOS 10.0-mitaka, ISO build #27.

See original description
Dmitriy Kruglov (dkruglov)
Changed in fuel:
assignee: Fuel QA telco (fuel-qa-telco) → Dmitriy Kruglov (dkruglov)
Sergey Novikov (snovikov)
Changed in fuel:
status: New → Confirmed
Dmitriy Kruglov (dkruglov)
summary: - [system-test] 'reconfigure_with_new_fields' test fails by timeout on
- applying configuration
+ 'reconfigure_with_new_fields' test fails by timeout on applying
+ configuration
Revision history for this message
Dmitriy Kruglov (dkruglov) wrote :
description: updated
Changed in fuel:
importance: Medium → High
assignee: Dmitriy Kruglov (dkruglov) → nobody
Dmitriy Kruglov (dkruglov)
Changed in fuel:
assignee: nobody → Fuel Mixed (fuel-mixed-team)
Revision history for this message
Bartosz Kupidura (zynzel) wrote :

This is caused because we switch node to maintenance mode. In maintenance mode pacemaker services cant restart.

2016-07-05 11:39:16 +0000 Scope(Class[main]) (notice): MODULAR: cluster_mm
2016-07-05 11:39:24 +0000 Scope(Class[Osnailyfacter::Apache::Apache]) (notice): MODULAR: apache/apache.pp
2016-07-05 11:39:44 +0000 Scope(Class[Openstack_tasks::Keystone::Openrc_delete]) (notice): MODULAR: keystone/openrc_delete.pp
2016-07-05 11:39:53 +0000 Scope(Class[Openstack_tasks::Keystone::Openrc_delete]) (notice): MODULAR: keystone/openrc_delete.pp
2016-07-05 11:39:59 +0000 Scope(Class[Openstack_tasks::Keystone::Keystone]) (notice): MODULAR: keystone/keystone.pp
2016-07-05 11:41:23 +0000 Scope(Class[Openstack_tasks::Openstack_controller::Openstack_controller]) (notice): MODULAR: openstack_controller/openstack_controller.pp
2016-07-05 11:41:52 +0000 Scope(Class[Openstack_tasks::Openstack_network::Server_config]) (notice): MODULAR: openstack_network/server_config.pp
2016-07-05 11:42:09 +0000 Scope(Class[Openstack_tasks::Openstack_network::Plugins::Ml2]) (notice): MODULAR: openstack_network/plugins/ml2.pp
2016-07-05 11:42:19 +0000 Scope(Class[Openstack_tasks::Openstack_network::Agents::L3]) (notice): MODULAR: openstack_network/agents/l3.pp

Modular cluster_mm enables MM, but ::openstack_tasks::openstack_network::agents::l3 tries to restart service. This ends with clash.

Revision history for this message
Bartosz Kupidura (zynzel) wrote :

Way to reproduce without uploading configuration:
1) Add to /etc/hiera/astute.yaml:
configuration:
  neutron_l3_agent_config:
    asd/dummy:
      value: 30
2) Put node in MM:
crm node maintenance
3) Execute manifest:
puppet apply --debug /etc/puppet/modules/osnailyfacter/modular/openstack-network/agents/l3.pp

Bogdan Dobrelya (bogdando)
Changed in fuel:
assignee: Fuel Mixed (fuel-mixed-team) → Fuel Sustaining (fuel-sustaining-team)
Michael Polenchuk (mpolenchuk)
Changed in fuel:
assignee: Fuel Sustaining (fuel-sustaining-team) → Michael Polenchuk (mpolenchuk)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/338111

Changed in fuel:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/338111
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=58412e8b5a459572df2456bcd486a42662adfd65
Submitter: Jenkins
Branch: master

commit 58412e8b5a459572df2456bcd486a42662adfd65
Author: Michael Polenchuk <email address hidden>
Date: Wed Jul 6 11:34:27 2016 +0300

    Skip maintenance mode if reconfig is in action

    In maintenance mode pacemaker services couldn't be restarted
    in order to new settings to take effect, so miss out this task.

    Change-Id: I0a45f5b3bf6edb608c33ce3e59a89aeb4495f594
    Closes-Bug: #1597321

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/fuel-library 10.0.0rc1

This issue was fixed in the openstack/fuel-library 10.0.0rc1 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/fuel-library 10.0.0

This issue was fixed in the openstack/fuel-library 10.0.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.