[octane] Control-plane upgrading fails when fuel doesn't have cached service tenant

Bug #1584746 reported by Andrey Volochay
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Committed
Medium
Anastasia Balobashina

Bug Description

Detailed bug description:
 During upgrade MOS6.0 to MOS8.0 via Octane(stable/8.0), I have faced with following bug.

By doing controller-plane upgrade via Octane (octane upgrade-control <orig_env_id> <seed_env_id>), it failed with an error 503. The error has been recieved from original environment.

Steps to reproduce:
 1) Have MOS6.0 environment
 2) Upgrade master node (fuel 6.0) to 6.1 and then to 7.0
 3) Backup environment settings, following documented process
 4) Prepare new master node (fuel 8.0)
 5) Restore the backup from third step
 5) Upgrade restored environment via Octane
 6) Upgrade primary controller via Octane
 7) Upgrade DB via Octane
 8) Upgrade Control Plane via Octane

Expected results:
 Control-plane upgrade is successfully done

Actual result:
 Octane tries to get data from original environment and fails.

Reproducibility:
 Clean /tmp directory at fuel before control-plane upgrade

Workaround:
 Turn off maintenance mode for MySQL in HAproxy on original environment.

Description of the environment:
 Versions of components: MOS8.0, Octane(stable/8.0)
 Network model: Environment uses network bonding
 Related projects installed: Octane

Additional information:
 It happens in octane/commands/upgrade_controlplane.py

 In function "update_neutron_config" at line:
 tenant_id = env_util.cache_service_tenant_id(orig_env)

 Function cache_service_tenant_id can't find cached service tenant data and tries to get it from original environment. But it fails because mysql is locked and keystone returns 503 error.

Changed in fuel:
milestone: none → 8.0-updates
assignee: nobody → Fuel Octane (fuel-octane-team)
importance: Undecided → High
status: New → Confirmed
tags: added: team-upgrades
Revision history for this message
Oleg S. Gelbukh (gelbuhos) wrote :

I believe, that should not be done at that point. We can try to devise some alternative method to get the service tenant ID, I suppose. For example, query the cluster's database directly.

Changed in fuel:
importance: High → Medium
summary: - [octane] Control-plane upgradiung fails when fuel has not cached service
- tenant
+ [octane] Control-plane upgrading fails when fuel doesn't have cached
+ service tenant
Changed in fuel:
assignee: Fuel Octane (fuel-octane-team) → Anastasiya (atolochkova)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-octane (master)

Fix proposed to branch: master
Review: https://review.openstack.org/333788

Changed in fuel:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-octane (master)

Change abandoned by Anastasia Tolochkova (<email address hidden>) on branch: master
Review: https://review.openstack.org/333788

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-octane (master)

Fix proposed to branch: master
Review: https://review.openstack.org/333959

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-octane (master)

Reviewed: https://review.openstack.org/333959
Committed: https://git.openstack.org/cgit/openstack/fuel-octane/commit/?id=c3ba24c8d5fab55410791c064d6ade49a25a1f57
Submitter: Jenkins
Branch: master

commit c3ba24c8d5fab55410791c064d6ade49a25a1f57
Author: Anastasiya <email address hidden>
Date: Fri Jun 24 17:59:06 2016 +0300

    Delete update_neutron_config function

    nova_admin_tenant_id doesn't need to update
    because it doesn't exist in neutron.conf since version 8.0

    Change-Id: Ifa906d6fe34b4ce8cdb5fab5d76a0138469101f1
    Closes-bug: 1584746

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-octane (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/334861

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-octane (stable/mitaka)

Change abandoned by Anastasia Tolochkova (<email address hidden>) on branch: stable/mitaka
Review: https://review.openstack.org/334861

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to fuel-octane (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/343602

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to fuel-octane (master)

Reviewed: https://review.openstack.org/343602
Committed: https://git.openstack.org/cgit/openstack/fuel-octane/commit/?id=c01c4d92d343745e136cbafc233e336f5fcef5f2
Submitter: Jenkins
Branch: master

commit c01c4d92d343745e136cbafc233e336f5fcef5f2
Author: Oleg Gelbukh <email address hidden>
Date: Mon Jul 4 14:33:24 2016 +0000

    Remove leftover neutron restart from controller post-deploy stage

    Since neutron.conf file is not updated anymore with valid value
    of nova_admin_tenant_id, the restart of neutron server in post
    deployment stage of upgrade operation is not needed.

    Remove command that restarts neutron server.

    Change-Id: I4e2ebe49221c0d0f02609d68cb343e2a707c5bad
    Related-bug: 1584746

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to fuel-octane (stable/8.0)

Reviewed: https://review.openstack.org/337257
Committed: https://git.openstack.org/cgit/openstack/fuel-octane/commit/?id=393247633e7afe64988548cd68491a5922a31c00
Submitter: Jenkins
Branch: stable/8.0

commit 393247633e7afe64988548cd68491a5922a31c00
Author: Oleg Gelbukh <email address hidden>
Date: Mon Jul 4 14:33:24 2016 +0000

    Remove leftover neutron restart from controller post-deploy stage

    Since neutron.conf file is not updated anymore with valid value
    of nova_admin_tenant_id, the restart of neutron server in post
    deployment stage of upgrade operation is not needed.

    Remove command that restarts neutron server.

    Change-Id: I4e2ebe49221c0d0f02609d68cb343e2a707c5bad
    Related-bug: 1584746
    (cherry picked from commit c01c4d92d343745e136cbafc233e336f5fcef5f2)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to fuel-octane (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/351721

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-octane (master)

Change abandoned by Ilya Kharin (<email address hidden>) on branch: master
Review: https://review.openstack.org/351721
Reason: This patch have to go in stable/mitaka instead of master.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to fuel-octane (stable/mitaka)

Related fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/351728

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to fuel-octane (stable/mitaka)
Download full text (17.8 KiB)

Reviewed: https://review.openstack.org/351728
Committed: https://git.openstack.org/cgit/openstack/fuel-octane/commit/?id=3101bf302ea0f2d501fd42a6ae6046c3587ac093
Submitter: Jenkins
Branch: stable/mitaka

commit a05d2ecf13e15c12f10e1517fae425e583e0d714
Author: Nikita Zubkov <email address hidden>
Date: Thu Aug 4 17:51:56 2016 +0300

    fix ssh connection auth

    According to spec [1] service user must use key authentication​ for ssh

    [1] https://github.com/openstack/fuel-specs/blame/ba4504/specs/9.0/fuel-nonroot-openstack-nodes.rst#L61-L64

    Change-Id: I461f659d4f0985283a43ec6a702bdd78af91a072

commit 7140398e501879a4e1f5605f3d1cc99195464850
Author: Sergey Abramov <email address hidden>
Date: Tue Aug 2 12:12:57 2016 +0300

    Setup upgrade levels for each fuel version

    Setup upgrade levels block in nova.conf for each fuel version,
    this is required for backward compatibility on RPC.

    Change-Id: I6ccbc34457d11616b015cdb1e46e733b0b49ce58

commit 9e2a419ac79ad591faee9afe16419bf4497eacad
Author: Vladimir Khlyunev <email address hidden>
Date: Mon Aug 1 21:31:56 2016 +0300

    Fix several typos

    Copypasting is evil, lets remove typos

    Change-Id: Icf4bec1a3de954911a469fa98380af2b6175da08

commit 013a7ded2275245a38ee58289dac2bc8b90fa37e
Author: Sergey Abramov <email address hidden>
Date: Wed Jul 20 16:05:10 2016 +0300

    Host evacuation using nova

    Evacuate instance from compute node using
    nova host-evacuate-live except host_evacuation.sh script

    Change-Id: I01a04cfa1ed2aafce7987e523e6b589dea21fbb0

commit 00a1d0dce1f7ae5db6706a59353403516f7913dc
Author: Sergey Abramov <email address hidden>
Date: Tue Jul 19 17:10:44 2016 +0300

    Change shutoff vm command

    * run nova commands using nova util function
    * host filter using get nova node handle method(
        hostname for fuel < 6.1 and fqdn for newer version)

    Change-Id: Ib5c373714e359982b13910a225787167db7b35e9

commit 49357c98313c5f551b13d0877581c5fb5e108db1
Author: Nikita Zubkov <email address hidden>
Date: Mon Jun 27 18:40:13 2016 +0300

    Add `stderr_log_level` parameter to popen

    Now posible to set logging level in popen for logging stderr

    fuel-bootstrap build comand stderr now logged with INFO level

    Change-Id: Ibaa2be54c68ce7a428e07052c917f54e99b95a5e

commit fb582b271dab4bdd10d27149a77c0e1f6a580b9e
Author: Sergey Abramov <email address hidden>
Date: Thu Jul 21 17:56:41 2016 +0300

    Add run_nova_cmd function

    This is required for running nova command on contoller without
    manipulating manipulating with command.

    It makes code easier and more readable

    Usage:

        from fuelclient.objects import environment as environment_obj

        from octane.util import env
        from octane.util import nova

        env_id = ...
        my_env = environment_obj.Environment(env_id)
        controller = env.get_one_controller(my_env)
        nova_cmd_just_run = [...]
        nova_cmd_with_output = [...]
        nova.run_nova_cmd(nova_cmd_just_run, controller, False)
        result = nova.run_nova_cmd(nova_cmd_w...

tags: added: in-stable-mitaka
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.