Ironic

Rolling upgrades: backward incompatible changes in RPC methods

Bug #1502903 reported by Grzegorz Grasza
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ironic
Won't Fix
High
Jim Rollenhagen

Bug Description

In the Liberty release, incompatible changes were made in the RPC API, which breaks rolling upgrades (upgrades when Ironic services are run at different versions simultaneously).

One example is a refactoring of a return value:

https://github.com/openstack/ironic/commit/37bc3e1ca641e70eefa902d8f366a4369bd509ad

The way to address compatibility issues is to support both versions on the RPC client. Here is an example from Nova on how to do this:

https://github.com/openstack/nova/blob/stable/kilo/nova/compute/rpcapi.py#L357-L364

Maybe we could address this bug in a similar way as in
https://bugs.launchpad.net/bugs/1502895
(backward incompatible changes in versioned objects).

See original description
Grzegorz Grasza (xek)
description: updated
Revision history for this message
Grzegorz Grasza (xek) wrote :

https://www.youtube.com/watch?v=wVXiw09XP04 - a discussion about RPC versioning in Nova

Dmitry Tantsur (divius)
Changed in ironic:
status: New → Triaged
importance: Undecided → High
tags: added: liberty-rc-potential
Revision history for this message
Ruby Loo (rloo) wrote :

(I think Jim is working on this.)

Changed in ironic:
assignee: nobody → Jim Rollenhagen (jim-rollenhagen)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ironic (master)

Fix proposed to branch: master
Review: https://review.openstack.org/234450

Changed in ironic:
status: Triaged → In Progress
Revision history for this message
Jim Rollenhagen (jim-rollenhagen) wrote :

I'm having trouble finding a way to do this on the conductor side - if we do it on the API side, it's not useful as you'd need the updated code on the API to get the fix, and at that point you've completed the upgrade.

Revision history for this message
Tan Lin (tan-lin-good) wrote :

I think during upgrades, services who are direct talk to DB should upgrade first, because of the change of DB schema. So in this case, conductor is the one should always the latest and it is able to talk with different version of other services like API.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on ironic (master)

Change abandoned by Jim Rollenhagen (<email address hidden>) on branch: master
Review: https://review.openstack.org/234450
Reason: Rolling upgrades will still be broken if we fix this bug. So, I'm dropping it.

Revision history for this message
Jim Rollenhagen (jim-rollenhagen) wrote :

So, I did find a way to accomplish this, however fixing this specific instance won't actually fix rolling upgrades for Liberty. So I don't think this is worth fixing.

I've opened a blueprint for truly supporting rolling upgrades; there's a lot of work to do. https://blueprints.launchpad.net/ironic/+spec/online-upgrade-support

Marking this WONTFIX.

Changed in ironic:
status: In Progress → Won't Fix
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Related blueprints

Remote bug watches

Bug watches keep track of this bug in other bug trackers.