Last night, the Juju 2 controllers in PS4.5 were upgraded to Juju 2.1 (I think 2.1.1). Some of the models are still on 2.0.3. I don't know for sure that this bug only affects the older models, but that's all the data I have so far.
My first attempted deployment after the controller upgrade failed, with "juju status --format=yaml" reporting (elided the silly byte-by-byte UserData serialisations for length; more complete version in https://pastebin.canonical.com/181839/):
"359":
juju-status:
current: down
message: agent is not communicating with the server
since: 08 Mar 2017 13:52:50Z
instance-id: pending
machine-status:
current: provisioning error
message: |-
cannot run instance: failed to run a server with nova.RunServerOpts{Name:"juju-83af10-stg-ols-snap-build-359", FlavorId:"e0ec0afc-6baa-4dda-aeae-164eed4590e3", ImageId:"df6158c5-1733-4972-84d8-480ead5af756", UserData:[]uint8{...}, SecurityGroupNames:[]nova.SecurityGroupName{nova.SecurityGroupName{Name:"juju-4955af7b-a72f-4076-88e4-99854e92d50f-ffaf8edc-1e06-490e-8991-1700fd83af10"}, nova.SecurityGroupName{Name:"juju-4955af7b-a72f-4076-88e4-99854e92d50f-ffaf8edc-1e06-490e-8991-1700fd83af10-359"}}, Networks:[]nova.ServerNetworks{nova.ServerNetworks{NetworkId:"cf014c32-f1ab-4447-a12e-5a2832f61452", FixedIp:"", PortId:""}}, AvailabilityZone:"prodstack-zone-2", Metadata:map[string]string{"juju-controller-uuid":"4955af7b-a72f-4076-88e4-99854e92d50f", "juju-model-uuid":"ffaf8edc-1e06-490e-8991-1700fd83af10", "juju-units-deployed":"snap-build-app-r6923965/0"}, ConfigDrive:false}
caused by: request (http://10.24.0.176:8774/v2/f7efb9e5b0a7457795db11179089c583/servers) returned unexpected status: 409; error info: {"conflictingRequest": {"message": "Multiple security_group matches found for name 'juju-4955af7b-a72f-4076-88e4-99854e92d50f-ffaf8edc-1e06-490e-8991-1700fd83af10', use an ID to be more specific.", "code": 409}}
since: 08 Mar 2017 13:48:50Z
series: xenial
"360":
juju-status:
current: down
message: agent is not communicating with the server
since: 08 Mar 2017 13:52:52Z
instance-id: pending
machine-status:
current: provisioning error
message: |-
cannot run instance: failed to run a server with nova.RunServerOpts{Name:"juju-83af10-stg-ols-snap-build-360", FlavorId:"e0ec0afc-6baa-4dda-aeae-164eed4590e3", ImageId:"df6158c5-1733-4972-84d8-480ead5af756", UserData:[]uint8{...}, SecurityGroupNames:[]nova.SecurityGroupName{nova.SecurityGroupName{Name:"juju-4955af7b-a72f-4076-88e4-99854e92d50f-ffaf8edc-1e06-490e-8991-1700fd83af10"}, nova.SecurityGroupName{Name:"juju-4955af7b-a72f-4076-88e4-99854e92d50f-ffaf8edc-1e06-490e-8991-1700fd83af10-360"}}, Networks:[]nova.ServerNetworks{nova.ServerNetworks{NetworkId:"cf014c32-f1ab-4447-a12e-5a2832f61452", FixedIp:"", PortId:""}}, AvailabilityZone:"prodstack-zone-2", Metadata:map[string]string{"juju-controller-uuid":"4955af7b-a72f-4076-88e4-99854e92d50f", "juju-model-uuid":"ffaf8edc-1e06-490e-8991-1700fd83af10", "juju-units-deployed":"snap-build-app-r6923965/1"}, ConfigDrive:false}
caused by: request (http://10.24.0.176:8774/v2/f7efb9e5b0a7457795db11179089c583/servers) returned unexpected status: 409; error info: {"conflictingRequest": {"message": "Multiple security_group matches found for name 'juju-4955af7b-a72f-4076-88e4-99854e92d50f-ffaf8edc-1e06-490e-8991-1700fd83af10', use an ID to be more specific.", "code": 409}}
since: 08 Mar 2017 13:49:30Z
series: xenial
I tried removing the new secgroup, but that didn't help; juju just put it right back on the next run. I then tried removing the old secgroup. That required replacing it on all machines, and for some reason this mysteriously caused the units within the deployment to be unable to send any packets to each other. Chris Stratford eventually suggested the workaround of running "juju add-unit" on a random application, which apparently caused juju to fix up the inter-unit networking although we couldn't work out exactly what it had done.
Since all that, I've heard the same report from two other people, so I investigated further. I tracked down commit 3e8e8e5c9c68237fb2cc7d10588711a900eacc1b ("OpenStack Provider: Use Neutron networking instead of Nova for FloatingIPs, SecurityGroups, SecurityGroupRules and Networks"), which I'm pretty sure is responsible for this. Some portions of the diff jump out at me:
- defaultGroup, err := c.environ.nova().SecurityGroupByName("default")
+ // Security Group Names in Neutron do not have to be unique. This
+ // function returns an array
+ defaultGroups, err := c.environ.neutron().SecurityGroupByNameV2("default")
- group, err := novaClient.SecurityGroupByName(name)
+ groupsFound, err := neutronClient.SecurityGroupByNameV2(name)
if err == nil {
- // Group exists, so assume it is correctly set up and return it.
- // TODO(jam): 2013-09-18 http://pad.lv/121795
- // We really should verify the group is set up correctly,
- // because deleting and re-creating environments can get us bad
- // groups (especially if they were set up under Python)
- return *group, nil
+ for _, group := range groupsFound {
+ if c.verifyGroupRules(group.Rules, rules) {
+ return group, nil
+ }
+ }
}
So I suspect that a better workaround in this situation would be to delete the new secgroup and add the correct rules (possibly the IPv6 rules, which weren't present in the old secgroup) to the old secgroup using neutron. However, this seems to be a pretty bad incompatibility. Perhaps juju should be more conservative and use the old "find existing group and append to it" behaviour with neutron just as it used to do with nova, even if neutron itself is happy with duplicate secgroup names?
I believe that even Juju 2.1 runs nova with the security group name rather than its ID, so it seems to me that this will probably break even with 2.1 on the model, but I can't prove that at the moment. My guess is that you can reproduce by creating an OpenStack-based model with 2.0, then upgrading the controller to 2.1, then trying to deploy anything more to the model.