Problems using Juju multi-cloud controller with multiple Openstack clouds having the same region name

Hi team,

Please fix this as this has been impacting us for some time and it is getting more urgent now.

Thanks.

Any updates here - no update since March 2nd and it's impacting customer

Do you have any logs and output to share on this issue.

Hi, I'm the end customer who worked with Nikolay to identify this bug affecting my multi-cloud OpenStack environment. Here's a sanitized version of the Juju CLI output that I shared with him previously via internal e-mail:

# A new model 'site1-model' created on the same OpenStack site 'site1-cloud' as the controller
# inherits the valid Glance image ID that was provided when bootstrapping Juju in that site
holmesb5@runner1:~$ juju add-model site1-model site1-cloud/RegionOne
Added 'site1-model' model on site1-cloud/RegionOne with credential 'my-credential' for user 'admin'

holmesb5@runner1:~$ juju metadata list-images -m site1-model
Source Version Arch Region Image ID Stream Virt Type Storage Type
custom 20.04 amd64 RegionOne fbc3a64c-bd93-48d5-8e6f-cf8d9f1f55bb released

# A second new model 'site2-model' in the distinct OpenStack cloud 'site2-cloud' (which uses
# the same region name as site1-cloud) unexpectedly picks up the image ID from site1-cloud,
# which of course does not exist in site2-cloud's Glance
holmesb5@runner1:~$ juju add-model site2-model site2-cloud/RegionOne
Added 'site2-model' model on site2-cloud/RegionOne with credential 'my-credential' for user 'admin'

holmesb5@runner1:~$ juju metadata list-images -m site2-model
Source Version Arch Region Image ID Stream Virt Type Storage Type
custom 20.04 amd64 RegionOne fbc3a64c-bd93-48d5-8e6f-cf8d9f1f55bb released

# Attempt to fix this by putting a valid site2-cloud image ID in manually
holmesb5@runner1:~$ juju metadata add-image -m site2-model --series focal --region RegionOne dd63d4be-8a9e-4a64-a4db-bc7f307ece1f

holmesb5@runner1:~$ juju metadata list-images -m site2-model
Source Version Arch Region Image ID Stream Virt Type Storage Type
custom 20.04 amd64 RegionOne dd63d4be-8a9e-4a64-a4db-bc7f307ece1f released

# This overwrote the Glance image ID stored for site1-cloud, leaving *that* site with an
# invalid image reference and blocking the creation of new machines there
holmesb5@runner1:~$ juju metadata list-images -m site1-model
Source Version Arch Region Image ID Stream Virt Type Storage Type
custom 20.04 amd64 RegionOne dd63d4be-8a9e-4a64-a4db-bc7f307ece1f released

# Looking at the Juju controller's actual MongoDB content, the 'cloudimagemetadata'
# collection only accounts for the cloud's region name in each image record. The cloud's
# endpoint URL needs to be stored as well to properly disambiguate images when
# multiple clouds share a common region name.
root@juju-c37988-controller-0:/var/snap/juju-db/common# juju-db.mongo localhost:37017/juju --authenticationDatabase admin --ssl --sslAllowInvalidCertificates --username machine-0 –password xxxxxxxx

juju:PRIMARY> db.cloudimagemetadata.find({})
{ "_id" : "released:RegionOne:focal:amd64:::custom", "image_id" : "fbc3a64c-bd93-48d5-8e6f-cf8d9f1f55bb", "stream" : "released", "region" : "RegionOne", "version" : "20.04", "series" : "focal", "arch" : "amd64", "root_storage_size" : NumberLong(0), "date_created" : NumberLong("1...

This issue is caused by juju not storing image metadata with a cloud name nor a model uuid. There are a few
other bugs associated with this as well; juju metadata add-image -m <model>, isn't actually model specific etc.

Fixing this is non trivial within juju due to surrounding details and will require a juju upgrade once the fix is implemented.

I've asked for help with a POC to verify a potential work around in the mean time is possible.

@nickolay, can you please verify the exact method being using in this config to supply cloud image metadata to juju? Glance-simplestreams-sync charm? image-agent-url model config? juju metadata add-image? A different one?