[Feature] Add option to use filter_scheduler weights

Hi Diko. What version of OpenStack is the customer using please? Just for info on what this might need to be backported to if implemented. Thanks!

The release is Ussuri.

Also, the customer's interest is stacking of GPU pass-through instances instead spreading across the fleet. Let's say 10 VMs (with 1 GPU) are requested first with 10 hypervisors (with 4 GPUs each), and another VM with 4 GPU. Then, each hypervisor would have 1 VM each with 1 GPU consumed out of 4, then the next 4-GPU VM request would fail. That's the scenario they want to avoid.

The description for Stacking CPU resources is clear to me:
https://docs.openstack.org/nova/latest/admin/scheduling.html#cpuweigher
> If the multiplier, filter_scheduler.cpu_weight_multiplier, is negative,
> the host with least CPUs available will win (useful for stacking hosts,
> instead of spreading).

However, nagative value for PCIWeigher is not allowed on purpose.
https://docs.openstack.org/nova/latest/admin/scheduling.html#pciweigher
> Only positive values are allowed for the multiplier of this weigher as
> a negative value would force non-PCI instances away from non-PCI hosts,
> thus, causing future scheduling issues.

So I'm not so sure if the stacking behavior of GPUs is actually possible.

Or we could set a huge weight like 9999 to achieve the following behavior?

https://docs.openstack.org/nova/latest/admin/scheduling.html#pciweigher
> For example, given three hosts - one with a single PCI device, one with
> many PCI devices, and one with no PCI devices - nova should prioritise
> these differently based on the demands of the instance. If the instance
> requests a single PCI device, then the first of the hosts should be
> preferred. Similarly, if the instance requests multiple PCI devices,
> then the second of these hosts would be preferred. Finally, if the
> instance does not request a PCI device, then the last of these hosts
> should be preferred.

Anyway, it has to be tested first.

Here is a PoC patch for configuring the required options here: https://review.opendev.org/c/openstack/charm-nova-cloud-controller/+/835920

We're putting this on hold for now because:

- for the customer mentioned in the bug description, we found a workaround by using openstack host aggregates to set the required weigher multipliers, so this is not urgent

- we may go a different direction in the implemention (eg. add support for configuring all weighers and multipliers, not just two; or go for a higher level configuration in juju such as `enable-stacking` which would auto-set appropriate weigher multipliers)

FTR, this bug may also be related to this wishlist bug: https://bugs.launchpad.net/charm-nova-cloud-controller/+bug/1831112