qemu-system-amd64 max cpus is too low for latest processors

Hi Jeff,
thanks for the request, that is a known limit that is being worked on by various upstream projects.

The limit of 288 [1] was deliberately chosen for being the limits of testing at the time and limits of xapic [2].

There recently ~5.15 (which is jammy and later) has been a lift of thelimit on the kernel side [3][4], but that is only the first step.

You also need other components to be ready, like the smbios 3.0 entry point which is in seabios 1.16 (Kinetic and later) and edk2 (there it is rather old and should be ok for longer).

The work / discussions in qemu is ongoing as you might see in [5], but those haven't completed or landed yet - it is work in progress that has to complete and stabilize. You see here that would be a post 7.2 change anyway.

There are more things in the stack which might need patching e.g. in libvirt or even higher parts, I haven't checked those yet - but overall this isn't a "change a number and done" change :-/

I hope that the upstream projects can continue their great work and complete it all, but right now despite looking like a simple number there is not enough confidence for all the implications yet to just bump up that number.

[1]: https://gitlab.com/qemu-project/qemu/-/commit/00d0f9fd6602a27b204f672ef5bc8e69736c7ff1
[2]: https://lists.gnu.org/archive/html/qemu-devel/2016-11/msg02266.html
[3]: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=074c82c8f7cf8a46c3b81965f122599e3a133450
[4]: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=da1bfd52b930726288d58f066bd668df9ce15260
[5]: https://<email address hidden>/

Thanks Christian. The tester reporting it was from one of the OEM labs during cert testing on the newer CPUs... I don't think this is really any sort of show-stopper, just one of those things noticed in the output that looked concerning to them (They report in anything that looks out of the ordinary).

So in the context of the details you provided I think it's safe on our end then to just know it's going to be a limitation and then wait for the various bits to update naturally.

This causes QEMU to be unusable on systems with more than 288 cores, notably recent AMD CPUs and is affecting certifications

Hmm,
"unusable" - really. Isn't it just limiting you to have each guest at max 288 vcpus?
Or did I miss that, due to that, it won't work to create any guest at all?

Our test is failing to run, not simply running with fewer than the requested number of cores. From the test output (which includes test script output and formatting, not just QEMU output):

DEBUG:root:Start VM:
ERROR:root:Command lxc start testbed returned a code of 1
ERROR:root: STDOUT:
ERROR:root: STDERR: Error: Failed to run: forklimits limit=memlock:unlimited:unlimited fd=3 -- /snap/lxd/24322/bin/qemu-system-x86_64 -S -name testbed -uuid e149a6e6-ce67-4b5b-ab56-94c740521c0e -daemonize -cpu host,hv_passthrough -nographic -serial chardev:console -nodefaults -no-user-config -sandbox on,obsolete=deny,elevateprivileges=allow,spawn=allow,resourcecontrol=deny -readconfig /var/snap/lxd/common/lxd/logs/testbed/qemu.conf -spice unix=on,disable-ticketing=on,addr=/var/snap/lxd/common/lxd/logs/testbed/qemu.spice -pidfile /var/snap/lxd/common/lxd/logs/testbed/qemu.pid -D /var/snap/lxd/common/lxd/logs/testbed/qemu.log -smbios type=2,manufacturer=Canonical Ltd.,product=LXD -runas lxd: : Process exited with non-zero value 1
Try `lxc info --show-log testbed` for more info

I know that may not be the error logs or output you need to fully debug this, but it's what I have on hand. (The system in question belongs to a Canonical partner.) We can work to produce more logs or output, but it would be helpful to know what you need.

Also, I did some digging, we use LXD to kick off KVMs and this exists int he LXD docs:
https://linuxcontainers.org/lxd/docs/stable-4.0/instances/

limits.cpu string - yes - Number or range of CPUs to expose to the instance (defaults to 1 CPU for VMs)

I had hoped that the issue was that kicking off that single VM was somehow going crazy and attaching to every CPU core.

BUT it looks like LXD defaults to 1 CPU for VMs, meaning it's not coming anywhere near close to that limit of 288. If that's the case that means QEMU itself is unsable on these new high-core-count CPUs

We can try to explicitly use limts.cpu with LXD but if that doesn't work, we need some help sorting out exactly what's happening here and how to work around it.

I launched a VM via LXC using qemu and verified that it does only create / attach a single CPU core:
root@maximum-porpoise:~# lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
Address sizes: 39 bits physical, 48 bits virtual
CPU(s): 1
On-line CPU(s) list: 0
Thread(s) per core: 1
Core(s) per socket: 1
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 165
Model name: Intel(R) Core(TM) i9-10900F CPU @ 2.80GHz

Note my machine has a single 10 core CPU with HT enabled:
Architecture: x86_64
  CPU op-mode(s): 32-bit, 64-bit
  Address sizes: 39 bits physical, 48 bits virtual
  Byte Order: Little Endian
CPU(s): 20
  On-line CPU(s) list: 0-19
Vendor ID: GenuineIntel
  Model name: Intel(R) Core(TM) i9-10900F CPU @ 2.80GHz
    CPU family: 6
    Model: 165
    Thread(s) per core: 2
    Core(s) per socket: 10

So I do suspect that qemu itself simply fails on systems with more than 288 cores regardless of the config of the VM...

The servers that are failing have dual 96 core AMD EPYC 9654 96-Core Processor, which, with hyperthreading provides 384 CPU cores to the system.

I've gone back and asked them to disable hyperthreading to get the CPU count down to 192 cores to see if qemu works then or not... if the same test succeeds with that config, I think that would certainly confirm the issue.

So just to update/reconfirm something, qemu-system-amd64 fails on systems with more than 288 cores, regardless of how you've configured the KVM Guest.

We have had them test both the default (which defaults to 1 vCPU), and by explicitly setting the config to a single vCPU. We have NEVER launched a KVM guest that was handed more than 1 CPU core, as we have always used the default config for simplicity.

Currently, this causes certification tests on systems with high end AMD CPUs to fail, as those have far more than 288 cores.

We do not have a system currently in house to test this with, but we can get our OEM partner to test patched versions of packages that address this. I have also raised this directly with AMD.

Oh wow, Sorry but I didn't read that in between the lines of the report yet.
I expected that to only block extra large guests which is where we would have waited for upstream.

Indeed guests up to the size limit should work (almost) no matter how many CPUs the system has.
Could you please work with Sergio (assigned now) to provide him access to the system so that he can have a look and potential debugging in the real thing.

> So just to update/reconfirm something, qemu-system-amd64 fails on systems with more than
> 288 cores, regardless of how you've configured the KVM Guest.

This really should be a guest size limit, I wonder if the system is picking up any default like "but it could be 384 via hotplugging" that one needs to configure.

@Jeff
Could you - in preparation - please provide the most simple libvirt-xml or qemu commandline that you expect to work but fails when the host count it >288.

> I launched a VM via LXC using qemu and verified that it does only create / attach a single CPU core

They also just use qemu, so that shouldn't be different...
Have you done that test
a) on a different system to check how many CPUs it configures by default?
b) on the 384 cpu system and you are saying "it works with the LXD snaps qemu, but not with the qemu in the Archive"?

If it was (a) that test isn't sufficient as qemu has the concept current and max cpus (available for hot-plug). And the Limit counts against the max-cpus.

I checked LXD myself on my laptop

$ lxc launch ubuntu-minimal-daily:j j-vm --ephemeral --vm
$ lxc exec j-vm lscpu | grep '^CPU(s):'
CPU(s): 1
=> Yes it is one by default, but it just doesn't give any arguments at all
$ ps axlf | grep qemu | grep j-vm
7 999 2014958 1 20 0 1776840 480184 - Sl ? 0:33 /snap/lxd/24918/bin/qemu-system-x86_64 -S -name j-vm -uuid 6e58b1c8-9484-4131-b4f4-d61e32556d28 -daemonize -cpu host,hv_passthrough -nographic -serial chardev:console -nodefaults -no-user-config -sandbox on,obsolete=deny,elevateprivileges=allow,spawn=allow,resourcecontrol=deny -readconfig /var/snap/lxd/common/lxd/logs/j-vm/qemu.conf -spice unix=on,disable-ticketing=on,addr=/var/snap/lxd/common/lxd/logs/j-vm/qemu.spice -pidfile /var/snap/lxd/common/lxd/logs/j-vm/qemu.pid -D /var/snap/lxd/common/lxd/logs/j-vm/qemu.log -smbios type=2,manufacturer=Canonical Ltd.,product=LXD -runas lxd

And at first it looks like LXD does limit things via cpusets only
https://linuxcontainers.org/lxd/docs/stable-4.0/instances/#cpu-limits

Even with that set explicitly it behaves the same:

$ lxc launch ubuntu-minimal-daily:j j-vm --ephemeral --vm -c limits.cpu=1
Creating j-vm
Starting j-vm
$ lxc exec j-vm lscpu | grep '^CPU(s):'
CPU(s): 1
$ ps axlf | grep qemu | grep j-vm
7 999 2033243 1 20 0 1777348 477060 - Sl ? 0:12 /snap/lxd/24918/bin/qemu-system-x86_64 -S -name j-vm -uuid 4c469ad8-136e-422a-9366-3503f072cddd -daemonize -cpu host,hv_passthrough -nographic -serial chardev:console -nodefaults -no-user-config -sandbox on,obsolete=deny,elevateprivileges=allow,spawn=allow,resourcecontrol=deny -readconfig /var/snap/lxd/common/lxd/logs/j-vm/qemu.conf -spice unix=on,disable-ticketing=on,addr=/var/snap/lxd/common/lxd/logs/j-vm/qemu.spice -pidfile /var/snap/lxd/common/lxd/logs/j-vm/qemu.pid -D /var/snap/lxd/common/lxd/logs/j-vm/qemu.log -smbios type=2,manufacturer=Canonical Ltd.,product=LXD -runas lxd

$ lxc launch ubuntu-minimal-daily:j j-vm --ephemeral --vm -c limits.cpu=2
Creating j-vm
Starting j-vm
$ lxc exec j-vm lscpu | grep '^CPU(s):'
CPU(s): 2
$ ps axlf | grep qemu | grep j-vm
7 999 2036838 1 20 0 1984268 481300 - Sl ? 0:15 /snap/lxd/24918/bin/qemu-system-x86_64 -S -name j-vm -uuid 73ed3b5b-c1f9-4d8f-bed3-dc763a4329e2 -daemonize -cpu host,hv_passthrough -nographic -serial chardev:console -nodefaults -no-user-config -sandbox on,obsolete=deny,elevateprivileges=allow,spawn=allow,resourcecontrol=deny -readconfig /var/snap/lxd/common/lxd/logs/j-vm/qemu.conf -spice unix=on,disable-ticketing=on,addr=/var/snap/lxd/common/lxd/logs/j-vm/qemu.spice -pidfile /var/snap/lxd/common/lxd/logs/j-vm/qemu.pid -D /var/snap/lxd/common/lxd/logs/j-vm/qemu.log -smbios type=2,manufacturer=Canonical Ltd.,product=LXD -runas lxd

For a start to rule out a real bug...
And to rule out any other smartness let us start a very very small qemu that does almost nothing. Does the following stumble over the 384 cpu error as well?

$ sudo qemu-system-x86_64 -smp cpus=1,maxcpus=1 -enable-kvm -net none -m 512M -nographic -kernel /boot/vmlinuz -initrd /boot/initrd.img -chardev stdio,mux=on,id=char0 -mon chardev=char0,mode=readline -serial chardev:char0 -append "console=ttyS0"

That will load a kernel from your host disk, after kernel load it will fail missing a root disk but that is fine. This way we would quickly know if really "everything fails" (bug) or if there might be just a argument needed in your way to spawn guests (configuration).

Pleas let us know if this works

AFAIC - If you insist/depend on LXD - you need to go all-in and use raw.qemu to add commandline parameters ignoring LXDs intentional opinionated use:

$ lxc launch ubuntu-minimal-daily:j j-vm --ephemeral --vm -c raw.qemu="-smp cpus=1,maxcpus=1"
Creating j-vm
Starting j-vm
$ ps axlf | grep qemu | grep j-vm | grep smp
7 999 2043048 1 20 0 1777460 323388 - Sl ? 0:07 /snap/lxd/24918/bin/qemu-system-x86_64 -S -name j-vm -uuid 9346be46-67fa-4931-ba2d-529cbc268190 -daemonize -cpu host,hv_passthrough -nographic -serial chardev:console -nodefaults -no-user-config -sandbox on,obsolete=deny,elevateprivileges=allow,spawn=allow,resourcecontrol=deny -readconfig /var/snap/lxd/common/lxd/logs/j-vm/qemu.conf -spice unix=on,disable-ticketing=on,addr=/var/snap/lxd/common/lxd/logs/j-vm/qemu.spice -pidfile /var/snap/lxd/common/lxd/logs/j-vm/qemu.pid -D /var/snap/lxd/common/lxd/logs/j-vm/qemu.log -smbios type=2,manufacturer=Canonical Ltd.,product=LXD -runas lxd -smp cpus=1,maxcpus=1

P.S. if only cpu is set maxcpu is the same and if nowing else is there cpu is implied. So I know that raw.qemu="-smp 1" does the same, but I wanted to be explicit while debugging here.

There seems to have been some movement on this upstream:

https://lore.kernel<email address hidden>/T/#m4f61669a283a87623e4b8ce484e65c1bbaa76935

The exact commands we use typically are:
lxc init ubuntu:22.04 testbed --vm
# lxc config set testbed limits.cpu 1
lxc start testbed

and assume defaults on everything. (the commented config line was added in later as an experiement)

I don't have direct access to a system with that many cores, but I'll ask them to try all your suggestions and update the bug with results.

On our XD225v AMD server with 2P 9754 Bergamo 128c (512 vcpus) on Ubuntu 22.04.2LTS, I ran the command from comment #12, see attached output comment12.txt.

So this was apparently fixed in qemu 8.1.0:

commit e0001297eb2f8569e950e55dbda8ad686e4155fb
Author: Suravee Suthikulpanit <email address hidden>
Date: Wed Jun 7 15:57:17 2023 -0500

    pc: q35: Bump max_cpus to 1024

    Since KVM_MAX_VCPUS is currently defined to 1024 for x86 as shown in
    arch/x86/include/asm/kvm_host.h, update QEMU limits to the same number.

    In case KVM could not support the specified number of vcpus, QEMU would
    return the following error message:

      qemu-system-x86_64: kvm_init_vcpu: kvm_get_vcpu failed (xxx): Invalid argument

    Also, keep max_cpus at 288 for machine version 8.0 and older.

    Cc: Igor Mammedov <email address hidden>
    Cc: Daniel P. Berrangé <email address hidden>
    Cc: Michael S. Tsirkin <email address hidden>
    Cc: Julia Suvorova <email address hidden>
    Reviewed-by: Igor Mammedov <email address hidden>
    Signed-off-by: Suravee Suthikulpanit <email address hidden>
    Message-Id: <email address hidden>
    Reviewed-by: Michael S. Tsirkin <email address hidden>
    Signed-off-by: Michael S. Tsirkin <email address hidden>
    Reviewed-by: Daniel P. Berrangé <email address hidden>

$ git tag --contains e0001297eb2
v8.1.0
v8.1.0-rc0
v8.1.0-rc1
v8.1.0-rc2
v8.1.0-rc3
v8.1.0-rc4

Looking at rmadison, mantic only has 8.0.4:

 qemu | 1:8.0.4+dfsg-1ubuntu1 | mantic | source

Would it be possible to:

A: get mantic bumped to 8.1.0
B: work on getting this back to Jammy to unblock 22.04 certs? (well, for now we are just accepting failed VM tests because these larger CPUs have no support in Jammy due to the qemu-system-x86_64 max_cpu limitation.

Thanks for the update.

It's not possible to bump QEMU to 8.1.0 on Mantic anymore (we're already on Feature Freeze), but it is possible to backport the patch above. It's also possible to backport this patch to Jammy as part of an SRU.

I'm assigning the bug to myself, but I'll likely only have time to work on this bug next week. Also, it's possible that I'll need your help to test the fix.

Thanks.

Jeff et al,

I worked to create new machine types for Jammy which support up to 1024 CPUs, which is exactly what the upstream patch pointed to by Jeff does. We decided to implement this via new machine types because, as Christian said, it is not entirely clear what kind of side effects this (apparent simple) setting can have, and also (perhaps most importantly) because it is much easier to justify SRUing such change if it's as contained as possible.

You can find a PPA with the proposed change here:

https://launchpad.net/~sergiodj/+archive/ubuntu/qemu

The qemu version is 1:6.2+dfsg-2ubuntu6.16~ppa2. The new machine types are named:

pc-i440fx-jammy-maxcpus Ubuntu 22.04 PC (i440FX + PIIX, maxcpus=1024, 1996)
pc-i440fx-jammy-hpb-maxcpus Ubuntu 22.04 PC (i440FX + PIIX +host-phys-bits=true, maxcpus=1024, 1996)
pc-q35-jammy-maxcpus Ubuntu 22.04 PC (Q35 + ICH9, maxcpus=1024, 2009)
pc-q35-jammy-hpb-maxcpus Ubuntu 22.04 PC (Q35 + ICH9, +host-phys-bits=true, maxcpus=1024, 2009)

Would it be possible for you to give this a try and let me know if it works? I still don't have access to a machine with that number of CPUs, so the amount of testing I can do is limited.

Thanks.

Untagging server-todo since this is awaiting feedback. It can be re-added through triage if needed.

I reached to one of the server OEMS who has access to a failing system (AMD
Genoa 2S with 384 total cores, IIRC) to test the patched qemu packages.
Hopefully they'll respond with results in the next week or so

On Wednesday, October 25 2023, Jeff Lane  wrote:

> I reached to one of the server OEMS who has access to a failing system (AMD
> Genoa 2S with 384 total cores, IIRC) to test the patched qemu packages.
> Hopefully they'll respond with results in the next week or so

Thanks, Jeff.

--
Sergio
GPG key ID: E92F D0B3 6B14 F1F4 D8E0 EB2F 106D A1C8 C3CB BF14

Hi, Sergio and Jeff, after the upgrade of QEMU and tried again, we got the same errors as before, please help refer to attached screenshots for detail.

The configuration of SUT:

Product name: ThinkSystem SR675 V3, which is based on AMD Genoa Platform

CPU: 2x AMD EPYC 9754 128-Core Processor, total 256 Cores and 512 threads.

Mem: 8x 128G DIMMs DDR5

Errors:

# lxc start testbed
error: Failed to run: forklimits limit=memlock:unlimited:unlimited fd-3 -- /snap/Ixd/24322bin/gem-system-xX86-64 -S -name testbed -uuid 55914767-a334-4acb-aac1-9c544b05497e -daemonize -Cpu host,hy passthrough -nographic -serlal chardey:console -nodefaults -no-use-config -sandbox on,absolete=deny,elevateprivileges=allow,spaun=allow,resourcecontrol=deny -readconfig /var/snap/Ixd/cmon/1xd/los/testhed/emu.conf -spice unix-on.disale-ticket ine-on.addr-/var/snan/1xd/common/lxd/1s/testhed/oemwu.snice -nidfe/yar/snan/1xd/common/xd/ logs/testhed/qemu.nid -0 /yar/snap/1xd/common/1xd/ogs/testhed/cemu.logjanonical Ltd.,product=LXD -runas lxd: : Process exited with non-zero valuery Ixc info --show-log testbed for more info
buntu@SR675V3-2204:~$ Ix info --show-log testbedName: testbed

# lxc info --show-log testbed
Name: tesetbed
Status: STOPPED
Type: virtual-machine
Architecture: x86_64

Log:

qemu-system-x86_64: Invalid SMP CPUs 512. The max CPUs supported by machine 'pc-q35-7.1' is 288.

screenshot1

screenshot2

Hi Amy,

Thanks for the feedback. I see from the snapshots you posted that you're using the 'pc-q35-7.1' machine type when launching the VM. As explained on comment #19, you have to use the new machine types in order to enable support for more than 288 vCPUs. In this case, you can use the machine type 'pc-q35-jammy-maxcpus'.

On top of that, you're using LXD to create the VM which means that it'll use its own copy of qemu-system-x86_64 (/snap/lxd/current/bin/qemu-system-x86_64), and not the system one. Can you try invoking qemu directly with the machine type mentioned above?

I'm almost sure it won't work out of the box, and there may be more patches that need to be backported in order to make this work on qemu 6.2, but I'd like to take a look at the output you get.

Thank you.

Hi, Sergio, could you help to share me the way to switch machine type from 'pc-q35-7.1' to 'pc-q35-jammy-maxcpus'?

Many thanks.

Hi Amy,

I believe the option to add to quemu-system-x86_64 is "-M"

qemu-system-x86_64 -M help

will output the list of all the machine types you can use, and I believe you can specify that exact one like this:

qemu-system-x68_64 -M pc-q35-jammy-maxcpus

Now, that said, I don't believe you can test this using our test-virtualization launcher, nor the virtualizaation.py test script as the test script uses LXD and AFAIK there's no way to specify the machine type to the lxc command. I'll follow up if I can find something different, but for now, I think just using the -M option with qemu-system-x86_64 will work.

I've now added LXD to this as today I learned that LXD (which is what we use for launching VMs) doesn't use Ubuntu qemu but rather pulls directly from upstream when building the snap. So patching Ubuntu will help for those uses, but won't fix the broken certification test as that will never pick up the patched Ubuntu qemu. So we'll hae to sort out some sort of solution there as well.

Dear Jeff Lane,

I can't run the "qemu-system-x68_64 -M pc-q35-jammy-maxcpus" command due to my system doesn't have the "pc-q35-jammy-maxcpus".

Can you provide it?

ubuntu@xd295v:~$ qemu-system-x86_64 -M help | grep pc-q35-jammy
ubuntu-q35 Ubuntu 22.04 PC (Q35 + ICH9, 2009) (alias of pc-q35-jammy)
pc-q35-jammy Ubuntu 22.04 PC (Q35 + ICH9, 2009)
pc-q35-jammy-hpb Ubuntu 22.04 PC (Q35 + ICH9, +host-phys-bits=true, 2009)

Thank you.
Jack Kim

Thank you for replying to Amy, Jeff.

@Amy, you can specify the machine by using the -M option, as Jeff said. You can try running a quick&dirty test by doing what Christian said above:

$ sudo qemu-system-x86_64 -smp cpus=1,maxcpus=1 -enable-kvm -net none -m 512M -nographic -kernel /boot/vmlinuz -initrd /boot/initrd.img -chardev stdio,mux=on,id=char0 -mon chardev=char0,mode=readline -serial chardev:char0 -append "console=ttyS0"

Note that you have to adjust the -smp parameter accordingly.

@JUNG, you need to install the qemu package from https://launchpad.net/~sergiodj/+archive/ubuntu/qemu.

The qemu version is 1:6.2+dfsg-2ubuntu6.16~ppa2.

I have used 8.1.2 and I am still getting error max 255 cpu can be added

root@us-ash-r1-c1-m2:~# qemu-system-x86_64 -version
QEMU emulator version 8.1.2 (Debian 1:8.1.2+ds-1)
Copyright (c) 2003-2023 Fabrice Bellard and the QEMU Project developers
root@us-ash-r1-c1-m2:~#

Unable to complete install: 'unsupported configuration: more than 255 vCPUs require extended interrupt mode enabled on the iommu device'

I modified the machine type to 'pc-q35-jammy-maxcpus' and revised the maxcpus to 512, I could see the vm could start, but also had some info like 'smpboot: native_cpu_up: bad cpu 297', could you help to look into the attached log for analysis?

The command I used:

$ sudo qemu-system-x86_64 -M pc-q35-jammy-maxcpus -smp cpus=512,maxcpus=512 -enable-kvm -net none -m 128G -nographic -kernel /boot/vmlinuz -initrd /boot/initrd.img -chardev stdio,mux=on,id=char0 -mon chardev=char0,mode=readline -serial chardev:char0 -append "console=ttyS0"

Hi Amy...

When you specify -smp cpus=512,maxcpus=512 you're creating a single VM with 512 vCPUs from the start. Could you try this but maybe set it to something smaller like `-smp cpus=2,maxcpus=64` to see what happens?

Or maybe also `-smp cpus-1,maxcpus=1` which is the default when launching these (single vCPU per VM)?

./src/qemu/qemu_validate.c:34:#define QEMU_MAX_VCPUS_WITHOUT_EIM 255

I think we need to change this value in libvirtd also to 1024

2023-11-02 14:26:06.060 7 ERROR nova.compute.manager [instance: 5d88d297-8c98-4dbc-b92f-b6a7f3ec882d] libvirt.libvirtError: unsupported configuration: more than 255 vCPUs require extended interrupt mode enabled on the iommu device

after fixing libvirtd able to launch the instance but getting following error
[ 3.694170] smpboot: native_kick_ap: bad cpu 477
[ 3.698119] smpboot: native_kick_ap: bad cpu 478
[ 3.702113] smpboot: native_kick_ap: bad cpu 479
[ 3.706216] smpboot: native_kick_ap: bad cpu 480
[ 3.710113] smpboot: native_kick_ap: bad cpu 481
[ 3.714111] smpboot: native_kick_ap: bad cpu 482
[ 3.718173] smpboot: native_kick_ap: bad cpu 483
[ 3.722113] smpboot: native_kick_ap: bad cpu 484
[ 3.726108] smpboot: native_kick_ap: bad cpu 485
[ 3.730225] smpboot: native_kick_ap: bad cpu 486
[ 3.734115] smpboot: native_kick_ap: bad cpu 487
[ 3.738110] smpboot: native_kick_ap: bad cpu 488
[ 3.742136] smpboot: native_kick_ap: bad cpu 489
[ 3.749814] smpboot: native_kick_ap: bad cpu 490
[ 3.754200] smpboot: native_kick_ap: bad cpu 491
[ 3.758115] smpboot: native_kick_ap: bad cpu 492
[ 3.762117] smpboot: native_kick_ap: bad cpu 493
[ 3.766202] smpboot: native_kick_ap: bad cpu 494
[ 3.770116] smpboot: native_kick_ap: bad cpu 495
[ 3.774108] smpboot: native_kick_ap: bad cpu 496
[ 3.778179] smpboot: native_kick_ap: bad cpu 497
[ 3.782117] smpboot: native_kick_ap: bad cpu 498
[ 3.786113] smpboot: native_kick_ap: bad cpu 499

also mpstat dint show any cpu after 255
root@test500:~# mpstat -P 255
Linux 6.5.7-vdx (test500) 11/03/2023 _x86_64_ (500 CPU)

01:05:56 AM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
01:05:56 AM 255 0.00 0.00 0.03 0.01 0.00 0.00 1.04 0.00 0.00 98.92
root@test500:~# mpstat -P 256
Linux 6.5.7-vdx (test500) 11/03/2023 _x86_64_ (500 CPU)

01:05:59 AM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
root@test500:~# mpstat -P 257
Linux 6.5.7-vdx (test500) 11/03/2023 _x86_64_ (500 CPU)

01:06:01 AM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
root@test500:~#

root@test500:~# lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
Address sizes: 52 bits physical, 57 bits virtual
CPU(s): 500
On-line CPU(s) list: 0-255
Off-line CPU(s) list: 256-499
Thread(s) per core: 1
Core(s) per socket: 1
Socket(s): 256
NUMA node(s): 1
Vendor ID: AuthenticAMD
CPU family: 25
Model: 160
Model name: AMD EPYC 9754 128-Core Processor

it says Off-line CPU(s) list: 256-499

Hi, Jeff and all, update my log, the parameters changed like below:

& sudo qemu-system-x86_64 -M pc-q35-jammy-maxcpus -smp cpus=2,maxcpus=64 -enable-kvm -net none -m 128G -nographic -kernel /boot/vmlinuz -initrd /boot/initrd.img -chardev stdio,mux=on,id=char0 -mon chardev=char0,mode=readline -serial chardev:char0 -append "console=ttyS0"

Thanks.

I got this separately from Lenovo:
We tested the qemu 6.2+dfsg-2ubuntu6 , creating a vm with only 1vcpu, and the results are in the attachment

Here's a further test
We tested two scenarios with qemu Debian 1:6.2+dfsg-2ubuntu6.16~ppa2:

1 qemu-system-x86_64 -smp cpus=1,maxcpus=1 the results are in Attachment 2 (qemu_1cpu_1maxcpu.log)

2 qemu-system-x86_64 -M pc-i440fx-jammy-maxcpus -smp cpus=300,maxcpus=300 the results are in Attachment 3(qemu_300cpu_300maxcpu.log)

And there's the log with cpu=300,max_cpu=300

Thanks for the feedback.

I was kind of expecting this change to *not* be enough, so that confirms my suspicions (unfortunately). I'll have to dive deeper and take a better look at what's going on.