# cat /etc/tmpfiles.d/nova-ovs-vhost-user.conf
# Create libvirt writeable directory for vhost-user sockets
d /run/libvirt-vhost-user 0770 libvirt-qemu kvm - -
In fact, none of the compute hosts have that file: https://paste.ubuntu.com/p/XJRFypbMQf/ (however, the error from this issue doesn't appear on non-DPDK hosts).
After doing the below command, that missing /run/... file has appeared and VM could have been scheduled and started. However, although it have been started, it wasn't reachable over the network.
== Env /paste. ubuntu. com/p/2725tV47y m/
focal/ussuri + ovn, latest stable charms
juju status: https:/
== Problem description
DPDK instance can't be launched after the fresh deployment (focal/ussuri + OVN, latest stable charms), raising a below error:
$ os server show dpdk-test-instance -f yaml AZ:availability _zone: '' SRV-ATTR: host: null SRV-ATTR: hypervisor_ hostname: null SRV-ATTR: instance_ name: instance-00000218 STS:power_ state: NOSTATE STS:task_ state: null STS:vm_ state: error USG:launched_ at: null USG:terminated_ at: null 15T18:51: 00Z' 15T18:52: 01Z' lib/python3/ dist-packages/ nova/conductor/ manager. py\"\ utils.populate_ retry(\ n File \"\ lib/python3/ dist-packages/ nova/scheduler/ utils.py\ ", line 919, in populate_retry\n\ MaxRetriesExcee ded(reason= msg)\nnova. exception. MaxRetriesExcee ded:\ e2e9-4d76- a346-a9b06ff22c 73. Last exception: internal error: process\ 15T18:51: 53.485265Z qemu-system- x86_64: \ id=charnet0, path=/run/ libvirt- vhost-user/ vhu3ba44fdc- 7c,server: \ vhost-user/ vhu3ba44fdc- 7c: No such file\ e2e9-4d76- a346-a9b06ff22c 73. Last exception: internal error: 15T18:51: 53.485265Z qemu-system-x86_64: project. dpdk (4f452aa3- 2b2c-4f2e- 8465-5e3c2d8ec3 f1) e2e9-4d76- a346-a9b06ff22c 73 ubuntu- bionic- 18.04-amd64- server- 20210907- disk1.img (3851450e- e73d-489b- a356-33650690ed 7a) 89e2f0199373a0d 95 15T18:52: 01Z' eaaebbde1ae096f 9e
OS-DCF:diskConfig: MANUAL
OS-EXT-
OS-EXT-
OS-EXT-
OS-EXT-
OS-EXT-
OS-EXT-
OS-EXT-
OS-SRV-
OS-SRV-
accessIPv4: ''
accessIPv6: ''
addresses: ''
config_drive: 'True'
created: '2021-09-
fault:
code: 500
created: '2021-09-
details: "Traceback (most recent call last):\n File \"/usr/
, line 651, in build_instances\n scheduler_
/usr/
\ raise exception.
\ Exceeded maximum number of retries. Exceeded max scheduling attempts 3 for instance\
\ 1bb2d1b7-
\ exited while connecting to monitor: 2021-09-
\ -chardev socket,
\ Failed to bind socket to /run/libvirt-
\ or directory\n"
message: 'Exceeded maximum number of retries. Exceeded max scheduling attempts 3
for instance 1bb2d1b7-
process exited while connecting to monitor: 2021-09-
-chardev '
flavor: m1.medium.
hostId: ''
id: 1bb2d1b7-
image: auto-sync/
key_name: ubuntu-keypair
name: dpdk-test-instance
project_id: cdade870811447a
properties: ''
status: ERROR
updated: '2021-09-
user_id: 13a0e7862c6641e
volumes_attached: ''
For the record, a "generic" instances (e.g non-DPDK/non-SRIOV) are scheduling/starting without any issues.
== Steps to reproduce
openstack network create --external --provider- network- type vlan --provider-segment xxx --provider- physical- network dpdkfabric ext_net_dpdk redacted> ,end=<redacted> --network ext_net_dpdk --subnet-range <redacted>/23 --gateway <redacted> --no-dhcp ext_net_dpdk_subnet
openstack subnet create --allocation-pool start=<
openstack aggregate create --zone nova dpdk
openstack aggregate set --property dpdk=true dpdk
openstack aggregate add host dpdk <fqdn>
openstack aggregate show dpdk --max-width=80
openstack flavor set --property aggregate_ instance_ extra_specs: dpdk=true --property hw:mem_ page_size= large m1.medium.dpdk
openstack server create --config-drive true --network ext_net_dpdk --key-name ubuntu-keypair --image focal --flavor m1.medium.dpdk dpdk-test-instance
== Analysis /pastebin. canonical. com/p/FgPYNb3bP j/ /drive. google. com/file/ d/1W_w3CAUq4ggp 4alDnpCk08mSaCL 6Uaxk/view? usp=sharing
[before redeployment] nova-compute log : https:/
[fresh deployment] juju crashdump: https:/
<on hypervisor>
# ovs-vsctl get open_vswitch . other_config "--pci- whitelist 0000:3e:00.0 --pci-whitelist 0000:40:00.0", dpdk-init="true", dpdk-lcore- mask="0x1000001 ", dpdk-socket- mem="4096, 4096"}
{dpdk-extra=
# cat /etc/tmpfiles. d/nova- ovs-vhost- user.conf vhost-user 0770 libvirt-qemu kvm - -
# Create libvirt writeable directory for vhost-user sockets
d /run/libvirt-
In fact, none of the compute hosts have that file: https:/ /paste. ubuntu. com/p/XJRFypbMQ f/ (however, the error from this issue doesn't appear on non-DPDK hosts).
After doing the below command, that missing /run/... file has appeared and VM could have been scheduled and started. However, although it have been started, it wasn't reachable over the network.
# systemd-tmpfiles --create vhost-user vhost-user
# stat /run/libvirt-
File: /run/libvirt-
Size: 40 Blocks: 0 IO Block: 4096 directory