fedora28 standalone failing on tempest

review to revert the voting status https://review.openstack.org/#/c/633087/

Arx will be looking into this..

removing the alert, as the job isn't voting anymore.

For this particular case, I found this error:

2019-01-24 19:01:33.969 9 ERROR vif_plug_ovs.ovsdb.impl_vsctl [req-9ba4d327-4bad-4a45-99c1-9dcf22b37193 f14d275e92cb4384ba0502ca1738c8cc 3879640c10fe4962a8ca60cd0be8dbd9 - default default] Unable to execute ['ovs-vsctl', '--timeout=120', '--oneline', '--format=json', '--', '--may-exist', 'add-br', u'br-int', '--', 'set', 'Bridge', u'br-int', u'datapath_type=system']. Exception: Unexpected error while running command.
Command: ovs-vsctl --timeout=120 --oneline --format=json -- --may-exist add-br br-int -- set Bridge br-int datapath_type=system
Exit code: 1
Stdout: u''
Stderr: u'ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory)\n': ProcessExecutionError: Unexpected error while running command.
2019-01-24 19:01:33.995 9 ERROR vif_plug_ovs.ovsdb.impl_vsctl [req-9ba4d327-4bad-4a45-99c1-9dcf22b37193 f14d275e92cb4384ba0502ca1738c8cc 3879640c10fe4962a8ca60cd0be8dbd9 - default default] Unable to execute ['ovs-vsctl', '--timeout=120', '--oneline', '--format=json', '--', '--may-exist', 'add-port', u'br-int', u'qvo94387b07-f4', '--', 'set', 'Interface', u'qvo94387b07-f4', u'external_ids:iface-id=94387b07-f47c-4529-a398-4e09a6f7c7ce', 'external_ids:iface-status=active', u'external_ids:vm-uuid=67d0ffd8-13ca-4271-9eeb-fd16ae96bbad', u'external_ids:attached-mac=fa:16:3e:9e:cb:9f']. Exception: Unexpected error while running command.
Command: ovs-vsctl --timeout=120 --oneline --format=json -- --may-exist add-port br-int qvo94387b07-f4 -- set Interface qvo94387b07-f4 external_ids:iface-id=94387b07-f47c-4529-a398-4e09a6f7c7ce external_ids:iface-status=active external_ids:vm-uuid=67d0ffd8-13ca-4271-9eeb-fd16ae96bbad external_ids:attached-mac=fa:16:3e:9e:cb:9f
Exit code: 1
Stdout: u''
Stderr: u'ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory)\n': ProcessExecutionError: Unexpected error while running command.

http://logs.openstack.org/31/626631/11/check/tripleo-ci-fedora-28-standalone/cf314a4/logs/undercloud/var/log/containers/nova/nova-compute.log.txt.gz#_2019-01-24_19_01_33_969

Checking other jobs, I also found this:

2019-01-27 20:05:04.762 9 ERROR vif_plug_ovs.ovsdb.impl_vsctl [req-c0565eb3-be73-4d2a-b59e-a7cc37f759d5 c9745094b10545779ae32bf53dd8cca9 7afb8674582846bc82865e1a71d6e67c - default default] Unable to execute ['ovs-vsctl', '--timeout=120', '--oneline', '--format=json', '--', '--may-exist', 'add-port', u'br-int', u'qvo1a3b80e5-55', '--', 'set', 'Interface', u'qvo1a3b80e5-55', u'external_ids:iface-id=1a3b80e5-5521-4134-bad5-3841ec13eead', 'external_ids:iface-status=active', u'external_ids:vm-uuid=4d420cb6-dc79-48d6-9df3-2d7041f8ca05', u'external_ids:attached-mac=fa:16:3e:78:78:8f']. Exception: Unexpected error while running command.
Command: ovs-vsctl --timeout=120 --oneline --format=json -- --may-exist add-port br-int qvo1a3b80e5-55 -- set Interface qvo1a3b80e5-55 external_ids:iface-id=1a3b80e5-5521-4134-bad5-3841ec13eead external_ids:iface-status=active external_ids:vm-uuid=4d420cb6-dc79-48d6-9df3-2d7041f8ca05 external_ids:attached-mac=fa:16:3e:78:78:8f
Exit code: -14
Stdout: u''
Stderr: u'2019-01-27T...

Openvswitch version installed: openvswitch-2.10.1-1.fc28.x86_64

There are lots of warnings showing up in ovs-vswitchd log

2019-01-24T18:40:59.070Z|00058|connmgr|INFO|br-ex<->tcp:127.0.0.1:6633: 2 flow_mods 10 s ago (2 adds)
2019-01-24T18:40:59.184Z|00059|connmgr|INFO|br-tun<->tcp:127.0.0.1:6633: 10 flow_mods 10 s ago (10 adds)
2019-01-24T18:46:19.496Z|00060|bridge|INFO|bridge br-int: added interface tapd0b4fd96-39 on port 3
2019-01-24T18:46:19.609Z|00061|netdev_linux|INFO|ioctl(SIOCGIFHWADDR) on tapd0b4fd96-39 device failed: No such device
2019-01-24T18:46:25.160Z|00062|bridge|INFO|bridge br-int: added interface qr-32fda31e-3a on port 4
2019-01-24T18:46:25.272Z|00063|netdev_linux|INFO|ioctl(SIOCGIFHWADDR) on qr-32fda31e-3a device failed: No such device
2019-01-24T18:46:25.645Z|00064|bridge|INFO|bridge br-int: added interface qg-193495cd-3d on port 5
2019-01-24T18:46:25.701Z|00065|netdev_linux|INFO|ioctl(SIOCGIFHWADDR) on qg-193495cd-3d device failed: No such device

So I see errors like this in one of the logs, http://logs.openstack.org/97/631297/3/check/tripleo-ci-fedora-28-standalone/7fe7dc1/logs/undercloud/var/log/containers/neutron/openvswitch-agent.log.txt.gz

2019-01-24 17:01:14.064 56271 ERROR neutron.agent.common.ovsdb_monitor [req-6c83cba5-d914-4ad1-a4de-6368051abd6b - - - - -] Interface monitor is not active
...
2019-01-24 17:01:14.534 56271 ERROR neutron.agent.common.async_process [-] Error received from [ovsdb-client monitor tcp:127.0.0.1:6640 Interface name,ofport,external_ids --format=json]: PMD: net_mlx5: cannot load glue library: libibverbs.so.1: cannot open shared object file: No such file or directory
2019-01-24 17:01:14.536 56271 ERROR neutron.agent.common.async_process [-] Error received from [ovsdb-client monitor tcp:127.0.0.1:6640 Interface name,ofport,external_ids --format=json]: PMD: net_mlx5: cannot initialize PMD due to missing run-time dependency on rdma-core libraries (libibverbs, libmlx5)

And things like this in the l3-agent, http://logs.openstack.org/97/631297/3/check/tripleo-ci-fedora-28-standalone/7fe7dc1/logs/undercloud/var/log/containers/neutron/l3-agent.log.txt.gz

19-01-24 17:01:11.623 55617 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1337 command(idx=1): DbSetCommand(table=Port, col_values=(('tag', 4095),), record=qg-49c21fc0-4e) do_commit /usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:84
2019-01-24 17:01:11.623 55617 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1337 command(idx=2): DbSetCommand(table=Interface, col_values=(('type', 'internal'), ('external_ids', {'iface-id': u'49c21fc0-4ee5-4c45-a0f6-971c74601d17', 'iface-status': 'active', 'attached-mac': u'fa:16:3e:7c:c1:fe'})), record=qg-49c21fc0-4e) do_commit /usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:84
2019-01-24 17:01:11.624 55617 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] OVSDB transaction returned TRY_AGAIN, retrying do_commit /usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:93

Are the agents being run in containers?

The libipverbs.so error might be a red herring according to a downstream bug I found, but the others I'm not sure.

http://zuul.openstack.org/builds?job_name=tripleo-ci-fedora-28-standalone
closing