kinetic ppc64le reporting Failed to send WATCHDOG=1 notification message: Transport endpoint is not connected

It could be related to #1991691 if you're using the kernel currently in release.

Getting the same issue on the kinetic server daily image during the install phase, see attached

Using power9 arch, attached is the xml config

<domain type="qemu">
  <name>ubuntu22.10-kinetic-ppc64le</name>
  <uuid>e410f72a-6555-4b47-bb18-8bb61eb52de2</uuid>
  <metadata>
    <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
      <libosinfo:os id="http://ubuntu.com/ubuntu/22.04"/>
    </libosinfo:libosinfo>
  </metadata>
  <memory unit="KiB">8388608</memory>
  <currentMemory unit="KiB">8388608</currentMemory>
  <vcpu placement="static">1</vcpu>
  <os>
    <type arch="ppc64le" machine="pseries-jammy">hvm</type>
  </os>
  <cpu mode="custom" match="exact" check="none">
    <model fallback="forbid">POWER9</model>
  </cpu>
  <clock offset="utc"/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-system-ppc64le</emulator>
    <disk type="file" device="cdrom">
      <driver name="qemu" type="raw" cache="unsafe"/>
      <source file="/home/cking/virt/ubuntu-22.04-live-server-ppc64el.iso"/>
      <target dev="sdb" bus="scsi"/>
      <readonly/>
      <boot order="2"/>
      <address type="drive" controller="0" bus="0" target="0" unit="1"/>
    </disk>
    <disk type="file" device="disk">
      <driver name="qemu" type="qcow2" cache="unsafe"/>
      <source file="/home/cking/virt/ubuntu22.04-ppc64le-clone"/>
      <target dev="vda" bus="virtio"/>
      <boot order="1"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x07" function="0x0"/>
    </disk>
    <controller type="usb" index="0" model="qemu-xhci">
      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x0"/>
    </controller>
    <controller type="pci" index="0" model="pci-root">
      <model name="spapr-pci-host-bridge"/>
      <target index="0"/>
    </controller>
    <controller type="scsi" index="0" model="ibmvscsi">
      <address type="spapr-vio" reg="0x00002000"/>
    </controller>
    <controller type="virtio-serial" index="0">
      <address type="pci" domain="0x0000" bus="0x00" slot="0x04" function="0x0"/>
    </controller>
    <interface type="network">
      <mac address="52:54:00:27:88:a3"/>
      <source network="default"/>
      <model type="rtl8139"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x0"/>
    </interface>
    <serial type="pty">
      <target type="spapr-vio-serial" port="0">
        <model name="spapr-vty"/>
      </target>
      <address type="spapr-vio" reg="0x30000000"/>
    </serial>
    <console type="pty">
      <target type="serial" port="0"/>
      <address type="spapr-vio" reg="0x30000000"/>
    </console>
    <channel type="spicevmc">
      <target type="virtio" name="com.redhat.spice.0"/>
      <address type="virtio-serial" controller="0" bus="0" port="1"/>
    </channel>
    <input type="keyboard" bus="usb">
      <address type="usb" bus="0" port="1"/>
    </input>
    <input type="mouse" bus="usb">
      <address type="usb" bus="0" port="2"/>
    </input>
    <graphics type="spice" autoport="yes">
      <listen type="address"/>
    </graphics>
    <sound model="ich6">
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x0"/>
    </sound...

This makes me think pid1 is dead, since the notification socket is apparently down. Can you please try booting with these options appended to the kernel command line?

systemd.log_level=debug systemd.log_target=console console=ttyS0,38400 console=tty1

Hopefully that will give us more information to debug this.

I can't install Kinetic server from the LiveCD because of this issue. It's just broken.

This looks related: https://bugs.launchpad.net/ubuntu/+source/ubiquity/+bug/1754328

systemd error is being emitted from dispatch_notify_event() in src/journal/journald-server.c, which is called from server_connect_notify(). This function contains a large hunk of comment block describing a potential race condition. I wonder if this is some kind of race condition that only occurs on some systems due to boot speed, in this case, emulated powerpc in QEMU.

Is this issue also present in Lunar? At this point I do not think we will do another systemd SRU for Kinetic.

Status changed to 'Confirmed' because the bug affects multiple users.

Why won't fix? It occurs on other releases too.

Only marked won't fix for Kinetic. Thanks for confirming it happens on other releases.

Ubuntu 23.04 (Lunar Lobster) has reached end of life, so this bug will not be fixed for that specific release.