opal-prd not installed by default on ppc64el systems

I think this is related to ubuntu-meta that provides the different meta packages for the different ubuntu flavors

Could you review the proposed patch based on the following :
some people recently ran into several issues on Minsky where they found that Opal PRD
was not running out on customer systems. "This is critical to our ability to
collect logs, determine what is causing failures, and for hostboot to take
appropriate action in regards to a failure."

The attachment "1555904.debdiff" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

I would like to mention that this is a very important package for OpenPower machine. We need to have it installed on every OpenPower machine that has Ubuntu.

The proposed patch (which we wouldn't take directly, since ubuntu-meta is generated from the seeds, but the intent is clear) would cause opal-prd to be installed on all ppc64el server installs, including cloud images (booted in KVM), PowerNV installs, and PowerVM installs. This doesn't sound quite like what we want here, is it?

What would be the right install-time platform check to decide whether or not we should have the opal-prd package included?

Hi Steve,
indeed, not all the machines would be concerned, even if the daemon won't be loaded as it already does
this check itself, which is why I was not sure to do the check at install time or later by the daemon itself.
It's better though to target only the specific configuration.
The criteria is the existence of /sys/firmware/devicetree/base/ibm,opal/diagnostics directory.

That is an openpower baremetal machine (ibmpower do not have that diagnostic feature accessible from the host as the fsp does the diagnotic work)
So yes the fix is to be done somewhere else then but I'm not sure where. Some udeb, tasksel, the installer?
Let me know and I can do a patch that you could review.
Thanks,

F.

> The criteria is the existence of
> /sys/firmware/devicetree/base/ibm,opal/diagnostics directory.

> That is an openpower baremetal machine (ibmpower do not have that
> diagnostic feature accessible from the host as the fsp does the diagnotic
> work) So yes the fix is to be done somewhere else then but I'm not sure
> where. Some udeb, tasksel, the installer? Let me know and I can do a
> patch that you could review. Thanks,

I believe this belongs in the hw-detect udeb for the installer.

I've filed https://bugs.launchpad.net/ubuntu/+source/skiboot/+bug/1691557 which will be a prerequisite for getting this seeded and installed via hw-detect.

The MIR in https://bugs.launchpad.net/ubuntu/+source/skiboot/+bug/1691557 has been approved, so this is now unblocked for the changes to be made in hw-detect and the seed. I'll update the seed immediately.

This bug was fixed in the package hw-detect - 1.117ubuntu5

---------------
hw-detect (1.117ubuntu5) bionic; urgency=medium

  * hw-detect.sh: install opal-prd on OpenPOWER machines. LP: #1555904

 -- Dimitri John Ledkov <email address hidden> Wed, 29 Nov 2017 12:55:19 +0000

This bug was fixed in the package debian-installer - 20101020ubuntu525

---------------
debian-installer (20101020ubuntu525) bionic; urgency=medium

  * Move master kernels to 4.13.0-17.
  * Rebuild with new hw-detect LP: #1555904.

 -- Dimitri John Ledkov <email address hidden> Thu, 30 Nov 2017 11:01:35 +0000

Please test d-i netboot installer 20101020ubuntu525 or later; or tomorrow's server images with a timestamp of 1st of December or later. It should resolve your issue. Please comment if your testing was successful. Then we can look into cherry-picking this fix for xenial upcoming point release too.

Just attempted with bionic 20101020ubuntu525 netboot installer. However, this build fails to install on my machine:

  │ │ Debootstrap warning │ │
  │ │ Warning: Failure trying to run: chroot /target dpkg-deb -f │ │
  │ │ /var/cache/apt/archives/dpkg_1.19.0.4ubuntu1_ppc64el.deb Version │ │

syslog tells me:

  Dec 11 08:58:46 debootstrap: chroot: can't execute 'dpkg-deb': No such file or directory

I can't run anything under /target:

  ~ # chroot /target /bin/bash
  chroot: can't execute '/bin/bash': No such file or directory
  ~ # ls -l /target/bin/bash
  -rwxr-xr-x 1 root root 1544376 May 17 2017 /target/bin/bash

Looks like that build has an invalid dynamic linker path, or the installer hasn't provided the requested linker yet:

  bash: ELF 64-bit LSB executable, 64-bit PowerPC or cisco 7500, version 1 (SYSV), dynamically linked, interpreter /lib64/ld64.so.2, for GNU/Linux 3.2.0, BuildID[sha1]=8b1909e1534e08ac77ea8f8f1c8a120c6d02d631, stripped

  ~ # ls -al /target/lib64/
  drwxr-xr-x 2 root root 4096 Oct 11 20:21 .
  drwxr-xr-x 20 root root 4096 May 23 2017 ..

(I do see a suitable linker under /target/lib/powerpc64le-linux-gnu/ though)

I see this earlier in the installer syslog:

Dec 11 08:58:45 debootstrap: tar: skipping unsafe symlink to '/lib/powerpc64le-linux-gnu/ld-2.26.so' in archive, set EXTRACT_UNSAFE_SYMLINKS=1 to extract

(along with another bunch of the same error for different symlinks).

I can provide the full syslog if required.

I've filed that bug as https://bugs.launchpad.net/ubuntu/+source/debian-installer/+bug/1737661

Hello Jeremy, or anyone else affected,

Accepted hw-detect into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/hw-detect/1.117ubuntu2.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

@IBM

Do you have any systems that have /sys/firmware/devicetree/base/ibm,opal/diagnostics and can test this on?

You can use the 16.04 xenial proposed d-i, and boot that with 'apt-setup/proposed=true' extra kernel cmdline this will use hw-detect from xenial-proposed, and should have this issue resolved, and automatically detect /diagnostics and thus install opal-prd in the target systems.

URL to xenial-proposed d-i is http://ports.ubuntu.com/dists/xenial-proposed/main/installer-ppc64el/

Unable to verify if the issue at hand is fixed or not, however code wise it does not have options to regress installations. The installation went fine on systems, that do not expose said diagnostics directory with hw-detect 1.117ubuntu2.3. Further testing will confirm if this issue is fixed, and can be closed.

Hi Dimitri,
I did the test with the netboot vmlinux/initrd.gz from xenial-proposed and 'apt-setup/proposed=true'
and it worked :

---
ubuntu@garri:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.4 LTS
Release: 16.04
Codename: xenial
ubuntu@garri:~$ apt-cache policy opal-prd
opal-prd:
  Installed: 5.4.3-1ubuntu0.16.04.1
  Candidate: 5.4.3-1ubuntu0.16.04.1
  Version table:
 *** 5.4.3-1ubuntu0.16.04.1 500
        500 http://de.ports.ubuntu.com/ubuntu-ports xenial-updates/main ppc64el Packages
        100 /var/lib/dpkg/status
     5.1.13-0ubuntu2 500
        500 http://de.ports.ubuntu.com/ubuntu-ports xenial/universe ppc64el Packages
ubuntu@garri:~$ ps axf|grep opal-prd
  1837 ? Ss 0:00 /usr/sbin/opal-prd --pnor /dev/mtd0
  2067 hvc0 S+ 0:00 \_ grep --color=auto opal-prd
ubuntu@garri:~$ ls -ld /sys/firmware/devicetree/base/ibm,opal/diagnostics
drwxr-xr-x 2 root root 0 Feb 23 14:55 /sys/firmware/devicetree/base/ibm,opal/diagnostics
---

Thanks,

F.

This bug was fixed in the package hw-detect - 1.117ubuntu2.3

---------------
hw-detect (1.117ubuntu2.3) xenial; urgency=medium

  * hw-detect.sh: install opal-prd on OpenPOWER machines. LP: #1555904
  * Drop hw-detect.pre-pkgsel.d/20install-hwpackages, as it installs
    universe package, which is probably in itself not useful on
    Ubuntu. LP: #1577833

 -- Dimitri John Ledkov <email address hidden> Wed, 21 Feb 2018 15:36:54 +0000

The verification of the Stable Release Update for hw-detect has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Hello Jeremy, or anyone else affected,

Accepted debian-installer into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/debian-installer/20101020ubuntu451.23 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

This bug was fixed in the package debian-installer - 20101020ubuntu451.23

---------------
debian-installer (20101020ubuntu451.23) xenial; urgency=medium

  * No change rebuild to pick up the hw-detect changes (LP: #1555904).

 -- Łukasz 'sil2100' Zemczak <email address hidden> Wed, 28 Feb 2018 14:54:36 +0100