[needs-packaging] [Ubuntu 18.10] Include cxlflash package in Ubuntu

This is a bug that should be assigned to the foundation team.

Other bugs impacting the cxlflash driver have been handled by the kernel team, e.g. https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1702521 and https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1605405.

Is this a request to package a userspace application that leverages the cxlflash driver's functionality?

I have commented on the packaging at <https://mentors.debian.net/package/cxlflash>. Some changes are needed before this package should be included in Ubuntu.

On Thu, Sep 14, 2017 at 10:14 PM, Steve Langasek
<email address hidden> wrote:
> I have commented on the packaging at
> <https://mentors.debian.net/package/cxlflash>. Some changes are needed
> before this package should be included in Ubuntu.
...
> RFS 870909: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=870909
> package on mentors: https://mentors.debian.net/package/cxlflash

There are also many comments about the packaging in the RFS bug for the record.
Resolving them and making the package ready for Debian would most
likely ensure that
the package is ready for Ubuntu, too.

*** This is an automated message ***

This bug is tagged needs-packaging which identifies it as a request for a new package in Ubuntu. As a part of the managing needs-packaging bug reports specification, https://wiki.ubuntu.com/QATeam/Specs/NeedsPackagingBugs, all needs-packaging bug reports have Wishlist importance. Subsequently, I'm setting this bug's status to Wishlist.

------- Comment From <email address hidden> 2017-09-18 15:22 EDT-------
Hi,

Some of the things commented on mentors.debian were already fixed before, but it seems that you've got the wrong release, which you pointed as an issue in the mentors comment.
We fixed this issue as well as the ones pointed in the RFS.

Could you check these changes, please?

@Steve: There is an updated version (4.3.2560-1) on mentors if you would like to continue the review. I can take a look, too, if needed.

Updating title based on change in target release milestone.

------- Comment From <email address hidden> 2018-02-14 14:18 EDT-------
*** Bug 143338 has been marked as a duplicate of this bug. ***

------- Comment From <email address hidden> 2018-03-07 10:30 EDT-------
I can't tell what the next step is here. What is still needed/outstanding. This is a request for 18.04 only at this point, not 17.10. Please update.

Hey! I am looking at the package that has been once previously submitted to Debian mentors. It looks ok, but I was wondering about the libraries that cxlflash is shipping - there seems to be a symbols file for all those that are bundled in this one binary package. Are those meant for external consumption? Usually for libraries that are to be used by external packages/applications we tend to have separate binary packages for each lib. Who would be/are the consumers of the cxlflash-shipped libraries?

------- Comment From <email address hidden> 2018-03-20 14:04 EDT-------
The consumers of the shared libraries are those who purchase a compatible IBM Adapter card on an IBM POWER system.

My question is more about if those libraries will be linked by external programs in external Debian packages, or are only used by the binaries included in the cxlflash package. Will other packages depend/build-depend on the cxlflash package for those libraries?

------- Comment From <email address hidden> 2018-03-20 16:15 EDT-------
I think you are talking about managing the library version dependencies with the library package version. Yes, external developers may use dynamic linking to our shared libraries and create their own packages. Those developers may set their package dependency to our package version.

In that case I am splitting out all the shared libraries into one libcflsh0 (and it's libcflsh-dev counterpart) package to make sure we have an easy way to handle ABI breakages and transitions. Normally we'd split out each library into its own package but after inspecting the source I saw that all of the generated libraries are handled by a single SOVER, so during an ABI break it would just bump the soname for all libraries anyway. After some additional minor cleanup and sanity testing I would expect it to be good to go for review.

------- Comment From <email address hidden> 2018-03-23 15:07 EDT-------
Since we have current customers with dependencies upon our existing package name and version, we do not like that solution.

> Since we have current customers with dependencies upon
> our existing package name and version, we do not like that
> solution.

The user-facing programs would still remain in a binary package named 'cxlflash' and installing this package should give the expected combination of runtime components. But a package accepted into the Debian or Ubuntu archives needs to comply with Debian/Ubuntu policy for libraries. There are sound technical reasons that this policy exists.

It's as commented by Steve. The libcflsh0 library binary package will be automatically pulled in as a dependency of cfxflash, ensuring that people upgrading from previous PPA-based versions get the same set of runtime dependencies. The source headers have been moved to libcflsh-dev though, so for development purposes the development package should be used.

------- Comment From <email address hidden> 2018-03-26 11:24 EDT-------
We appreciate the effort, but changing the package names wont work for us. It would require existing customers to make changes for dependencies based on the new package names.

Could you explain how it will not work for you? I suspect customers are depending on the cxlflash binary package name to get the required tools and libraries, correct? If that's the case, everything should work as with the new package naming layout installing cxlflash will also automatically install libcflsh0, pulling in all the shared libraries as previously. So generally any users and their applications depending on the .so files should not see any difference, without requiring any manual intervention.

Or maybe do you mean the addition of the libcflsh-dev package for developers?

I have prepared a proof-of-concept upload into this PPA:
https://launchpad.net/~sil2100/+archive/ubuntu/cxlflash

Thanks!

One question regarding the packaging itself - the description of the cxlflash package mentions cxlflashimage. I saw this package in one of the cxlflash versions in some PPA, but the main upstream tarball's debian/control does not define that binary package. Should it exist? What should be its contents?

------- Comment From <email address hidden> 2018-03-27 11:22 EDT-------
The cxlflash-test and cxlflashimage packages contain binary files which are not open-source, and therefore won't be included in a linux distro.

------- Comment From <email address hidden> 2018-03-27 18:41 EDT-------
I couldn't get the 18.04 debuild to run, it doesn't do the compile:
> fakeroot debian/rules build
dh build --with=systemd
> echo $?
0

*any idea? can you try it on 18.04?

It ran on 16.04, with warnings:
.....
dh_gencontrol
dpkg-gencontrol: warning: Depends field of package cxlflash: unknown substitution variable ${perl:Depends}
dpkg-gencontrol: warning: Depends field of package cxlflash: unknown substitution variable ${shlibs:Depends}
dpkg-gencontrol: warning: Depends field of package libcflsh-dev: unknown substitution variable ${shlibs:Depends}
dh_md5sums
dh_builddeb
dpkg-deb: building package 'cxlflash' in '../cxlflash_5.0.2661-1_ppc64el.deb'.
dpkg-deb: building package 'libcflsh0' in '../libcflsh0_5.0.2661-1_ppc64el.deb'.
dpkg-deb: building package 'libcflsh-dev' in '../libcflsh-dev_5.0.2661-1_ppc64el.deb'.
dpkg-genchanges >../cxlflash_5.0.2661-1_ppc64el.changes
dpkg-genchanges: including full source code in upload
dpkg-source --after-build cxlflash_5.0.2661
dpkg-buildpackage: full upload (original source is included)
Now running lintian...
W: cxlflash source: intra-source-package-circular-dependency cxlflash libcflsh0
W: cxlflash source: newer-standards-version 4.0.1 (current is 3.9.7)
W: libcflsh-dev: new-package-should-close-itp-bug
W: cxlflash: new-package-should-close-itp-bug
W: cxlflash: empty-binary-package
W: libcflsh0: new-package-should-close-itp-bug
W: libcflsh0: dev-pkg-without-shlib-symlink usr/lib/libcflsh_usfs-0.so usr/lib/libcflsh_usfs.so
W: libcflsh0: symbols-declares-dependency-on-other-package cxlflash #MINVER#
N: 1 tag overridden (1 warning)
Finished running lintian.

Locally I only tested source-package builds and those worked fine. But as seen in the test-package PPA the package seems to build fine on bionic:
https://launchpad.net/~sil2100/+archive/ubuntu/cxlflash/+build/14497603

What packaging are you using? None of the warnings I see from your pasted output were visible on my test packages. cxlflash: empty-binary-package is worrying, a binary package shouldn't be made empty if it's not virtual - what's in the .install file? For "W: libcflsh0: symbols-declares-dependency-on-other-package cxlflash #MINVER#" remember that in the symbols file you have to use the libcflsh0 package name.
Where is 5.0.2661 coming from, is that an upcoming RC? I didn't see it released on the github project page.

------- Comment From <email address hidden> 2018-03-28 18:08 EDT-------
5.0.* is the latest dev, and it needs to be in 18.04. That update comes when I push these packaging changes, very soon.
I have the new packaging working, and it built in my PPA.

We want to name the packages:

cxlflash_5.0.2662-1_ppc64el.deb
libcxlflash0_5.0.2662-1_ppc64el.deb
libcxlflash-dev_5.0.2662-1_ppc64el.deb

so all the packages have "cxlflash" in their name.

------- Comment From <email address hidden> 2018-03-29 17:56 EDT-------
I pushed the 5.0.2663 level to github with the libcxlflash chgs. Pls review for 18.04 inclusion. thx.

------- Comment From <email address hidden> 2018-03-30 11:14 EDT-------
https://github.com/open-power/capiflash/releases
5.0.2663

I was able to talk to Mike and he is OK with most of the changes, we just need two minor fixes, if possible:

1) Upgrade the package to version 5, which was just released and contains important fixes.

2) Rename libcflsh0 and libcflsh-dev to libcxlflash0 and libcxlflash-dev

That done, we can upload the file to the archive.

Thank you,
Breno

On Wed, Apr 04, 2018 at 06:44:24PM -0000, Breno Leitão wrote:
> I was able to talk to Mike and he is OK with most of the changes, we
> just need two minor fixes, if possible:

> 1) Upgrade the package to version 5, which was just released and
> contains important fixes.

> 2) Rename libcflsh0 and libcflsh-dev to libcxlflash0 and libcxlflash-dev

Thanks, Breno. Since the library packages include a number of shared
libraries, neither of these names is more "correct" than the other in terms
of archive policy, so I think it's fine for us to follow suit on the
upstream names.

------- Comment From <email address hidden> 2018-04-05 11:26 EDT-------
I published a new release which removed the debian files, and has a couple of minor fixes.

https://github.com/open-power/capiflash/releases/tag/5.0.2669

I have pushed cxlflash 5.0.2669-0ubuntu1~test1 to the earlier PPA for a test-build. The rename of libcflsh to libcxlflash is acceptable, so I did that in the version I prepared, although I preferred the former as usually it's good if the library binary package name corresponds to an soname. Of course in this case we have bundled many libraries into one binary package so the binary name doesn't matter so much.

One thing that might be good to get fixed with with versions 5.0.* is that there seems to be a binary file present in the tarball that wasn't around in earlier versions:
src/build/install/resources/cxl_flash_vm looks like a binary file. It doesn't seem to be installed in the current package (which is good), but generally binary files in source tarballs are hm, not ideal. I see it was probably usually removed through the use of the build/tools/create_tarball script - was it not run this time by some chance? Should that binary be included in the tarball? Why wasn't it around in the earlier 4.0.* versions?

I see the cxl_flash_vm file is being used by capi_flash script which seems to be installed in the package. I guess it's not a hard requirement, right?

------- Comment From <email address hidden> 2018-04-06 11:54 EDT-------
you are correct, cxl_flash_vm does not belong. I removed the file, pushed that commit to github, and re-released 5.0.2669.

It has been validly pointed out to me by a fellow archive-admin that the current cxlflash sources seem to ship a non-free license in src/build/install/resources/license/ of the tarball contents. This can cause some legal confusion, as the sources are apparently Apache 2.0.

From what I see those actually also shouldn't be in the tarball IIUC, as the /src/build/tools/create_tarball script seems to remove them at the distro_scrub() stage. Is this script deprecated?

Could we get rid of those?

------- Comment From <email address hidden> 2018-04-10 12:06 EDT-------
The license agreement is used when a customer has purchased and installed our adapter card, to detail a liability and support agreement with IBM. The customer runs a configuration script after installing our adapter card, and the script has a "click to accept" agreement based upon the license text.
The source code itself is bound by the apache 2.0 license.

This might be problematic in that case. The license that the user is asked to accept when running one of the cxlflash scripts seems to be written in a way that makes the resulting binaries no longer Apache 2.0 licensed. I am not a licensing expert, but when users need to accept a license that seems to prohibit the user of using the 'Program' with any other hardware, as well as seemingly limit the ability to redistribute or modify (and others), the resulting software doesn't seem to be free software anymore. For us to consider the package and its binaries being Apache 2.0 would mean the binaries being licensed in the same (or compatible) way. I might be reading this wrong, but the shipped license files do not seem to address the 'hardware usage' but the "Program" usage - which I guess in this case would mean the binaries that are present in cxlflash. Although I might be misunderstanding something.

I would certainly like to get this clarified with someone more experienced in free-software licensing and legal.

As the package stands right now I guess it could still be accepted into multiverse, if needed, since it right now it doesn't comply to universe/main.

------- Comment From <email address hidden> 2018-04-11 15:28 EDT-------
We agree that multiverse works.

For either universe or multiverse, we need a clear debian/copyright that explains what the user's rights are on the software. Currently the packaging declares an Apache license, but it's not clear that these are the rights that the user has on this software; the EULA imposes other conditions and asserts that they are binding by simply downloading or installing the package. So what are the effective license terms of this software?

If an Ubuntu community developer modifies this package to remove the code that requires EULA acceptance, is that ok from IBM's POV? If not, then I would say this code is not actually Apache licensed, since removal of a EULA prompt would be a legitimate modification under the terms of the Apache license.

Also, in the course of digging deeper into the EULA handling (in the hopes of finding a path by which this could still be accepted into universe), I noticed that the maintainer scripts for this package (postinst) use 'mv' to stow the libraries into a directory outside of the linker path until the license is accepted. This isn't an acceptable technical mechanism from the point of view of distribution integration. Preferred technical mechanism would be either:

 - a debconf prompt requiring the user to accept the EULA prior to unpacking (I believe there are examples of this in the Ubuntu archive in the flashplugin packages, or the msttcorefonts package)
 - using dpkg-divert instead of mv to move the libraries out of the path (i.e. not bypassing the package manager in a way indistinguishable from filesystem corruption)

------- Comment From <email address hidden> 2018-04-11 18:30 EDT-------
-Replacing the current EULA handling with a EULA prompt before install sounds great.
-The thought of a developer removing a EULA prompt from a package sounds very strange.

changed title to 18.10, according to targetmilestone change after comment #39

Marking as incomplete until the EULA handling is resolved.

Hi,

it's been a while since last comment on this. I believe, we are still awaiting for EULA changes as per most recent sync calls. If there was more progress, maybe you can comment about it on Thursday call this week?

Mostly looking for timelines to possibly plan when this ticket will become unblocked again for further work.

------- Comment From <email address hidden> 2019-02-20 19:22 EDT-------
I will close this bug - the utility won't be needed now. If it's needed in the future we'll still need to address the IBM EULA issue.

Thanks Michael. I'll close this bug out. Please reopen if it's required in the future.