Ubuntu Pro APT integration is a bit much

Status changed to 'Confirmed' because the bug affects multiple users.

Reproduced on Kubuntu 22.04. I heartily agree that this needs fixed (may I add "quickly please"). I'll dig into it some hopefully today and see if I can patch this out.

Actually, me attempting to patch it out might not be a great idea. Anyway, I've put my support behind removing this notice. We can see it's making users upset and it doesn't seem to be all that useful IMO (there's better places to put a notice like that, like at the Livepatch screen immediately after installation). Hopefully it will be removed. If not, I guess we'll just roll with it.

The team is looking into this.
We have contacted Lech, which is the requester for this change, to get his input on how to proceed from here.

This change broke our monitoring for new available updates in our complete infrastructure, since the monitoring tool can not recognize the output of apt-get anymore. This makes this change ultra critical to be fixed.

Removing the file /etc/apt/apt.conf.d/20apt-esm-hook.conf seems to help. One could also remove the package ubuntu-advantage-tools but since packages like ubuntu-minimal depend on them they would also be removed which is not what one wants here.

Lukas, to be fair this is not machine readable output, you should not be parsing it. Please use proper tools.

After aligning with Product Management, our team is working in the following fixes:

1. We will move the phrase to the end of the APT output, separated by a newline, so it does not get in the middle of anything expected now

2. We will provide a command to remove any messages that the Pro Client is inserting into the APT output.

Keep in mind that while 2 is permanent and will be there as an opt-out mechanism, 1 is temporary, as when Ubuntu Pro goes GA the regular package counts for ESM will be back and this message will be gone.

In addition to what already said please note that the Ubuntu Pro banner does not honor apt -qq option, meaning that "apt-get -qq upgrade" will only show that banner:

root@ubuntu:/tmp# apt-get upgrade -qq
Try Ubuntu Pro beta with a free personal subscription on up to 5 machines.
Learn more at https://ubuntu.com/pro

In addition to the opt-out mechanism, please honor apt quiet option.
Thanks.

@juliank: while I understand your point, this is not a reason to let such regressions unfixed.

@juliank:
a simple diff of the output compared to a stored "no updates" output is what I use to let me know which machines need manual intervention. No advanced parsing at all. On a rare occasion when the apt-get output changes (and every machine suddenly is listed in the aggregation), I can redo the stored "no updates" output. But the addition of these lines which can change with potentially random intervals means such a simple setup will result in false positives far more often than once or twice per machine lifetime.
"Proper tools" were mentioned as a class. What is recommended as proper if not "apt-get -s dist-upgrade"?

@renanrodrigo:
regarding 2: this is happening on machines which do not have Ubuntu Pro client installed (ubuntu-advantage-pro correct?)
regarding 1: Moving these messages to the bottom of the output still keeps the variable messages. Are they guaranteed to always be exactly two lines tall going forward?

@shodanshok:
Unfortunately, at the time there is no mechanism for us to leverage so APT will hide the lines when the quiet option is there. I would ask you to please run the command we will provide to hide the message when it is there (and then you'll have all quiet again).

@david-tauriainen:
on 1: Yes, guaranteed this message will not change. This message will be removed once Ubuntu Pro is released as GA. Only the security updates count message (which is already there for quite some time) will persist.

on 2: ubuntu-advantage-pro is a cloud-specific package; the package which delivers the Pro Client is ubuntu-advantage-tools (and yes, it is confusing, we are very sorry for that). ubuntu-advantage-tools is an ubuntu-minimal dependency, so it should be in every Ubuntu machine. We are working with the team to consolidate the packaging but this will take time (and it's a separate thing anyway).

Running /usr/lib/update-notifier/apt-check will give you machine readable "how many updates are available" numbers for security and non-security updates. It's called by /usr/lib/update-notifier/update-motd-updates-available which is called from /etc/apt/apt.conf.d/99update-notifier in a APT::Update::Post-Invoke-Success hook, and then presented to the user through /etc/update-motd.d/90-updates-available. This may be an internal implementation detail of update-notifier. I'm not sure to what extent it's a public API, but in practice calling /usr/lib/update-notifier/apt-check for machine readable information has been consistently stable over many releases. If you want to use something that isn't officially stable but works in practice and this gives you the information you need, using this is probably the best option.

On -qq not being honoured by the hook, this is a good point, but I suggest that you file a separate bug for it - the fix would need to be a separate effort, so tracking it separately makes sense.

I would suggest that this not be implemented until the mechanism to comply
with the utility's stated functionality is there (-qq), and not vice versa,
as this is quite antithetical to the philosophy of a toolchain in general.

On Mon, Oct 10, 2022 at 8:55 AM Robie Basak <email address hidden>
wrote:

> Running /usr/lib/update-notifier/apt-check will give you machine
> readable "how many updates are available" numbers for security and non-
> security updates. It's called by /usr/lib/update-notifier/update-motd-
> updates-available which is called from /etc/apt/apt.conf.d/99update-
> notifier in a APT::Update::Post-Invoke-Success hook, and then presented
> to the user through /etc/update-motd.d/90-updates-available. This may be
> an internal implementation detail of update-notifier. I'm not sure to
> what extent it's a public API, but in practice calling /usr/lib/update-
> notifier/apt-check for machine readable information has been
> consistently stable over many releases. If you want to use something
> that isn't officially stable but works in practice and this gives you
> the information you need, using this is probably the best option.
>
> On -qq not being honoured by the hook, this is a good point, but I
> suggest that you file a separate bug for it - the fix would need to be a
> separate effort, so tracking it separately makes sense.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1992026
>
> Title:
> Ubuntu Pro APT integration is a bit much
>
> Status in ubuntu-advantage-tools package in Ubuntu:
> In Progress
>
> Bug description:
> When we designed the whole hook mechanism for what was ua esm back in
> the day, the intent was for this to be an interface to tastefully
> display information about additional updates you are missing out on
> without a subscription.
>
> The new Ubuntu Pro message crosses the line of a tasteful additional
> information and is negatively impacting apt user experience. The top
> message I saw today on /r/Ubuntu was
>
>
> https://www.reddit.com/r/Ubuntu/comments/xxafiu/how_to_remove_advertisements_from_apt/
>
> I'd vastly prefer hooks not to use their freedom to display
> information irrelevant at the specific point. At this hook, the user
> should be notified about updates, like updates in Ubuntu pro they are
> missing out on, or I guess pending blocked snap refreshes are valid
> options.
>
> For messages like the Ubuntu Pro beta announcement, we already have
> the motd mechanism.
>
> To manage notifications about this bug go to:
>
> https://bugs.launchpad.net/ubuntu/+source/ubuntu-advantage-tools/+bug/1992026/+subscriptions
>
>

--
Ken MacInnis - kcm at clueful dot org - http://www.clueful.org/

@shodanshok, @racb, @kcm :
For what is worth it, a bug was filed against APT to improve on the support of the -q flag on hooks.

https://bugs.launchpad.net/ubuntu/+source/apt/+bug/1992472

Once this is implemented, we can adapt any further messaging in the UA side. This is the best we can do so far.

Can't even remove the package ubuntu-advantage-tools as it removes
ttf-mscorefonts-installer* ubuntu-advantage-tools* ubuntu-minimal* update-manager-core* update-notifier-common*

> Can't even remove the package ubuntu-advantage-tools ...

Hi Mark Fraser,
this aspect is not what is discussed in the bug here.
The discussion about your statement has happened a while ago in bug 1950692 and been closed in [1].
Feel free to chime in there as it is waiting for a technical reason to become a recommends instead.

[1]: https://bugs.launchpad.net/ubuntu/+source/ubuntu-advantage-tools/+bug/1950692/comments/12

I've been asked to review the changes present in 27.11.3 for inclusion in Ubuntu.

This comment relates to the proposed "APT news" feature in 27.11.3, which I think also partly existed in 27.11.2 but had not been individually called out in the review for that until after it was released, when this bug was filed.

My understanding is that the apt hook was added so that the Ubuntu Pro tooling would be able to provide system-specific information about updates available were the user to subscribe. For example, something to the effect of "5 packages have updates available in ESM were you to subscribe; here's how to do that" (but better phrased). It does seem appropriate to me to provide this information to users at the time of running apt.

But I'm not sure about extending this to make it a general messaging mechanism as suggested by "APT news". Given the concerns raised in this bug and the wide community feedback received, and speaking as a member of the Ubuntu Technical Board, I'm not sure I would approve.

In Ubuntu we value having a clear path to leadership to make a decision when it is needed, and then that's something we can all stand behind. I think it's appropriate to take that path here. In this case this kind of decision seems like a matter for the Ubuntu Technical Board rather than individual Ubuntu developers and the SRU team alone.

Can we escalate to the Ubuntu Technical Board to determine if this is appropriate please, or directly to sabdfl if you think an immediate decision is required?

The output of apt (and apt-get) now need to reflect differential availability of updates, so a certain level of disruption is unavoidable. We did have that in -proposed for a while to shake out any issues.

It was slightly cheeky to use that change to also give people a heads-up of new capabilities, but I think it better to have a format change land everywhere rather than scatter-shot based on specific package update patterns (i.e. it would be worse to have a different layout appear on a specific machine only when there is a differentially available update for the first time). On reflection, based on feedback, we'd like to generalise that to provide contextual information about updates to CLI users, much as we have in the GUI. That's the 'apt news' mechanism, which will give us an out-of-band ability to raise awareness of update-related matters without a full apt package update.

For example, if we have another heartbleed, we will use this mechanism to make CLI users aware of it.

The proposal does include a way to suppress news, which will survive upgrades, and also the ability for apt plugins to learn about a 'quiet' mode, which will be valuable for many plugins other than the news one.

I do think it would be reasonable to expect the news to be related to updates and changes in the archive that are interesting for someone using apt, rather than general news, so can happily agree to set that policy and expectation amongst the folk who have write access to the news.

This kind of spam is unwelcome, unwanted and unacceptable to me.

I switched away from Windows over (among other things) ads in start menu.

Ads in CLI logs are also not at all OK to me.

> It was slightly cheeky to use that change to also give people a heads-up of new capabilities

No, it was a boring spam and marketing.

> For example, if we have another heartbleed, we will use this mechanism to make CLI users aware of it.

The complain is not about mechanism, it is about abusing it. Though this outcome was quite obvious risk of adding this new ad vector and new ad attack surface.

PLEASE no advertising by default.

This is the kind of incredibly annoying thing that gets people to dump Ubuntu and switch distros. Your users have no interest in playing whack-a-mole with ads. Please check some of the popular forums if you are in doubt on how unpopular this decision is (I was linked here from one).

Constructively, I would like to suggest instead that the message is ONLY shown to relevant systems and include why. IE, instead of sending everyone "Try our commercial offering for free!" spam, instead only send to EOL systems: "Your system is currently not receiving security updates as the end of life for (Version) has been reached -- security updates are available via (url). This message can be dismissed with (command)." I don't think people would be nearly so annoyed with that approach.

Advertising belongs in motd where it doesn't matter, changing the output of basic system commands for advertising purposes affects people in real ways when they are scripting them. Making excuses for putting it into apt is just that.

The original bug report mentioned the following:

"- There is product interest in keeping the message there."

"When we designed the whole hook mechanism for what was ua esm back in the day, the intent was for this to be an interface to **tastefully** display information about additional updates you are missing out on without a subscription." (emphasis added)

The patch which was just pushed switches the message to the following:

#
# News about significant security updates, features and services will
# appear here to raise awareness and perhaps tease /r/Linux ;)
# Use 'pro config set apt_news=false' to hide this and future APT news.
#

Is there product interest in advertising Reddit? Also, I don't think this is a "tasteful" way to display such information, instead it seems juvenile.

I appreciate being able to switch off such messages entirely with 'pro config set apt_news=false'. But say I have now read the message above for the 20th time (each time I run 'apt upgrade'). I would really like a way to dismiss the *latest* message, so I don't have to keep seeing it, but if there is a new message, I would like to see it.

Please make this default off. It was worrying to see a command you run under sudo having unexpected outputs like this.

For reference, disabling this output is now responsible for generating conffile prompts on upgrade: https://bugs.launchpad.net/ubuntu/+source/ubuntu-advantage-tools/+bug/2003977

A number of users have been complaining lately about how Universe updates are showing in apt updates with the assumption that updates are not available without an ESM / Pro subscription, independent of the shell login MOTD prompts.

Unfortunately, this causes problems with users who are not familiar with the whole Ubuntu OS structure, and causes concerns that Ubuntu isn't actually free anymore. It is *strongly* suggested by me as a Community Council member that the default be changed to disable this kind of output on apt and MOTD because of the level of "over-integrated" that Pro is becoming vs. the updates/notifications, which impacts User Experience.

I have to agree with Thomas above. Sadly, I'm pretty sure I do know who is responsible for the heavy-handed scare-tactic messaging lately that is extremely off-putting to the community, and this person needs to start making sure such messaging at the very least gets approval from the Community Team prior to commit, because it's clear this person lacks tact in their messaging and has lost sight of what it means to be truly "Ubuntu" in the sense of the word. In fact, I'd almost consider it to a Code of Conduct violation in that it has attempted manipulate and coerce people into doing something they wouldn't otherwise do.

I understand this is targeted at enterprise, but individuals in the community are getting scared that their systems are now insecure when, in reality, they're not. This advertisement looks like a typical "packages being held back" response when it's not, and that's scaring people that something is wrong with their system and that they need to pay to fix it. That is heavy-handed and a scare tactic, and that is something that I cannot say "I am who I am because of who we all are". It is not in the giving spirit or philosophy of "Ubuntu".

Thomas, Erich, there seems to be some misunderstanding in the comments. The systems are (partially) insecure. Before Pro, patches for universe were occasionally made available by the community. Systems were (partially) insecure then, but nobody told you.

With Pro, universe gets security support. If these users do not enable pro, and the community does not provide security updates, then these users are on insecure systems.

Also, the messaging and documentation is clear that the individuals do not need to pay for it.

juliank, Thomas is a Core Dev and Erich is a MOTU. Both of them (as well as myself, a Lubuntu developer) are very well aware of the fact that there are insecurities in Universe. But there's a difference between some insecure packages being on a system, and a system being insecure in general. People have been using Ubuntu with these Universe packages being left in a somewhat insecure state for years and years (ever since Ubuntu has been an OS, AFAIK). And people's systems have been fine. The chances of someone loading malicious gunk into a Universe package are slim.

The fact that users think that we're trying to extort them into paying indicates that something has been done wrong. We absolutely should give users the option of using Ubuntu Pro. It should be clear that it's free on a small scale. And we should even help users be able to see what packages could get extra updates if they were using Ubuntu Pro. But what we're doing now almost looks like malware, warning that there are unpatched security holes that one must sign up for a special service in order to get patched. That's actually true in this particular instance, and for a good reason, but this is what malware usually looks like. We don't want to look like malware.

I personally ran into this notification earlier yesterday. I actually didn't mind the "ads" for Ubuntu Pro and apt_news in apt - I found them helpful and beneficial. But even though I knew exactly what the "missing security updates" notification meant, it even made me a tiny bit uneasy when updating a 22.04 system.

Hi Aaron,

the difference really depend on how you use. If you installed a web browser from universe for example (I'm not sure this exists), this might matter more to you than a security update for grep in main.

This is not about malicious gunk in the package, security issues are usually accidents.

I'm just trying to make clear what the situation for universe updates is, because I do see a lot of people saying "now they want to make us pay for universe updates when they were free before" and then you clarify it and they say "I did not know universe did not receive security support".

When I built my VPS years ago I was like "which packages can I afford to install from universe - which can I provide security updates for and which are not security critical". I had to be very careful in selecting my packages. I'm glad that Pro solves that issue.

I generally agree though that we need to be careful with the integration and not over-integrate too much, personally, as an apt developer you can imagine I don't particularly enjoy large intrusions into the APT UX because it might not fit with the language or style apt normally uses.

> and that's scaring people that something is wrong with their system
> and that they need to pay to fix it.

Having an ads emitted by system commands means that something IS wrong
with their system.

Nowadays Ubuntu is an adware ( https://en.wikipedia.org/wiki/Adware ).

To be more specific

```
sudo apt dist-upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
Get more security updates through Ubuntu Pro with 'esm-apps' enabled:
  libgraphicsmagick-q16-3 librpmsign8 python2.7-dev rpm2cpio liburiparser1
  libmaven3-core-java libopenexr-dev php-symfony-expression-language
  libgegl-0.4-0 librpmbuild8 lynx-common libzmq5 python2.7-minimal libhdf5-103
  libgegl-common libpython2.7 debugedit python2.7 libpython2.7-dev lynx
  libzmq3-dev phpmyadmin librpmio8 rpm-common maven rpm librpm8
  libhdf5-cpp-103 cpanminus libjs-jquery-ui libopenexr24 libsdl2-2.0-0
  libmysofa1 libpython2.7-minimal php-symfony-cache libgraphicsmagick++-q16-12
  php-symfony-var-exporter libpython2.7-stdlib
Learn more about Ubuntu Pro at https://ubuntu.com/pro
#
# "Phased updates" allow careful rollout of deb package changes at scale.
# You may see these as packages that have been "kept back". Learn more at
# https://ubuntu.com/server/docs/about-apt-upgrade-and-phased-updates
#
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
```

that I just got is a clear ad for Ubuntu Pro, that claims that updates
were deliberately withheld from being available because I am not subscribed
to some product by Canonical.

For now `sudo apt purge ubuntu-advantage-tools` seems to work to get rid of ads.

Though it appears that long term solution is migrating to OS which is not adware.

https://askubuntu.com/questions/1434512/how-to-get-rid-of-ubuntu-pro-advertisement-when-updating-apt has some input

I get the need for revenue. A tip for livepatch is one thing. But, literally advertising that your distro won't install security updates unless you pay for something.... that's just poor marketing and an open invitation to move to the vast array of distros that don't!

> For now `sudo apt purge ubuntu-advantage-tools` seems to work to get rid of ads.

This does not work because it also removes core packages.

eg ubuntu-minimal* gnome-software* xubuntu-desktop*

Likewise `sudo apt purge ubuntu-pro-client`

Trying to squeeze cash by utilising FOMO or lack of security and and worrying users who then bombard me with questions is not a great look.

There should be a simple switch to disable this marketing nonsense.

Ironically I am a target for your sales pitch, but bully boy tactics have the reverse effect. It's one of the reasons I dumped the Windows world years ago. Sad to say you have steadily become more and more like M$.

You have clearly CHOSEN to monetise 'security' updates. It's by design, not accident.

(Lets not mention the debacle that is snaps that you spent years ignoring issues with, despite choosing to foist them on users when they broke systems and prevented timely upgrades - I have little more confidence in your 'security' than I do snaps)

It therefore gives me intense pleasure to disable it all and steadfastly refuse to play your stupid games.

I can only assume it is making you lots more money because there is no other reason to keep foisting this annoying junk on users otherwise.

I wonder which will come first. Retirement or changing OS again? The latter seems to be approaching much more rapidly than the former.

If your push is 'these packages are insecure but we provide secure ones' (haha - see above) then users may as well just install Debian and be done with it (I have already started moving in this direction for some installs)

All very sad.

I understand the need for advertising Ubuntu Pro, but having confusing popups is a grave accessibility bug. This should never be default behavior and is very intrusive.

I also don't want ubuntu-pro-client to be a hard dependency of ubuntu-minimal. Move it to recommends if you have to, but give the user an option to delete it without touching the rest of the system.

I'm running Ubuntu as an open source OS on a production server and don't want to hear about any "pro" offers from Canonical in any way, shape, or form.