Anacron service and timer disabled after upgrading from 2.3-33

Bug #2006589 reported by Adrien Nader
24
This bug affects 3 people
Affects Status Importance Assigned to Milestone
anacron (Ubuntu)
Fix Released
Undecided
Unassigned
Kinetic
Fix Committed
Undecided
Unassigned

Bug Description

As hinted by https://lists.debian.org/debian-devel-announce/2022/11/msg00001.html , anacron 2.3-33's packaging has a bug which disables both the timer and service when its postrm script is invoked and that includes upgrades from this package.

We need to work-around this issue since it is the version in kinetic.

=== SRU request below ===

[ Impact ]

 * Due to an issue in the postrm script of anacron 2.3-33, upgrades
   *from* that version disable the anacron service and timer.
   Timeline:
   - 2.3-32*: not buggy
   - 2.3-33*: buggy
   - 2.3-34*: buggy
   - 2.3-35*: not buggy, no work-around (Debian decided against further
       actions due to the affected versions only hitting unstable)
   - 2.3-36: not buggy, no work-around
   - 2.3-36ubuntu1: not buggy, no work-around
   - 2.3-36ubuntu2: not buggy, work-around added

 * We need to make sure that users do not upgrade to any version that is
   less than 2.3-36ubuntu2 which contains a work-around for the bug in
   2.3-33's postrm. This means that security updates must contain the
   fix and the work-around, hence this SRU.
   There are two aspects to this SRU:
   - fixing the bug introduced in 2.3-33 and removed in 2.3-35
     (responsible code was reverted back),
   - working around the fact that the bug gets triggered when updating
     from 2.3-33.

 * The fix has been discussed as part of
   https://bugs.launchpad.net/ubuntu/+source/anacron/+bug/2006589
   The work-around uses 1) preinst to backup the service state before
   the buggy postrm script runs, 2) postinst to restore the backup.

[ Test Plan ]

 * NB: if you're running Lunar, your anacron.service and anacron.timer
   are likely disabled due to this issue.

 * Steps to reproduce the issue:
   - install anacron-2.3-33* or anacron-2.3-34* (kinetic uses 2.3-33)
   - check whether anacron.service (same for anacron.timer) is enabled:
       systemctl show anacron.service --property=UnitFileState
   - update anacron to any other version before 2.3-36ubuntu2 (including
     previous version and including reinstalling the same version);
   - check whether anacron.service (same for anacron.timer) is enabled:
       systemctl show anacron.service --property=UnitFileState
     if the service or timer were enabled before, they will now be
     disabled; if they are not disabled, run "systemctl daemon-reload"
     and check again

 * Steps to test the fix:
   - install anacron-2.3-33* or anacron-2.3-34*
   - enable anacron.service and anacron.timer:
       systemctl enable anacron.service anacron.timer
   - update to anacron >= 2.3-36ubuntu2; if you pay attention to the
     console output, you will see messages related to the issue
   - check that anacron.service and anacron.timer are still enabled:
       systemctl show anacron.service --property=UnitFileState
       systemctl show anacron.timer --property=UnitFileState

[ Where problems could occur ]

 * The fixed version is in lunar and is installed on a number of
   machines already with no report so far. Unfortunately there has been
   a window of opportunity for people to upgrade to an intermediate and
   trigger the issue, therefore preventing the work-around to run. This
   is not a specifically a problem with this update but it is a
   limitation.

 * There is always the possibility that this change breaks anacron.
   However, without the work-around, anacron will be disabled silently.
   As such, this update cannot make things worse in this regard.

 * There is always the possibility of a grave bug that removes
   everything on the machine. This is made more likely because the
   postinst and postrm scripts are shell scripts and shell script makes
   all errors more likely.
   The code has however been written defensively, has been reviewed,
   has been analyzed with shellcheck and has been tested on several
   machines thanks to the update being in lunar. Moreover it does not
   use destructive operations like "rm" (the exception is "rmdir
   --ignore-fail-on-non-empty" which is zero-risk).

[ Other Info ]

 * We should communicate to LL users about this issue once the SRU is
   done.
   Why wait for the SRU to communicate to users of an unreleased
   version? Because it makes the message simpler: "all versions now are
   fixed; update and check the service/timer status, you can't stumble
   on the issue anymore".
   While preparing the SRU, I was also reminded that we've been
   including this NEWS file from Debian:
     https://salsa.debian.org/debian/anacron/-/raw/0ce23b/debian/NEWS
   I've tweaked it but the change isn't present in the lunar package

 * Since the only requirement for this SRU is to be present before any
   other update of anacron, this should be blocked forever. The bug
   report on launchpad has the block-proposed-kinetic tag.

Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in anacron (Ubuntu):
status: New → Confirmed
Revision history for this message
Adrien Nader (adrien-n) wrote :

I'm attaching a patch from 2.3.26ubuntu1 (kinetic's) to 2.3-36ubuntu2(~ppa2) which uses a heuristic to re-enable the service and/or the timer if they were previously enabled. There are comments inside the postinst/preinst scripts.

A typical test I used on my machine looked like the following:

  sudo dpkg -i ../lp/anacron_2.3-33ubuntu1_amd64.deb
  sudo systemctl enable anacron.timer
  sudo systemctl disable anacron.service
  sudo dpkg -i ../anacron_2.3-36ubuntu2\~ppa2_amd64.deb

The following displays the enabled/disabled status:

  systemctl show --property=UnitFileState anacron.timer
  systemctl show --property=UnitFileState anacron.service

As I mention in my changes, systemctl show/status have side-effects and I have no idea why.

Revision history for this message
Ubuntu Foundations Team Bug Bot (crichton) wrote :

The attachment "anacron_2.3-36ubuntu1-to-2.3-36ubuntu2~ppa2.diff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

tags: added: patch
Revision history for this message
Adrien Nader (adrien-n) wrote :

Target should be lunar instead of kinetic.

If we end up making a new version for kinetic, we would have to include these change

Revision history for this message
Michael Hudson-Doyle (mwhudson) wrote :

Ugh what a mess. Well done on fighting to a solution! I have some peanut gallery level observations:

1) can you explain in a bit more detail what happens without the systemctl show calls in preinst? "goes wrong" is a bit ambiguous

2) Did you consider recording the results of systemctl show --property=UnitFileState anacron.{timer,service} in preinst (guarded by checks on $1 and $2) into /run or something and then acting on that in postinst? It feels like it might be a bit less delicate than the solution you have here. But maybe I'm missing something.

Revision history for this message
Adrien Nader (adrien-n) wrote :

1) Without the "systemctl show" in preinst, the "systemctl show" in postinst is wrong. Yes, that makes no sense.
Basically, the steps would be:
- systemctl enable anacron.service
- dpkg -i anacron 2.3-33
- (magic step)
- dpkg -i anacron 2.3-36 + my changes
  - in postinst, systemctl show anacron.service returns disabled unless "systemctl show anacron.service" is called during (magic step), which could also be in the new anacron's postinst.

TBH, I'm thinking "select isn't broken" but I also feel like I've tried everything and this is the only change. I might also describe this improperly. Maybe there's some cache effect, especially since the whole issue here is related to changing configuration behind systemd's back.

2) I didn't consider doing that, mostly because I changed postinst late. Basically I was debugging and somehow noticed the pattern that results in postinst were right when I was debugging and only when I was debugging. Probing only from preinst could be cleaner and it could also make more sense overall since there would be less opportunities for changes to have happened to the system. I need to look at that and review the exact order in which the package scripts are being run.

I'll see what I can do following your idea in 2). Thanks.

Revision history for this message
Steve Langasek (vorlon) wrote :

+ if [ -n "${units_to_enable}" ]; then
+ echo "Some systemd units were probably just disabled mistakenly by anacron 2.3-33; re-enabling ${units_to_enable}"
+ for unit in ${units_to_enable}; do
+ systemctl enable "${unit}" || true
+ done
+ fi
+fi

Is there any deb-systemd-helper state to worry about here in addition to the systemd state?

WRT recording results in preinst and acting on them in postinst, if this is necessary, please use /var/lib/ not /run. It is possible under undesirable circumstances that a reboot happens between the preinst and the postinst being executed.

Revision history for this message
Michael Hudson-Doyle (mwhudson) wrote :

I did review the order the maintainer scripts run before making my preinst / postinst suggestion but of course checking that carefully again is a good idea here! Steve makes a good point about them needing to persist.

Revision history for this message
Adrien Nader (adrien-n) wrote :

I'm attaching another patch.

It backups the symlinks in preinst and restores them in postinst. Overall it feels cleaner than my previous changes and the symlinks should be unaltered. I output warnings (although on stdout now that I think of it) from both preinst and postinst in order to warn about what will be done and to warn about what has been done, along with the symlink targets for debug (and now that I think of it, I shouldn't assume they are symlinks).

The changelog isn't cleaned up; I'll do that once the review is (mostly) cleared.

Revision history for this message
Adrien Nader (adrien-n) wrote :

I had an issue with my mail sync and didn't notice your comments here.

The issue with persisting changes is that there is a window of time during which we could end up overwriting user changes. The more time between preinst and postinst, the more likely we are to un-do something done by the user. It might be possible to protect more against that by re-using my logic from the first iteration of the patch and check whether the on-disk and in-memory states differ. There would still be a window of time during which we would be blind (namely a reboot and/or another systemctl enable/disable anacron.{timer,service}) but it would be smaller.

That being said, I don't think we can have a perfect solution without overwriting 2.3-33's postrm file on disk (I've been tempted to do that more than once) and there is almost certainly a decision to make between possibly being blind to user changes, and not restoring the state in all situations.

About deb-systemd-helper, I don't think this is an issue with it. I need to audit this again in order to be sure.

Revision history for this message
Adrien Nader (adrien-n) wrote :

It's a bit difficult to understand deb-systemd-helper fully well but this doesn't matter much.

Luckily, we don't need to look at the whole code. No script is called between the new preinst, the old postrm and the new postinst. As such, the only code we have to look at is what is inserted through the #DEBHELPER placeholder. The only cases during which deb-systemd-helper can be called are on "purge" and "remove" events which are not a concern here.

As for the configuration step being interrupted, I think I prefer to keep this simpler and not try to take into account voluntary changes between preinst and postinst, even if the two are separate by a month. Having anacron installed but disabled makes fairly little sense and disabling it in the middle of an upgrades makes even less sense. What do you think about this?

Revision history for this message
Michael Hudson-Doyle (mwhudson) wrote :

I think your change looks good.

On the topic of where the units/symlinks are persisted, I don't really understand quite what you are trying to say (possibly my problem).

I guess the situations where there is an unexpected delay between new-preinst and new-postinst are when the system crashes or a maintainer script in a different package fails (I think?). The risk here is that an explicit systemctl disable anacron.service before "apt install -f" or whatever gets run will be ignored. I think given the general edge-case-y-ness of all this I'm OK with this?

The issue of persisting to /run or /var/lib is that the behaviour depends on if the system is rebooted between preinst and postinst. This frankly seems a bit surprising -- so I'd prefer you to change to persisting in /var/lib.

Revision history for this message
Adrien Nader (adrien-n) wrote :

Thanks. I'll post an updated patch on tuesday or wednesday.

What I meant for with the persistence change is that it lengthen the time during which extra things can happen but we'll still be trying to restore the configuration. Obviously tt's not the only way configuration changes happen but in my mind it changes that timeframe from "a couple seconds at most" to "maybe hours or days".

Anyway, I also think it's an edge-case and you have answered my concern. Thanks. :)

Revision history for this message
Adrien Nader (adrien-n) wrote :

It's Thursday, here's my latest patch. The debian/changelog is not clean but I've edited it so that it's only a matter of dropping all the ~ppa versions.

Attached is patch from 2.3-36ubuntu1 to 2.3-36-ubuntu2~ppa5: anacron_2.3-36ubuntu1.dsc-to-anacron_2.3-36ubuntu2~ppa5.dsc.diff

Revision history for this message
Adrien Nader (adrien-n) wrote :

And here is the diff from the previous patch that I posted, from 2.3-36ubuntu2~ppa4 to 2.3-36ubuntu2~ppa5: anacron_2.3-36ubuntu2~ppa4.dsc-to-anacron_2.3-36ubuntu2~ppa5.dsc.diff .

Revision history for this message
Michael Hudson-Doyle (mwhudson) wrote :

I think this looks pretty good. Only one quibble (apart from the indentation) is that you have a message

    echo "Warning: found a backup for ${name} but original file still exist; please check its status and configuration; backup is available at \`${backup}'"

but it looks to me as if the backup file will be deleted shortly after this. I don't know how much it is worth complicating things to handle this edge case on an edge case but this contradiction should be resolved somehow I think (maybe just don't do anything silently if the original file is still there?)

Revision history for this message
Adrien Nader (adrien-n) wrote :

I'm attaching an updated debdiff. I can't (easily) produce an debdiff between patch revisions unfortunately.

Changes:
- fixed the indentation
- improved messages a bit
- made it so that the backup directory is not removed anymore in case a file exists at the original location of the backup: replace "rm -rf $dir" with "rmdir --ignore-fail-on-non-empty" in order to not fail configuration but maybe we prefer it to fail.

For reference, the (updated) message in case a file exists at the original location of the backup:
> Warning: found a backup for anacron.timer but a file exists at the original location; please check its status and configuration; backup is available at `/var/lib/dpkg/anacron-service-timer-status/anacron-timer-backup'

The changelog still has all the clutter from the several revisions here but it should be simple to remove all the ppa entries before applying.

Revision history for this message
Michael Hudson-Doyle (mwhudson) wrote :

OK, this version looks good and I've just uploaded it:

Uploading to ubuntu (via ftp to upload.ubuntu.com):
  Uploading anacron_2.3-36ubuntu2.dsc: done.
  Uploading anacron_2.3-36ubuntu2.debian.tar.xz: done.
  Uploading anacron_2.3-36ubuntu2_source.buildinfo: done.
  Uploading anacron_2.3-36ubuntu2_source.changes: done.
Successfully uploaded packages.

Revision history for this message
Michael Hudson-Doyle (mwhudson) wrote :

Thanks for the perseverance!

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package anacron - 2.3-36ubuntu2

---------------
anacron (2.3-36ubuntu2) lunar; urgency=medium

  * Work around upgrades from 2.3-33 disabling service and timer (LP: #2006589)

 -- Adrien Nader <email address hidden> Wed, 08 Feb 2023 09:53:26 +0100

Changed in anacron (Ubuntu):
status: Confirmed → Fix Released
Adrien Nader (adrien-n)
tags: added: block-proposed-kinetic
Revision history for this message
Adrien Nader (adrien-n) wrote (last edit ):

[Removed wrong patch]

Adrien Nader (adrien-n)
description: updated
Revision history for this message
Adrien Nader (adrien-n) wrote :

Here's the patch for the SRU

Revision history for this message
Steve Langasek (vorlon) wrote :

+anacron (2.3-33ubuntu2) kinetic; urgency=low
+
+ If you have installed Ubuntu Kinetic Kudu and then upgraded to Lunar
+ Lobster during its development cycle before mid March 2023, then anacron
+ might no longer be enabled and the daily/weekly/monthly cron jobs might
+ not be run until it is.
+
[...]

This doesn't seem correct to include in the SRU. (I don't know that anyone will ever *see* this NEWS file, but let's not include a larger than necessary delta that contains irrelevant information, shall we?)

The SRU also includes no changes to the postrm. Isn't this the part in 33ubuntu1 where the bug lies? (Seen in delta between the proposed SRU and current lunar)

-# Close bug #993348, remove dangling symlinks if they exist
-if [ -f /etc/systemd/system/multi-user.target.wants/anacron.service ]; then
- rm /etc/systemd/system/multi-user.target.wants/anacron.service
-fi
-
-if [ -f /etc/systemd/system/timers.target.wants/anacron.timer ]; then
- rm /etc/systemd/system/timers.target.wants/anacron.timer
-fi
-

Revision history for this message
Adrien Nader (adrien-n) wrote :

Thanks a lot for the review.

I wasn't sure about the NEWS entry. I tweaked it compared to the one in Debian. In hindsight, I think I had two releases in mind at the same time and the result doesn't make much sense indeed. There's indeed no version path that would lead to what I described there. I'll remove the NEWS file.

About the postrm, I'm not sure I understand what you mean. The diff you've posted is included in my patch. I have posted two patches, the older of the two being wrong/obsolete and I've removed it now. Did you look at the one posted on 2023-03-22 which is the most recent?

Revision history for this message
Steve Langasek (vorlon) wrote : Re: [Bug 2006589] Re: Anacron service and timer disabled after upgrading from 2.3-33

> About the postrm, I'm not sure I understand what you mean. The diff
> you've posted is included in my patch. I have posted two patches, the
> older of the two being wrong/obsolete and I've removed it now. Did you
> look at the one posted on 2023-03-22 which is the most recent?

Apparently I did have the wrong one, sorry!

I'll re-review tomorrow after you've had a chance to remove the NEWS file.

Revision history for this message
Steve Langasek (vorlon) wrote : Please test proposed package

Hello Adrien, or anyone else affected,

Accepted anacron into kinetic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/anacron/2.3-33ubuntu2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-kinetic to verification-done-kinetic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-kinetic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in anacron (Ubuntu Kinetic):
status: New → Fix Committed
tags: added: verification-needed verification-needed-kinetic
Revision history for this message
Łukasz Zemczak (sil2100) wrote :

I assume everything that was needed has been sponsored here. Removing ubuntu-sponsors.

Revision history for this message
Adrien Nader (adrien-n) wrote :
Download full text (4.1 KiB)

Sorry, I hadn't yet done the verification step after proposed. Here is the log for a kinetic LXD which installs anacron and then updates it to the version in -proposed ((2.3-33ubuntu2) over (2.3-33ubuntu1)).

As can be seen, the service and timer are enabled both before and after the upgrade. We can also see that the work-around I added is triggered and points out that the service and timer have been disabled until the work-around kicks in.

======

RUN lxc launch ubuntu:kinetic anacron-sru
Creating anacron-sru
Starting anacron-sru
RUN@anacron-sru sh -c apt-get -qq update >/dev/null
RUN@anacron-sru apt-get -y -qq -o=Dpkg::Use-Pty=0 upgrade --autoremove
(Reading database ... 34189 files and directories currently installed.)
Removing libfreetype6:amd64 (2.12.1+dfsg-3ubuntu0.1) ...
Processing triggers for libc-bin (2.36-0ubuntu4) ...
RUN@anacron-sru apt-get -y install anacron
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Suggested packages:
  default-mta | mail-transport-agent
The following NEW packages will be installed:
  anacron
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 25.9 kB of archives.
After this operation, 100 kB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu kinetic/main amd64 anacron amd64 2.3-33ubuntu1 [25.9 kB]
Fetched 25.9 kB in 0s (274 kB/s)
Selecting previously unselected package anacron.
(Reading database ... 34183 files and directories currently installed.)
Preparing to unpack .../anacron_2.3-33ubuntu1_amd64.deb ...
Unpacking anacron (2.3-33ubuntu1) ...
Setting up anacron (2.3-33ubuntu1) ...
Created symlink /etc/systemd/system/multi-user.target.wants/anacron.service → /lib/systemd/system/anacron.service.
Created symlink /etc/systemd/system/timers.target.wants/anacron.timer → /lib/systemd/system/anacron.timer.
Processing triggers for man-db (2.10.2-2) ...
Scanning processes...

No services need to be restarted.

No containers need to be restarted.

No user sessions are running outdated binaries.

No VM guests are running outdated hypervisor (qemu) binaries on this host.
RUN@anacron-sru sh -c echo deb http://fr.archive.ubuntu.com/ubuntu kinetic-proposed main restricted >> /etc/apt/sources.list
RUN@anacron-sru sh -c apt-get -qq update >/dev/null
RUN@anacron-sru apt-get install anacron
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Suggested packages:
  default-mta | mail-transport-agent
The following packages will be upgraded:
  anacron
1 upgraded, 0 newly installed, 0 to remove and 24 not upgraded.
Need to get 26.7 kB of archives.
After this operation, 4096 B of additional disk space will be used.
Get:1 http://fr.archive.ubuntu.com/ubuntu kinetic-proposed/main amd64 anacron amd64 2.3-33ubuntu2 [26.7 kB]
Fetched 26.7 kB in 0s (292 kB/s)
(Reading database ... 34202 files and directories currently installed.)
Preparing to unpack .../anacron_2.3-33ubuntu2_amd64.deb ...
Warning: anacron.service will be disabled during the upgrade from anacron 2.3-33ubuntu1; we'll do our best to re-enable it (currently points to `/lib/systemd/system/anacron.service')
Warning: anacron.timer...

Read more...

tags: added: verification-done-kinetic
removed: verification-needed-kinetic
tags: removed: verification-needed
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.