IMSM fakeraid handled by mdadm: unclean mounted volumes on shutdown/reboot

annoying.

/lib/systemd/system-shutdown/mdadm.shutdown should be syncing everything up, but of course that's only in unstable / yakkety.

I have spare drives finally again, so I will setup ISMS on my xenial desktop and play around with it. Hopefully package from yakkety is "good enough" for SRU into xenial.

I can do some testing as well. Admittedly any unsuccessful run is a bit of a pain as the re-sync takes 1-2hrs.

I did a quick test and created a mdadm.shutdown in /lib/systemd/system-shutdown like the template in Yakkety (adapted path to binary and made it executable). Unfortunately that did not help. However there might be some step missing to activate it in systemd. I guess I need to make output more verbose and add some delay to be sure that this indeed is executed and when.

Status changed to 'Confirmed' because the bug affects multiple users.

See also 1320402 and 1587142

I have a reproducer test case, and a partial fix for xenial I think.
I will start a bileto PPA with test packages for this issue.

as part of https://bugs.launchpad.net/ubuntu/+source/mdadm/+bug/1722491 there is currently mdadm update available from xenial-proposed and zesty-proposed that might resolve this issue.

To test that solution please perform the following:

1) Install mdadm from xenial-proposed/zesty-proposed
   - See https://wiki.ubuntu.com/Testing/EnableProposed
   - Or download & install packages from
xenial
https://launchpad.net/ubuntu/+source/mdadm/3.4-4ubuntu0.1/+build/13596415
zesty
https://launchpad.net/ubuntu/+source/mdadm/3.3-2ubuntu7.5/+build/13596431

2) $ sudo apt install dracut-core

3) $ sudo systemctl enable mdadm-shutdown.service

4) $ sudo systemctl start mdadm-shutdown.service

After this the expectation is for shutdown/reboots to perform clean a shutdown, maintaining the raid array in a synced state, such that it comes up clean.

Please let me know if above resolves shutdown/reboot issues for you.

Regards,

Dimitri.

Just a quick clarification, first boot/shutdown will still not be clean, but subsequent ones (those that are booted with the updated mdadm package) should be clean.

Followed the instructions from comment #7:
Base installation: Xenial/16.04
ii mdadm 3.4-4ubuntu0.1
ii dracut-core 044+3-3

Using IMSM based mdadm mounted on /home:
/dev/mapper/datavg01-home 197G 121G 67G 65% /home

PV VG Fmt Attr PSize PFree
/dev/md126p6 datavg01 lvm2 a-- 831,50g 353,85g

dev/md126p6:
      Container : /dev/md/imsm0, member 0
     Raid Level : raid5
     Array Size : 871895713 (831.50 GiB 892.82 GB)
  Used Dev Size : unknown
   Raid Devices : 3
  Total Devices : 3

          State : clean
 Active Devices : 3
Working Devices : 3
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-asymmetric
     Chunk Size : 64K

           UUID : cc707f7d:77869bd6:de8d52a2:ca21e329
    Number Major Minor RaidDevice State
       2 8 16 0 active sync /dev/sdb
       1 8 32 1 active sync /dev/sdc
       0 8 48 2 active sync /dev/sdd

Now done 2 reboots, one from text console directly after installing the new packages and one from the GUI. In both cases the array came up in sync.

I saw this fix & installed it before your update went out. I did it by hand, so I didn't know about the dracut-core package or enabling/starting the mdadm-shutdown.service. Even without dracut-core, it worked for me.

I did make some changes to 64-md-raid-assembly.rules. I commented out the ENV{ANACONDA}=="?*", GOTO="md_inc_end" line, because I didn't think this applied to Ubuntu. I also made the following change:

ACTION=="add|change", IMPORT{program}="/sbin/mdadm --incremental --export $devnode --offroot $env{DEVLINKS}"

I saw this was updated on the linux-raid list:
[PATCH 2/4] Use correct syntax for passing DEVLINKS to mdadm from udev

I also did 2 reboots & everything seems to be working correctly. One question: Does this mean that Ubuntu 18.04 will use mdadm by default during installation instead of dmraid?

@john-center

I believe mdadm is already used by default during installation. However, when reinstalling my desktop it was not smooth at all. It seems like underlying devices got probed / mounted by something before mdadm-udeb was available in the d-i and thus i had to manualy unmount things and assemble the array from the d-i shell before continuing the installation. Apart from that ftab / bootloaders / initramfs / etc were all setup correctly.

w.r.t. changing / updating udev rules. Things are odd on xenial vs later releases, due to not using mdadm units correctly. =/ thus using offroot is currently broken on xenial, where rootfs is on the intel matrix raid, but I can double check if anything else can be improved.

I don't think it's used by default for IMSM raid installation. That's why your devices were likely set up by dmraid first. When you mentioned in a previous comment about installing dracut-core, this automatically installs dmraid with kpartx. Mdadm never gets involved. When I've manually installed mdadm for IMSM raid, I first remove dmraid. That's the only way I was able to install Ubuntu 16.04 on top of IMSM raid1 via mdadm reliably.

I am having this issue using IMSM / VROC 6.2 on Ubuntu 18.04 LTS, Kernel 5.3.0-28, mdadm - v4.1-rc1 - 2018-03-22.

I am running a bcache cache device on one of the RAID 1 IMSM (VROC) arrays. At the very minimum, every reboot the device needs to resync (not shutting down clean).

What other information is required to help us debug this issue?

TO correctly shutdown IMSM/VROC intel matrix raid devices, I expect you to need to install finalrd & add mdadm finalrd hook.

$ sudo apt install finalrd

Download the attached hook, place it into /usr/share/finalrd/mdadm.finalrd, mark it executable (sudo chmod +x /usr/share/finalrd/mdadm.finalrd)

Then reboot twice.

After the second reboot, i expect the raid to come up clean. Please report back if that helps, and then I should schedule to SRU this change to bionic.

This is how clean shutdown for IMSM/VROC is implemented in later releases.

thank you so much, Dimitri. I will test and confirm

Anything special needed for bcache for finalrd?

On Fri, 7 Feb 2020, 17:25 Joshua Diamant, <email address hidden>
wrote:

> Anything special needed for bcache for finalrd?

I don't know.

First reboot: IMSM/VROC RAID1 resynced (as you stated)

I waited for the RAID volume to become clean and then rebooted.
Second reboot: Raid did not resync and bcache volume successfully mounted!
Third reboot: Raid did not resync and bcache volume successfully mounted!

Thank you so much, Dimitri! You fixed my issue

It seems like the fix stopped working - On every reboot now, the IMSM VROC raid drive is being resynced

Is this related to a bug that produces this error? I see this a few times right before reboot

systemd shutdown [1] Failed to wait for process : protocol error

I find that the issue is resolved if I run "/usr/bin/finalrd" after I boot so that the /run/initramfs directory is populated.

Do I need to run "/usr/bin/finalrd" on start (I see it runs on shutdown)?

Do I need any "/etc/initramfs-tools" hooks or scripts?

I think this issue is occuring because I am running a bcache cache device on a VROC IMSM 'fake raid' device. Once I followed Dimitri's finalrd steps in post #14 (this is required), I also had to add the following script '/lib/systemd/system-shutdown/bcache_stop'

Please make sure you run 'chmod +x /lib/systemd/system-shutdown/bcache_stop' after creating the file with the contents below:

#!/bin/bash

for stop in /sys/block/bcache[0-9]*/bcache/stop
do
        [ -f "$stop" ] || continue
        #echo "Stopping $stop"
        echo 1 > $stop
        echo 1 > $stop
done

for stop in /sys/fs/bcache/*/stop
do
        [ -f "$stop" ] || continue
        #echo "Stopping $stop"
        echo 1 > $stop
        echo 1 > $stop
done