Ubuntu
zfs-linux package

ZFS import cause panic

Bug #1881810 reported by Andreas Mattisson
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
zfs-linux (Ubuntu)
Invalid
High
Unassigned

Bug Description

1)
Ubuntu 20.04 LTS

2)
apt policy zfsutils-linux
zfsutils-linux:
Installed: 0.8.3-1ubuntu12
Candidate: 0.8.3-1ubuntu12

3)
ZFS pool not possible to import, unless using read only.
If trying to import normal way as write/read, casing panic message

Expected to be able to import zpool without panic.
Expected that scrub should not be able to run in read only.
Expected that somehow be able to restore the zpool somehow.

4)
# zpool status
no pools available
# zpool import
   pool: datastore
     id: 3190464655986727485
  state: ONLINE
 status: Some supported features are not enabled on the pool.
 action: The pool can be imported using its name or numeric identifier, though
        some features will not be available without an explicit 'zpool upgrade'.
 config:

        datastore ONLINE
          raidz2-0 ONLINE
            sdb ONLINE
            sdc ONLINE
            sdd ONLINE
            sde ONLINE
            sdf ONLINE
            sdg ONLINE

# zpool import -o readonly=on datastore
# zpool status
  pool: datastore
 state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
        still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(5) for details.
  scan: scrub in progress since Sun May 10 00:24:01 2020
        15,1T scanned at 0B/s, 15,1T issued at 0B/s, 15,1T total
        1,44M repaired, 100,01% done, no estimated completion time
config:

        NAME STATE READ WRITE CKSUM
        datastore ONLINE 0 0 0
          raidz2-0 ONLINE 0 0 0
            sdb ONLINE 0 0 0
            sdc ONLINE 0 0 0
            sdd ONLINE 0 0 0
            sde ONLINE 0 0 0
            sdf ONLINE 0 0 0
            sdg ONLINE 0 0 0

errors: No known data errors

Running zdb it always stops ad 2,23T and then aborts.
# zdb -e -bcsvL datastore
Traversing all blocks to verify checksums ...
2.23T completed ( 149MB/s) estimated time remaining: 25hr 06min 25sec free(): invalid pointer
Abort (SIGABRT)

#zpool import datastore
VERIFY3(c < SPA_MAXBLOCKSIZE >> SPA_MINBLOCKSHIFT) failed (36028797018963967 < 32768)
PANIC at zio.c:293:zio_buf_alloc()

Revision history for this message
Colin Ian King (colin-king) wrote :

Can you attach to the bug report the full output from dmesg, the full panic message is more instructive when trying to debug this.

Changed in zfs-linux (Ubuntu):
status: New → In Progress
importance: Undecided → High
assignee: nobody → Colin Ian King (colin-king)
status: In Progress → Triaged
Revision history for this message
Andreas Mattisson (andreasmatt) wrote :

Added dmesg output

Revision history for this message
Colin Ian King (colin-king) wrote :

Looks like you may have data corruption. One can temporarily disable the spa verification to see if this allows the import to at least complete to see if this helps.

echo 0 | sudo tee /sys/module/zfs/parameters/spa_load_verify_metadata
echo 0 | sudo tee /sys/module/zfs/parameters/spa_load_verify_data

do the import, and then re-enable:

echo 1 | sudo tee /sys/module/zfs/parameters/spa_load_verify_metadata
echo 1 | sudo tee /sys/module/zfs/parameters/spa_load_verify_data

Revision history for this message
Andreas Mattisson (andreasmatt) wrote :

Thanks for you fast reply.

This is the same behaviour from bug: 1880580, that I wrote.
I have updated and can only continue verify on the latest 20.04, not 18.04 anymore.
This is what I have done so far now.

I tried you suggested solution. I first had to remove /etc/zfs/zpool.cache in order to be able to do this.

Then I tried your solution 2 times, once with no flags, and then with -f flag.
The panic still occurs when doing this.

Should I try zdb again?
Any other suggestions ?

# cat /sys/module/zfs/parameters/spa_load_verify_metadata
1
# echo 0 | sudo tee /sys/module/zfs/parameters/spa_load_verify_metadata
0
# echo 0 | sudo tee /sys/module/zfs/parameters/spa_load_verify_data
0
# zpool import
   pool: datastore
     id: 3190464655986727485
  state: ONLINE
 status: Some supported features are not enabled on the pool.
 action: The pool can be imported using its name or numeric identifier, though
        some features will not be available without an explicit 'zpool upgrade'.
 config:

        datastore ONLINE
          raidz2-0 ONLINE
            sdb ONLINE
            sdc ONLINE
            sdd ONLINE
            sde ONLINE
            sdf ONLINE
            sdg ONLINE
# zpool import datastore
or
# zpool import -f datastore

Cause this output:
VERIFY3(c < SPA_MAXBLOCKSIZE >> SPA_MINBLOCKSHIFT) failed (...)
PANIC at zio.c:293:zio_buf_alloc()

Revision history for this message
Colin Ian King (colin-king) wrote :

I does seem that something on disc is corrupted. I'm still trying to find any viable solutions.

Colin Ian King (colin-king)
Changed in zfs-linux (Ubuntu):
assignee: Colin Ian King (colin-king) → Dimitri John Ledkov (xnox)
Dimitri John Ledkov (xnox)
Changed in zfs-linux (Ubuntu):
assignee: Dimitri John Ledkov (xnox) → nobody
status: Triaged → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Bug attachments

Remote bug watches

Bug watches keep track of this bug in other bug trackers.