The dump file parsing issue arises from structural changes in Linux kernel 6.2

debdiff for Lunar

debdiff for Mantic

The attachment "lp2038249-crash-lunar.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

Marking Jammy as affected due to the 6.2+ HWE kernel from Lunar (and later).

As this will need fixing in the development release, I'll re-subscribe the ubuntu-sponsors team. After this has been fixed there, we can take a look at the SRU for the other stable releases.

Thanks, Chengen!

Hi, so AIUI then plan here is the following:

- Wait for Noble to be open for development;
- You'll prepare a Noble debdiff to be sponsored;
- Once fixed in Noble, we'll move to review/sponsor the SRUs.

Given that Noble is still frozen, and no Noble debdiff is present, I take there is nothing to sponsor here for now.

Updating the bug description with the crash versions/tags which _contains_ the commit IDs.

Verified with the provided commit list:

$ cat <<EOF | while read id subject; do echo; echo $id $subject; git describe --contains $id 2>/dev/null || { echo -n '(*) '; git describe --tags $id; }; done
14f8c460473c memory: Handle struct slab changes on Linux 5.17-rc1 and later
5f390ed811b0 Fix for "kmem -s|-S" and "bt -F[F]" on Linux 5.17-rc1
b89f9ccf511a Fix for "kmem -s|-S" on Linux 5.17+ with CONFIG_SLAB
f02c8e87fccb arm64: use TCR_EL1_T1SZ to get the correct info if vabits_actual is missing
d83df2fb66cd SLUB: Fix for offset change of struct slab members on Linux 6.2-rc1
df1f0cba729f x86_64: Fix for move of per-cpu variables into struct pcpu_hot
120d6e89fc14 SLAB: Fix for "kmem -s|-S" options on Linux 6.1 and later
ac96e17d1de5 SLAB: Fix for "kmem -s|-S" options on Linux 6.2-rc1 and later
7750e61fdb2a Support module memory layout change on Linux 6.4
88580068b7dd Fix failure of gathering task table on Linux 6.5-rc1 and later
4ee56105881d Fix compilation error due to new strlcpy function that glibc added
EOF

14f8c460473c memory: Handle struct slab changes on Linux 5.17-rc1 and later
8.0.1~34

5f390ed811b0 Fix for "kmem -s|-S" and "bt -F[F]" on Linux 5.17-rc1
8.0.1~28

b89f9ccf511a Fix for "kmem -s|-S" on Linux 5.17+ with CONFIG_SLAB
8.0.1~4

f02c8e87fccb arm64: use TCR_EL1_T1SZ to get the correct info if vabits_actual is missing
8.0.2~14

d83df2fb66cd SLUB: Fix for offset change of struct slab members on Linux 6.2-rc1
8.0.3~32

df1f0cba729f x86_64: Fix for move of per-cpu variables into struct pcpu_hot
8.0.3~44

120d6e89fc14 SLAB: Fix for "kmem -s|-S" options on Linux 6.1 and later
8.0.3~28

ac96e17d1de5 SLAB: Fix for "kmem -s|-S" options on Linux 6.2-rc1 and later
8.0.3~27

7750e61fdb2a Support module memory layout change on Linux 6.4
(*) 8.0.3-14-g7750e61fdb2a

88580068b7dd Fix failure of gathering task table on Linux 6.5-rc1 and later
(*) 8.0.3-15-g88580068b7dd

4ee56105881d Fix compilation error due to new strlcpy function that glibc added
(*) 8.0.3-16-g4ee56105881d

Submitted bug [1] (wishlist) to Debian maintainer to update crash from 8.0.2 to 8.0.3 (released April/2023) in sid/unstable.

With this we could merge a smaller set of changes later on.

[1] https://bugs.debian.org/1054805

The version in Noble is still the same as in Mantic, and Debian does not have a newer version either to sync/merge [1], so it apparently will not change soon.

Let's proceed with the Mantic debdiff plus changes for Noble.

Changes:
- Modified d/changelog style to '* Description (LP: #)' entry with '- d/p/.patch' sub-entries (more common).
- Numbered the patches (1-7).

Great work on DEP-3 headers, thank you! (Bug-Ubuntu, Origin, and backport notes; all matching.)

I checked there have been no changes from these patches upstream (i.e., fix-up commits to add; except for patch 5 mentioned later, with another strategy).

I'll attach the updated debdiff for now (keeping your name in signature line, if you don't mind, as I haven't done serious changes).

...

$ rmadison -a source crash
 crash | 7.0.3-3ubuntu2 | trusty | source
 crash | 7.0.3-3ubuntu4.5 | trusty-updates | source
 crash | 7.1.4-1ubuntu4 | xenial | source
 crash | 7.2.1-1 | bionic | source
 crash | 7.2.3+real-1~16.04.1 | xenial-updates | source
 crash | 7.2.8-1ubuntu0.18.04.2 | bionic-updates | source
 crash | 7.2.8-1ubuntu1 | focal | source
 crash | 7.2.8-1ubuntu1.20.04.1 | focal-updates | source
 crash | 8.0.0-1ubuntu1 | jammy | source
 crash | 8.0.0-1ubuntu1 | lunar | source
 crash | 8.0.2-1ubuntu1 | mantic | source
 crash | 8.0.2-1ubuntu1 | noble | source

$ date
Fri Oct 27 05:10:39 PM -03 2023

Hi Chengen,

Thanks for the great work for these SRUs.

I'd like to suggest an update/improvement to the Test Plan section.
Even though crash is so broken that it can't even open a file,
once it starts working at all, it would be important to check
that it is working _correctly_.

So, please, could you add verifications for basic correctness
of the commands being addressed per patch (for GA and HWE)?
e.g., kmem -s|-S, module memory layout, etc.

And I had questions on 2 patches:

Patch 3:
---

 31 + * commit e36ce448a08d removed kmem_cache.freelist_cache in 6.1,
 32 + * so use freelist_size instead.
 33 */
 34 - if (MEMBER_EXISTS("kmem_cache", "freelist_cache")) {
 35 + if (MEMBER_EXISTS("kmem_cache", "freelist_size")) {

This is an inconditional change before/after 6.1, which thus could impact Jammy, as it ships both 5.15 and 6.2 kernels.

However, it seems the (new value) attribute 'freelist_size' already exists, so it should be fine in Jammy _if_ 5.15 has it too.

Could you please confirm?

Patch 5:
---

Some of this backport's context update is because this patch is not included,
and it would also fit the 6.2 criteria (changes in 6.1). It seems only code
adds though, not sure how it changes any behavior (or more patches needed).

Can you clarify if it's not really needed for 6.5?
(I haven't followed the maple tree closely, but the commit message suggests it's important.)

If it's not needed _right_ now, i.e., if this SRU is priority, and crash
at least _works_ (which is a good improvement), I think it would be fine
to add it later.

 commit 872cad2d63b3a07f65323fe80a7abb29ea276b44
 Author: Tao Liu <email address hidden>
 Date: Tue Jan 10 14:56:27 2023 +0800

     Port the maple tree data structures and functions

     There have been two ways to iterate vm_area_struct until Linux 6.0:
      1) by rbtree, aka vma.vm_rb;
      2) by linked list, aka vma.vm_{next,prev}.
     However with the maple tree patches[1][2] in Linux 6.1, vm_rb and
     vm_{next,prev} are removed from vm_area_struct. The vm_area_dump()
     in crash mainly uses the linked list for vma iteration, which will
     not work for this case. So the maple tree iteration needs to be
     ported to crash.

This patch 5 is also big, adding a lot, anyway, but it goes to gdb-10.2.patch,
which only changed context lines for the backport, indeed.

A pure-code review against upstream seems a big effort (~2000 lines, I managed
up to ~1000, and it seems to match upstream).

I guess this patch will be reasonably verified if the crash commands to show
module memory layout on kernel 6.4+ (6.5 in our case) run correctly.

(Comment #8)

> Submitted bug [1] (wishlist) to Debian maintainer to update crash from 8.0.2 to 8.0.3 (released April/2023) in sid/unstable.
> With this we could merge a smaller set of changes later on.
> [1] https://bugs.debian.org/1054805

Crash 8.0.3 should soon be available in Debian, and we can merge it into Noble
with a few more recent fixes for the 6.x kernels (Noble is 6.5+), which we'll
need to SRU in older releases too (Mantic is 6.5, Lunar is 6.2, Jammy HWE is 6.2 now, will be 6.5).

https://tracker.debian.org/news/1474801/accepted-crash-803-1-source-into-unstable/

Debian's crash 8.0.3-1 FTBFS, waiting on 8.0.3-2 [1].

[1] https://bugs.debian.org/1055117

Hi, I'm focusing on resolving all issues caused by structural changes rather than solely ensuring the crash utility functions across different kernel versions. The patch set will be extensive and require additional time for backporting and testing. I'll provide the new patch as soon as possible.

debdiff for Jammy HWE

Test plan for Jammy HWE

debdiff for Lunar

Test plan for Lunar

debdiff for Mantic

Test plan for Mantic

Regarding the swapped page read failure issue, I've sent a patch upstream for review. Visit the link if you're interested:
https://<email address hidden>/thread/76EA25KPIRKQY5JDFFNYQY2C4CDRLZP2/