2022-03-02 16:20:48

by Abdul Haleem

[permalink] [raw]
Subject: [next-20220225][Oops][ppc] lvm snapshot merge results kernel panics (throtl_pending_timer_fn)

Greeting's

Linux next kernel 5.17.0-rc5-next-20220225 crashed on my power 10 LPAR
when merge lvm snapshot on nvme disk

console logs
------------

fdisk -l /dev/nvme1n1

Disk /dev/nvme1n1: 372.6 GiB, 400088457216 bytes, 97677846 sectors
Units: sectors of 1 * 4096 = 4096 bytes
Sector size (logical/physical): 4096 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
vgcreate avocado_vg /dev/nvme1n1
  Volume group "avocado_vg" successfully created

lvcreate --name avocado_lv --size 18432.0 avocado_vg -y
  Wiping ext2 signature on /dev/avocado_vg/avocado_lv.
  Logical volume "avocado_lv" created.

lvdisplay avocado_vg
  --- Logical volume ---
  LV Path                /dev/avocado_vg/avocado_lv
  LV Name                avocado_lv
  VG Name                avocado_vg
  LV UUID nkhkFh-Oofl-GKH1-1055-3B47-0gup-yQtI1s
  LV Write Access        read/write
  LV Creation host, time ltc-zz1b-lp4.aus.stglabs.ibm.com, 2022-03-02
01:32:19 -0600
  LV Status              available
  # open                 0
  LV Size                18.00 GiB
  Current LE             4608
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     8192
  Block device           253:0

mkfs.ext2 /dev/avocado_vg/avocado_lv

mke2fs 1.44.6 (5-Mar-2019)
Creating filesystem with 4718592 4k blocks and 1179648 inodes
Filesystem UUID: 5ed3c335-bac2-4b64-a827-222d287af0b2
Superblock backups stored on blocks:
    32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632,
2654208,
    4096000

Allocating group tables: done
Writing inode tables: done
Writing superblocks and filesystem accounting information: done

mount /dev/avocado_vg/avocado_lv /mnt

[ 1053.517019] EXT4-fs (dm-0): mounting ext2 file system using the ext4
subsystem
[ 1053.518270] EXT4-fs (dm-0): mounted filesystem without journal. Quota
mode: none.

umount /dev/avocado_vg/avocado_lv

lvdisplay avocado_vg
  --- Logical volume ---
  LV Path                /dev/avocado_vg/avocado_lv
  LV Name                avocado_lv
  VG Name                avocado_vg
  LV UUID nkhkFh-Oofl-GKH1-1055-3B47-0gup-yQtI1s
  LV Write Access        read/write
  LV Creation host, time ltc-zz1b-lp4.aus.stglabs.ibm.com, 2022-03-02
01:32:19 -0600
  LV Status              available
  # open                 0
  LV Size                18.00 GiB
  Current LE             4608
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     8192
  Block device           253:0

lvcreate --snapshot --name avocado_sn /dev/avocado_vg/avocado_lv
--ignoreactivationskip --size 18432.0

  Logical volume "avocado_sn" created.

vgdisplay avocado_vg

  --- Volume group ---
  VG Name               avocado_vg
  System ID
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  4
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                2
  Open LV               0
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               <372.61 GiB
  PE Size               4.00 MiB
  Total PE              95388
  Alloc PE / Size       9216 / 36.00 GiB
  Free  PE / Size       86172 / <336.61 GiB
  VG UUID               jobi1f-kHw4-2ovw-nR3E-Eml5-tFYR-mJc3WT

lvdisplay avocado_vg
  --- Logical volume ---
  LV Path                /dev/avocado_vg/avocado_lv
  LV Name                avocado_lv
  VG Name                avocado_vg
  LV UUID nkhkFh-Oofl-GKH1-1055-3B47-0gup-yQtI1s
  LV Write Access        read/write
  LV Creation host, time ltc-zz1b-lp4.aus.stglabs.ibm.com, 2022-03-02
01:32:19 -0600
  LV snapshot status     source of
                         avocado_sn [active]
  LV Status              available
  # open                 0
  LV Size                18.00 GiB
  Current LE             4608
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:0

  --- Logical volume ---
  LV Path                /dev/avocado_vg/avocado_sn
  LV Name                avocado_sn
  VG Name                avocado_vg
  LV UUID QL14R9-aVa9-nD8z-DTyg-87qL-b2yY-O30G5P
  LV Write Access        read/write
  LV Creation host, time ltc-zz1b-lp4.aus.stglabs.ibm.com, 2022-03-02
01:32:20 -0600
  LV snapshot status     active destination for avocado_lv
  LV Status              available
  # open                 0
  LV Size                18.00 GiB
  Current LE             4608
  COW-table size         18.00 GiB
  COW-table LE           4608
  Allocated to snapshot  0.00%
  Snapshot chunk size    4.00 KiB
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     8192
  Block device           253:3

lvconvert --merge --interval 1 /dev/avocado_vg/avocado_sn
  Merging of volume avocado_vg/avocado_sn started.
  avocado_vg/avocado_lv: Merged: 100.00%
BUG: Unable to handle kernel data access on write at 0x3335323d524f4ab1
Faulting instruction address: 0xc000000000cb3a2c
Oops: Kernel access of bad area, sig: 11 [#1]
LE PAGE_SIZE=64K MMU=Hash SMP NR_CPUS=2048 NUMA pSeries
Modules linked in: dm_snapshot dm_bufio rpadlpar_io rpaphp kvm_pr kvm
nf_tables libcrc32c
 nfnetlink tcp_diag udp_diag inet_diag unix_diag af_packet_diag
netlink_diag rfkill dm_mod sg pseries_rn
g xts uio_pdrv_genirq vmx_crypto gf128mul uio nfsd auth_rpcgss nfs_acl
lockd grace sunrpc binfmt_misc sc
h_fq_codel ip_tables ext4 mbcache jbd2 sd_mod ibmvscsi ibmvnic
scsi_transport_srp ibmveth nvme nvme_core
 t10_pi
 CPU: 17 PID: 0 Comm: swapper/17 Kdump: loaded Not tainted
5.17.0-rc5-next-20220225-autote

 NIP:  c000000000cb3a2c LR: c0000000006740b4 CTR: c000000000674030
 REGS: c0000000034ff390 TRAP: 0380   Not tainted
(5.17.0-rc5-next-20220225-autotest)
 MSR:  8000000000009033 <SF,EE,ME,IR,DR,RI,LE>  CR: 82008228  XER: 20040000
 CFAR: c0000000006740b0 IRQMASK: 1
 GPR00: c000000000220290 c0000000034ff630 c0000000028d3200
3335323d524f4ab1
 GPR04: c000000000674030 00000001000126d5 0000000000000200
0000000000000000
 GPR08: 0000000000000000 0000000000000000 0000000000000001
0000000000000001
 GPR12: 0000000022000222 c00000000ffa6680 0000000000000100
0000000000000010
 GPR16: 0000000000000100 0000000000000002 c000000002902108
c0000000020e7880
 GPR20: c000000000674030 5deadbeef0000122 c00000077c908668
c0000000b4dce000
 GPR24: c0000000acb6a800 c0000000034ff7c8 3335323d524f4a41
0000000000000100
 GPR28: c000000000674030 c0000000acb6a880 3335323d524f4ab1
c0000000acb6a830
 NIP [c000000000cb3a2c] _raw_spin_lock_irq+0x1c/0xa0
 LR [c0000000006740b4] throtl_pending_timer_fn+0x84/0x450
 Call Trace:
 [c0000000034ff630] [c0000000034ff670] 0xc0000000034ff670 (unreliable)
 [c0000000034ff650] [c000000002903a00] jiffies+0x0/0x80
 [c0000000034ff710] [c000000000220290] call_timer_fn+0x50/0x200
 [c0000000034ff7a0] [c000000000221c30] run_timer_softirq+0x340/0x7c0
 [c0000000034ff870] [c000000000cb3cb8] __do_softirq+0x158/0x3e0
 [c0000000034ff970] [c00000000014ef78] irq_exit+0x168/0x1b0
 [c0000000034ff9a0] [c0000000000271c4] timer_interrupt+0x1a4/0x3e0
 [c0000000034ffa00] [c000000000009a08] decrementer_common_virt+0x208/0x210
 --- interrupt: 900 at plpar_hcall_norets_notrace+0x18/0x2c
 NIP:  c0000000000ece58 LR: c0000000009d58c8 CTR: 0000000000000000
 REGS: c0000000034ffa70 TRAP: 0900   Not tainted
(5.17.0-rc5-next-20220225-autotest)
 MSR:  800000000280b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE>  CR:
22000224  XER: 20040000
 CFAR: 0000000000000000 IRQMASK: 0
 GPR00: 0000000000000000 c0000000034ffd10 c0000000028d3200
0000000000000000
 GPR04: 000000000000ffff 0000000000000000 0000000000000000
0000000000000000
 GPR08: 0000000000000000 0000000000000000 800000140fc3fc00
ffffffffffffffff
 GPR12: 0000000000000000 c00000000ffa6680 0000000000010000
0000000000000800
 GPR16: c0000000016a20e8 c00000077fff1678 ffffffffffffffff
0000000000000001
 GPR20: 0000000000000001 0000000000000002 0000000000000010
c00000000282ff88
 GPR24: 0000000000000001 000000f59a421dce 0000000000000000
0000000000000000
 GPR28: 0000000000000001 0000000000000000 c0000000020e2098
c0000000020e20a0
 NIP [c0000000000ece58] plpar_hcall_norets_notrace+0x18/0x2c
 LR [c0000000009d58c8] check_and_cede_processor+0x48/0x60
 --- interrupt: 900
 [c0000000034ffd10] [c0000000034ffd40] 0xc0000000034ffd40 (unreliable)
 [c0000000034ffd70] [c0000000009d5d04] dedicated_cede_loop+0x94/0x1a0
 [c0000000034ffdc0] [c0000000009d2634] cpuidle_enter_state+0x2d4/0x4e0
 [c0000000034ffe20] [c0000000009d28d8] cpuidle_enter+0x48/0x70
 [c0000000034ffe60] [c00000000019db74] call_cpuidle+0x44/0x80
 [c0000000034ffe80] [c00000000019e150] do_idle+0x340/0x390
 [c0000000034fff00] [c00000000019e3d4] cpu_startup_entry+0x34/0x40
 [c0000000034fff30] [c0000000000607e0] start_secondary+0x290/0x2b0
 [c0000000034fff90] [c00000000000d154] start_secondary_prolog+0x10/0x14
 Instruction dump:
 7c0803a6 4bffffa4 60000000 60000000 60000000 3c4c01c2 3842f7f0 f821ffe1
 39200001 992d0932 39200000 39400001 <7c801829> 7c044800 40c20010 7d40192d
 ---[ end trace 0000000000000000 ]---

--
Regard's

Abdul Haleem
IBM Linux Technology Center


2022-03-02 23:47:29

by Ming Lei

[permalink] [raw]
Subject: Re: [next-20220225][Oops][ppc] lvm snapshot merge results kernel panics (throtl_pending_timer_fn)

On Wed, Mar 02, 2022 at 01:31:39PM +0530, Abdul Haleem wrote:
> Greeting's
>
> Linux next kernel 5.17.0-rc5-next-20220225 crashed on my power 10 LPAR when
> merge lvm snapshot on nvme disk

Please try next-20220301, in which the "bad" patch of 'block: cancel all
throttled bios in del_gendisk()' is dropped.


Thanks,
Ming