Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <d583adf0-2d98-60b6-620c-722912c05852@linux.vnet.ibm.com>
Date:   Wed, 2 Mar 2022 13:31:39 +0530
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101
 Thunderbird/91.3.0
Content-Language: en-US
To:     linux-scsi <linux-scsi@vger.kernel.org>
From:   Abdul Haleem <abdhalee@linux.vnet.ibm.com>
Subject: [next-20220225][Oops][ppc] lvm snapshot merge results kernel panics
 (throtl_pending_timer_fn)
Cc:     yukuai3@huawei.com, ming.lei@redhat.com,
        linux-next <linux-next@vger.kernel.org>, axboe@kernel.dk,
        linux-block@vger.kernel.org,
        linuxppc-dev <linuxppc-dev@lists.ozlabs.org>,
        linux-kernel <linux-kernel@vger.kernel.org>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Precedence: bulk

Greeting's

Linux next kernel 5.17.0-rc5-next-20220225 crashed on my power 10 LPAR 
when merge lvm snapshot on nvme disk

console logs
------------

fdisk -l /dev/nvme1n1

Disk /dev/nvme1n1: 372.6 GiB, 400088457216 bytes, 97677846 sectors
Units: sectors of 1 * 4096 = 4096 bytes
Sector size (logical/physical): 4096 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
vgcreate avocado_vg /dev/nvme1n1
   Volume group "avocado_vg" successfully created

lvcreate --name avocado_lv --size 18432.0 avocado_vg -y
   Wiping ext2 signature on /dev/avocado_vg/avocado_lv.
   Logical volume "avocado_lv" created.

lvdisplay avocado_vg
   --- Logical volume ---
   LV Path                /dev/avocado_vg/avocado_lv
   LV Name                avocado_lv
   VG Name                avocado_vg
   LV UUID nkhkFh-Oofl-GKH1-1055-3B47-0gup-yQtI1s
   LV Write Access        read/write
   LV Creation host, time ltc-zz1b-lp4.aus.stglabs.ibm.com, 2022-03-02 
01:32:19 -0600
   LV Status              available
   # open                 0
   LV Size                18.00 GiB
   Current LE             4608
   Segments               1
   Allocation             inherit
   Read ahead sectors     auto
   - currently set to     8192
   Block device           253:0

mkfs.ext2 /dev/avocado_vg/avocado_lv

mke2fs 1.44.6 (5-Mar-2019)
Creating filesystem with 4718592 4k blocks and 1179648 inodes
Filesystem UUID: 5ed3c335-bac2-4b64-a827-222d287af0b2
Superblock backups stored on blocks:
     32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 
2654208,
     4096000

Allocating group tables: done
Writing inode tables: done
Writing superblocks and filesystem accounting information: done

mount /dev/avocado_vg/avocado_lv /mnt

[ 1053.517019] EXT4-fs (dm-0): mounting ext2 file system using the ext4 
subsystem
[ 1053.518270] EXT4-fs (dm-0): mounted filesystem without journal. Quota 
mode: none.

umount /dev/avocado_vg/avocado_lv

lvdisplay avocado_vg
   --- Logical volume ---
   LV Path                /dev/avocado_vg/avocado_lv
   LV Name                avocado_lv
   VG Name                avocado_vg
   LV UUID nkhkFh-Oofl-GKH1-1055-3B47-0gup-yQtI1s
   LV Write Access        read/write
   LV Creation host, time ltc-zz1b-lp4.aus.stglabs.ibm.com, 2022-03-02 
01:32:19 -0600
   LV Status              available
   # open                 0
   LV Size                18.00 GiB
   Current LE             4608
   Segments               1
   Allocation             inherit
   Read ahead sectors     auto
   - currently set to     8192
   Block device           253:0

lvcreate --snapshot --name avocado_sn /dev/avocado_vg/avocado_lv 
--ignoreactivationskip --size 18432.0

   Logical volume "avocado_sn" created.

vgdisplay avocado_vg

   --- Volume group ---
   VG Name               avocado_vg
   System ID
   Format                lvm2
   Metadata Areas        1
   Metadata Sequence No  4
   VG Access             read/write
   VG Status             resizable
   MAX LV                0
   Cur LV                2
   Open LV               0
   Max PV                0
   Cur PV                1
   Act PV                1
   VG Size               <372.61 GiB
   PE Size               4.00 MiB
   Total PE              95388
   Alloc PE / Size       9216 / 36.00 GiB
   Free  PE / Size       86172 / <336.61 GiB
   VG UUID               jobi1f-kHw4-2ovw-nR3E-Eml5-tFYR-mJc3WT

lvdisplay avocado_vg
   --- Logical volume ---
   LV Path                /dev/avocado_vg/avocado_lv
   LV Name                avocado_lv
   VG Name                avocado_vg
   LV UUID nkhkFh-Oofl-GKH1-1055-3B47-0gup-yQtI1s
   LV Write Access        read/write
   LV Creation host, time ltc-zz1b-lp4.aus.stglabs.ibm.com, 2022-03-02 
01:32:19 -0600
   LV snapshot status     source of
                          avocado_sn [active]
   LV Status              available
   # open                 0
   LV Size                18.00 GiB
   Current LE             4608
   Segments               1
   Allocation             inherit
   Read ahead sectors     auto
   - currently set to     256
   Block device           253:0

   --- Logical volume ---
   LV Path                /dev/avocado_vg/avocado_sn
   LV Name                avocado_sn
   VG Name                avocado_vg
   LV UUID QL14R9-aVa9-nD8z-DTyg-87qL-b2yY-O30G5P
   LV Write Access        read/write
   LV Creation host, time ltc-zz1b-lp4.aus.stglabs.ibm.com, 2022-03-02 
01:32:20 -0600
   LV snapshot status     active destination for avocado_lv
   LV Status              available
   # open                 0
   LV Size                18.00 GiB
   Current LE             4608
   COW-table size         18.00 GiB
   COW-table LE           4608
   Allocated to snapshot  0.00%
   Snapshot chunk size    4.00 KiB
   Segments               1
   Allocation             inherit
   Read ahead sectors     auto
   - currently set to     8192
   Block device           253:3

lvconvert --merge --interval 1 /dev/avocado_vg/avocado_sn
   Merging of volume avocado_vg/avocado_sn started.
   avocado_vg/avocado_lv: Merged: 100.00%
BUG: Unable to handle kernel data access on write at 0x3335323d524f4ab1
Faulting instruction address: 0xc000000000cb3a2c
Oops: Kernel access of bad area, sig: 11 [#1]
LE PAGE_SIZE=64K MMU=Hash SMP NR_CPUS=2048 NUMA pSeries
Modules linked in: dm_snapshot dm_bufio rpadlpar_io rpaphp kvm_pr kvm 
nf_tables libcrc32c
  nfnetlink tcp_diag udp_diag inet_diag unix_diag af_packet_diag 
netlink_diag rfkill dm_mod sg pseries_rn
g xts uio_pdrv_genirq vmx_crypto gf128mul uio nfsd auth_rpcgss nfs_acl 
lockd grace sunrpc binfmt_misc sc
h_fq_codel ip_tables ext4 mbcache jbd2 sd_mod ibmvscsi ibmvnic 
scsi_transport_srp ibmveth nvme nvme_core
  t10_pi
  CPU: 17 PID: 0 Comm: swapper/17 Kdump: loaded Not tainted 
5.17.0-rc5-next-20220225-autote

  NIP:  c000000000cb3a2c LR: c0000000006740b4 CTR: c000000000674030
  REGS: c0000000034ff390 TRAP: 0380   Not tainted 
(5.17.0-rc5-next-20220225-autotest)
  MSR:  8000000000009033 <SF,EE,ME,IR,DR,RI,LE>  CR: 82008228  XER: 20040000
  CFAR: c0000000006740b0 IRQMASK: 1
  GPR00: c000000000220290 c0000000034ff630 c0000000028d3200 
3335323d524f4ab1
  GPR04: c000000000674030 00000001000126d5 0000000000000200 
0000000000000000
  GPR08: 0000000000000000 0000000000000000 0000000000000001 
0000000000000001
  GPR12: 0000000022000222 c00000000ffa6680 0000000000000100 
0000000000000010
  GPR16: 0000000000000100 0000000000000002 c000000002902108 
c0000000020e7880
  GPR20: c000000000674030 5deadbeef0000122 c00000077c908668 
c0000000b4dce000
  GPR24: c0000000acb6a800 c0000000034ff7c8 3335323d524f4a41 
0000000000000100
  GPR28: c000000000674030 c0000000acb6a880 3335323d524f4ab1 
c0000000acb6a830
  NIP [c000000000cb3a2c] _raw_spin_lock_irq+0x1c/0xa0
  LR [c0000000006740b4] throtl_pending_timer_fn+0x84/0x450
  Call Trace:
  [c0000000034ff630] [c0000000034ff670] 0xc0000000034ff670 (unreliable)
  [c0000000034ff650] [c000000002903a00] jiffies+0x0/0x80
  [c0000000034ff710] [c000000000220290] call_timer_fn+0x50/0x200
  [c0000000034ff7a0] [c000000000221c30] run_timer_softirq+0x340/0x7c0
  [c0000000034ff870] [c000000000cb3cb8] __do_softirq+0x158/0x3e0
  [c0000000034ff970] [c00000000014ef78] irq_exit+0x168/0x1b0
  [c0000000034ff9a0] [c0000000000271c4] timer_interrupt+0x1a4/0x3e0
  [c0000000034ffa00] [c000000000009a08] decrementer_common_virt+0x208/0x210
  --- interrupt: 900 at plpar_hcall_norets_notrace+0x18/0x2c
  NIP:  c0000000000ece58 LR: c0000000009d58c8 CTR: 0000000000000000
  REGS: c0000000034ffa70 TRAP: 0900   Not tainted 
(5.17.0-rc5-next-20220225-autotest)
  MSR:  800000000280b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE>  CR: 
22000224  XER: 20040000
  CFAR: 0000000000000000 IRQMASK: 0
  GPR00: 0000000000000000 c0000000034ffd10 c0000000028d3200 
0000000000000000
  GPR04: 000000000000ffff 0000000000000000 0000000000000000 
0000000000000000
  GPR08: 0000000000000000 0000000000000000 800000140fc3fc00 
ffffffffffffffff
  GPR12: 0000000000000000 c00000000ffa6680 0000000000010000 
0000000000000800
  GPR16: c0000000016a20e8 c00000077fff1678 ffffffffffffffff 
0000000000000001
  GPR20: 0000000000000001 0000000000000002 0000000000000010 
c00000000282ff88
  GPR24: 0000000000000001 000000f59a421dce 0000000000000000 
0000000000000000
  GPR28: 0000000000000001 0000000000000000 c0000000020e2098 
c0000000020e20a0
  NIP [c0000000000ece58] plpar_hcall_norets_notrace+0x18/0x2c
  LR [c0000000009d58c8] check_and_cede_processor+0x48/0x60
  --- interrupt: 900
  [c0000000034ffd10] [c0000000034ffd40] 0xc0000000034ffd40 (unreliable)
  [c0000000034ffd70] [c0000000009d5d04] dedicated_cede_loop+0x94/0x1a0
  [c0000000034ffdc0] [c0000000009d2634] cpuidle_enter_state+0x2d4/0x4e0
  [c0000000034ffe20] [c0000000009d28d8] cpuidle_enter+0x48/0x70
  [c0000000034ffe60] [c00000000019db74] call_cpuidle+0x44/0x80
  [c0000000034ffe80] [c00000000019e150] do_idle+0x340/0x390
  [c0000000034fff00] [c00000000019e3d4] cpu_startup_entry+0x34/0x40
  [c0000000034fff30] [c0000000000607e0] start_secondary+0x290/0x2b0
  [c0000000034fff90] [c00000000000d154] start_secondary_prolog+0x10/0x14
  Instruction dump:
  7c0803a6 4bffffa4 60000000 60000000 60000000 3c4c01c2 3842f7f0 f821ffe1
  39200001 992d0932 39200000 39400001 <7c801829> 7c044800 40c20010 7d40192d
  ---[ end trace 0000000000000000 ]---

-- 
Regard's

Abdul Haleem
IBM Linux Technology Center