Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Content-Type: text/plain;
        charset=utf-8
Mime-Version: 1.0 (Mac OS X Mail 15.0 \(3693.60.0.1.1\))
Subject: Re: [powerpc]Kernel crash while running xfstests (generic/250)
 [next-20220404]
From:   Sachin Sant <sachinp@linux.ibm.com>
In-Reply-To: <A3B628C9-A20B-4621-BC8C-C79B270297E1@linux.ibm.com>
Date:   Thu, 7 Apr 2022 10:19:44 +0530
Cc:     dm-devel@redhat.com, riteshh@linux.ibm.com,
        linuxppc-dev@lists.ozlabs.org,
        open list <linux-kernel@vger.kernel.org>
Content-Transfer-Encoding: quoted-printable
Message-Id: <B34CE5E9-8635-4DF0-AC0D-FB887462DFD6@linux.ibm.com>
References: <A3B628C9-A20B-4621-BC8C-C79B270297E1@linux.ibm.com>
To:     linux-block@vger.kernel.org, snitzer@redhat.com
Precedence: bulk


> On 04-Apr-2022, at 5:04 PM, Sachin Sant <sachinp@linux.ibm.com> wrote:
>=20
> While running xfstests(ext4 or XFS as fs) on a Power10 LPAR booted =
with today=E2=80=99s
> next (5.18.0-rc1-next-20220404) following crash is seen.=20
>=20
> This problem was possibly introduced with 5.17.0-next-20220330.=20
> Git bisect leads me to following patch
> commit 1d158814db8e7b3cbca0f2c8d9242fbec4fbc57e
>    dm: conditionally enable BIOSET_PERCPU_CACHE for dm_io bioset
>=20

Continue to see this problem with latest next.=20

[ 2388.091152] EXT4-fs (dm-0): mounted filesystem with ordered data =
mode. Quota mode: none.
[ 2388.091173] ext4 filesystem being mounted at /mnt/scratch supports =
timestamps until 2038 (0x7fffffff)
[ 2388.287138] BUG: Unable to handle kernel data access at =
0x5deadbeef0000122
[ 2388.287154] Faulting instruction address: 0xc000000000154a6c
[ 2388.287160] Oops: Kernel access of bad area, sig: 11 [#1]
[ 2388.287164] LE PAGE_SIZE=3D64K MMU=3DHash SMP NR_CPUS=3D2048 NUMA =
pSeries
[ 2388.287172] Modules linked in: xfs dm_flakey dm_snapshot dm_bufio =
dm_zero loop dm_mod ip_set bonding rfkill nf_tables libcrc32c nfnetlink =
sunrpc pseries_rng xts vmx_crypto uio_pdrv_genirq uio sch_fq_codel ext4 =
mbcache jbd2 lpfc nvmet_fc nvmet sr_mod sd_mod cdrom nvme_fc sg nvme =
nvme_fabrics tg3 nvme_core ptp ibmvscsi t10_pi crc64_rocksoft ibmveth =
scsi_transport_srp scsi_transport_fc pps_core crc64 ipmi_devintf =
ipmi_msghandler fuse [last unloaded: scsi_debug]
[ 2388.287236] CPU: 16 PID: 1043652 Comm: dmsetup Not tainted =
5.18.0-rc1-next-20220406 #1
[ 2388.287244] NIP:  c000000000154a6c LR: c000000000154998 CTR: =
c000000000674690
[ 2388.287249] REGS: c000000145fb3610 TRAP: 0380   Not tainted  =
(5.18.0-rc1-next-20220406)
[ 2388.287255] MSR:  800000000280b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE>  =
CR: 28024824  XER: 00000000
[ 2388.287271] CFAR: c0000000001549a0 IRQMASK: 0=20
[ 2388.287271] GPR00: c000000000154998 c000000145fb38b0 c000000002a1f400 =
0000000000000000=20
[ 2388.287271] GPR04: c00000004aa2c378 0000000000000000 c000000048fdf060 =
c00000015387b600=20
[ 2388.287271] GPR08: c00000015387b600 5deadbeef0000100 5deadbeef0000122 =
c00800000988a4e8=20
[ 2388.287271] GPR12: c000000000674690 c00000001ec28a80 0000010014bf0b40 =
00007fff9f1b9da8=20
[ 2388.287271] GPR16: 00007fff9f1b9da8 00007fff9f1b9da8 00007fff9f1f3670 =
0000000000000000=20
[ 2388.287271] GPR20: 00007fff9f1c3388 00007fff9f1f2040 0000010014bf0b70 =
0000000000000131=20
[ 2388.287271] GPR24: c00000000254d768 ffffffffffff0000 c00000000254d730 =
c0000000f5103a00=20
[ 2388.287271] GPR28: c0000000029b0570 c00000004aa2c378 0000000000000017 =
0000000000000000=20
[ 2388.287332] NIP [c000000000154a6c] =
__cpuhp_state_remove_instance+0x19c/0x2c0
[ 2388.287344] LR [c000000000154998] =
__cpuhp_state_remove_instance+0xc8/0x2c0
[ 2388.287351] Call Trace:
[ 2388.287353] [c000000145fb38b0] [c000000000154998] =
__cpuhp_state_remove_instance+0xc8/0x2c0 (unreliable)
[ 2388.287362] [c000000145fb3920] [c0000000006746ec] =
bioset_exit+0x5c/0x280
[ 2388.287369] [c000000145fb39c0] [c0080000098733f4] =
cleanup_mapped_device+0x4c/0x1a0 [dm_mod]
[ 2388.287385] [c000000145fb3a00] [c0080000098736f0] =
__dm_destroy+0x1a8/0x360 [dm_mod]
[ 2388.287397] [c000000145fb3aa0] [c008000009880e90] =
dev_remove+0x1a8/0x280 [dm_mod]
[ 2388.287409] [c000000145fb3b30] [c00800000988115c] =
ctl_ioctl+0x1f4/0x7c0 [dm_mod]
[ 2388.287422] [c000000145fb3d40] [c008000009881748] =
dm_ctl_ioctl+0x20/0x40 [dm_mod]
[ 2388.287434] [c000000145fb3d60] [c0000000004b2c08] =
sys_ioctl+0xf8/0x150
[ 2388.287441] [c000000145fb3db0] [c0000000000324e8] =
system_call_exception+0x178/0x380
[ 2388.287449] [c000000145fb3e10] [c00000000000c64c] =
system_call_common+0xec/0x250
[ 2388.287457] --- interrupt: c00 at 0x7fff9ec991a0
[ 2388.287461] NIP:  00007fff9ec991a0 LR: 00007fff9f1b6824 CTR: =
0000000000000000
[ 2388.287466] REGS: c000000145fb3e80 TRAP: 0c00   Not tainted  =
(5.18.0-rc1-next-20220406)
[ 2388.287471] MSR:  800000000280f033 =
<SF,VEC,VSX,EE,PR,FP,ME,IR,DR,RI,LE>  CR: 28004484  XER: 00000000
[ 2388.287486] IRQMASK: 0=20
[ 2388.287486] GPR00: 0000000000000036 00007fffe5635be0 00007fff9ed77300 =
0000000000000003=20
[ 2388.287486] GPR04: 00000000c138fd04 0000010014bf0b40 0000000000000004 =
00007fff9f1c3f98=20
[ 2388.287486] GPR08: 0000000000000003 0000000000000000 0000000000000000 =
0000000000000000=20
[ 2388.287486] GPR12: 0000000000000000 00007fff9f29fa00 0000010014bf0b40 =
00007fff9f1b9da8=20
[ 2388.287486] GPR16: 00007fff9f1b9da8 00007fff9f1b9da8 00007fff9f1f3670 =
0000000000000000=20
[ 2388.287486] GPR20: 00007fff9f1c3388 00007fff9f1f2040 0000010014bf0b70 =
0000010014bf0940=20
[ 2388.287486] GPR24: 00007fff9f1b9da8 00007fff9f1b9da8 00007fff9f1b9da8 =
00007fff9f1b9da8=20
[ 2388.287486] GPR28: 0000000000000001 00007fff9f1b9da8 0000000000000000 =
00007fff9f1b9da8=20
[ 2388.287543] NIP [00007fff9ec991a0] 0x7fff9ec991a0
[ 2388.287547] LR [00007fff9f1b6824] 0x7fff9f1b6824
[ 2388.287551] --- interrupt: c00
[ 2388.287554] Instruction dump:
[ 2388.287558] 60000000 7f69db78 7f83e040 7c7f07b4 7bea1f24 419cffb4 =
eae10028 eb210038=20
[ 2388.287569] eb610048 e93d0000 e95d0008 2fa90000 <f92a0000> 419e0008 =
f9490008 3d405dea=20
[ 2388.287581] ---[ end trace 0000000000000000 ]---
[ 2388.403785]=20
[ 2389.403791] Kernel panic - not syncing: Fatal exception

Let me know if any additional information is required.

-Sachin