Received-SPF: pass (google.com: domain of linux-ext4-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Content-Type: text/plain;
        charset=utf-8
Subject: Re: [powerpc] Kernel oops while running xfstests w/ext4
 (5.18-rc6-next-20220510)
From:   Sachin Sant <sachinp@linux.ibm.com>
In-Reply-To: <849697D9-0BF2-435F-B4F0-BC971269A9AA@linux.ibm.com>
Date:   Wed, 11 May 2022 20:05:27 +0530
Cc:     linux-next@vger.kernel.org, linuxppc-dev@lists.ozlabs.org,
        riteshh@linux.ibm.com
Message-Id: <78688136-AE1B-4D57-8E94-C18AE8C4E006@linux.ibm.com>
References: <849697D9-0BF2-435F-B4F0-BC971269A9AA@linux.ibm.com>
To:     Ext4 Developers List <linux-ext4@vger.kernel.org>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
Precedence: bulk


> On 11-May-2022, at 2:56 PM, Sachin Sant <sachinp@linux.ibm.com> wrote:
>=20
> While running xfstests (specifically ext4/032) w/ext4 on a POWER9 LPAR ru=
nning
> linux-next version 5.18.0-rc6-next-20220510 following crash is seen:
>=20
> [  472.486440] EXT4-fs (loop0): resized filesystem to 41943040
> [  472.760888] BUG: Kernel NULL pointer dereference at 0x0000002c
> [  472.760891] Faulting instruction address: 0xc0000000007729f4
> [  472.760894] Oops: Kernel access of bad area, sig: 11 [#1]
> [  472.760913] LE PAGE_SIZE=3D64K MMU=3DHash SMP NR_CPUS=3D2048 NUMA pSer=
ies
> [  472.760921] Modules linked in: loop(E) dm_mod(E) nft_fib_inet(E) nft_f=
ib_ipv4(E) nft_fib_ipv6(E) nft_fib(E) nft_reject_inet(E) nf_reject_ipv4(E) =
nf_reject_ipv6(E) nft_reject(E) nft_ct(E) nft_chain_nat(E) nf_nat(E) nf_con=
ntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) ip_set(E) bonding(E) rfkill(E=
) tls(E) nf_tables(E) libcrc32c(E) nfnetlink(E) sunrpc(E) pseries_rng(E) vm=
x_crypto(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sd_mod(E) sg(E) l=
pfc(E) nvmet_fc(E) nvmet(E) ibmvscsi(E) scsi_transport_srp(E) ibmveth(E) nv=
me_fc(E) nvme(E) nvme_fabrics(E) nvme_core(E) t10_pi(E) scsi_transport_fc(E=
) crc64_rocksoft(E) crc64(E) tg3(E) ipmi_devintf(E) ipmi_msghandler(E) fuse=
(E)
> [  472.761006] CPU: 8 PID: 5139 Comm: kworker/u193:0 Tainted: G          =
  E     5.18.0-rc6-next-20220510 #2
> [  472.761013] Workqueue: loop0 loop_rootcg_workfn [loop]
> [  472.761027] NIP:  c0000000007729f4 LR: c00000000077331c CTR: c00000000=
09e9ac0
> [  472.761032] REGS: c00000002d95b3a0 TRAP: 0380   Tainted: G            =
E      (5.18.0-rc6-next-20220510)
> [  472.761038] MSR:  800000000280b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE>  =
CR: 24008822  XER: 00000000
> [  472.761057] CFAR: c000000000772b80 IRQMASK: 0=20
> [  472.761057] GPR00: c00000000077331c c00000002d95b640 c000000002a7cf00 =
c00000002d95b8e0=20
> [  472.761057] GPR04: c00000006fd58200 0000000000000001 0000000000000010 =
0000000000000040=20
> [  472.761057] GPR08: 0000000000000020 0000000000000000 0000000000010000 =
c0080000089570f8=20
> [  472.761057] GPR12: 0000000000008000 c00000001ec46300 0000000000000000 =
c000000054e32200=20
> [  472.761057] GPR16: 5deadbeef0000100 0000000000000000 0000000000000000 =
0000000000000000=20
> [  472.761057] GPR20: 000000007fffffff c009fffffc817a00 c00000002d95b748 =
c00000002d95b8e0=20
> [  472.761057] GPR24: 0000000000000001 0000000000000000 c0000000842b1c00 =
0000000000000000=20
> [  472.761057] GPR28: 0000000000000000 0000000000000000 c00000006fd58200 =
c00000002d95b8e0=20
> [  472.761126] NIP [c0000000007729f4] blk_add_rq_to_plug+0x74/0x1d0
> [  472.761135] LR [c00000000077331c] blk_mq_try_issue_list_directly+0x18c=
/0x1d0
> [  472.761141] Call Trace:
> [  472.761144] [c00000002d95b640] [c0000000842b1c00] 0xc0000000842b1c00 (=
unreliable)
> [  472.761153] [c00000002d95b680] [c000000000773244] blk_mq_try_issue_lis=
t_directly+0xb4/0x1d0
> [  472.761160] [c00000002d95b6d0] [c00000000077b38c] blk_mq_sched_insert_=
requests+0x13c/0x240
> [  472.761168] [c00000002d95b720] [c000000000772658] blk_mq_flush_plug_li=
st+0x118/0x440
> [  472.761175] [c00000002d95b7c0] [c00000000075ecbc] __blk_flush_plug+0x1=
7c/0x200
> [  472.761183] [c00000002d95b840] [c00000000075efe0] blk_finish_plug+0x50=
/0x70
> [  472.761190] [c00000002d95b870] [c00000000061a2a4] __iomap_dio_rw+0x444=
/0x960
> [  472.761200] [c00000002d95ba60] [c00000000061a7e0] iomap_dio_rw+0x20/0x=
90
> [  472.761208] [c00000002d95ba80] [c008000008c56424] ext4_file_read_iter+=
0x17c/0x2d0 [ext4]
> [  472.761237] [c00000002d95bac0] [c008000009822aa8] lo_rw_aio.isra.36+0x=
260/0x320 [loop]
> [  472.761245] [c00000002d95bb40] [c008000009824030] loop_process_work+0x=
448/0xb70 [loop]
> [  472.761253] [c00000002d95bc90] [c000000000183744] process_one_work+0x2=
b4/0x5b0
> [  472.761262] [c00000002d95bd30] [c000000000183ab8] worker_thread+0x78/0=
x600
> [  472.761269] [c00000002d95bdc0] [c0000000001901d4] kthread+0x124/0x130
> [  472.761276] [c00000002d95be10] [c00000000000ce04] ret_from_kernel_thre=
ad+0x5c/0x64
> [  472.761284] Instruction dump:
> [  472.761288] 893f0014 38e00040 39000020 2fa90000 7d283f9e 7e8a4840 4094=
00b4 e93e0000=20
> [  472.761300] e9290068 71290008 40820024 3d400001 <813d002c> 614affff 7e=
895040 41950090=20
> [  472.761314] ---[ end trace 0000000000000000 ]---
> [  472.769088]=20
> [  473.769091] Kernel panic - not syncing: Fatal exception
>=20
> 5.18.0-rc6-next-20220509 build did not exhibit this problem.
> Will try git bisect and report back with results.
>=20

Unfortunately git bisect doesn=E2=80=99t seem to help.
first bad commit: [3aedd17333a514c6f2542ed305d940e7a970a6f2]=20
          Merge branch 'next' of git://git.kernel.org/pub/scm/linux/kernel/=
git/ulfh/mmc.git

# git bisect log
git bisect start
# bad: [3bf222d317a20170ee17f082626c1e0f83537e13] Add linux-next specific f=
iles for 20220510
git bisect bad 3bf222d317a20170ee17f082626c1e0f83537e13
# good: [c5eb0a61238dd6faf37f58c9ce61c9980aaffd7a] Linux 5.18-rc6
git bisect good c5eb0a61238dd6faf37f58c9ce61c9980aaffd7a
# good: [5b0c7020c4bf2cfed914323a7d58777c47df6bf8] Merge branch 'drm-next' =
of git://git.freedesktop.org/git/drm/drm.git
git bisect good 5b0c7020c4bf2cfed914323a7d58777c47df6bf8
# bad: [d11e84253a18f5afa5bb6e7da8509ba249435c0c] Merge branch 'next' of gi=
t://git.kernel.org/pub/scm/virt/kvm/kvm.git
git bisect bad d11e84253a18f5afa5bb6e7da8509ba249435c0c
# good: [24f8d248db54c383b01e373355d8655810832192] Merge branch 'next' of g=
it://git.kernel.org/pub/scm/linux/kernel/git/dtor/input.git
git bisect good 24f8d248db54c383b01e373355d8655810832192
# bad: [52be5c3c2431cb82789bd9cdb5169f7a2538a8af] Merge branch 'for-next' o=
f git://git.kernel.org/pub/scm/linux/kernel/git/robh/linux.git
git bisect bad 52be5c3c2431cb82789bd9cdb5169f7a2538a8af
# good: [020a5624b05bb35b24beb1e405ecf38a37c3c238] Merge branch 'pcmcia-nex=
t' of git://git.kernel.org/pub/scm/linux/kernel/git/brodo/linux.git
git bisect good 020a5624b05bb35b24beb1e405ecf38a37c3c238
# bad: [528beacf1e5a082c15738652115e8eed516e925d] Merge branch 'master' of =
https://scm.osdn.net/gitroot/tomoyo/tomoyo-test1.git
git bisect bad 528beacf1e5a082c15738652115e8eed516e925d
# bad: [e8f0eb6a4d1b9775d1ba3ed5b95bbb9cdbe663a0] Merge branch 'for-next' o=
f git://git.kernel.org/pub/scm/linux/kernel/git/sre/linux-power-supply.git
git bisect bad e8f0eb6a4d1b9775d1ba3ed5b95bbb9cdbe663a0
# good: [7a0587496a6233b9ffa8441573d38f8844751066] dt-bindings: mmc: sdhci-=
am654: Add flag to force setting of TESTCD bit
git bisect good 7a0587496a6233b9ffa8441573d38f8844751066
# good: [17a9f73d45ea74b2beb009c29ad38569990c3453] dt-bindings: mmc: sdhci-=
msm: Add compatible string for sm8150
git bisect good 17a9f73d45ea74b2beb009c29ad38569990c3453
# good: [3474b838f4204c21d108183d9268611d961a428f] dt-bindings: Drop undocu=
mented i.MX iomuxc-gpr bindings in examples
git bisect good 3474b838f4204c21d108183d9268611d961a428f
# good: [0c9ee5ba7555016afd1efc9598c3f83de5d83470] mmc: sdhci-brcmstb: Fix =
compiler warning
git bisect good 0c9ee5ba7555016afd1efc9598c3f83de5d83470
# good: [d96a89407e5f682d1cb22569d91784506c784863] power: supply: bq24190_c=
harger: using pm_runtime_resume_and_get instead of pm_runtime_get_sync
git bisect good d96a89407e5f682d1cb22569d91784506c784863
# bad: [6d4c27440f59a40aa8df6ad5ed3ff8ae50048870] Merge branch 'for-mfd-nex=
t' of git://git.kernel.org/pub/scm/linux/kernel/git/lee/mfd.git
git bisect bad 6d4c27440f59a40aa8df6ad5ed3ff8ae50048870
# bad: [3aedd17333a514c6f2542ed305d940e7a970a6f2] Merge branch 'next' of gi=
t://git.kernel.org/pub/scm/linux/kernel/git/ulfh/mmc.git
git bisect bad 3aedd17333a514c6f2542ed305d940e7a970a6f2
# first bad commit: [3aedd17333a514c6f2542ed305d940e7a970a6f2] Merge branch=
 'next' of git://git.kernel.org/pub/scm/linux/kernel/git/ulfh/mmc.git

-Sachin