Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Sender: Guenter Roeck <groeck7@gmail.com>
Date:   Wed, 12 Oct 2022 09:49:32 -0700
From:   Guenter Roeck <linux@roeck-us.net>
To:     "Jason A. Donenfeld" <Jason@zx2c4.com>
Cc:     Michael Ellerman <mpe@ellerman.id.au>,
        Linus Torvalds <torvalds@linux-foundation.org>,
        ajd@linux.ibm.com, aneesh.kumar@linux.ibm.com,
        atrajeev@linux.vnet.ibm.com, christophe.leroy@csgroup.eu,
        cuigaosheng1@huawei.com, david@redhat.com, farosas@linux.ibm.com,
        geoff@infradead.org, gustavoars@kernel.org, haren@linux.ibm.com,
        hbathini@linux.ibm.com, joel@jms.id.au, lihuafei1@huawei.com,
        linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org,
        lukas.bulwahn@gmail.com, mikey@neuling.org, nathan@kernel.org,
        nathanl@linux.ibm.com, nicholas@linux.ibm.com, npiggin@gmail.com,
        pali@kernel.org, paul@paul-moore.com, rmclure@linux.ibm.com,
        ruscur@russell.cc, windhl@126.com,
        wsa+renesas@sang-engineering.com, ye.xingchen@zte.com.cn,
        yuanjilin@cdjrlc.com, zhengyongjun3@huawei.com
Subject: Re: [GIT PULL] Please pull powerpc/linux.git powerpc-6.1-1 tag
Message-ID: <20221012164932.GB2990467@roeck-us.net>
References: <87edvhntv0.fsf@mpe.ellerman.id.au>
 <20221012141827.GA2405914@roeck-us.net>
 <Y0biBtCUtc2mowbQ@zx2c4.com>
 <Y0bvOreqH2BdyVqt@zx2c4.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <Y0bvOreqH2BdyVqt@zx2c4.com>
Precedence: bulk

On Wed, Oct 12, 2022 at 10:45:46AM -0600, Jason A. Donenfeld wrote:
> On Wed, Oct 12, 2022 at 09:49:26AM -0600, Jason A. Donenfeld wrote:
> > On Wed, Oct 12, 2022 at 07:18:27AM -0700, Guenter Roeck wrote:
> > > NIP [c000000000031630] .replay_soft_interrupts+0x60/0x300
> > > LR [c000000000031964] .arch_local_irq_restore+0x94/0x1c0
> > > Call Trace:
> > > [c000000007df3870] [c000000000031964] .arch_local_irq_restore+0x94/0x1c0 (unreliable)
> > > [c000000007df38f0] [c000000000f8a444] .__schedule+0x664/0xa50
> > > [c000000007df39d0] [c000000000f8a8b0] .schedule+0x80/0x140
> > > [c000000007df3a50] [c00000000092f0dc] .try_to_generate_entropy+0x118/0x174
> > > [c000000007df3b40] [c00000000092e2e4] .urandom_read_iter+0x74/0x140
> > > [c000000007df3bc0] [c0000000003b0044] .vfs_read+0x284/0x2d0
> > > [c000000007df3cd0] [c0000000003b0d2c] .ksys_read+0xdc/0x130
> > > [c000000007df3d80] [c00000000002a88c] .system_call_exception+0x19c/0x330
> > > [c000000007df3e10] [c00000000000c1d4] system_call_common+0xf4/0x258
> > 
> > Obviously the first couple lines of this concern me a bit. But I think
> > actually this might just be a catalyst for another bug. You could view
> > that function as basically just:
> > 
> >     while (something)
> >     	schedule();
> > 
> > And I guess in the process of calling the scheduler a lot, which toggles
> > interrupts a lot, something got wedged.
> > 
> > Curious, though, I did try to reproduce this, to no avail. My .config is
> > https://xn--4db.cc/rBvHWfDZ . What's yours?
> 
> I also just tried using your github linux-build-test scripts as a guide
> for construction a config -- https://xn--4db.cc/B0HpEQDQ -- and loaded
> up your rootfs over sdhci and such, and still couldn't manage to
> reproduce. I tried commenting out the line "if (!bits)" in
> _credit_init_bits(), so that the rng would never initialize, so that the
> schedule() loop would just keep on running indefinitely, but still no
> dice.
> 
> But also, I'm running Linus' tree. From your log, I see
> "6.0.0-rc2-00163-ga5edf9815dd7". So maybe these bugs got fixed
> elsewhere?
> 

Blame me for not attaching the latest crash report.

Guenter

---
BUG: soft lockup - CPU#0 stuck for 23s! [dd:111]
Modules linked in:
CPU: 0 PID: 111 Comm: dd Not tainted 6.0.0-11414-g49da07006239 #1
Hardware name: PowerMac3,1 PPC970FX 0x3c0301 PowerMac
NIP:  c000000000031630 LR: c000000000031964 CTR: 0000000000000000
REGS: c000000007d5b6a8 TRAP: 0900   Not tainted  (6.0.0-11414-g49da07006239)
MSR:  8000000000009032 <SF,EE,ME,IR,DR,RI>  CR: 28002228  XER: 00000000
IRQMASK: 0
GPR00: c000000000031964 c000000007d5b870 c0000000013e5500 c000000007d5b6a8
GPR04: c00000000125e1c0 0000000000000000 c000000007d5b814 c00000000291d018
GPR08: c000000002d4bbb8 0000000000000000 c000000007356400 c000000002d21098
GPR12: 0000000028002222 c000000002e20000 00000000100d32e0 00000000100d32b4
GPR16: 00000000100d3301 00000000100d32b9 00000000100d3358 00000000100d32bf
GPR20: 0000000000002000 00000000100d3372 00000000100d331e c000000007356c18
GPR24: 0000000000000000 0000000000000e60 0000000000000900 0000000000000500
GPR28: 0000000000000a00 0000000000000f00 0000000000000002 0000000000000003
NIP [c000000000031630] .replay_soft_interrupts+0x60/0x300
LR [c000000000031964] .arch_local_irq_restore+0x94/0x1c0
Call Trace:
[c000000007d5b870] [c000000000031964] .arch_local_irq_restore+0x94/0x1c0 (unreliable)
[c000000007d5b8f0] [c000000000f8bac4] .__schedule+0x664/0xa50
[c000000007d5b9d0] [c000000000f8bf30] .schedule+0x80/0x140
[c000000007d5ba50] [c00000000093085c] .try_to_generate_entropy+0x118/0x174
[c000000007d5bb40] [c00000000092fa64] .urandom_read_iter+0x74/0x140
[c000000007d5bbc0] [c0000000003b0044] .vfs_read+0x284/0x2d0
[c000000007d5bcd0] [c0000000003b0d2c] .ksys_read+0xdc/0x130
[c000000007d5bd80] [c00000000002a88c] .system_call_exception+0x19c/0x330
[c000000007d5be10] [c00000000000c1d4] system_call_common+0xf4/0x258
--- interrupt: c00 at 0x7fffb5c9d49c
NIP:  00007fffb5c9d49c LR: 000000001000da90 CTR: 0000000000000000
REGS: c000000007d5be80 TRAP: 0c00   Not tainted  (6.0.0-11414-g49da07006239)
MSR:  800000000000f032 <SF,EE,PR,FP,ME,IR,DR,RI>  CR: 22002422  XER: 00000000
IRQMASK: 0
GPR00: 0000000000000003 00007ffff6dcc220 00007fffb5d97300 0000000000000000
GPR04: 00000000101102a0 0000000000000020 0000000000000000 0000000000000000
GPR08: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
GPR12: 0000000000000000 00007fffb5e6aac0 00000000100d32e0 00000000100d32b4
GPR16: 00000000100d3301 00000000100d32b9 00000000100d3358 00000000100d32bf
GPR20: 0000000000002000 00000000100d3372 00000000100d331e 0000000000000000
GPR24: 7fffffffffffffff 00000000100b3a9c 00000000101102a0 0000000000000020
GPR28: 00000000101025c0 0000000000000020 0000000000000000 0000000000000000
NIP [00007fffb5c9d49c] 0x7fffb5c9d49c
LR [000000001000da90] 0x1000da90
--- interrupt: c00
Instruction dump:
3b600500 3b800a00 3ba00f00 f8010010 f821fdc1 60000000 60000000 38610078
e92d0af8 f92101f8 39200000 48039745 <60000000> 39000000 e9410180 892d0933
Kernel panic - not syncing: softlockup: hung tasks
CPU: 0 PID: 111 Comm: dd Tainted: G             L     6.0.0-11414-g49da07006239 #1
Hardware name: PowerMac3,1 PPC970FX 0x3c0301 PowerMac
Call Trace:
[c000000007d5b180] [c000000000f59b40] .dump_stack_lvl+0x7c/0xc4 (unreliable)
[c000000007d5b210] [c0000000000d5b58] .panic+0x174/0x42c
[c000000007d5b2c0] [c000000000231054] .watchdog_timer_fn+0x3a4/0x3e0
[c000000007d5b380] [c0000000001cc65c] .__hrtimer_run_queues+0x1fc/0x650
[c000000007d5b490] [c0000000001cd5cc] .hrtimer_interrupt+0x11c/0x320
[c000000007d5b550] [c000000000021f5c] .timer_interrupt+0x1cc/0x600
[c000000007d5b630] [c0000000000316c8] .replay_soft_interrupts+0xf8/0x300
[c000000007d5b870] [c000000000031964] .arch_local_irq_restore+0x94/0x1c0
[c000000007d5b8f0] [c000000000f8bac4] .__schedule+0x664/0xa50
[c000000007d5b9d0] [c000000000f8bf30] .schedule+0x80/0x140
[c000000007d5ba50] [c00000000093085c] .try_to_generate_entropy+0x118/0x174
[c000000007d5bb40] [c00000000092fa64] .urandom_read_iter+0x74/0x140
[c000000007d5bbc0] [c0000000003b0044] .vfs_read+0x284/0x2d0
[c000000007d5bcd0] [c0000000003b0d2c] .ksys_read+0xdc/0x130
[c000000007d5bd80] [c00000000002a88c] .system_call_exception+0x19c/0x330
[c000000007d5be10] [c00000000000c1d4] system_call_common+0xf4/0x258
--- interrupt: c00 at 0x7fffb5c9d49c
NIP:  00007fffb5c9d49c LR: 000000001000da90 CTR: 0000000000000000
REGS: c000000007d5be80 TRAP: 0c00   Tainted: G             L      (6.0.0-11414-g49da07006239)
MSR:  800000000000f032 <SF,EE,PR,FP,ME,IR,DR,RI>  CR: 22002422  XER: 00000000
IRQMASK: 0
GPR00: 0000000000000003 00007ffff6dcc220 00007fffb5d97300 0000000000000000
GPR04: 00000000101102a0 0000000000000020 0000000000000000 0000000000000000
GPR08: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
GPR12: 0000000000000000 00007fffb5e6aac0 00000000100d32e0 00000000100d32b4
GPR16: 00000000100d3301 00000000100d32b9 00000000100d3358 00000000100d32bf
GPR20: 0000000000002000 00000000100d3372 00000000100d331e 0000000000000000
GPR24: 7fffffffffffffff 00000000100b3a9c 00000000101102a0 0000000000000020
GPR28: 00000000101025c0 0000000000000020 0000000000000000 0000000000000000
NIP [00007fffb5c9d49c] 0x7fffb5c9d49c
LR [000000001000da90] 0x1000da90
--- interrupt: c00