Hi,
it's a late reply but I didn't find enough determination earlier.
On 8.09.2023 10:10, Linus Walleij wrote:
> On Mon, Sep 4, 2023 at 10:34 AM Rafał Miłecki <[email protected]> wrote:
>
>> I'm clueless at this point.
>> Maybe someone can come up with an idea of actual issue & ideally a
>> solution.
>
> Damn this is frustrating.
>
>> 2. Clock (arm,armv7-timer)
>>
>> While comparing main clock in Broadcom's SDK with upstream one I noticed
>> a tiny difference: mask value. I don't know it it makes any sense but
>> switching from CLOCKSOURCE_MASK(56) to CLOCKSOURCE_MASK(64) in
>> arm_arch_timer.c (to match SDK) increases average uptime (time before a
>> hang/lockup happens) from 4 minutes to 36 minutes.
>
> This could be related to how often the system goes to idle.
>
>> + if (cpu_idle_force_poll == 1234)
>> + arch_cpu_idle();
>> + if (cpu_idle_force_poll == 5678)
>> + arch_cpu_idle();
>> + if (cpu_idle_force_poll == 1234)
>> + arch_cpu_idle();
>> + if (cpu_idle_force_poll == 5678)
>> + arch_cpu_idle();
>> + if (cpu_idle_force_poll == 1234)
>> + arch_cpu_idle();
>> + if (cpu_idle_force_poll == 5678)
>> + arch_cpu_idle();
>> + if (cpu_idle_force_poll == 1234)
>> + arch_cpu_idle();
>
> Idle again.
>
> I would have tried to see what arch_cpu_idle() is doing.
>
> arm_pm_idle() or cpu_do_idle()?
In my case arm_pm_idle is NULL.
> What happens if you just put return in arch_cpu_idle()
> so it does nothing?
Doesn't help. I also tried putting:
udelay(10);
and
udelay(1000);
at the arch_cpu_idle() beginning. None helped.
Here comes more interesting experiment though. Putting there:
if (!(foo++ % 10000)) {
pr_info("[%s] arm_pm_idle:%ps\n", __func__, arm_pm_idle);
}
doesn't seem to help.
Putting following however seems to make kernel/device stable:
if (!(foo++ % 100)) {
pr_info("[%s] arm_pm_idle:%ps\n", __func__, arm_pm_idle);
}
I think I'm just going to assume those chipsets are simply hw broken.
On Wed, Nov 29, 2023 at 10:20 PM Rafał Miłecki <[email protected]> wrote:
> Here comes more interesting experiment though. Putting there:
>
> if (!(foo++ % 10000)) {
> pr_info("[%s] arm_pm_idle:%ps\n", __func__, arm_pm_idle);
> }
>
> doesn't seem to help.
>
>
> Putting following however seems to make kernel/device stable:
>
> if (!(foo++ % 100)) {
> pr_info("[%s] arm_pm_idle:%ps\n", __func__, arm_pm_idle);
> }
That's just too weird.
> I think I'm just going to assume those chipsets are simply hw broken.
If disabling CPU idle on these altogether stabilize them, then maybe that
is what we need to do?
Yours,
Linus Walleij
On 11/29/23 13:33, Linus Walleij wrote:
> On Wed, Nov 29, 2023 at 10:20 PM Rafał Miłecki <[email protected]> wrote:
>
>> Here comes more interesting experiment though. Putting there:
>>
>> if (!(foo++ % 10000)) {
>> pr_info("[%s] arm_pm_idle:%ps\n", __func__, arm_pm_idle);
>> }
>>
>> doesn't seem to help.
>>
>>
>> Putting following however seems to make kernel/device stable:
>>
>> if (!(foo++ % 100)) {
>> pr_info("[%s] arm_pm_idle:%ps\n", __func__, arm_pm_idle);
>> }
>
> That's just too weird.
It does seem to indicate that idling for too long wrecks havoc, but it
is indeed not making much sense. Not having proper documentation for
this SoC, it is hard to figure out what impact does stopping the ARM CPU
clock has on the rest of the memory subsystem, especially outside of the
CPU. I do not believe that this SoC has any form of PLL clock gating or
pulse skipping.
>
>> I think I'm just going to assume those chipsets are simply hw broken.
>
> If disabling CPU idle on these altogether stabilize them, then maybe that
> is what we need to do?
Yes, please try booting with "nohlt" set on the kernel command line and
see how that fares.
Also useful would be to dump the L2 CTLR and L2 ECTLR, this is a
complete shot in the dark, though was initially wondering if there could
be some retention issues, and would have recommended disabling the L2
retention policy completely just for testing.
MRC p15, 1, <Rt>, c9, c0, 2;
of particular interest here would be bit at position 0, try to see if
changing it to 1 (3 cycles) or 0 (2 cycles) changes anything.
MRC p15, 1, <Rt>, c9, c0, 3;
the lower bits are reserved, so I would not necessarily expect them to
be mapping to configurable latencies, but if you see non-zero values in
bits [28:0], try changing them to 0 and see if that changes anything.
Thanks for your persistence!
--
Florian