Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
MIME-Version: 1.0
References: <20230104181701.43224-1-wander@redhat.com> <xhsmh4jsmieqx.mognet@vschneid.remote.csb>
In-Reply-To: <xhsmh4jsmieqx.mognet@vschneid.remote.csb>
From:   Wander Lairson Costa <wander@redhat.com>
Date:   Thu, 19 Jan 2023 16:53:44 -0300
Message-ID: <CAAq0SUmBCdMVgEqb0U8nAc8cNCc9YKwLp0YtCBtJncQOPi-5qg@mail.gmail.com>
Subject: Re: [PATCH] sched/deadline: fix inactive_task_timer splat with CONFIG_PREEMPT_RT
To:     Valentin Schneider <vschneid@redhat.com>
Cc:     Ingo Molnar <mingo@redhat.com>,
        Peter Zijlstra <peterz@infradead.org>,
        Juri Lelli <juri.lelli@redhat.com>,
        Vincent Guittot <vincent.guittot@linaro.org>,
        Dietmar Eggemann <dietmar.eggemann@arm.com>,
        Steven Rostedt <rostedt@goodmis.org>,
        Ben Segall <bsegall@google.com>, Mel Gorman <mgorman@suse.de>,
        Daniel Bristot de Oliveira <bristot@redhat.com>,
        "open list:SCHEDULER" <linux-kernel@vger.kernel.org>,
        Paul McKenney <paulmck@kernel.org>,
        Thomas Gleixner <tglx@linutronix.de>
Content-Type: text/plain; charset="UTF-8"
Precedence: bulk

On Thu, Jan 19, 2023 at 3:03 PM Valentin Schneider <vschneid@redhat.com> wrote:
>
> On 04/01/23 15:17, Wander Lairson Costa wrote:
> > inactive_task_timer() executes in interrupt (atomic) context. It calls
> > put_task_struct(), which indirectly acquires sleeping locks under
> > PREEMPT_RT.
> >
> > Below is an example of a splat that happened in a test environment:
> >
> >  CPU: 1 PID: 2848 Comm: life Kdump: loaded Tainted: G W ---------
> >  Hardware name: HP ProLiant DL388p Gen8, BIOS P70 07/15/2012
> >  Call Trace:
> >  dump_stack_lvl+0x57/0x7d
> >  mark_lock_irq.cold+0x33/0xba
> >  ? stack_trace_save+0x4b/0x70
> >  ? save_trace+0x55/0x150
> >  mark_lock+0x1e7/0x400
> >  mark_usage+0x11d/0x140
> >  __lock_acquire+0x30d/0x930
> >  lock_acquire.part.0+0x9c/0x210
> >  ? refill_obj_stock+0x3d/0x3a0
> >  ? rcu_read_lock_sched_held+0x3f/0x70
> >  ? trace_lock_acquire+0x38/0x140
> >  ? lock_acquire+0x30/0x80
> >  ? refill_obj_stock+0x3d/0x3a0
> >  rt_spin_lock+0x27/0xe0
> >  ? refill_obj_stock+0x3d/0x3a0
> >  refill_obj_stock+0x3d/0x3a0
> >  ? inactive_task_timer+0x1ad/0x340
> >  kmem_cache_free+0x357/0x560
> >  inactive_task_timer+0x1ad/0x340
> >  ? switched_from_dl+0x2d0/0x2d0
> >  __run_hrtimer+0x8a/0x1a0
> >  __hrtimer_run_queues+0x91/0x130
> >  hrtimer_interrupt+0x10f/0x220
> >  __sysvec_apic_timer_interrupt+0x7b/0xd0
> >  sysvec_apic_timer_interrupt+0x4f/0xd0
> >  ? asm_sysvec_apic_timer_interrupt+0xa/0x20
> >  asm_sysvec_apic_timer_interrupt+0x12/0x20
> >  RIP: 0033:0x7fff196bf6f5
> >
> > Instead of calling put_task_struct() directly, we defer it using
> > call_rcu(). A more natural approach would use a workqueue, but since
> > in PREEMPT_RT, we can't allocate dynamic memory from atomic context,
> > the code would become more complex because we would need to put the
> > work_struct instance in the task_struct and initialize it when we
> > allocate a new task_struct.
> >
>
> Sorry to come back on this; Juri reminded me offline that put_task_struct()
> is invoked in other non-sleepable contexts, not just inactive_task_timer().
>

I guess there is no splat because the usage count doesn't reach zero
in those code paths.

> e.g.
>
>   rto_push_irq_work_func() // hard irq work so hardirq context
>   `\
>     push_rt_task()
>     `\
>        put_task_struct()
>

This is paired with a get_task_struct() a few lines above inside in
the same function.

> Or
>
>   cpu_stopper_thread() // stopper callbacks must not sleep
>   `\
>     push_cpu_stop()
>     `\
>       put_task_struct()
>

This is paired with a get_task_struct() from get_push_task()

> ... But then again I'm not aware of any splats happening in these paths. Is
> there something special about inactive_task_timer(), or could it be the
> issue is there for those other paths but we just haven't had them reported
> yet?
>

Given that those calls have corresponding get_task_struct() calls that
are close in time, there is a low probability of the usage count
reaching zero and triggering the splat. In any case, I will work in a
v2 that also addresses those call sites.