Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Thu, 7 Jul 2022 12:19:56 -0400
From:   Steven Rostedt <rostedt@goodmis.org>
To:     John Keeping <john@metanate.com>
Cc:     linux-kernel@vger.kernel.org, linux-rt-users@vger.kernel.org,
        Sebastian Andrzej Siewior <bigeasy@linutronix.de>,
        Peter Zijlstra <peterz@infradead.org>,
        Thomas Gleixner <tglx@linutronix.de>,
        Ingo Molnar <mingo@redhat.com>,
        Juri Lelli <juri.lelli@redhat.com>,
        Vincent Guittot <vincent.guittot@linaro.org>,
        Dietmar Eggemann <dietmar.eggemann@arm.com>,
        Ben Segall <bsegall@google.com>, Mel Gorman <mgorman@suse.de>,
        Daniel Bristot de Oliveira <bristot@redhat.com>,
        Valentin Schneider <vschneid@redhat.com>
Subject: Re: [PATCH] sched/core: Always flush pending blk_plug
Message-ID: <20220707121956.0ab19fd8@gandalf.local.home>
In-Reply-To: <20220707143902.529938-1-john@metanate.com>
References: <20220707143902.529938-1-john@metanate.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
Precedence: bulk

On Thu,  7 Jul 2022 15:39:02 +0100
John Keeping <john@metanate.com> wrote:

> Here the kworker is waiting on msdos_sb_info::s_lock which is held by
> tar which is in turn waiting for a buffer which is locked waiting to be
> flushed, but this operation is plugged in the kworker.
> 
> The lock is a normal struct mutex, so tsk_is_pi_blocked() will always
> return false on !RT and thus the behaviour changes for RT.
> 
> It seems that the intent here is to skip blk_flush_plug() in the case
> where a non-preemptible lock (such as a spinlock) has been converted to
> a rtmutex on RT, which is the case covered by the SM_RTLOCK_WAIT
> schedule flag.  But sched_submit_work() is only called from schedule()
> which is never called in this scenario, so the check can simply be
> deleted.
> 
> Looking at the history of the -rt patchset, in fact this change was
> present from v5.9.1-rt20 until being dropped in v5.13-rt1 as it was part
> of a larger patch [1] most of which was replaced by commit b4bfa3fcfe3b
> ("sched/core: Rework the __schedule() preempt argument").
> 

Nice investigation.

So basically what you are saying is that commit b4bfa3fcfe3b was the
implementation of [1], but left out the removal of the tsk_is_pi_blocked(),
and that what you are seeing is the problem that is described in [1].

Can you add this in the change log:

"As described in [1]:

   The schedule process must distinguish between blocking on a regular
   sleeping lock (rwsem and mutex) and a RT-only sleeping lock (spinlock
   and rwlock):
   - rwsem and mutex must flush block requests (blk_schedule_flush_plug())
     even if blocked on a lock. This can not deadlock because this also
     happens for non-RT.
     There should be a warning if the scheduling point is within a RCU read
     section.

   - spinlock and rwlock must not flush block requests. This will deadlock
     if the callback attempts to acquire a lock which is already acquired.
     Similarly to being preempted, there should be no warning if the
     scheduling point is within a RCU read section.

and with the tsk_is_pi_blocked() in the scheduler path, we hit the first
issue."

Reviewed-by: Steven Rostedt (Google) <rostedt@goodmis.org>

-- Steve