Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   Sebastian Andrzej Siewior <bigeasy@linutronix.de>
To:     linux-kernel@vger.kernel.org
Cc:     Ben Segall <bsegall@google.com>,
        Daniel Bristot de Oliveira <bristot@redhat.com>,
        Dietmar Eggemann <dietmar.eggemann@arm.com>,
        Frederic Weisbecker <frederic@kernel.org>,
        Ingo Molnar <mingo@redhat.com>,
        John Stultz <jstultz@google.com>,
        Juri Lelli <juri.lelli@redhat.com>,
        Mel Gorman <mgorman@suse.de>,
        Peter Zijlstra <peterz@infradead.org>,
        Stephen Boyd <sboyd@kernel.org>,
        Steven Rostedt <rostedt@goodmis.org>,
        Thomas Gleixner <tglx@linutronix.de>,
        Valentin Schneider <vschneid@redhat.com>,
        Vincent Guittot <vincent.guittot@linaro.org>
Subject: [RFC PATCH 0/3] Allow to preempt a timer softirq on PREEMPT_RT.
Date:   Fri,  4 Aug 2023 13:30:36 +0200
Message-Id: <20230804113039.419794-1-bigeasy@linutronix.de>
MIME-Version: 1.0
Content-Transfer-Encoding: quoted-printable
Precedence: bulk

Hi,

while the softirqs are served, bottom halves are disabled. By disabling
bottom halves (as per local_bh_disable()) PREEMPT_RT acquires a
local_lock_t. This lock ensures that the softirq is synchronized against
other softirq user on that CPU while keeping the context preemptible.

This leads to a scenario where context itself is preemptible but needs
to "complete" before the system can make progress. For instance, the
timer callback (in TIMER_SOFTIRQ) gets preempted because a
force-threaded interrupt thread, with higher priority, gets woken up.
Before the handler of the forced-threaded interrupt can be invoked,
bottom halves get disabled and this blocks on the same per-CPU lock.
This in turn leads to a PI-boost and the preempted timer softirq is back
on the CPU with higher priority completing its job (not just the timer,
all pending softirqs).

In the end the force threaded interrupt is blocked until all pending
softirqs have been served.

The PI-boost is usually intended to allow the thread with lower priority
to "quickly" finish what it was doing and leave the critical section
ASAP. This is not the case with softirqs and how this is handled by the
individual callbacks. Additionally the need_resched() check in
__do_softirq() is never true due to the boost. This means in worst case
this can run for MAX_SOFTIRQ_TIME or MAX_SOFTIRQ_RESTART.

One way of out would be to add preemption within the softirq handling at
which point the softirq-BKL can be dropped. This can be after all
softirqs have been served (__do_softirq() where the need_resched() check
is located), after each softirq handler or within the softirq handler
where it is considered safe to do so.

This series adds as an example such a preemption point to the timer
softirq handler. Should this fly then it would be needed the remaining
handlers as well.

Sebastian