Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   "Chang, Junxiao" <junxiao.chang@intel.com>
To:     "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
CC:     "linux-rt-users@vger.kernel.org" <linux-rt-users@vger.kernel.org>,
        "bigeasy@linutronix.de" <bigeasy@linutronix.de>,
        "tglx@linutronix.de" <tglx@linutronix.de>,
        "rostedt@goodmis.org" <rostedt@goodmis.org>,
        "Peh, Hock Zhang" <hock.zhang.peh@intel.com>
Subject: RE: [PATCH] softirq: wake up ktimer thread in softirq context
Thread-Topic: [PATCH] softirq: wake up ktimer thread in softirq context
Thread-Index: AQHZCtu9FgSt2BLnhEqT/Df9j5aCWK52jjbA
Date:   Tue, 20 Dec 2022 10:44:07 +0000
Message-ID: <BN9PR11MB5370BA8A506EB8519DC879C1ECEA9@BN9PR11MB5370.namprd11.prod.outlook.com>
References: <20221208075604.811710-1-junxiao.chang@intel.com>
In-Reply-To: <20221208075604.811710-1-junxiao.chang@intel.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?PN/GBqKUgUqmCdqOd8jG9dHByHXF0NuIPWUjG2bpcI/jY6aTxporCJF5zyzv?=
 =?us-ascii?Q?bt3FbaHQyquFTVOq+0aBFl7Eq8bAFUACYqCzYzbuO+WzpTciBkBmvmOGz1vs?=
 =?us-ascii?Q?VJey24FFrRQGl4bVoq3HHuJJbnxWMIy8CpY0ji7I36ubOzdE8cBZinKS27Iv?=
 =?us-ascii?Q?3yV+y8pFP2rVv+SoOgxOD0ryi2gQkplnrmHDTv+6i6EdlZDdA+NBCdhIxw/2?=
 =?us-ascii?Q?SY/iN/vE9Y+KZ4J3uo5I3u7g2R3e62on3hDBSNoLcEWLKU6v2DL0b+JqzeB7?=
 =?us-ascii?Q?ENK4r6Qe3VdK4pshGVZGWX6WMfobJV8+rySiOwPDz4gvEvjby/oV/sNq1NM+?=
 =?us-ascii?Q?TpCpXvs48Cdyuc6Ud+ceApjUm707Vpb6fP0/kMny9jAHCkzBFyGNmCklEt53?=
 =?us-ascii?Q?Jqu1AIUK1F9GX9BnvEFeOhvM3mjyZdWmrrpEnhdmn1bVAqUEZ8mK2bjO+C1w?=
 =?us-ascii?Q?l2HUHlH3MDaDUISyljjThD0JfLZT50A6GYe/Ua0zFaHf6xvLn8bFwiAo+BKF?=
 =?us-ascii?Q?kTACWwOlsgmf2emWyA2npp8uNTOkUITbVrQbb4QwUBhxO/byAaVp8tZoj9/W?=
 =?us-ascii?Q?lU0+KTdx/PdVQAHZDrQ0dCct+HsG8SXUDGZgOlmnOSst7uxwMIxRQXU+Lq7I?=
 =?us-ascii?Q?/VFesbKeqaZuWAHOnace2sPfSsq4nMAV441xJ/zxaWGUxqmLcwwRJaKND4bI?=
 =?us-ascii?Q?huPRWIkGny7KH22GsKdvpt7XDNqPlK2O/hvx8JC+qrUnUsRBabnqAIFiZ6/P?=
 =?us-ascii?Q?aIarhvG23ajZhbPVB3PO5VCSzs9m8UPaELV9ZAq09fhlLDENTvE747EF65JD?=
 =?us-ascii?Q?9Bo0LZqJ5o7AkJ2i0VArwSB4c7X4TXMs+nY/fylIRs5cIwgsr2PjgHZaF5E2?=
 =?us-ascii?Q?jhQOg8QxEJnIExVTyx8chaN3Ke/K43K2hm6GmPfaaXKVbDmQxDBipgR0MWk0?=
 =?us-ascii?Q?/zZEsN+ITGQFi1SUwulUYDsjZK2IKgxqXybS9ZnnfoRWkmodRT7fVSIsDfXW?=
 =?us-ascii?Q?QVqzwnYC3QoKYnFbI2ejyVKKmiFmGhASEntM24sKGnJ6pEjn3y7LseXOjjev?=
 =?us-ascii?Q?UDD1L34kGVOt5cvUUafRrBiQ99w97VdLe5z+CT4mxUEet+jf5alJ/C94hOce?=
 =?us-ascii?Q?JhtuojwB5u05hlUNfmrRubPWQAaE2aPP4pTyIV7pq1lq4B2cunIJ8EqPVv9+?=
 =?us-ascii?Q?AiC6hLfUsFOQBR+WzWAEnVK7z0uHmEA3gr1PoQmzvPgSbm34/qIUXXSmCz6t?=
 =?us-ascii?Q?889YaUkKHOu24/agrtucBulHGGwWdb2dfTyyN/qoWGIMxjgPI6P6JKqY1K6U?=
 =?us-ascii?Q?eM58AJE2VMQJj4DfMjk1OHuc1iYwRaVGnuYN8LqkLWQjealRC9IzrtNU4S8u?=
 =?us-ascii?Q?++gNe1v7DomoNmqncRBuO5PSOlWDmMg3iSOLlPlWTdQ/33GsIF2MMfVd1kKw?=
 =?us-ascii?Q?LGK5xK/UqjZr6LNMfyHNCUUgrtgQdD2urgPI0ovZxsl2jIlMmgY8QuevYjsP?=
 =?us-ascii?Q?yTRIcr2I0BxdcymM1z76+DVKxq6nTiCFSGF2waflkqWYhjAF25kWgSAwUC0P?=
 =?us-ascii?Q?0ub+XQFu/oEE120PBDwt7QAPf/Sfx4zAJcMv9J3I?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN9PR11MB5370.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 721eb6fd-57e8-46fc-0602-08dae2771eca
X-MS-Exchange-CrossTenant-originalarrivaltime: 20 Dec 2022 10:44:07.1575
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: N+Rr8xoJB8rx21bN/LhYzGZ3dkKdJNAzKxUOgxaqq2YCoUVkL/EYs0c0ElCI7IrwfNbZvB2qw60rKNotr1VaHA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR11MB6399
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-7.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI,
        SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Any comment? This patch is for 6.1-rt, issue could be reproduced with 5.19-=
rt kernel as well.

This issue is easier to reproduced when there is heavy network workload whi=
ch introduces a lot of softirq events. If hrtimer interrupt is triggered in=
 softirq context, with current RT kernel, it will not wake up ktimers threa=
d which handles hrtimer event because in function __irq_exit_rcu, "in_inter=
rupt()" is true:

static inline void __irq_exit_rcu(void)
{
...
        preempt_count_sub(HARDIRQ_OFFSET);
        if (!in_interrupt()) {
                if (local_softirq_pending())
                        invoke_softirq();

                if (IS_ENABLED(CONFIG_PREEMPT_RT) && local_pending_timers()=
)
                        wake_timersd();
        }
...
}

Then ktimers threads stays in sleep state, hrtimer function will not be cal=
led although hrtimer interrupt has been triggered. Ktimers thread might be =
woken up in next timer interrupt which introduces long delay.

Any comments are welcome.

Regards,
Junxiao

-----Original Message-----
From: Chang, Junxiao <junxiao.chang@intel.com>=20
Sent: Thursday, December 8, 2022 3:56 PM
To: linux-kernel@vger.kernel.org
Cc: linux-rt-users@vger.kernel.org; bigeasy@linutronix.de; tglx@linutronix.=
de; rostedt@goodmis.org; Chang, Junxiao <junxiao.chang@intel.com>; Peh, Hoc=
k Zhang <hock.zhang.peh@intel.com>
Subject: [PATCH] softirq: wake up ktimer thread in softirq context

Occiaionally timer interrupt might be triggered in softirq context, ktimer =
thread should be woken up with RT kernel, or else ktimer thread might stay =
in sleep state although timer interrupt has been triggered.

This change fixes a latency issue that timer handler is delayed for more th=
an 4ms in network related test.

Fixes: 2165d27554e8 ("softirq: Use a dedicated thread for timer wakeups.")
Reported-by: Peh, Hock Zhang <hock.zhang.peh@intel.com>
Signed-off-by: Junxiao Chang <junxiao.chang@intel.com>
---
 kernel/softirq.c | 11 ++++-------
 1 file changed, 4 insertions(+), 7 deletions(-)

diff --git a/kernel/softirq.c b/kernel/softirq.c index ab1fe34326bab..34ae3=
9e4a3d10 100644
--- a/kernel/softirq.c
+++ b/kernel/softirq.c
@@ -664,13 +664,10 @@ static inline void __irq_exit_rcu(void)  #endif
 	account_hardirq_exit(current);
 	preempt_count_sub(HARDIRQ_OFFSET);
-	if (!in_interrupt()) {
-		if (local_softirq_pending())
-			invoke_softirq();
-
-		if (IS_ENABLED(CONFIG_PREEMPT_RT) && local_pending_timers())
-			wake_timersd();
-	}
+	if (!in_interrupt() && local_softirq_pending())
+		invoke_softirq();
+	if (!(in_nmi() || in_hardirq()) && local_pending_timers())
+		wake_timersd();
=20
 	tick_irq_exit();
 }
--
2.25.1