Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   Libo Chen <libo.chen@oracle.com>
To:     peterz@infradead.org, vincent.guittot@linaro.org, mgorman@suse.de,
        tim.c.chen@linux.intel.com, 21cnbao@gmail.com,
        dietmar.eggemann@arm.com
Cc:     linux-kernel@vger.kernel.org, tglx@linutronix.de
Subject: [PATCH] sched/fair: no sync wakeup from interrupt context
Date:   Mon, 11 Jul 2022 15:47:04 -0700
Message-Id: <20220711224704.1672831-1-libo.chen@oracle.com>
Content-Transfer-Encoding: 8bit
Content-Type: text/plain
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?+GaIEyeAHsd7+Gp9SXZ8e/ygVUQFxFI0l28gaQ9ZsKGKBDoi/jqezJ86yL4i?=
 =?us-ascii?Q?QJYXrk34TIMNU9tmRwfMnXmk+uODrE6FdMOu3ZE2uMUoJd0qFHXTLE7k3+XZ?=
 =?us-ascii?Q?9qnTjfYgNPSBdCsUiTnKYHXEogehq0j+QHpzJjKnfD/vL+8FKEGD2JzuZ4b4?=
 =?us-ascii?Q?wrCk3tuvP/F/NWZ9NZGR4/8Mpy77blW5whM3lItCHDdpug/B4MaAFb5OQ06E?=
 =?us-ascii?Q?Onhk2Yk4FHzAgKWkls82NgcACLwupqgnrQ2j3fjfb5KJePUTaNGhrP92T14A?=
 =?us-ascii?Q?yozjdqfx9RaPdp10nktdoDe/Or12IdeS/mDlKP9++BYo2HDC0dOCliLtw8E7?=
 =?us-ascii?Q?USwMce0ZiFLoqDYVcyiaPLDYMGsCG8HYN1B+1ibSNbFoBZAAY8LBZcNu6Ovq?=
 =?us-ascii?Q?bXXcmy19QCdH7oYx5YaNKQkksTQrT+n+9VRa1Zt8OAUmQJ27Q9tM2KSPAfQ3?=
 =?us-ascii?Q?EDvnIjKt0kj+6AI9Ybdp7WXVJ+VgN3W7O5eftxKBV9YHMTSaYEV9NiMhJpml?=
 =?us-ascii?Q?hATE8a6F9smSaQiNTn3YiQF3nY7++8TfeWFNvIDHW7m1vyxC1u7tybTthJRq?=
 =?us-ascii?Q?rZJKxzz4v9R0Tf4+nf8o7hK8OYKp5wH/xiNsLKqM16mx2yXP0Ta5N8rZ6JPh?=
 =?us-ascii?Q?t4eT4qRg4v33dw3UANsyJn17KkfKL62Afy8lbxbz0LkPFI1yumlxM8Hr4jF9?=
 =?us-ascii?Q?qyN/qr/rYRR2CDl32AaD9vs/uA6kqdKFj0m2KS7XTXwGUHS3RyTzFg0RpTQv?=
 =?us-ascii?Q?uSniM5NdDZSw+dPt0/BMBrNLTJHNXEgHPN9yszAgXdQ8jQVKO45Nsz4RZ9Bg?=
 =?us-ascii?Q?mpQhPQPAoFSSMAJaCC7kbWCpubwLdHxNfUaV7p08zfMsdEbEXT8N+bLAvHvy?=
 =?us-ascii?Q?br2sAiN0uT9PAl7I4Kad4fusuwAoThZtHWShm/0Th5Zs+1MbEAUy4LTpxkwu?=
 =?us-ascii?Q?Qnzil0vMsSRu58z/inl9E7e6/+1UeJUs3ErEKrEo/7m97VyHtf9Ek4Sjv0tY?=
 =?us-ascii?Q?8QWJwXDelGEbyS4zQaT5QCROGLz7L7Znk/Mk8SXDwBCM5YcaPOIXtQRL0D1f?=
 =?us-ascii?Q?CEaJ6H7s9xIHEgQ7dUhxuu2JFxLY2ttdMI6gQ2roeIRG19zP4vLuzzwldArI?=
 =?us-ascii?Q?bOli9TNgh16a4hRx/8fPkDxewYGzXu/qVEQcKQP54BqQDqqcyEwzBB6WP4Wu?=
 =?us-ascii?Q?efjXQkFTC8zKaMU9WWtnHoetPWXzd3HuOvU3ys5mG8nO6fUN5hwHIOKY3SnE?=
 =?us-ascii?Q?PHsdrIzC9tty5CHfG5m+FT/rhplu0KRmlzk23GAcv9op8wx+/UhwfhXKNTsC?=
 =?us-ascii?Q?cKXxadfGKYLfxYkgXggUPlbUqk/Vvd6JJXZXpsssWgW9mW3RxHfRYGP1xDhk?=
 =?us-ascii?Q?802iIHBVoGDI27v7CJsGf3o2+QgCiQidJLLgsT+ctoPjyn+ZcEOqyXCjsRNu?=
 =?us-ascii?Q?26LqtpC4sr2jwZZsrSmhwhx//aVBB3lg0V67DCiCgQxOSHGDIjhZJVCv5+2y?=
 =?us-ascii?Q?cCrEqpkYXR13RG1SEYL4MUkVcCACsHISfka12nng6tWMG+TrvbOm6OSRgJoX?=
 =?us-ascii?Q?HUo2Fa5q3FAGika/k0rkraSFjABveSITeV5hgiiV?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: f69fad82-6d86-4077-690b-08da638f4ade
X-MS-Exchange-CrossTenant-AuthSource: SJ0PR10MB4576.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 11 Jul 2022 22:47:11.5316
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 5kKCy+Lh8uY+m0IUXLr2kModOH+9mJ1NcbIrcofQLk3dywCjo2oYyezg1e5ZKykeKUN6k+fLSyfKHda05eUcFA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL0PR10MB2964
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.517,18.0.883
 definitions=2022-07-11_25:2022-07-08,2022-07-11 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 mlxlogscore=999
 malwarescore=0 suspectscore=0 mlxscore=0 spamscore=0 adultscore=0
 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2206140000 definitions=main-2207110092
X-Proofpoint-GUID: Au8asF5-xmBfJ6vv_h6duXBQrI5FTdsC
X-Proofpoint-ORIG-GUID: Au8asF5-xmBfJ6vv_h6duXBQrI5FTdsC
X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_LOW,
        RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE
        autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Barry Song first pointed out that replacing sync wakeup with regular wakeup
seems to reduce overeager wakeup pulling and shows noticeable performance
improvement.[1]

This patch argues that allowing sync for wakeups from interrupt context
is a bug and fixing it can improve performance even when irq/softirq is
evenly spread out.

For wakeups from ISR, the waking CPU is just the CPU of ISR and the so-called
waker can be any random task that happens to be running on that CPU when the
interrupt comes in. This is completely different from other wakups where the
task running on the waking CPU is the actual waker. For example, two tasks
communicate through a pipe or mutiple tasks access the same critical section,
etc. This difference is important because with sync we assume the waker will
get off the runqueue and go to sleep immedately after the wakeup. The
assumption is built into wake_affine() where it discounts the waker's presence
from the runqueue when sync is true. The random waker from interrupts bears no
relation to the wakee and don't usually go to sleep immediately afterwards
unless wakeup granularity is reached. Plus the scheduler no longer enforces the
preepmtion of waker for sync wakeup as it used to before
patch f2e74eeac03ffb7 ("sched: Remove WAKEUP_SYNC feature"). Enforcing sync
wakeup preemption for wakeups from interrupt contexts doesn't seem to be
appropriate too but at least sync wakeup will do what it's supposed to do.

Add a check to make sure that sync can only be set when in_task() is true. If
a wakeup is from interrupt context, sync flag will be 0 because in_task()
returns 0.

Tested in two scenarios: wakeups from 1) task contexts, expected to see no
performance changes; 2) interrupt contexts, expected to see better performance
under low/medium load and no regression under heavy load.

Use hackbench for scenario 1 and pgbench for scenarios 2 both from mmtests on
a 2-socket Xeon E5-2699v3 box with 256G memory in total. Running 5.18 kernel
with SELinux disabled. Scheduler/MM tunables are all default. Irqbalance
daemon is active.

Hackbench (config-scheduler-unbound)
=========
process-pipes:
                Baseline                Patched
Amean   1       0.4300  ( 0.00%)        0.4420  ( -2.79%)
Amean   4       0.8757  ( 0.00%)        0.8774  ( -0.20%)
Amean   7       1.3712  ( 0.00%)        1.3789  ( -0.56%)
Amean   12      2.3541  ( 0.00%)        2.3714  ( -0.73%)
Amean   21      4.2229  ( 0.00%)        4.2439  ( -0.50%)
Amean   30      5.9113  ( 0.00%)        5.9451  ( -0.57%)
Amean   48      9.3873  ( 0.00%)        9.4898  ( -1.09%)
Amean   79      15.9281 ( 0.00%)        16.1385 ( -1.32%)
Amean   110     22.0961 ( 0.00%)        22.3433 ( -1.12%)
Amean   141     28.2973 ( 0.00%)        28.6209 ( -1.14%)
Amean   172     34.4709 ( 0.00%)        34.9347 ( -1.35%)
Amean   203     40.7621 ( 0.00%)        41.2497 ( -1.20%)
Amean   234     47.0416 ( 0.00%)        47.6470 ( -1.29%)
Amean   265     53.3048 ( 0.00%)        54.1625 ( -1.61%)
Amean   288     58.0595 ( 0.00%)        58.8096 ( -1.29%)

process-sockets:
                Baseline                Patched
Amean   1       0.6776   ( 0.00%)       0.6611   ( 2.43%)
Amean   4       2.6183   ( 0.00%)       2.5769   ( 1.58%)
Amean   7       4.5662   ( 0.00%)       4.4801   ( 1.89%)
Amean   12      7.7638   ( 0.00%)       7.6201   ( 1.85%)
Amean   21      13.5335  ( 0.00%)       13.2915  ( 1.79%)
Amean   30      19.3369  ( 0.00%)       18.9811  ( 1.84%)
Amean   48      31.0724  ( 0.00%)       30.6015  ( 1.52%)
Amean   79      51.1881  ( 0.00%)       50.4251  ( 1.49%)
Amean   110     71.3399  ( 0.00%)       70.4578  ( 1.24%)
Amean   141     91.4675  ( 0.00%)       90.3769  ( 1.19%)
Amean   172     111.7463 ( 0.00%)       110.3947 ( 1.21%)
Amean   203     131.6927 ( 0.00%)       130.3270 ( 1.04%)
Amean   234     151.7459 ( 0.00%)       150.1320 ( 1.06%)
Amean   265     171.9101 ( 0.00%)       169.9751 ( 1.13%)
Amean   288     186.9231 ( 0.00%)       184.7706 ( 1.15%)

thread-pipes:
                Baseline                Patched
Amean   1       0.4523  ( 0.00%)        0.4535  ( -0.28%)
Amean   4       0.9041  ( 0.00%)        0.9085  ( -0.48%)
Amean   7       1.4111  ( 0.00%)        1.4146  ( -0.25%)
Amean   12      2.3532  ( 0.00%)        2.3688  ( -0.66%)
Amean   21      4.1550  ( 0.00%)        4.1701  ( -0.36%)
Amean   30      6.1043  ( 0.00%)        6.2391  ( -2.21%)
Amean   48      10.2077 ( 0.00%)        10.3511 ( -1.40%)
Amean   79      16.7922 ( 0.00%)        17.0427 ( -1.49%)
Amean   110     23.3350 ( 0.00%)        23.6522 ( -1.36%)
Amean   141     29.6458 ( 0.00%)        30.2617 ( -2.08%)
Amean   172     35.8649 ( 0.00%)        36.4225 ( -1.55%)
Amean   203     42.4477 ( 0.00%)        42.8332 ( -0.91%)
Amean   234     48.7117 ( 0.00%)        49.4042 ( -1.42%)
Amean   265     54.9171 ( 0.00%)        55.6551 ( -1.34%)
Amean   288     59.5282 ( 0.00%)        60.2903 ( -1.28%)

thread-sockets:
                Baseline                Patched
Amean   1       0.6917   ( 0.00%)       0.6892   ( 0.37%)
Amean   4       2.6651   ( 0.00%)       2.6017   ( 2.38%)
Amean   7       4.6734   ( 0.00%)       4.5637   ( 2.35%)
Amean   12      8.0156   ( 0.00%)       7.8079   ( 2.59%)
Amean   21      14.0451  ( 0.00%)       13.6679  ( 2.69%)
Amean   30      20.0963  ( 0.00%)       19.5657  ( 2.64%)
Amean   48      32.4115  ( 0.00%)       31.6001  ( 2.50%)
Amean   79      53.1104  ( 0.00%)       51.8395  ( 2.39%)
Amean   110     74.0929  ( 0.00%)       72.4391  ( 2.23%)
Amean   141     95.1506  ( 0.00%)       93.0992  ( 2.16%)
Amean   172     116.1969 ( 0.00%)       113.8307 ( 2.04%)
Amean   203     137.4413 ( 0.00%)       134.5247 ( 2.12%)
Amean   234     158.5395 ( 0.00%)       155.2793 ( 2.06%)
Amean   265     179.7729 ( 0.00%)       176.1099 ( 2.04%)
Amean   288     195.5573 ( 0.00%)       191.3977 ( 2.13%)

Pgbench (config-db-pgbench-timed-ro-small)
=======
                Baseline            Patched
Hmean   1       68.54    ( 0.00%)   69.72    ( 1.71%)
Hmean   6       27725.78 ( 0.00%)   34119.11 ( 23.06%)
Hmean   12      55724.26 ( 0.00%)   63158.22 ( 13.34%)
Hmean   22      72806.26 ( 0.00%)   73389.98 ( 0.80%)
Hmean   30      79000.45 ( 0.00%)   75197.02 ( -4.81%)
Hmean   48      78180.16 ( 0.00%)   75074.09 ( -3.97%)
Hmean   80      75001.93 ( 0.00%)   70590.72 ( -5.88%)
Hmean   110     74812.25 ( 0.00%)   74128.57 ( -0.91%)
Hmean   142     74261.05 ( 0.00%)   72910.48 ( -1.82%)
Hmean   144     75375.35 ( 0.00%)   71295.72 ( -5.41%)

For hackbench, +-3% fluctuation is normal on this two-socket box, it's safe to
conclude that there are no performance differences for wakeups from task context.
For pgbench, after many runs, 10~30% gains are very consistent at lower
client counts (< #cores per socket). For higher client counts, both kernels are
very close, +-5% swings are quite common. Also NET_TX|RX softirq load
does spread out across both NUMA nodes in this test so there is no need to do
any explicit RPS/RFS.

[1]: https://lkml.org/lkml/2021/11/5/234

Signed-off-by: Libo Chen <libo.chen@oracle.com>
---
 kernel/sched/fair.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 794c2cb945f8..59b210d2cdb5 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -6704,7 +6704,9 @@ static int find_energy_efficient_cpu(struct task_struct *p, int prev_cpu)
 static int
 select_task_rq_fair(struct task_struct *p, int prev_cpu, int wake_flags)
 {
-	int sync = (wake_flags & WF_SYNC) && !(current->flags & PF_EXITING);
+	/* Don't set sync for wakeup from irq/soft ctx */
+	int sync = in_task() && (wake_flags & WF_SYNC)
+		   && !(current->flags & PF_EXITING);
 	struct sched_domain *tmp, *sd = NULL;
 	int cpu = smp_processor_id();
 	int new_cpu = prev_cpu;
--
2.31.1