Received: by 2002:a05:7412:3b8b:b0:fc:a2b0:25d7 with SMTP id nd11csp1268478rdb; Fri, 9 Feb 2024 16:26:14 -0800 (PST) X-Google-Smtp-Source: AGHT+IFLZQjoT6qLK7kAapsavO8qkO8JULp8D9CMEo0emBt46d+OthxvMJ9ylbs1ktDWuWP9HG5x X-Received: by 2002:a05:620a:22cb:b0:785:bdb5:c0df with SMTP id o11-20020a05620a22cb00b00785bdb5c0dfmr687480qki.1.1707524773851; Fri, 09 Feb 2024 16:26:13 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707524773; cv=pass; d=google.com; s=arc-20160816; b=P2nZesGjaaqvSN3YGmyyQqGALUknpEQ9K+i2Az+CVdgLADCpxm4kb5+B+y5tGgsaU9 7kxwFtNVVq5pBbaRs3+Vi2UWewUNRRG627ZJz1Mr8ekWRvUnNee47B4Lf+9lVd8W3rEN za3xl7yeIIZUoOGrPBeRonb4c82hd2Ga6aZLquzX047J1ryE1nHVaLxdHwGvU8PCnRba PKMIlU9ot1jCU0rIAF1vKBCiq0a+nfArc2xpmWu1BNUKCmtpYXfZ1qoHFxu2RiWEIun6 S0WxT4EOna4azZaFZEkubxQjllJg/vcArcXmN9Sbi4g67urgPZc0xt7e5B5x01TtWylw DyCw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:from:subject:message-id:references:mime-version :list-unsubscribe:list-subscribe:list-id:precedence:in-reply-to:date :dkim-signature; bh=nVQs41j45YpWDUG3rQkDZ0HYMJ4B7s1Xw+YLIcCiuX4=; fh=vKbTfaDgZuGBnqlV/ERNKfyfsQk1OEpH0SLL972DMbI=; b=kKIBUWsrUesXA5JKk9M2119D23aBGU8mDNTEkPJS9QuscKd/1QY/l1lZaq1nRLR3uZ BPzfeXgQALcY8R6j8aZ666ceqR1yWlIq618LofniT1jXbeEkVOWOfPyRDetSnYD/L7qv yAbnoMetyzEYMhV7RjSmIMcRLV4q6cfqMFbMRSHWqcpcaPXih7vLnTbFTWZLoFD3qk5l 9ps2fc/qiRRWZxsrMeCUCOoRt6NjnHwtZNpMtfczzHpUzJgtkOwko8IKmTZY7ZHKxt1o hmtOnD7g16DvYYGnakfO6tI72uZVxa3H5UQD+VX7Dnvk5zRDiBX7f8mPIjgxfG+Z/C3t xpxg==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=CSggqmaJ; arc=pass (i=1 spf=pass spfdomain=flex--jstultz.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-60184-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) smtp.mailfrom="linux-kernel+bounces-60184-linux.lists.archive=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com X-Forwarded-Encrypted: i=2; AJvYcCVJhc7i4iN+MLUYvFbyr+Y76fuBroH5Fk3bxJmAZf5oeSq+efT3j7IgvyE8WEpqLp/Dvy0GBTnttb6U2RyLAH50utx0zhxK82jL6Txh4g== Return-Path: Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [147.75.199.223]) by mx.google.com with ESMTPS id j19-20020a37ef13000000b0078316dbc426si625551qkk.551.2024.02.09.16.26.13 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Feb 2024 16:26:13 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-60184-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=CSggqmaJ; arc=pass (i=1 spf=pass spfdomain=flex--jstultz.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-60184-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) smtp.mailfrom="linux-kernel+bounces-60184-linux.lists.archive=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id 80A3F1C25E74 for ; Sat, 10 Feb 2024 00:25:19 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 9F433125D2; Sat, 10 Feb 2024 00:23:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="CSggqmaJ" Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DC29B8F4A for ; Sat, 10 Feb 2024 00:23:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.201 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707524629; cv=none; b=WIXi/I7d2ca0oqlyJjSkqKetml0u2ijqsCe31oT+j52BmPN6ubFwOudlUhs6MAonClFhiBwAo4grRU6xzgMdUA2eMdkR5vpOVXzm8sOIr6yd9hd5t6r0yopMoPoyNb7/pWwWVYRLbyFt48aojdF57JTrbBbArWu585Mu+RO/bQ0= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707524629; c=relaxed/simple; bh=kmDH6T7WV6qaFmXYpUOVLT7kQ2NUoOiS1/tXVBLKpdk=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=ofGFdKPddh7THS9/63YxksSLOf+lPpDINwJX8scHjtcRpeuzRZ7tlb9FA6vB9knDsS3LGF3r7kVRT24wTfcOlqD4UNzE/qNiK+NgP+idG5dUNUR93oRnHztKLAtRV+hxpKCrwJ3kKUDUZE30Yw69GZamHVVvBiFwDTr7s2YayhU= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--jstultz.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=CSggqmaJ; arc=none smtp.client-ip=209.85.128.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--jstultz.bounces.google.com Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-604a52664d9so30414867b3.1 for ; Fri, 09 Feb 2024 16:23:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707524627; x=1708129427; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=nVQs41j45YpWDUG3rQkDZ0HYMJ4B7s1Xw+YLIcCiuX4=; b=CSggqmaJPga+u/QpYrYbWxufvr+thfGw9IiqVrp7+pncZr+GAS4pp6vvaHCPw+urxn Yiu6bKQn576VjBH0N/HiOWNmZFCC4Dg00QdoUNzAG+o4soaZaW1ewt0c004kGg1eQ4wR AsGLYxxR9sRAJkmajqgfVcakL4zmTWEawhCPEz07Gvn/eA7+i+e27SHB9xjVig5oJxx3 UyQIrh0wTlsm8qSLuen4bs1lkg3Q+/H9fMFWx8MxaGhGqpl9OuiVn9Q9j9eWDJReOLvy 1peW+rwMPhn8t4ysdxFDMKkvHYKK/7X4OOxWaWbGl3smCl3sElemumH6dTAlFJqKZmhJ 4SOQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707524627; x=1708129427; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=nVQs41j45YpWDUG3rQkDZ0HYMJ4B7s1Xw+YLIcCiuX4=; b=iJXujQlDWhuRuMYz13R+rG/yvSbloRC7zjczJcbz95YnZQQQogrisS4vJ0aIPH9S81 DEbLY5NJcjJirpPp62ItcwusagcIhWZr4ln3MM3g1GdHXC9rn2dl74p1I6NRS4o25nnn kb1fJ9L8WyYE51r66NTqaeVpqchMxiabnD8XYLLE8eYAAcDEbnH8PwHiwTbmK8UEpzu2 dj5hMu8AHgrLgsFVuLjNpmq9E/FokLObLwBLHxpwjsVGjnfzcChi9e9CW97JENFZiaQT O6Mh4g8hU4LO0qoq8IkLKH9Be+oLsCIh07jeOkkucd/ZpkljX957U4iuzBoIKDjQtXru Dv4g== X-Gm-Message-State: AOJu0YzkjSU2BIvWUb8Jux4IyEVG1xFrh5H1WClB9wM0K0JDyAdKoye4 SRmNwRAY1JsP52N/rDk/evC3yOWA5QiEvWWx7cXuJxQoFWVen3NevBZRfe7yAT/Xw82iThr50I/ 9wD0LCPv3auzWpcjvEh+KFzDt5bg4/KHEuvlKre5AEuQpE5Mub8gUcdlnfNdEIAggkWzIn+E7KE 5sXVe5q1hgw2azk0MA32PawhpZ6SSW2XITzPDb1yYHPNqd X-Received: from jstultz-noogler2.c.googlers.com ([fda3:e722:ac3:cc00:24:72f4:c0a8:600]) (user=jstultz job=sendgmr) by 2002:a05:690c:c07:b0:604:42a3:3adc with SMTP id cl7-20020a05690c0c0700b0060442a33adcmr199082ywb.10.1707524626873; Fri, 09 Feb 2024 16:23:46 -0800 (PST) Date: Fri, 9 Feb 2024 16:23:15 -0800 In-Reply-To: <20240210002328.4126422-1-jstultz@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240210002328.4126422-1-jstultz@google.com> X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240210002328.4126422-7-jstultz@google.com> Subject: [PATCH v8 6/7] sched: Split out __schedule() deactivate task logic into a helper From: John Stultz To: LKML Cc: John Stultz , Joel Fernandes , Qais Yousef , Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Valentin Schneider , Steven Rostedt , Ben Segall , Zimuzo Ezeozue , Youssef Esmat , Mel Gorman , Daniel Bristot de Oliveira , Will Deacon , Waiman Long , Boqun Feng , "Paul E. McKenney" , Metin Kaya , Xuewen Yan , K Prateek Nayak , Thomas Gleixner , kernel-team@android.com Content-Type: text/plain; charset="UTF-8" As we're going to re-use the deactivation logic, split it into a helper. Cc: Joel Fernandes Cc: Qais Yousef Cc: Ingo Molnar Cc: Peter Zijlstra Cc: Juri Lelli Cc: Vincent Guittot Cc: Dietmar Eggemann Cc: Valentin Schneider Cc: Steven Rostedt Cc: Ben Segall Cc: Zimuzo Ezeozue Cc: Youssef Esmat Cc: Mel Gorman Cc: Daniel Bristot de Oliveira Cc: Will Deacon Cc: Waiman Long Cc: Boqun Feng Cc: "Paul E. McKenney" Cc: Metin Kaya Cc: Xuewen Yan Cc: K Prateek Nayak Cc: Thomas Gleixner Cc: kernel-team@android.com Signed-off-by: John Stultz --- v6: * Define function as static to avoid "no previous prototype" warnings as Reported-by: kernel test robot v7: * Rename state task_state to be more clear, as suggested by Metin Kaya --- kernel/sched/core.c | 72 +++++++++++++++++++++++++++------------------ 1 file changed, 43 insertions(+), 29 deletions(-) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index ad4748327651..b537e5f501ea 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -6563,6 +6563,48 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf) # define SM_MASK_PREEMPT SM_PREEMPT #endif +/* + * Helper function for __schedule() + * + * If a task does not have signals pending, deactivate it and return true + * Otherwise marks the task's __state as RUNNING and returns false + */ +static bool try_to_deactivate_task(struct rq *rq, struct task_struct *p, + unsigned long task_state) +{ + if (signal_pending_state(task_state, p)) { + WRITE_ONCE(p->__state, TASK_RUNNING); + } else { + p->sched_contributes_to_load = + (task_state & TASK_UNINTERRUPTIBLE) && + !(task_state & TASK_NOLOAD) && + !(task_state & TASK_FROZEN); + + if (p->sched_contributes_to_load) + rq->nr_uninterruptible++; + + /* + * __schedule() ttwu() + * prev_state = prev->state; if (p->on_rq && ...) + * if (prev_state) goto out; + * p->on_rq = 0; smp_acquire__after_ctrl_dep(); + * p->state = TASK_WAKING + * + * Where __schedule() and ttwu() have matching control dependencies. + * + * After this, schedule() must not care about p->state any more. + */ + deactivate_task(rq, p, DEQUEUE_SLEEP | DEQUEUE_NOCLOCK); + + if (p->in_iowait) { + atomic_inc(&rq->nr_iowait); + delayacct_blkio_start(); + } + return true; + } + return false; +} + /* * __schedule() is the main scheduler function. * @@ -6654,35 +6696,7 @@ static void __sched notrace __schedule(unsigned int sched_mode) */ prev_state = READ_ONCE(prev->__state); if (!(sched_mode & SM_MASK_PREEMPT) && prev_state) { - if (signal_pending_state(prev_state, prev)) { - WRITE_ONCE(prev->__state, TASK_RUNNING); - } else { - prev->sched_contributes_to_load = - (prev_state & TASK_UNINTERRUPTIBLE) && - !(prev_state & TASK_NOLOAD) && - !(prev_state & TASK_FROZEN); - - if (prev->sched_contributes_to_load) - rq->nr_uninterruptible++; - - /* - * __schedule() ttwu() - * prev_state = prev->state; if (p->on_rq && ...) - * if (prev_state) goto out; - * p->on_rq = 0; smp_acquire__after_ctrl_dep(); - * p->state = TASK_WAKING - * - * Where __schedule() and ttwu() have matching control dependencies. - * - * After this, schedule() must not care about p->state any more. - */ - deactivate_task(rq, prev, DEQUEUE_SLEEP | DEQUEUE_NOCLOCK); - - if (prev->in_iowait) { - atomic_inc(&rq->nr_iowait); - delayacct_blkio_start(); - } - } + try_to_deactivate_task(rq, prev, prev_state); switch_count = &prev->nvcsw; } -- 2.43.0.687.g38aa6559b0-goog