Received: by 2002:a05:6a10:a0d1:0:0:0:0 with SMTP id j17csp1301519pxa; Sat, 15 Aug 2020 15:03:02 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwOxcmM8nFeuvUt733M1tfO8OXSzdlE0kWk4N9E74DSTxhsAKprWX/cuGmpqDYd0/tTs4NH X-Received: by 2002:a17:907:213c:: with SMTP id qo28mr8359747ejb.174.1597528982555; Sat, 15 Aug 2020 15:03:02 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1597528982; cv=none; d=google.com; s=arc-20160816; b=wxgePmuILq677ZG4j9HXUBqXL0WQ+wla2oHqJgwYkSNRJ8JMtA9mBUD8MRx0yZifaY m4hc5GSj9N8R9JrjM8SfeoMFcWtwbiabRup1FGmSkGCIG5648GjDPVOokEQiNdTwmxwA fs30EEpals2uxpAOnGly8zNdi3gFmNJKQDxpdNnZ5MiGekSyj43bAnd5xMZQjAILNXdK gLk5qLmcCUBGZee/1vu6jrtYbdyaZC0CRJl3Zs6yTiQQffWQPZwl6nWCEmFw4ntwrFQj LC9BpsnFt1ojQSmGslWvKo0iDUh1Ao5vStLa6McLxG20izGG3ay5N/v3jqWi8kde0sfy 7M+A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=jOuJJ23oEouPSGg3lZDZzJM6phxa5MifEp0ZsmuPH0A=; b=RFJryLNTojhmL8XBtfnGhd6ceT4JXMnNmfaBZo0/oDZMDGt0AE3WeyUNTfyBpKUZuP 9HIeiQEoeRQ5HWKwu0p4LDi5oJIrt1raF6iMatfsbEAkLYjTv8XeeUrmcflO+tYwidNo 4C7NUhy3YrN4Bjs9HE1hG3PuQecTyzOD1lvFgNH2GTZsg4BFyE1B/4GEghZqsr0Twh7K zApkxPO35GK9Tgryt7U1oQ5Z8nnTa8/TFa49C2EMl4YEuhlSlIq0LmKfIRCBobR6CWIH UKu1DAiDLZwevemVd8Tt0SQPsw6xI8T02B0tZhDntF2H/ZMfjnMxNS/RBQ3Bx5dNloWT Ding== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@joelfernandes.org header.s=google header.b=P3cbT+5i; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id a18si7014348ejr.184.2020.08.15.15.02.39; Sat, 15 Aug 2020 15:03:02 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@joelfernandes.org header.s=google header.b=P3cbT+5i; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728810AbgHOWBQ (ORCPT + 99 others); Sat, 15 Aug 2020 18:01:16 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45612 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726021AbgHOVvd (ORCPT ); Sat, 15 Aug 2020 17:51:33 -0400 Received: from mail-qk1-x743.google.com (mail-qk1-x743.google.com [IPv6:2607:f8b0:4864:20::743]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CFCF3C06137B for ; Fri, 14 Aug 2020 20:19:34 -0700 (PDT) Received: by mail-qk1-x743.google.com with SMTP id p4so10241977qkf.0 for ; Fri, 14 Aug 2020 20:19:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=jOuJJ23oEouPSGg3lZDZzJM6phxa5MifEp0ZsmuPH0A=; b=P3cbT+5ibY0DarOGRd8+2HzKTmoLWqb9/Qp47CMqSclIzbGT9hbhvUuTT/lH20T0vD pzf2wmWRPH4Nw5sPERqFv/fkMj3kSTmYwCpzfKaz+DbnAafQBQIE4etQbvbITbT7IA5Y 7VOuQanq3Pq5EoJ5jgJoDluGf06TyjduSX2zM= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=jOuJJ23oEouPSGg3lZDZzJM6phxa5MifEp0ZsmuPH0A=; b=aFh43ZOtio3w7/GsmwWphjEjyelaCbtekcpC/wYVw2RhtM1LP10mrPgtijHthBiL+P ak3tu5FWrWgrfV+aIkWpzqPf816CBY+1fg/Fjotd+EvfXzgnfnShP3Od4TEm1t4YlbJq nxqHQs9CUCccbw5uE9TxztNcq1yrbJ6mDha7vIWSevQFqkkCwzQ4vPY6j8CNQMe/Bh9x r8Ye8Q1owm/LM0FboG+WVnmRsLrnmKXAlvWXT3L+Vfj09vyBDzjPifvJFW4/ntW55Mmr /ceDqyfUfPlKH7PG5erInoNEdPrTQEfDqcOcoZYzkh2UK0P85F/BiILKzNX4sBtiDr9J 4eXA== X-Gm-Message-State: AOAM532ozqjwkV94Ax0Lb2OFslAfycOlHrLxjfKpTpcUKjXhBL9jICVS Qbo3WFfeF+JcY55r/t185w0WgEb3maEDKg== X-Received: by 2002:a37:6583:: with SMTP id z125mr4698518qkb.297.1597461573799; Fri, 14 Aug 2020 20:19:33 -0700 (PDT) Received: from joelaf.cam.corp.google.com ([2620:15c:6:12:cad3:ffff:feb3:bd59]) by smtp.gmail.com with ESMTPSA id p12sm10055199qkk.118.2020.08.14.20.19.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 14 Aug 2020 20:19:33 -0700 (PDT) From: "Joel Fernandes (Google)" To: linux-kernel@vger.kernel.org Cc: Vineeth Pillai , Joel Fernandes , Aaron Lu , Aubrey Li , Julien Desfossez , Kees Cook , "Paul E. McKenney" , Paul Turner , Peter Zijlstra , Steven Rostedt , Thomas Gleixner , Tim Chen , Tim Chen , Vincent Guittot , x86@kernel.org (maintainer:X86 ARCHITECTURE (32-BIT AND 64-BIT)), fweisbec@gmail.com, kerrnel@google.com, Phil Auld , Valentin Schneider , Paolo Bonzini , Chen Yu , Christian Brauner Subject: [PATCH RFC 10/12] sched/coresched: Make core_pick_seq per run-queue Date: Fri, 14 Aug 2020 23:19:06 -0400 Message-Id: <20200815031908.1015049-11-joel@joelfernandes.org> X-Mailer: git-send-email 2.28.0.220.ged08abb693-goog In-Reply-To: <20200815031908.1015049-1-joel@joelfernandes.org> References: <20200815031908.1015049-1-joel@joelfernandes.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Vineeth Pillai core_pick_seq is a core wide counter to identify if a task pick has been made for a CPU by its sibling. But during hotplug scenarios, pick cannot be made for CPUs that are offline and when they come up, they get tricked because the counter is incremented. So, make core_pick_seq per run-queue. Signed-off-by: Vineeth Pillai Signed-off-by: Joel Fernandes (Google) --- kernel/sched/core.c | 19 ++++++++++--------- kernel/sched/sched.h | 2 +- 2 files changed, 11 insertions(+), 10 deletions(-) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 3e9df8221c62..48a49168e57f 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -4623,9 +4623,9 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf) * pointers are all still valid), and we haven't scheduled the last * pick yet, do so now. */ - if (rq->core->core_pick_seq == rq->core->core_task_seq && - rq->core->core_pick_seq != rq->core_sched_seq) { - WRITE_ONCE(rq->core_sched_seq, rq->core->core_pick_seq); + if (rq->core_pick_seq == rq->core->core_task_seq && + rq->core_pick_seq != rq->core_sched_seq) { + WRITE_ONCE(rq->core_sched_seq, rq->core_pick_seq); next = rq->core_pick; if (next != prev) { @@ -4635,7 +4635,7 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf) trace_printk("pick pre selected (%u %u %u): %s/%d %lx\n", rq->core->core_task_seq, - rq->core->core_pick_seq, + rq->core_pick_seq, rq->core_sched_seq, next->comm, next->pid, next->core_cookie); @@ -4649,7 +4649,7 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf) smt_mask = cpu_smt_mask(cpu); /* - * core->core_task_seq, core->core_pick_seq, rq->core_sched_seq + * core->core_task_seq, rq->core_pick_seq, rq->core_sched_seq * * @task_seq guards the task state ({en,de}queues) * @pick_seq is the @task_seq we did a selection on @@ -4667,8 +4667,10 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf) struct rq *rq_i = cpu_rq(i); trace_printk("CPU %d is in smt_mask, resetting\n", i); - - rq_i->core_pick = NULL; + if (rq_i->core_pick) { + WRITE_ONCE(rq_i->core_sched_seq, rq_i->core_pick_seq); + rq_i->core_pick = NULL; + } if (rq_i->core_forceidle) { need_sync = true; @@ -4771,9 +4773,7 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf) next_class:; } - rq->core->core_pick_seq = rq->core->core_task_seq; next = rq->core_pick; - rq->core_sched_seq = rq->core->core_pick_seq; /* Something should have been selected for current CPU */ WARN_ON_ONCE(!next); @@ -4801,6 +4801,7 @@ next_class:; continue; if (rq_i->curr != rq_i->core_pick) { + WRITE_ONCE(rq_i->core_pick_seq, rq->core->core_task_seq); trace_printk("IPI(%d)\n", i); resched_curr(rq_i); } diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 2922e171a1f0..c7caece2df6e 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -1036,6 +1036,7 @@ struct rq { /* per rq */ struct rq *core; struct task_struct *core_pick; + unsigned int core_pick_seq; unsigned int core_enabled; unsigned int core_sched_seq; struct rb_root core_tree; @@ -1045,7 +1046,6 @@ struct rq { /* shared state */ unsigned int core_task_seq; - unsigned int core_pick_seq; unsigned long core_cookie; unsigned int core_unsafe_nest; #endif -- 2.28.0.220.ged08abb693-goog