Received: by 2002:a25:868d:0:0:0:0:0 with SMTP id z13csp90763ybk; Fri, 8 May 2020 19:43:24 -0700 (PDT) X-Google-Smtp-Source: APiQypI7CPeDkk2eT/wz2RJOol5N1CoZixjtIgphRb9EppBl++66oIcpxX8XhmuF788GsAngoqaM X-Received: by 2002:a17:907:435d:: with SMTP id oc21mr4579803ejb.100.1588992204748; Fri, 08 May 2020 19:43:24 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1588992204; cv=none; d=google.com; s=arc-20160816; b=PJqwh3a/Hgv2VhZaCskKpFMsSxm/JzYD8jzZNwLL1cdBEea3woo2ulh0Lw/r7pBUpr gOy67RLXdUT18YcH8D0eRNbDQDSZimG/tBI1cZ5+EVzuPPWwzv06QWq/2wJYr+dsgghw m5GgRG5vCnRZW7n/a2UnZKsx1hw2HIuUfxpqLIVzl8Y4DELaOHCcgjsv4AI85ilRUuQx m7xIqlcztvdHUqF96m8Jfeg1aRmSjIIFVtTmQc2VDYvRzUCg74vd7+wn/zK4SM2ofbzd BwXca0GSV9ARnm+CwMQ8wqK1qdEdsOuGrFf/wtlMcZGIyCIjsj4XwkStNdRzM88BaAAx OA+g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:user-agent:in-reply-to :content-disposition:mime-version:references:message-id:subject:cc :to:from:date:dmarc-filter:dkim-signature; bh=JJ9AKZ/PmehK3UlrXKD3f/suIB/gF43b9Bgi6O+K0fU=; b=ZpecIOgTsFKEVkW/q3cUZgplgZsvpLgVHBqD8MrU/AXwNzIYHzZCvI9Dlv/1oDMEUM k9bHrRzPcVP89EjsFKxj59nZyX0onOqIJ2/C7z3rl0K3Uf0yeCMEhsOHkFxHQyFeP8vi jVVJexa0Mp3lJPszoCld1R19l4YHnOe+qXYbd8XHrLb+t2qZhUN9HknrZMX8Nj+tyA4l h8l/oacdMxX1TvhJzTe+1Zk4IOq+ySyknIVoiU0A32hT6ZkpiZhCRFiDjKTuT2z9C2a6 quRy7OCLQZ8Zf6PykRT0jkjrp2KsmY6huGmk03yuhDOz5sAb3iByraMqc4r/ajOg2fFD s82Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@mg.codeaurora.org header.s=smtp header.b="pbGfnK/V"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id c5si381356edk.226.2020.05.08.19.43.01; Fri, 08 May 2020 19:43:24 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=fail header.i=@mg.codeaurora.org header.s=smtp header.b="pbGfnK/V"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728632AbgEICjZ (ORCPT + 99 others); Fri, 8 May 2020 22:39:25 -0400 Received: from mail27.static.mailgun.info ([104.130.122.27]:17184 "EHLO mail27.static.mailgun.info" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728158AbgEICjZ (ORCPT ); Fri, 8 May 2020 22:39:25 -0400 DKIM-Signature: a=rsa-sha256; v=1; c=relaxed/relaxed; d=mg.codeaurora.org; q=dns/txt; s=smtp; t=1588991964; h=In-Reply-To: Content-Type: MIME-Version: References: Message-ID: Subject: Cc: To: From: Date: Sender; bh=JJ9AKZ/PmehK3UlrXKD3f/suIB/gF43b9Bgi6O+K0fU=; b=pbGfnK/V32zaw4Cfh98PyIdUeL7G1sdYWeQEVTHIaKjQMvze3cu3+yni8TYtT5wt18fMJm2d YgKvl0w7USOchZie0x2lmq8DDZKeg/l4qYfWNMwbRG73dEARDk+bp2i3LCt2NtfbtHKTo+WL G10JfXMSpDCAUrRuggup7u2TRdA= X-Mailgun-Sending-Ip: 104.130.122.27 X-Mailgun-Sid: WyI0MWYwYSIsICJsaW51eC1rZXJuZWxAdmdlci5rZXJuZWwub3JnIiwgImJlOWU0YSJd Received: from smtp.codeaurora.org (ec2-35-166-182-171.us-west-2.compute.amazonaws.com [35.166.182.171]) by mxa.mailgun.org with ESMTP id 5eb617da.7fc1940492d0-smtp-out-n02; Sat, 09 May 2020 02:39:22 -0000 (UTC) Received: by smtp.codeaurora.org (Postfix, from userid 1001) id ADB16C433D2; Sat, 9 May 2020 02:39:22 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-caf-mail-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.0 required=2.0 tests=ALL_TRUSTED,SPF_NONE autolearn=unavailable autolearn_force=no version=3.4.0 Received: from codeaurora.org (blr-c-bdr-fw-01_GlobalNAT_AllZones-Outside.qualcomm.com [103.229.19.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: pkondeti) by smtp.codeaurora.org (Postfix) with ESMTPSA id 905A2C433F2; Sat, 9 May 2020 02:39:18 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 smtp.codeaurora.org 905A2C433F2 Authentication-Results: aws-us-west-2-caf-mail-1.web.codeaurora.org; dmarc=none (p=none dis=none) header.from=codeaurora.org Authentication-Results: aws-us-west-2-caf-mail-1.web.codeaurora.org; spf=none smtp.mailfrom=pkondeti@codeaurora.org Date: Sat, 9 May 2020 08:09:15 +0530 From: Pavan Kondeti To: Parth Shah Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, mingo@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, qais.yousef@arm.com, chris.hyser@oracle.com, valentin.schneider@arm.com, rjw@rjwysocki.net Subject: Re: [RFC 2/4] sched/core: Set nr_lat_sensitive counter at various scheduler entry/exit points Message-ID: <20200509023915.GN19464@codeaurora.org> References: <20200507133723.18325-1-parth@linux.ibm.com> <20200507133723.18325-3-parth@linux.ibm.com> <20200508083308.GI19464@codeaurora.org> <73506bba-7bcb-fd40-6866-d5d88d436fbf@linux.ibm.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <73506bba-7bcb-fd40-6866-d5d88d436fbf@linux.ibm.com> User-Agent: Mutt/1.5.24 (2015-08-30) Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Fri, May 08, 2020 at 04:45:16PM +0530, Parth Shah wrote: > Hi Pavan, > > Thanks for going through this patch-set. > > On 5/8/20 2:03 PM, Pavan Kondeti wrote: > > Hi Parth, > > > > On Thu, May 07, 2020 at 07:07:21PM +0530, Parth Shah wrote: > >> Monitor tasks at: > >> 1. wake_up_new_task() - forked tasks > >> > >> 2. set_task_cpu() - task migrations, Load balancer > >> > >> 3. __sched_setscheduler() - set/unset latency_nice value > >> Increment the nr_lat_sensitive count on the CPU with task marked with > >> latency_nice == -20. > >> Similarly, decrement the nr_lat_sensitive counter upon re-marking the task > >> with >-20 latency_nice task. > >> > >> 4. finish_task_switch() - dying task > >> > > > > > >> Signed-off-by: Parth Shah > >> --- > >> kernel/sched/core.c | 30 ++++++++++++++++++++++++++++-- > >> kernel/sched/sched.h | 5 +++++ > >> 2 files changed, 33 insertions(+), 2 deletions(-) > >> > >> diff --git a/kernel/sched/core.c b/kernel/sched/core.c > >> index 2d8b76f41d61..ad396c36eba6 100644 > >> --- a/kernel/sched/core.c > >> +++ b/kernel/sched/core.c > >> @@ -1744,6 +1744,11 @@ void set_task_cpu(struct task_struct *p, unsigned int new_cpu) > >> trace_sched_migrate_task(p, new_cpu); > >> > >> if (task_cpu(p) != new_cpu) { > >> + if (task_is_lat_sensitive(p)) { > >> + per_cpu(nr_lat_sensitive, task_cpu(p))--; > >> + per_cpu(nr_lat_sensitive, new_cpu)++; > >> + } > >> + > > > > Since we can come here without rq locks, there is a possibility > > of a race and incorrect updates can happen. Since the counters > > are used to prevent C-states, we don't want that to happen. > > I did tried using task_lock(p) wherever we do change refcount and when > latency_nice value is set. There I was using nr_lat_sensitive with atomic_t > type. > > After lots of thinking to optimize it and thinking that we anyways hold rq > lock, I thought of not using any lock here and see if scheduler community > has well known solution for this :-) > > But in brief, using atomic_t nr_lat_sensitive and task_lock(p) when changin > refcount should solve problem, right? > > If you or anyone have solution for this kind of pattern, then that surely > will be helpful. > I am not sure if task_lock() can help here, because we are operating the counter on per CPU basis here. May be cmpxchg based loop works here to make sure that increment/decrement operation happens atomically here. > > > >> if (p->sched_class->migrate_task_rq) > >> p->sched_class->migrate_task_rq(p, new_cpu); > >> p->se.nr_migrations++; [...] > >> @@ -4732,8 +4749,17 @@ static void __setscheduler_params(struct task_struct *p, > >> p->normal_prio = normal_prio(p); > >> set_load_weight(p, true); > >> > >> - if (attr->sched_flags & SCHED_FLAG_LATENCY_NICE) > >> + if (attr->sched_flags & SCHED_FLAG_LATENCY_NICE) { > >> + if (p->state != TASK_DEAD && > >> + attr->sched_latency_nice != p->latency_nice) { > >> + if (attr->sched_latency_nice == MIN_LATENCY_NICE) > >> + per_cpu(nr_lat_sensitive, task_cpu(p))++; > >> + else if (task_is_lat_sensitive(p)) > >> + per_cpu(nr_lat_sensitive, task_cpu(p))--; > >> + } > >> + > >> p->latency_nice = attr->sched_latency_nice; > >> + } > >> } > > > > There is a potential race here due to which we can mess up the refcount. > > > > - A latency sensitive task is marked TASK_DEAD > > > > - sched_setattr() called on the task to clear the latency nice. Since > > we check the task state here, we skip the decrement. > > - The task is finally context switched out and we skip the decrement again > > since it is not a latency senstivie task. > > if task is already marked TASK_DEAD then we should have already decremented > its refcount in finish_task_switch(). > am I missing something? There is a window (context switch and dropping rq lock) between marking a task DEAD (in do_task_dead()) and dropping the ref counter (in finish_task_switch()) during which we can run into here and skip the checking because task is marked as DEAD. Thanks, Pavan -- Qualcomm India Private Limited, on behalf of Qualcomm Innovation Center, Inc. Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, a Linux Foundation Collaborative Project.