Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Date:   Thu, 4 Feb 2021 14:59:58 +0100
From:   Peter Zijlstra <peterz@infradead.org>
To:     "Joel Fernandes (Google)" <joel@joelfernandes.org>
Cc:     Nishanth Aravamudan <naravamudan@digitalocean.com>,
        Julien Desfossez <jdesfossez@digitalocean.com>,
        Tim Chen <tim.c.chen@linux.intel.com>,
        Vineeth Pillai <viremana@linux.microsoft.com>,
        Aaron Lu <aaron.lwe@gmail.com>,
        Aubrey Li <aubrey.intel@gmail.com>, tglx@linutronix.de,
        linux-kernel@vger.kernel.org, mingo@kernel.org,
        torvalds@linux-foundation.org, fweisbec@gmail.com,
        keescook@chromium.org, kerrnel@google.com,
        Phil Auld <pauld@redhat.com>,
        Valentin Schneider <valentin.schneider@arm.com>,
        Mel Gorman <mgorman@techsingularity.net>,
        Pawan Gupta <pawan.kumar.gupta@linux.intel.com>,
        Paolo Bonzini <pbonzini@redhat.com>, vineeth@bitbyteword.org,
        Chen Yu <yu.c.chen@intel.com>,
        Christian Brauner <christian.brauner@ubuntu.com>,
        Agata Gruza <agata.gruza@intel.com>,
        Antonio Gomez Iglesias <antonio.gomez.iglesias@intel.com>,
        graf@amazon.com, konrad.wilk@oracle.com, dfaggioli@suse.com,
        pjt@google.com, rostedt@goodmis.org, derkling@google.com,
        benbjiang@tencent.com,
        Alexandre Chartre <alexandre.chartre@oracle.com>,
        James.Bottomley@hansenpartnership.com, OWeisse@umich.edu,
        Dhaval Giani <dhaval.giani@oracle.com>,
        Junaid Shahid <junaids@google.com>, jsbarnes@google.com,
        chris.hyser@oracle.com, Ben Segall <bsegall@google.com>,
        Josh Don <joshdon@google.com>, Hao Luo <haoluo@google.com>,
        Tom Lendacky <thomas.lendacky@amd.com>
Subject: Re: [PATCH v10 2/5] sched: CGroup tagging interface for core
 scheduling
Message-ID: <YBv93iXqI8UTw9tD@hirez.programming.kicks-ass.net>
References: <20210123011704.1901835-1-joel@joelfernandes.org>
 <20210123011704.1901835-3-joel@joelfernandes.org>
 <YBrUgxLfjcpjwgo6@hirez.programming.kicks-ass.net>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <YBrUgxLfjcpjwgo6@hirez.programming.kicks-ass.net>
Precedence: bulk

On Wed, Feb 03, 2021 at 05:51:15PM +0100, Peter Zijlstra wrote:
> 
> I'm slowly starting to go through this...
> 
> On Fri, Jan 22, 2021 at 08:17:01PM -0500, Joel Fernandes (Google) wrote:
> > +static bool sched_core_empty(struct rq *rq)
> > +{
> > +	return RB_EMPTY_ROOT(&rq->core_tree);
> > +}
> > +
> > +static struct task_struct *sched_core_first(struct rq *rq)
> > +{
> > +	struct task_struct *task;
> > +
> > +	task = container_of(rb_first(&rq->core_tree), struct task_struct, core_node);
> > +	return task;
> > +}
> 
> AFAICT you can do with:
> 
> static struct task_struct *sched_core_any(struct rq *rq)
> {
> 	return rb_entry(rq->core_tree.rb_node, struct task_struct, code_node);
> }
> 
> > +static void sched_core_flush(int cpu)
> > +{
> > +	struct rq *rq = cpu_rq(cpu);
> > +	struct task_struct *task;
> > +
> > +	while (!sched_core_empty(rq)) {
> > +		task = sched_core_first(rq);
> > +		rb_erase(&task->core_node, &rq->core_tree);
> > +		RB_CLEAR_NODE(&task->core_node);
> > +	}
> > +	rq->core->core_task_seq++;
> > +}
> 
> However,
> 
> > +	for_each_possible_cpu(cpu) {
> > +		struct rq *rq = cpu_rq(cpu);
> > +
> > +		WARN_ON_ONCE(enabled == rq->core_enabled);
> > +
> > +		if (!enabled || (enabled && cpumask_weight(cpu_smt_mask(cpu)) >= 2)) {
> > +			/*
> > +			 * All active and migrating tasks will have already
> > +			 * been removed from core queue when we clear the
> > +			 * cgroup tags. However, dying tasks could still be
> > +			 * left in core queue. Flush them here.
> > +			 */
> > +			if (!enabled)
> > +				sched_core_flush(cpu);
> > +
> > +			rq->core_enabled = enabled;
> > +		}
> > +	}
> 
> I'm not sure I understand. Is the problem that we're still schedulable
> during do_exit() after cgroup_exit() ? It could be argued that when we
> leave the cgroup there, we should definitely leave the tag group too.

That is, did you forget to implement cpu_cgroup_exit()?