Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Date:   Thu, 27 Jan 2022 13:19:43 +0100
From:   Peter Zijlstra <peterz@infradead.org>
To:     Tao Zhou <tao.zhou@linux.dev>
Cc:     mingo@redhat.com, tglx@linutronix.de, juri.lelli@redhat.com,
        vincent.guittot@linaro.org, dietmar.eggemann@arm.com,
        rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de,
        bristot@redhat.com, linux-kernel@vger.kernel.org,
        linux-mm@kvack.org, linux-api@vger.kernel.org, x86@kernel.org,
        pjt@google.com, posk@google.com, avagin@google.com,
        jannh@google.com, tdelisle@uwaterloo.ca, mark.rutland@arm.com,
        posk@posk.io
Subject: Re: [RFC][PATCH v2 5/5] sched: User Mode Concurency Groups
Message-ID: <YfKN3+84gtaIopHW@hirez.programming.kicks-ass.net>
References: <20220120155517.066795336@infradead.org>
 <20220120160822.914418096@infradead.org>
 <Ye67uQa4CwUuQJVY@geo.homenetwork>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <Ye67uQa4CwUuQJVY@geo.homenetwork>
Precedence: bulk

On Mon, Jan 24, 2022 at 10:46:17PM +0800, Tao Zhou wrote:
> Hi Peter,
> 
> On Thu, Jan 20, 2022 at 04:55:22PM +0100, Peter Zijlstra wrote:
> 
> [...]
> 
> > +/* pre-schedule() */
> > +void umcg_wq_worker_sleeping(struct task_struct *tsk)
> > +{
> > +	struct umcg_task __user *self = READ_ONCE(tsk->umcg_task);
> > +	int ret;
> > +
> > +	if (!tsk->umcg_server) {
> > +		/*
> > +		 * Already blocked before, the pages are unpinned.
> > +		 */
> > +		return;
> > +	}
> > +
> > +	/* Must not fault, mmap_sem might be held. */
> > +	pagefault_disable();
> > +
> > +	ret = umcg_update_state(tsk, self, UMCG_TASK_RUNNING, UMCG_TASK_BLOCKED);
> > +	if (ret == -EAGAIN) {
> > +		/*
> > +		 * Consider:
> > +		 *
> > +		 *   self->state = UMCG_TASK_RUNNABLE | UMCG_TF_COND_WAIT;
> > +		 *   ...
> > +		 *   sys_umcg_wait();
> > +		 *
> > +		 * and the '...' code doing a blocking syscall/fault. This
> > +		 * ensures that returns with UMCG_TASK_RUNNING, which will make
> 
> /UMCG_TASK_RUNNING/UMCG_TASK_RUNNABLE/

So the issue is that:

	self->state = UMCG_TASK_RUNNABLE | UMCG_TF_COND_WAIT;

	<#PF>
	  umcg_sys_enter()
	    umcg_pin_user_page()
	  schedule()
	    sched_submit_work()
	      umcg_wq_worker_sleeping()
	        umcg_update_state(tsk, self, UMCG_TASK_RUNNING, UMCG_TASK_BLOCKED) // -EAGAIN
		UMCG_DIE()

Which is clearly not desirable.

So this additinoal thing ensures that:

		umcg_update_state(tsk, self, UMCG_TASK_RUNNABLE, UMCG_TASK_BLOCKED) // 0

	  umcg_sys_exit()
	    umcg_update_state(tsk, self, UMCG_TASK_BLOCKED, UMCG_TASK_RUNNABLE);
	    umcg_enqueue_and_wake()

	  umcg_notify_resume()
	    umcg_wait()

	// must be UMCG_TASK_RUNNING here
	</#PF>

So when the pagefault finally does return, it will have:
UMCG_TASK_RUNNING.

Which will then make sys_umcg_wait() return -EAGAIN and around we go.

> > +		 * sys_umcg_wait() return with -EAGAIN.
> > +		 */
> > +		ret = umcg_update_state(tsk, self, UMCG_TASK_RUNNABLE, UMCG_TASK_BLOCKED);
> > +	}
> > +	if (ret)
> > +		UMCG_DIE_PF("state");
> > +
> > +	if (umcg_wake_server(tsk))
> > +		UMCG_DIE_PF("wake");
> > +
> > +	pagefault_enable();
> > +
> > +	/*
> > +	 * We're going to sleep, make sure to unpin the pages, this ensures
> > +	 * the pins are temporary. Also see umcg_sys_exit().
> > +	 */
> > +	umcg_unpin_pages();
> > +}