Date:   Thu, 9 Dec 2021 15:23:32 +0200
From:   Shachar Sharon <synarete@gmail.com>
To:     Miklos Szeredi <miklos@szeredi.hu>
Cc:     Peter Zijlstra <peterz@infradead.org>,
        Ingo Molnar <mingo@redhat.com>, quic_stummala@quicinc.com,
        linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org,
        quic_pkondeti@quicinc.com, quic_sayalil@quicinc.com,
        quic_aiquny@quicinc.com, quic_zljing@quicinc.com,
        quic_blong@quicinc.com, quic_richardp@quicinc.com,
        quic_cdevired@quicinc.com,
        Pradeep P V K <quic_pragalla@quicinc.com>
Subject: Re: [PATCH V1] fuse: give wakeup hints to the scheduler
Message-ID: <YbIDVMouwlHyjZT+@lpc>
References: <CAJfpegvDfc9JUo6VASRyYXzj1=j3t6oU9W3QGWO08vhfWHf-UA@mail.gmail.com>
 <Ya8ycLODlcvLx4xB@hirez.programming.kicks-ass.net>
 <CAJfpegsVg2K0CvrPvXGSu=Jz_R3VZZOy49Jw51rThQUJ1_9e6g@mail.gmail.com>
 <Ya86coKm4RuQDmVS@hirez.programming.kicks-ass.net>
 <CAJfpegumZ1RQLBCtbrOiOAT9ygDtDThpySwb8yCpWGBu1fRQmw@mail.gmail.com>
 <Ya9ljdrOkhBhhnJX@hirez.programming.kicks-ass.net>
 <Ya9m0ME1pom49b+D@hirez.programming.kicks-ass.net>
 <CAJfpegt2x1ztuzh0niY7fgx1UKxDGsAkJbS0wVPp5awxwyhRpA@mail.gmail.com>
 <Ya9uxHGo7UJikEte@hirez.programming.kicks-ass.net>
 <CAJfpegupPZMG2dv27ZkpQwTeUw-WcaRZbYXSH-i=+Rt=T+UaDg@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii; format=flowed
Content-Disposition: inline
In-Reply-To: <CAJfpegupPZMG2dv27ZkpQwTeUw-WcaRZbYXSH-i=+Rt=T+UaDg@mail.gmail.com>
Precedence: bulk

On Wed, Dec 08, 2021 at 04:46:06PM +0100, Miklos Szeredi wrote:
>On Tue, 7 Dec 2021 at 15:25, Peter Zijlstra <peterz@infradead.org> wrote:
>
>> FIFO means the thread used longest ago gets to go first. If your threads
>> are an idempotent workers, FIFO might not be the best option. But I'm
>> not much familiar with the FUSE code or it's design.
>
>Okay.  Did some experiments, but couldn't see
>wake_up_interruptible_sync() actually migrate the woken task, the
>behavior was identical to wake_up_interruptible().   I guess this is
>the "less" part in "more or less", but it would be good to see more
>clearly what is happening.
>
>I'll try to describe the design to give more context:
>
>- FUSE is similar to network filesystem in that there's a server and a
>client, except both are on the same host. The client lives in the
>kernel and the server lives in userspace.
>
>- Communication between them is done with read and write syscalls.
>
>- Usually the server has multiple threads.  When a server thread is
>idle it is blocking in sys_read -> ... -> fuse_dev_do_read ->
>wait_event_interruptible_exclusive(fiq->waitq,...).
>
>- When a filesystem request comes in (e.g. mkdir) a request is
>constructed, put on the input queue (fiq->pending) and fiq->waitq
>woken up.  After this the client task goes to sleep in
>request_wait_answer -> wait_event_interruptible(req->waitq, ...).
>
>- The server thread takes the request off the pending list, copies the
>data to the userspace buffer and puts the request on the processing
>list.
>
>- The userspace part interprets the read buffer, performs the fs
>operation, and writes the reply.
>
>- During the write(2) the reply is now copied to the kernel and the
>request is looked up on the processing list.  The client is woken up
>through req->waitq.  After returning from write(2) the server thread
>again calls read(2) to get the next request.
>
>- After being woken up, the client task now returns with the result of
>the operation.
>
>- The above example is for synchronous requests.  There are async
>requests like readahead or buffered writes.  In that case the client
>does not call request_wait_answer() but returns immediately and the
>result is processed from the server thread using a callback function
>(req->args->end()).
>
>From a scheduling prospective it would be ideal if the server thread's
>CPU was matched to the client thread's CPU, since that would make the
>data stay local, and for synchronous requests a _sync type wakeup is
>perfect, since the client goes to sleep just as the server starts
>processing and vice versa.
>
>Always migrating the woken server thread to the client's CPU is not
>going to be good, since this would result in too many migrations and
>would loose locality for the server's stack.
>
>Another idea is to add per-cpu input queues.  The client then would
>queue the request on the pending queue corresponding to its CPU and
>wake up the server thread blocked on that queue.
>
>What happens though if this particular queue has no servers?  Or if a
>queue is starved because it's served by less threads than another?
>Handing these cases seems really complicated.
>
Per-CPU input queue is a great idea. It was a key concept in ZUFS[1]
design. However, describing it as complicated would be an understatement,
to say the least.

>Is there a simper way?
Here is an idea: hint the userspace server on which cpu to execute the
_next_ request via the (unused) fuse_in_header.padding field in current
request. Naturally, this method would be effective only for cases where
queue-size > 1.

>
>Thanks,
>Miklos

[1] https://lwn.net/Articles/795996/