Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
MIME-Version: 1.0
References: <6ba14287-336d-cdcd-0d39-680f288ca776@ddn.com> <CAJfpegt=KZJKExpxPgGXoBEzWpzepL9cyaqS=dwW5AN6y2up_Q@mail.gmail.com>
 <d1955ffb-77ce-97a6-fcf2-b25960d389aa@fastmail.fm>
In-Reply-To: <d1955ffb-77ce-97a6-fcf2-b25960d389aa@fastmail.fm>
From:   Miklos Szeredi <miklos@szeredi.hu>
Date:   Mon, 25 Apr 2022 10:37:58 +0200
Message-ID: <CAJfpegsr3fqcFuNekLwf69v3mpNJyze741=L5KUJjvH758eE6g@mail.gmail.com>
Subject: Re: RFC fuse waitq latency
To:     Bernd Schubert <bernd.schubert@fastmail.fm>
Cc:     Bernd Schubert <bschubert@ddn.com>,
        Linux-FSDevel <linux-fsdevel@vger.kernel.org>,
        Linux Kernel <linux-kernel@vger.kernel.org>,
        Dharmendra Singh <dsingh@ddn.com>
Content-Type: text/plain; charset="UTF-8"
Precedence: bulk

On Fri, 22 Apr 2022 at 17:46, Bernd Schubert <bernd.schubert@fastmail.fm> wrote:
>
> [I removed the failing netapp/zufs CCs]
>
> On 4/22/22 14:25, Miklos Szeredi wrote:
> > On Mon, 28 Mar 2022 at 15:21, Bernd Schubert <bschubert@ddn.com> wrote:
> >>
> >> I would like to discuss the user thread wake up latency in
> >> fuse_dev_do_read(). Profiling fuse shows there is room for improvement
> >> regarding memory copies and splice. The basic profiling with flame graphs
> >> didn't reveal, though, why fuse is so much
> >> slower (with an overlay file system) than just accessing the underlying
> >> file system directly and also didn't reveal why a single threaded fuse
> >> uses less than 100% cpu, with the application on top of use also using
> >> less than 100% cpu (simple bonnie++ runs with 1B files).
> >> So I started to suspect the wait queues and indeed, keeping the thread
> >> that reads the fuse device for work running for some time gives quite
> >> some improvements.
> >
> > Might be related: I experimented with wake_up_sync() that didn't meet
> > my expectations.  See this thread:
> >
> > https://lore.kernel.org/all/1638780405-38026-1-git-send-email-quic_pragalla@quicinc.com/#r
> >
> > Possibly fuse needs some wake up tweaks due to its special scheduling
> > requirements.
>
> Thanks I will look at that as well. I have a patch with spinning and
> avoid of thread wake  that is almost complete and in my (still limited)
> testing almost does not take more CPU and improves meta data / bonnie
> performance in between factor ~1.9 and 3, depending on in which
> performance mode the cpu is.
>
> https://github.com/aakefbs/linux/commits/v5.17-fuse-scheduling3
>
> Missing is just another option for wake-queue-size trigger and handling
> of signals. Should be ready once I'm done with my other work.

Trying to understand what is being optimized here...  does the
following correctly describe your use case?

- an I/O thread is submitting synchronous requests (direct I/O?)

- the fuse thread always goes to sleep, because the request queue is
empty (there's always a single request on the queue)

- with this change the fuse thread spins for a jiffy before going to
sleep, and by that time the I/O thread will submit a new sync request.

- the I/O thread does not spin while the the fuse thread is processing
the request, so it still goes to sleep.

Thanks,
Miklos