Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
MIME-Version: 1.0
References: <20201112140752.1554-1-rach4x0r@gmail.com> <5a954c4e-aa84-834d-7d04-0ce3545d45c9@kernel.dk>
 <da0c7aea-d917-4f3a-5136-89c30d12ba1f@grimberg.me> <fd12993a-bcb7-7b45-5406-61da1979d49d@kernel.dk>
 <10993ce4-7048-a369-ea44-adf445acfca7@grimberg.me> <c4cb66f6-8a66-7973-dc03-0f4f61d0a1e4@kernel.dk>
 <cbe18a3d-8a6b-e775-81bb-3b3f11045183@grimberg.me> <26a1cd20-6b25-eaa6-7ab6-ba7f5afaf6dd@kernel.dk>
 <81cdcb58-9a23-8192-6213-7f2408a3b8ee@grimberg.me> <CAKeUqKKHg1wD19pnwJEd8whubnuGVic_ZhDjebaq3kKmY9TtsQ@mail.gmail.com>
 <CAKeUqKK3yykq8LNv1CCHZTHSz1=bzBaCwVQmi6bhpbYzqVJsqQ@mail.gmail.com>
In-Reply-To: <CAKeUqKK3yykq8LNv1CCHZTHSz1=bzBaCwVQmi6bhpbYzqVJsqQ@mail.gmail.com>
From:   Rachit Agarwal <rach4x0r@gmail.com>
Date:   Mon, 11 Jan 2021 13:15:28 -0500
Message-ID: <CAKeUqKJKXP8iJY1x=wXc7Vz+qQPhP=MaUmy71x2f2H_YdHf_8w@mail.gmail.com>
Subject: Re: [PATCH] iosched: Add i10 I/O Scheduler
To:     Sagi Grimberg <sagi@grimberg.me>
Cc:     Jens Axboe <axboe@kernel.dk>, Christoph Hellwig <hch@lst.de>,
        linux-block@vger.kernel.org, linux-nvme@lists.infradead.org,
        linux-kernel@vger.kernel.org, Keith Busch <kbusch@kernel.org>,
        Ming Lei <ming.lei@redhat.com>,
        Jaehyun Hwang <jaehyun.hwang@cornell.edu>,
        Qizhe Cai <qc228@cornell.edu>,
        Midhul Vuppalapati <mvv25@cornell.edu>,
        Sagi Grimberg <sagi@lightbitslabs.com>,
        Rachit Agarwal <ragarwal@cornell.edu>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk

[Resending the last message] Happy 2021 everyone!

> Dear all:
>
> Hope you are all well.
>
> Sagi and I were wondering if you have any additional feedback on the
> updated patch? (@Ming?) We have been receiving a lot of
> interest/questions from industry on incorporation of i10 in the
> kernel. If people do not have additional feedback, it would be nice to
> move this forward.
>
> Looking forward to hearing from you!
> ~Rachit
>
> On Sat, Nov 28, 2020 at 12:49 AM Rachit Agarwal <rach4x0r@gmail.com> wrot=
e:
> >
> >
> >
> > On Fri, Nov 13, 2020 at 4:56 PM Sagi Grimberg <sagi@grimberg.me> wrote:
> >>
> >>
> >> >>>> But if you think this has a better home, I'm assuming that the gu=
ys
> >> >>>> will be open to that.
> >> >>>
> >> >>> Also see the reply from Ming. It's a balancing act - don't want to=
 add
> >> >>> extra overhead to the core, but also don't want to carry an extra
> >> >>> scheduler if the main change is really just variable dispatch batc=
hing.
> >> >>> And since we already have a notion of that, seems worthwhile to ex=
plore
> >> >>> that venue.
> >> >>
> >> >> I agree,
> >> >>
> >> >> The main difference is that this balancing is not driven from devic=
e
> >> >> resource pressure, but rather from an assumption of device specific
> >> >> optimization (and also with a specific optimization target), hence =
a
> >> >> scheduler a user would need to opt-in seemed like a good compromise=
.
> >> >>
> >> >> But maybe Ming has some good ideas on a different way to add it..
> >> >
> >> > So here's another case - virtualized nvme. The commit overhead is
> >> > suitably large there that performance suffers quite a bit, similarly=
 to
> >> > your remote storage case. If we had suitable logic in the core, then=
 we
> >> > could easily propagate this knowledge when setting up the queue. The=
n it
> >> > could happen automatically, without needing a configuration to switc=
h to
> >> > a specific scheduler.
> >>
> >> Yes, these use-cases share characteristics. I'm not at all opposed to
> >> placing this in the core. I do think that in order to put something li=
ke
> >> this in the core, the bar needs to be higher such that an optimization
> >> target cannot be biased towards a workload (i.e. needs to be adaptive)=
.
> >>
> >> I'm still not sure how we would build this on top of what we already
> >> have as it is really centered around device being busy (which is not
> >> the case for nvme), but I didn't put enough thought into it yet.
> >>
> >
> > Dear all:
> >
> > Thanks, again, for the very constructive decisions.
> >
> > I am writing back with quite a few updates:
> >
> > 1. We have now included a detailed comparison of i10 scheduler with Kyb=
er with NVMe-over-TCP (https://github.com/i10-kernel/upstream-linux/blob/ma=
ster/i10-evaluation.pdf). In a nutshell, when operating with NVMe-over-TCP,=
 i10 demonstrates the core tradeoff: higher latency, but also higher throug=
hput. This seems to be the core tradeoff exposed by i10.
> >
> > 2. We have now implemented an adaptive version of i10 I/O scheduler, th=
at uses the number of outstanding requests at the time of batch dispatch (a=
nd whether the dispatch was triggered by timeout or not) to adaptively set =
the batching size. The new results (https://github.com/i10-kernel/upstream-=
linux/blob/master/i10-evaluation.pdf) show that i10-adaptive further improv=
es performance for low loads, while keeping the performance for high loads.=
 IMO, there is much to do on designing improved adaptation algorithms.
> >
> > 3. We have now updated the i10-evaluation document to include results f=
or local storage access. The core take-away here is that i10-adaptive can a=
chieve similar throughput and latency at low loads and at high loads when c=
ompared to noop, but still requires more work for lower loads. However, giv=
en that the tradeoff exposed by i10 scheduler is particularly useful for re=
mote storage devices (and as Jens suggested, perhaps for virtualized local =
storage access), I do agree with Sagi -- I think we should consider includi=
ng it in the core, since this may be useful for a broad range of new use ca=
ses.
> >
> > We have also created a second version of the patch that includes these =
updates: https://github.com/i10-kernel/upstream-linux/blob/master/0002-iosc=
hed-Add-i10-I-O-Scheduler.patch
> >
> > As always, thank you for the constructive discussion and I look forward=
 to working with you on this.
> >
> > Best,
> > ~Rachit
> >