Message-ID: <1515780530.21898.40.camel@gmx.de>
Subject: Re: [RFC 1/2] softirq: Defer net rx/tx processing to ksoftirqd
 context
From: Mike Galbraith <efault@gmx.de>
To: Frederic Weisbecker <frederic@kernel.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>,
        Eric Dumazet <edumazet@google.com>,
        Peter Zijlstra <peterz@infradead.org>,
        Dmitry Safonov <dima@arista.com>,
        LKML <linux-kernel@vger.kernel.org>,
        Dmitry Safonov <0x7f454c46@gmail.com>,
        Andrew Morton <akpm@linux-foundation.org>,
        David Miller <davem@davemloft.net>,
        Frederic Weisbecker <fweisbec@gmail.com>,
        Hannes Frederic Sowa <hannes@stressinduktion.org>,
        Ingo Molnar <mingo@kernel.org>,
        "Levin, Alexander (Sasha Levin)" <alexander.levin@verizon.com>,
        Paolo Abeni <pabeni@redhat.com>,
        "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>,
        Radu Rendec <rrendec@arista.com>,
        Rik van Riel <riel@redhat.com>,
        Stanislaw Gruszka <sgruszka@redhat.com>,
        Thomas Gleixner <tglx@linutronix.de>,
        Wanpeng Li <wanpeng.li@hotmail.com>
Date: Fri, 12 Jan 2018 19:08:50 +0100
In-Reply-To: <20180112170033.GD1950@lerouge>
References: <CA+55aFwc3CP-sKOyVvaLab3azmr3LnPfADnGJXDcxYz9dT75=A@mail.gmail.com>
         <CANn89i+ZTLtA5ZLRAbCgM_Cx-2xiwRbDXM4x=-QiM78r5ptcqg@mail.gmail.com>
         <CA+55aFyZPzkjwkLXWWXp3KUfLD7MUtGxSu1Q6vc0O5i9Ea6ZKw@mail.gmail.com>
         <CANn89iJzekwx_Hs0t0O==+gwAfqMyVHBg=gemayZZJXb4bYJdQ@mail.gmail.com>
         <CA+55aFx+1tFpnLBXjZKoYsMMVPakeP8nycyfMpF7agUXz_kGkQ@mail.gmail.com>
         <CANn89i+ehJg_7YfOCicgv_EuQWR6Xn7GHi+g5=atigeXDeNMHw@mail.gmail.com>
         <CA+55aFwA1skftujPWmuQJq_s-EG=PP+mFiuUiZNBar=deYNu3Q@mail.gmail.com>
         <1515734588.10188.36.camel@gmx.de> <20180112145819.GC1950@lerouge>
         <1515770104.21898.13.camel@gmx.de> <20180112170033.GD1950@lerouge>
Content-Type: text/plain; charset="ISO-8859-15"
Mime-Version: 1.0
Content-Transfer-Encoding: 8BIT
Sender: linux-kernel-owner@vger.kernel.org

On Fri, 2018-01-12 at 18:00 +0100, Frederic Weisbecker wrote:
> On Fri, Jan 12, 2018 at 04:15:04PM +0100, Mike Galbraith wrote:
> > On Fri, 2018-01-12 at 15:58 +0100, Frederic Weisbecker wrote:
> > > On Fri, Jan 12, 2018 at 06:23:08AM +0100, Mike Galbraith wrote:
> > > > On Thu, 2018-01-11 at 12:22 -0800, Linus Torvalds wrote:
> > > > > On Thu, Jan 11, 2018 at 12:16 PM, Eric Dumazet <edumazet@google.com> wrote:
> > > > > >
> > > > > > Note that when I implemented TCP Small queues, I did experiments between
> > > > > > using a work queue or a tasklet, and workqueues added unacceptable P99
> > > > > > latencies, when many user threads are competing with kernel threads.
> > > > > 
> > > > > Yes.
> > > > > 
> > > > > So I think one solution might be to have a hybrid system, where we do
> > > > > the softirq's synchronously normally (which is what you really want
> > > > > for good latency).
> > > > > 
> > > > > But then fall down on a threaded model - but that fallback case should
> > > > > be per-softirq, not global. So if one softirq uses a lot of CPU time,
> > > > > that shouldn't affect the latency of other softirqs.
> > > > > 
> > > > > So maybe we could get rid of the per-cpu ksoftirqd entirely, and
> > > > > replace it with with per-cpu and per-softirq workqueues?
> > > > 
> > > > How would that be better than what RT used to do, and I still do for my
> > > > RT kernels via boot option, namely split ksoftirqd into per-softirq
> > > > threads.
> > > 
> > > Workqueue are probably more simple. Unless you need to set specific prios
> > > to your ksoftirqds? Not sure if that's tunable on workqueues.
> > 
> > No, you can't prioritize workqueues, and they spawn threads whenever
> > they bloody well feel like.
> > 
> > I carry a hack to give users minimal control over kthread/workqueue
> > priority. ?Very handy thing to have, especially if you're doing high
> > utilization stuff, and would prefer your box actually survive it.
> 
> How useful system_highpri_wq can be in this regard?

Not at all. ?You could make the system protect itself via boosting, but
that just makes a noisy mess.

	-Mike