Message-ID: <1515702883.3039.27.camel@arista.com>
Subject: Re: [RFC 1/2] softirq: Defer net rx/tx processing to ksoftirqd
 context
From: Dmitry Safonov <dima@arista.com>
To: Linus Torvalds <torvalds@linux-foundation.org>,
        Eric Dumazet <edumazet@google.com>
Cc: Peter Zijlstra <peterz@infradead.org>,
        Frederic Weisbecker <frederic@kernel.org>,
        LKML <linux-kernel@vger.kernel.org>,
        Dmitry Safonov <0x7f454c46@gmail.com>,
        Andrew Morton <akpm@linux-foundation.org>,
        David Miller <davem@davemloft.net>,
        Frederic Weisbecker <fweisbec@gmail.com>,
        Hannes Frederic Sowa <hannes@stressinduktion.org>,
        Ingo Molnar <mingo@kernel.org>,
        "Levin, Alexander (Sasha Levin)" <alexander.levin@verizon.com>,
        Paolo Abeni <pabeni@redhat.com>,
        "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>,
        Radu Rendec <rrendec@arista.com>,
        Rik van Riel <riel@redhat.com>,
        Stanislaw Gruszka <sgruszka@redhat.com>,
        Thomas Gleixner <tglx@linutronix.de>,
        Wanpeng Li <wanpeng.li@hotmail.com>
Date: Thu, 11 Jan 2018 20:34:43 +0000
In-Reply-To: <CA+55aFwA1skftujPWmuQJq_s-EG=PP+mFiuUiZNBar=deYNu3Q@mail.gmail.com>
References: <20180109133623.10711-1-dima@arista.com>
         <20180109133623.10711-2-dima@arista.com>
         <CANn89iK3M97MN0Pf3nXb+UAqqhUWOdSthHRBTYCwP75Ax_hO8Q@mail.gmail.com>
         <1515620880.3350.44.camel@arista.com>
         <CA+55aFyKKt4_5RT9RT8ZH-W26hC8=AvRYf8YxBm98dGSWwFs8g@mail.gmail.com>
         <20180111032232.GA11633@lerouge>
         <CA+55aFx_3zwQJ0YbDCL4YxpWEWhcEZfJnn42LzWBWDi3h1VdGA@mail.gmail.com>
         <20180111044456.GC11633@lerouge> <1515681091.3039.21.camel@arista.com>
         <CANn89i+mVmzrZ14Kttt=J0wsDOMHhm8CHiMRLQwEZXMxiVpftg@mail.gmail.com>
         <20180111163204.GE6176@hirez.programming.kicks-ass.net>
         <CA+55aFwc3CP-sKOyVvaLab3azmr3LnPfADnGJXDcxYz9dT75=A@mail.gmail.com>
         <CANn89i+ZTLtA5ZLRAbCgM_Cx-2xiwRbDXM4x=-QiM78r5ptcqg@mail.gmail.com>
         <CA+55aFyZPzkjwkLXWWXp3KUfLD7MUtGxSu1Q6vc0O5i9Ea6ZKw@mail.gmail.com>
         <CANn89iJzekwx_Hs0t0O==+gwAfqMyVHBg=gemayZZJXb4bYJdQ@mail.gmail.com>
         <CA+55aFx+1tFpnLBXjZKoYsMMVPakeP8nycyfMpF7agUXz_kGkQ@mail.gmail.com>
         <CANn89i+ehJg_7YfOCicgv_EuQWR6Xn7GHi+g5=atigeXDeNMHw@mail.gmail.com>
         <CA+55aFwA1skftujPWmuQJq_s-EG=PP+mFiuUiZNBar=deYNu3Q@mail.gmail.com>
Content-Type: text/plain; charset="UTF-8"
Mime-Version: 1.0
Content-Transfer-Encoding: 7bit
Sender: linux-kernel-owner@vger.kernel.org

On Thu, 2018-01-11 at 12:22 -0800, Linus Torvalds wrote:
> On Thu, Jan 11, 2018 at 12:16 PM, Eric Dumazet <edumazet@google.com>
> wrote:
> > 
> > Note that when I implemented TCP Small queues, I did experiments
> > between
> > using a work queue or a tasklet, and workqueues added unacceptable
> > P99
> > latencies, when many user threads are competing with kernel
> > threads.
> 
> Yes.
> 
> So I think one solution might be to have a hybrid system, where we do
> the softirq's synchronously normally (which is what you really want
> for good latency).
> 
> But then fall down on a threaded model - but that fallback case
> should
> be per-softirq, not global. So if one softirq uses a lot of CPU time,
> that shouldn't affect the latency of other softirqs.
> 
> So maybe we could get rid of the per-cpu ksoftirqd entirely, and
> replace it with with per-cpu and per-softirq workqueues?
> 
> Would something like that sound sane?
> 
> Just a SMOP/SMOT (small matter of programming/testing).

I could try to write a PoC for that..
What should be the trigger to fall into workqueue?
How to tell if there're too many softirqs of the kind?
Current logic with if (pending) in the end of __do_softirq()
looks working selectively..
It looks to be still possible to starve a cpu.

-- 
Thanks,
             Dmitry