MIME-Version: 1.0
In-Reply-To: <CANn89i+ehJg_7YfOCicgv_EuQWR6Xn7GHi+g5=atigeXDeNMHw@mail.gmail.com>
References: <20180109133623.10711-1-dima@arista.com> <20180109133623.10711-2-dima@arista.com>
 <CANn89iK3M97MN0Pf3nXb+UAqqhUWOdSthHRBTYCwP75Ax_hO8Q@mail.gmail.com>
 <1515620880.3350.44.camel@arista.com> <CA+55aFyKKt4_5RT9RT8ZH-W26hC8=AvRYf8YxBm98dGSWwFs8g@mail.gmail.com>
 <20180111032232.GA11633@lerouge> <CA+55aFx_3zwQJ0YbDCL4YxpWEWhcEZfJnn42LzWBWDi3h1VdGA@mail.gmail.com>
 <20180111044456.GC11633@lerouge> <1515681091.3039.21.camel@arista.com>
 <CANn89i+mVmzrZ14Kttt=J0wsDOMHhm8CHiMRLQwEZXMxiVpftg@mail.gmail.com>
 <20180111163204.GE6176@hirez.programming.kicks-ass.net> <CA+55aFwc3CP-sKOyVvaLab3azmr3LnPfADnGJXDcxYz9dT75=A@mail.gmail.com>
 <CANn89i+ZTLtA5ZLRAbCgM_Cx-2xiwRbDXM4x=-QiM78r5ptcqg@mail.gmail.com>
 <CA+55aFyZPzkjwkLXWWXp3KUfLD7MUtGxSu1Q6vc0O5i9Ea6ZKw@mail.gmail.com>
 <CANn89iJzekwx_Hs0t0O==+gwAfqMyVHBg=gemayZZJXb4bYJdQ@mail.gmail.com>
 <CA+55aFx+1tFpnLBXjZKoYsMMVPakeP8nycyfMpF7agUXz_kGkQ@mail.gmail.com> <CANn89i+ehJg_7YfOCicgv_EuQWR6Xn7GHi+g5=atigeXDeNMHw@mail.gmail.com>
From: Linus Torvalds <torvalds@linux-foundation.org>
Date: Thu, 11 Jan 2018 12:22:38 -0800
Message-ID: <CA+55aFwA1skftujPWmuQJq_s-EG=PP+mFiuUiZNBar=deYNu3Q@mail.gmail.com>
Subject: Re: [RFC 1/2] softirq: Defer net rx/tx processing to ksoftirqd context
To: Eric Dumazet <edumazet@google.com>
Cc: Peter Zijlstra <peterz@infradead.org>,
        Dmitry Safonov <dima@arista.com>,
        Frederic Weisbecker <frederic@kernel.org>,
        LKML <linux-kernel@vger.kernel.org>,
        Dmitry Safonov <0x7f454c46@gmail.com>,
        Andrew Morton <akpm@linux-foundation.org>,
        David Miller <davem@davemloft.net>,
        Frederic Weisbecker <fweisbec@gmail.com>,
        Hannes Frederic Sowa <hannes@stressinduktion.org>,
        Ingo Molnar <mingo@kernel.org>,
        "Levin, Alexander (Sasha Levin)" <alexander.levin@verizon.com>,
        Paolo Abeni <pabeni@redhat.com>,
        "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>,
        Radu Rendec <rrendec@arista.com>,
        Rik van Riel <riel@redhat.com>,
        Stanislaw Gruszka <sgruszka@redhat.com>,
        Thomas Gleixner <tglx@linutronix.de>,
        Wanpeng Li <wanpeng.li@hotmail.com>
Content-Type: text/plain; charset="UTF-8"
Sender: linux-kernel-owner@vger.kernel.org

On Thu, Jan 11, 2018 at 12:16 PM, Eric Dumazet <edumazet@google.com> wrote:
>
> Note that when I implemented TCP Small queues, I did experiments between
> using a work queue or a tasklet, and workqueues added unacceptable P99
> latencies, when many user threads are competing with kernel threads.

Yes.

So I think one solution might be to have a hybrid system, where we do
the softirq's synchronously normally (which is what you really want
for good latency).

But then fall down on a threaded model - but that fallback case should
be per-softirq, not global. So if one softirq uses a lot of CPU time,
that shouldn't affect the latency of other softirqs.

So maybe we could get rid of the per-cpu ksoftirqd entirely, and
replace it with with per-cpu and per-softirq workqueues?

Would something like that sound sane?

Just a SMOP/SMOT (small matter of programming/testing).

               Linus