Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Subject: Re: [RFC] net;sched: Try to find idle cpu for RPS to handle packets
To:     Eric Dumazet <edumazet@google.com>
Cc:     Peter Zijlstra <peterz@infradead.org>,
        David Miller <davem@davemloft.net>,
        Daniel Borkmann <daniel@iogearbox.net>, tom@quantonium.net,
        netdev <netdev@vger.kernel.org>,
        LKML <linux-kernel@vger.kernel.org>
References: <153736009982.24033.13696245431713246950.stgit@localhost.localdomain>
 <CANn89iJ1UnwLFv5+AwXgeb1BUhYg8UJYTJLtiipavJee+2SWxQ@mail.gmail.com>
From:   Kirill Tkhai <ktkhai@virtuozzo.com>
Message-ID: <2fdf2bd7-1cc4-a1e1-15c2-e2badfcd4d59@virtuozzo.com>
Date:   Wed, 19 Sep 2018 18:41:10 +0300
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101
 Thunderbird/60.0
MIME-Version: 1.0
In-Reply-To: <CANn89iJ1UnwLFv5+AwXgeb1BUhYg8UJYTJLtiipavJee+2SWxQ@mail.gmail.com>
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: 7bit
Received-SPF: None (protection.outlook.com: virtuozzo.com does not designate
 permitted sender hosts)
SpamDiagnosticOutput: 1:99
SpamDiagnosticMetadata: NSPM
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Sep 2018 15:41:20.2399 (UTC)
X-MS-Exchange-CrossTenant-Network-Message-Id: 4e81f01d-f43e-482f-23bc-08d61e46590c
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 0bc7f26d-0264-416e-a6fc-8352af79c58f
X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM5PR0801MB2019
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On 19.09.2018 17:55, Eric Dumazet wrote:
> On Wed, Sep 19, 2018 at 5:29 AM Kirill Tkhai <ktkhai@virtuozzo.com> wrote:
>>
>> Many workloads have polling mode of work. The application
>> checks for incomming packets from time to time, but it also
>> has a work to do, when there is no packets. This RFC
>> tries to develop an idea to queue RPS packets on idle
>> CPU in the the L3 domain of the consumer, so backlog
>> processing of the packets and the application can execute
>> in parallel.
>>
>> We require this in case of network cards does not
>> have enough RX queues to cover all online CPUs (this seems
>> to be the most cards), and  get_rps_cpu() actually chooses
>> remote cpu, and SMP interrupt is sent. Here we may try
>> our best, and to find idle CPU nearly the consumer's CPU.
>> Note, that in case of consumer works in poll mode and it
>> does not waits for incomming packets, its CPU will be not
>> idle, while CPU of a sleeping consumer may be idle. So,
>> not polling consumers will still be able to have skb
>> handled on its CPU.
>>
>> In case of network card has many queues, the device
>> interrupts will come on consumer's CPU, and this patch
>> won't try to find idle cpu for them.
>>
>> I've tried simple netperf test for this:
>> netserver -p 1234
>> netperf -L 127.0.0.1 -p 1234 -l 100
>>
>> Before:
>>  87380  16384  16384    100.00   60323.56
>>  87380  16384  16384    100.00   60388.46
>>  87380  16384  16384    100.00   60217.68
>>  87380  16384  16384    100.00   57995.41
>>  87380  16384  16384    100.00   60659.00
>>
>> After:
>>  87380  16384  16384    100.00   64569.09
>>  87380  16384  16384    100.00   64569.25
>>  87380  16384  16384    100.00   64691.63
>>  87380  16384  16384    100.00   64930.14
>>  87380  16384  16384    100.00   62670.15
>>
>> The difference between best runs is +7%,
>> the worst runs differ +8%.
>>
>> What do you think about following somehow in this way?
> 
> Hi Kirill
> 
> In my experience, scheduler has a poor view of softirq processing
> happening on various cpus.
> A cpu spending 90% of its cycles processing IRQ might be considered 'idle'

Yes, in case of there is softirq on top of irq_exit(), the cpu is not
considered as busy. But after MAX_SOFTIRQ_TIME (=2ms), ksoftirqd are
waken up to execute the work in process context, and the processor is
considered as !idle. 2ms is 2 timer ticks in case of HZ=1000. So, we
don't restart softirq in case of it was executed for more then 2ms.

The similar way, single net_rx_action() can't be executed longer
than 2ms.

Having 90% load in softirq (called on top of irq_exit()) should be
very unlikely situation, when there are too many interrupts with small
amount of work, which related softirq calls are doing for each of them.
I think it had be a problem even in plain napi case, since it would
worked not like expected.

But anyway. You worry, that during handling of next portion of skbs,
we find that previous portion of skbs already woken ksoftirqd, and
we don't see this cpu as idle? Yeah, then we'll try to change cpu,
and this is not what we want. We want to continue use the cpu, where
previous portion was handler. Hm, not so fast I'll answer, but certainly,
this may be handled somehow in more creative way.

> So please run a real workload (it is _very_ uncommon anyone set up RPS
> on lo interface !)
>
> Like 400 or more concurrent netperf -t TCP_RR on a 10Gbit NIC.

Yeah, it's just a simulation of a single irq nic. I'll try on something
more real hardware.

How do you execute such the tests? I don't see the appropriate parameter
of netperf. Does this mean just to start 400 copies of netperf? How is
to aggregate their results in this case? 
 
> Thanks.
> 
> PS: Idea of playing with L3 domains is interesting, I have personally
> tried various strategies in the past but none of them
> demonstrated a clear win.

Thanks,
Kirill