Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Date:   Sat, 29 Jan 2022 12:24:47 +0800
From:   Tony Lu <tonylu@linux.alibaba.com>
To:     Guangguan Wang <guangguan.wang@linux.alibaba.com>
Cc:     Stefan Raspl <raspl@linux.ibm.com>, kgraul@linux.ibm.com,
        linux-s390@vger.kernel.org, netdev@vger.kernel.org,
        linux-kernel@vger.kernel.org, kuba@kernel.org, davem@davemloft.net
Subject: Re: [RFC PATCH net-next] net/smc: Introduce receive queue flow
 control support
Message-ID: <YfTBj6o8fpcWAfpL@TonyMac-Alibaba>
Reply-To: Tony Lu <tonylu@linux.alibaba.com>
References: <20220120065140.5385-1-guangguan.wang@linux.alibaba.com>
 <1f13f001-e4d7-fdcd-6575-caa1be1526e1@linux.ibm.com>
 <a297c8cf-384c-2184-aabb-49ee32476d99@linux.alibaba.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <a297c8cf-384c-2184-aabb-49ee32476d99@linux.alibaba.com>
Precedence: bulk

On Sat, Jan 29, 2022 at 11:43:22AM +0800, Guangguan Wang wrote:
> 
> Yes, there are alternative ways, as RNR caused by the missmatch of send rate and receive rate, which means sending too fast
> or receiving too slow. What I have done in this patch is to backpressure the sending side when sending too fast.
> 
> Another solution is to process and refill the receive queue as quickly as posibble, which requires no protocol-level change. 
> The fllowing modifications are needed:
> - Enqueue cdc msgs to backlog queues instead of processing in rx tasklet. llc msgs remain unchanged.

It's a good idea to use backlog to free the work in softirq. Rx backlog
can help move the heavy logic out of softirq, and let extra kthread or
workqueue to handle it, then let kernel scheduler to deal with the
balance between userspace process and kthread.

There are two things to be careful, one for introducing more latency,
this should trade off latency and throughput, the other for backlog full.

> - A mempool is needed as cdc msgs are processed asynchronously. Allocate new receive buffers from mempool when refill receive queue.

Yes, we need a elastically expanding RX buffer, also elastically
shrinking. This looks like tcp_mem with tree elements to limit the
memory usage. We also need to free memory automatically, based on memcg
pressure is a good idea.

> - Schedule backlog queues to other cpus, which are calculated by 4-tuple or 5-tuple hash of the connections, to process the cdc msgs,
>   in order to reduce the usage of the cpu where rx tasklet runs on.

I am wondering if it is need for now. In general, it should spread the
CPU usage to different cores. The memory usage or CPU usage which one
will reach its limitation before trigger RNR. Maybe there should some
data to support it?

Thank you,
Tony Lu