Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Message-ID: <6f1b9813-483b-ea2e-f6da-c349edd34003@linux.alibaba.com>
Date:   Sat, 22 Jan 2022 00:21:45 +0800
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0)
 Gecko/20100101 Thunderbird/91.5.0
Subject: Re: [RFC PATCH net-next] net/smc: Introduce receive queue flow
 control support
Content-Language: en-US
To:     dust.li@linux.alibaba.com, kgraul@linux.ibm.com,
        davem@davemloft.net, kuba@kernel.org
Cc:     linux-s390@vger.kernel.org, netdev@vger.kernel.org,
        linux-kernel@vger.kernel.org
References: <20220120065140.5385-1-guangguan.wang@linux.alibaba.com>
 <20220120095130.GB41938@linux.alibaba.com>
From:   Guangguan Wang <guangguan.wang@linux.alibaba.com>
In-Reply-To: <20220120095130.GB41938@linux.alibaba.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
Precedence: bulk

On 2022/1/20 17:51, dust.li wrote:
> On Thu, Jan 20, 2022 at 02:51:40PM +0800, Guangguan Wang wrote:
>> This implement rq flow control in smc-r link layer. QPs
>> communicating without rq flow control, in the previous
>> version, may result in RNR (reveive not ready) error, which
>> means when sq sends a message to the remote qp, but the
>> remote qp's rq has no valid rq entities to receive the message.
>> In RNR condition, the rdma transport layer may retransmit
>> the messages again and again until the rq has any entities,
>> which may lower the performance, especially in heavy traffic.
>> Using credits to do rq flow control can avoid the occurrence
>> of RNR.
> 
> I'm wondering if SRQ can be used to solve this problem ?
> 
> One of my concern on credit-base flow control is if the RTT is
> a bit longer, we may have to wait RTT/2 for peer to grant us credit
> before we can really send more data. That may decrease the maximium
> bandwidth we can achive in this case.

Longer RTT can result in more inflight messages and increase
the announcement latency indeed.

The following items are used in this patch to reduce the pact
of this situation.
- More rqe. (average 2 credits per smc_connection now, longer RTT is
  a good case for me to check whether an average of 2 is enough. As
  each additional rqe only increases the memory by 104 Bytes,
  SRQ may be an icing on the cake option to reduce memory usage)
- Announce frequenly. (credits carried by every cdc msg)
- Avoid credit accumulation. (announce as soon as the low watermark(1/3 rq entities) is reached)