Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <7ae922eb-511b-f5af-04fe-f68cd5b19237@talpey.com>
Date:   Fri, 9 Jun 2023 16:31:18 -0400
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101
 Thunderbird/102.11.2
Subject: Re: [PATCH v1 4/4] SUNRPC: Optimize page release in svc_rdma_sendto()
Content-Language: en-US
To:     Chuck Lever <cel@kernel.org>, linux-nfs@vger.kernel.org
Cc:     Chuck Lever <chuck.lever@oracle.com>, linux-rdma@vger.kernel.org
References: <168607259937.2076.15447551371235387735.stgit@manet.1015granger.net>
 <168607282316.2076.15999420482159168604.stgit@manet.1015granger.net>
From:   Tom Talpey <tom@talpey.com>
In-Reply-To: <168607282316.2076.15999420482159168604.stgit@manet.1015granger.net>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?RjNCclNCTzlTU1VBZ0ZvSWVaNGJtK2g3VTA2aXlVbUVWZVdOYW02Wk92OEtG?=
 =?utf-8?B?ekpWbUpQOHZIWkM0S3B1cFNGcVR4WjEzTExDMGNJWk4vMGp2OGVzWTc5QmJZ?=
 =?utf-8?B?R1lHaVNKSWVucXViK2s0eWpVZlZLekp5NzQvaW1tY0JIYlhWYUpKaW1JdjdM?=
 =?utf-8?B?NWJGVzRVbHZvQ3JIcnFmajE0UlRZcHhPTDY3eUxUZ0NLRisxYkp4NmRkUmF5?=
 =?utf-8?B?NWlCY2FiZmp2ZktpczRFWTVKYmtzZy8vbkJUNWlseTh3U3pMaUpNRU51cURm?=
 =?utf-8?B?cWZBNnoxM09HSHhiMnFUSlBJNGZlQUJPQ0cyaVJmcSs3dVg2N29YdFpFSUI5?=
 =?utf-8?B?em5uNGVmZUNkWUdiM1QxL2FYa2dNaEw3QjNpZmVFcW9DMTAweUxJUTVWM05X?=
 =?utf-8?B?VkU2ckU1NWpIQ2UvSmhnWDlySEFCcDFQTURUQndEQ0lIaVJQM1ZSSXBBZ2dC?=
 =?utf-8?B?ZEl4ODU2T1JwNGhPTzZ1Nm1lbmRyOE52UkhZUUpYN1JQdHNwTlk5S0EyTFpR?=
 =?utf-8?B?RGd1Rk9sajZPUlh3RTZSRmJlVm91MTFNbDdiZ082dGJBeTVrcjJYTERwaWRR?=
 =?utf-8?B?bStoVzlXOUVBaGRuOXRyUVBrYnZhWithZHZjWkIzZEhQREZYM2ZmQVVmNHRF?=
 =?utf-8?B?aWhuZkI0VG1iVlpMTU5PbzlVWFZheTRBaVZ0bmZEa2dZN1FvNFJGZFVPVjg4?=
 =?utf-8?B?blZobEV1WUtacFFRYmZtVWQ0YUVkUlFMek5JWS9QblF2M1VSRytRUXYvQzBm?=
 =?utf-8?B?S1BOMWhkdVBTQk0rS0lJZjdsNzlnUkZOQkRwdFhFTFArbFpCaUViV1laN3VN?=
 =?utf-8?B?QTE3UXN1bjZUTUp6aUlyUFh2dEpCblR0OHU1ZGxnWEFwc1EzUlZoZmI2V0xH?=
 =?utf-8?B?T1UvUlhPeTRxU3U2eURDbkVmOWNZejFicG9OdEZ2RW1NcmlrVnBBYTVCQ1Z4?=
 =?utf-8?B?Y1Z5T1pnNHV4ZkppYTNFSGZrWG1oRDhaLzg2STdFUm5FbUg1anZhUXk0VXhQ?=
 =?utf-8?B?RXpMcVlUNXlNa0t3S2czWUdCbmFVL003R3hnSk1zOFdNdXhhL293QlVnRzBL?=
 =?utf-8?B?dXJwQjhtSjZqWnExSTk4ZVRWaW1ZQmlVclo0aUs5WVZNVk03dTZTbFFMY2JM?=
 =?utf-8?B?SHR0K3p5bDUyU09uelpSaFB3T1FZYjVvTVkyRXJsKy9YTnRDQThqTHYvWWhS?=
 =?utf-8?B?MmM2dFNNbXFxam1JQ2VvaHRhTmU1aWRnTDVLaUlVNUNtdzN5S0lXMERESWwr?=
 =?utf-8?B?V3JsdlJqWWRsSUI5bWtYaFpqRUZjT3Z1QTEyVUVvN05Ham0wZmkyY0V0MFhT?=
 =?utf-8?B?U3dJTi9kc3dCamlkSTZmRkd3Ky9PWXNGUVJpQlhnUXRuWkpkRnkrZG56NU14?=
 =?utf-8?B?SCtGdFdBOEZkL2ZGUEp6d3ZEN1dKYnhHVlZrc1lrZDBTNXJiWTd2Q0JwN0hV?=
 =?utf-8?B?Q2M1czdDY0NLeUdUTWRaK2VoU2lzWHZzQzJhYUYyL21ZZlBBWkpjMXQvbFZB?=
 =?utf-8?B?Nmtvdk9XVy9yWWE1a3J6b0pNNk01YzRMb3hJMHpjZ0RERzV6SGxld2JtQXhC?=
 =?utf-8?B?Ky9kbmF0c0RYWXdPckh1U3NhVFk4MWVrVXpHZWRUNXJJQytFYXEyKys5MVRq?=
 =?utf-8?B?Q2FkR3RPcDhNNFdMUHliMG0wVklIOHRrSUV1b04vcjFOWDhLczh2S1dYMHZv?=
 =?utf-8?B?Z2hvMmNPV2gzdWFzYWdld3c1Tm9ESWJlMHNkN2o5c2IrcDNtOXI1MWpHZWtT?=
 =?utf-8?B?UGJxZ0YvaE5PYlVLRnBUZnFPV040ZnhnUHZxblRoa0VHZkVuT3pWeFNBU29k?=
 =?utf-8?B?L1U4N3kyYnJlSHVSeW51VkplNnpjZWE1d3FSWVdPYlBtSUsvdEtMdkJ0K0NZ?=
 =?utf-8?B?eitYVkZaaE43Z0lZL0gzYnRlcmpoMWFscGpsbi9oV09EWUZxVk9PQW1uYWIz?=
 =?utf-8?B?ajdWaE96dUo2aExQejh6Z2ZmUXA2b2EvQksvQk5MUXd4WXVvaVNzNFlwejBJ?=
 =?utf-8?B?UnJUSkFoNGFmVERDNncweE9ZanQyMGRsanQrdkUwakhZb1lpWWNOYWswK08w?=
 =?utf-8?B?MFg5ZkNYTmt2YXlZeW4rWU9DR0E0UGZRUEloQmtzSHJSTGwra0l5K0hmYWQ2?=
 =?utf-8?Q?wMZiJMq7bgye/lbbLb73DyCpx?=
X-OriginatorOrg: talpey.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 5940748a-47a1-499d-5edc-08db69287c11
X-MS-Exchange-CrossTenant-AuthSource: SN6PR01MB4445.prod.exchangelabs.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Jun 2023 20:31:20.7481
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 2b2dcae7-2555-4add-bc80-48756da031d5
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: WYgJnfPejKKWYw4WH1W6GilGn04I+Xtk8Ywe+I6kh61Pp4KsL/6AL82qbe6SmVGp
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR01MB8025
X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,NICE_REPLY_A,
        RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_PASS,
        T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org

On 6/6/2023 1:33 PM, Chuck Lever wrote:
> From: Chuck Lever <chuck.lever@oracle.com>
> 
> Now that we have bulk page allocation and release APIs, it's more
> efficient to use those than it is for nfsd threads to wait for send
> completions. Previous patches have eliminated the calls to
> wait_for_completion() and complete(), in order to avoid scheduler
> overhead.

Makes sense to me. Have you considered changing the send cq arming
to avoid completion notifications and simply poll them from the cq
at future convenient/efficient times?

Reviewed-by: Tom Talpey <tom@talpey.com>

Tom.

> Now release pages-under-I/O in the send completion handler using
> the efficient bulk release API.
> 
> I've measured a 7% reduction in cumulative CPU utilization in
> svc_rdma_sendto(), svc_rdma_wc_send(), and svc_xprt_release(). In
> particular, using release_pages() instead of complete() cuts the
> time per svc_rdma_wc_send() call by two-thirds. This helps improve
> scalability because svc_rdma_wc_send() is single-threaded per
> connection.
> 
> Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
> ---
>   net/sunrpc/xprtrdma/svc_rdma_sendto.c |    4 ++--
>   1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/net/sunrpc/xprtrdma/svc_rdma_sendto.c b/net/sunrpc/xprtrdma/svc_rdma_sendto.c
> index 1ae4236d04a3..24228f3611e8 100644
> --- a/net/sunrpc/xprtrdma/svc_rdma_sendto.c
> +++ b/net/sunrpc/xprtrdma/svc_rdma_sendto.c
> @@ -236,8 +236,8 @@ void svc_rdma_send_ctxt_put(struct svcxprt_rdma *rdma,
>   	struct ib_device *device = rdma->sc_cm_id->device;
>   	unsigned int i;
>   
> -	for (i = 0; i < ctxt->sc_page_count; ++i)
> -		put_page(ctxt->sc_pages[i]);
> +	if (ctxt->sc_page_count)
> +		release_pages(ctxt->sc_pages, ctxt->sc_page_count);
>   
>   	/* The first SGE contains the transport header, which
>   	 * remains mapped until @ctxt is destroyed.
> 
> 
>