Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   Chuck Lever III <chuck.lever@oracle.com>
To:     Tom Talpey <tom@talpey.com>
CC:     Chuck Lever <cel@kernel.org>,
        Linux NFS Mailing List <linux-nfs@vger.kernel.org>,
        "linux-rdma@vger.kernel.org" <linux-rdma@vger.kernel.org>
Subject: Re: [PATCH v1 4/4] SUNRPC: Optimize page release in svc_rdma_sendto()
Thread-Topic: [PATCH v1 4/4] SUNRPC: Optimize page release in
 svc_rdma_sendto()
Thread-Index: AQHZmJ0NA0ZICmeJcUqRKW78JwWSNa+C8XYAgAAFGgA=
Date:   Fri, 9 Jun 2023 20:49:44 +0000
Message-ID: <17F60F35-FA54-459B-BE48-07522D951675@oracle.com>
References: <168607259937.2076.15447551371235387735.stgit@manet.1015granger.net>
 <168607282316.2076.15999420482159168604.stgit@manet.1015granger.net>
 <7ae922eb-511b-f5af-04fe-f68cd5b19237@talpey.com>
In-Reply-To: <7ae922eb-511b-f5af-04fe-f68cd5b19237@talpey.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?FTLmHDXuBv2Z1jIJye/TL42PaDmf362a+LLRfDZK4HNMi7b+AppnW5e4USKC?=
 =?us-ascii?Q?1r2EguM4tJ4/Z0IkghNnIA5bczIdaouyD6nX0wMTouFTJUUa3IlayLpnZv/h?=
 =?us-ascii?Q?wwRtwozkYRQkxeMCuGAcFxk+wdIjHX/yJyUvwxPVnqg3FOmn3lFuGj2U+z+J?=
 =?us-ascii?Q?DOGsVhsmU+GLS6pHBcf/75AXNNNCgt+FkuoLN8qo6pyyUUoujgFffXTa3WbD?=
 =?us-ascii?Q?C3ruyzjesQniSvSs9AzKCHTQwkEkm6KToOMGSW1S7srFdwAEgpnB/8kSOh+Z?=
 =?us-ascii?Q?lZfeoma8jIGnLECeyMGbv2+bxEl/Rq2vmtO48T25FO/cRfweHLVeVX73Ae0+?=
 =?us-ascii?Q?NmJzfBKPgOdi2OnCELuPgFSxrUiolaEp3J1xGoVu1FEA9d73VlWaSMbP4A3s?=
 =?us-ascii?Q?LTUj164aMJw+dzGn4j4UPAzkWEHnhTpjrLS7ZuOir5RP9WZAfLsiD9rWyy0n?=
 =?us-ascii?Q?Ehk3sEduE42gGoYF5pIuUXlJSci5ebNwazCOxDffrBw7+BsaD5kysFhhzznL?=
 =?us-ascii?Q?yg9wl5kg50ORQo18UbcOnsBzvqaEjFzxPxRDv3M/N9PV9aXILvvuKILU56xd?=
 =?us-ascii?Q?kc7Cy4wfPqeyjd8bo0Z/6u0MlbtlYHSY0LU1oZieRN4/o/nqsfWGBIhU5PKk?=
 =?us-ascii?Q?+/GuXtqHuowNMtWAdbU9y+ce0/18EvgR1D0O1IaDW1DwyFgpk0LZ0SRPpdWR?=
 =?us-ascii?Q?bLnbnvD6FdnoVF096o//AqOs4ZukdXuIF7547OH/gtGC1Oib0pWSQLKZrRyN?=
 =?us-ascii?Q?CSC2YgkPLHojEXTegLJ0AC1IooFpkFZPzywR6NAdZKzLz0ZDK9l7WR5z6dLG?=
 =?us-ascii?Q?Sk9Y0PcoTDOYfd2ti1A7ER8fNhtdymq00VTF2tFmKfPRpCuJCR+gnM7JxeHI?=
 =?us-ascii?Q?6S1o5J4Y6Gj+G3POsaWkz8VfIGJnSIkVs2qtoWnoA6QjbawAUagKIfqY7Csz?=
 =?us-ascii?Q?jqJgzrXGnThYlqeq8Srh7UnAcOC1r4HDjDXh82UEK+DuPjCliXWTcYnMsz/D?=
 =?us-ascii?Q?pVAhvEpFbNINZb0+cExcFO+EzT8MyF0VIlhG7/Sgepioj1jpfRLE0PGawrQM?=
 =?us-ascii?Q?DoIzfjLSi8q6qkKHxnIDwSaLw3vtsaCwiCyBbAxy2q5oZ5uW1wkOYj7klFG1?=
 =?us-ascii?Q?0hDHVCR+hAdVmKELA6L5KF7UEWqRL/vjFHmWqsv37rBFrdiIwK1YbNYxbSGY?=
 =?us-ascii?Q?gVug8sZYPlDk8sRT3j7SFSnqogFhJ1vjryR1h3xMSfW9rF0O5JXELhA6jde/?=
 =?us-ascii?Q?GtxYP/lSHlpd/h+oV/6iIbD9SJ4w50DIg5DYzTWG6LsRrmqvLJHM3NANEXCO?=
 =?us-ascii?Q?TCR4yVoEdS25VIHJLOxQI9e2gUQ/XKPR5603volhKvcidmpNNNUTV9BAbpsH?=
 =?us-ascii?Q?Cy+d9X0LQ10AYSE259ZgXPv7nAJO1Y39kDCa7FM1j4kupnqhNh9dvqMs+Bk/?=
 =?us-ascii?Q?Da5IzNSXyuacuR1fAFbO7CF3ZWbyebirwI1ivLoTrFXgU7FqXV+xSvGhmWnU?=
 =?us-ascii?Q?Ci9WmMQi0eWAoeZH+CAdb6NIAGlmIjR1+PzV3W23LzVXsW+/ETEerqrnh5C/?=
 =?us-ascii?Q?Pl5REOT3beMbsatk48LKo5gSN4ZdwaEL1sQd7x4nPGprrQ6VwvPHlVdAaZ3c?=
 =?us-ascii?Q?Wg=3D=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-ID: <B35137E6F659F24281A1DF6F8EB38787@namprd10.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: q18xlkrH9RxOnnQEniaJro6YUJ8OllPmpMHXLNOnQH3xqpRSyc+6uiBO6zUzkQ5XxBW0MVnbSmmbwi70pr3/NKhKJX2QUXzC898MS4bPQstPz8n02152LNvakeB2+QCQkz4mQbMdtUvn1NTI9n/VgprcLg6kO7SdST0bSJj9WqlxN0dmsRK9Cm/kZCuaoOgDA2v3XkJRsdoz9baL3d1Pt0x/QoicTx3zLbDqJdki3YLyl/tuTfVYdxJDnOxne2JNcC01IMXI4pDt1X4G0lZxCVRRIkRy96UtgQH9qhiR8Xvo7d/qLkGtpmP5CqF11r4skKTx+HUHBqhkg72LbXivwdSuRoRyTgChlyaRG9Tl9htBr8G2cHXmAJte0CjAQ7fzFwKo6dXeG/jReqEIirvc03Uz0j3XqzLwZU/zgnGnUpZ8RmtKSNoeOuPv0sOWdJOcnAx00maTl7iHP0ER2Hp0qZuP15MHt1d5FSIWR7iP/twemIKTz98CXOsVIC6xuytt/Yahcq0hqBhEMVNrQhQdMMePlxJewbNLkQAxi/oVHweQqtrNzcGwzUoWL4GgbgQDBiA5jsxs0GubgMpZRiCpi1anB/MjE7fTNnzFLNbvKK+J91eQdaKzK9LJ5GS+MrSlb2CPUGb37DLgSCEs8HArrxv0Y6YbllMdDKDjVtciETnaqotj3O+k85LamiacUjFMfvPBaPoM9VWpK8A73Dgbhqaqp5QqhFh6nkgLK55/+lCMgflI6hAWCzjjPbhLZuxV2s+NdzlGp9bAZ2qIbDsUVHSHdXsoTklAQUgL8ap3LsfaTLZSEPTtz31eCwLb3W4C
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN0PR10MB5128.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: f7a9d911-d077-4769-3dca-08db692b0e69
X-MS-Exchange-CrossTenant-originalarrivaltime: 09 Jun 2023 20:49:44.8780
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: UZypubVFRCr9ej4NiS8XqpXxtDlFvKFkCvhurAy9TrrZfejOAQn1ZB7SguSztzRHS7qoQVUnUHo3wIpig4T+LQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR10MB6497
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.254,Aquarius:18.0.957,Hydra:6.0.573,FMLib:17.11.176.26
 definitions=2023-06-09_16,2023-06-09_01,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 malwarescore=0
 mlxscore=0 bulkscore=0 phishscore=0 mlxlogscore=826 spamscore=0
 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2305260000 definitions=main-2306090175
X-Proofpoint-ORIG-GUID: 2HKhNuL2vcwW8a9gLNy8ZgNCxknCq2w_
X-Proofpoint-GUID: 2HKhNuL2vcwW8a9gLNy8ZgNCxknCq2w_
X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_LOW,
        RCVD_IN_MSPIKE_H5,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE,
        T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org


> On Jun 9, 2023, at 4:31 PM, Tom Talpey <tom@talpey.com> wrote:
>=20
> On 6/6/2023 1:33 PM, Chuck Lever wrote:
>> From: Chuck Lever <chuck.lever@oracle.com>
>> Now that we have bulk page allocation and release APIs, it's more
>> efficient to use those than it is for nfsd threads to wait for send
>> completions. Previous patches have eliminated the calls to
>> wait_for_completion() and complete(), in order to avoid scheduler
>> overhead.
>=20
> Makes sense to me. Have you considered changing the send cq arming
> to avoid completion notifications and simply poll them from the cq
> at future convenient/efficient times?

That doesn't work for Read completions: those need to wake a new
nfsd as soon as they complete.

Send and Write completion don't need timely handling unless the
SQ is out of SQEs. It would be nice to move that processing out
of the CQ handler.

Otherwise, no, I haven't considered leaving the Send CQ disarmed.
I assumed that, at least on mlx5 hardware, there is some interrupt
mitigation that helps in this regard.


> Reviewed-by: Tom Talpey <tom@talpey.com>
>=20
> Tom.
>=20
>> Now release pages-under-I/O in the send completion handler using
>> the efficient bulk release API.
>> I've measured a 7% reduction in cumulative CPU utilization in
>> svc_rdma_sendto(), svc_rdma_wc_send(), and svc_xprt_release(). In
>> particular, using release_pages() instead of complete() cuts the
>> time per svc_rdma_wc_send() call by two-thirds. This helps improve
>> scalability because svc_rdma_wc_send() is single-threaded per
>> connection.
>> Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
>> ---
>>  net/sunrpc/xprtrdma/svc_rdma_sendto.c |    4 ++--
>>  1 file changed, 2 insertions(+), 2 deletions(-)
>> diff --git a/net/sunrpc/xprtrdma/svc_rdma_sendto.c b/net/sunrpc/xprtrdma=
/svc_rdma_sendto.c
>> index 1ae4236d04a3..24228f3611e8 100644
>> --- a/net/sunrpc/xprtrdma/svc_rdma_sendto.c
>> +++ b/net/sunrpc/xprtrdma/svc_rdma_sendto.c
>> @@ -236,8 +236,8 @@ void svc_rdma_send_ctxt_put(struct svcxprt_rdma *rdm=
a,
>>   struct ib_device *device =3D rdma->sc_cm_id->device;
>>   unsigned int i;
>>  - for (i =3D 0; i < ctxt->sc_page_count; ++i)
>> - put_page(ctxt->sc_pages[i]);
>> + if (ctxt->sc_page_count)
>> + release_pages(ctxt->sc_pages, ctxt->sc_page_count);
>>     /* The first SGE contains the transport header, which
>>   * remains mapped until @ctxt is destroyed.

--
Chuck Lever