Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
From:   Chuck Lever <chuck.lever@oracle.com>
To:     Tom Talpey <tom@talpey.com>
CC:     Anna Schumaker <anna.schumaker@netapp.com>,
        Linux NFS Mailing List <linux-nfs@vger.kernel.org>,
        "linux-rdma@vger.kernel.org" <linux-rdma@vger.kernel.org>
Subject: Re: [PATCH v1] xprtrdma: Simplify rpcrdma_convert_kvec() and
 frwr_map()
Thread-Topic: [PATCH v1] xprtrdma: Simplify rpcrdma_convert_kvec() and
 frwr_map()
Thread-Index: AQHW+XI3YU9+0Px6s0a/A4oYM4MFE6pFPVyAgAAAmgCAACoMAIAAATsA
Date:   Tue, 2 Feb 2021 21:55:07 +0000
Message-ID: <4A44C2FF-BFD6-423A-9FE5-F08CD2D75AA4@oracle.com>
References: <161227696787.3689758.305854118266206775.stgit@manet.1015granger.net>
 <e53cc3c2-2209-5f35-c487-9e59b9b9e526@talpey.com>
 <B0EED542-086B-4E76-9348-FAB8EBA612AB@oracle.com>
 <23f6893c-de31-0382-78a4-c09b1ceb2787@talpey.com>
In-Reply-To: <23f6893c-de31-0382-78a4-c09b1ceb2787@talpey.com>
Accept-Language: en-US
Content-Language: en-US
Content-Type: text/plain; charset="us-ascii"
Content-ID: <8564F7F56080734F859F917009593CB6@namprd10.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-Network-Message-Id: 599a7318-7e9a-4839-ba23-08d8c7c53486
X-MS-Exchange-CrossTenant-originalarrivaltime: 02 Feb 2021 21:55:07.5529
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: Ks6rAGRFcrk3cW+RulnDyq4ksXj74YS5SMquUOhSx8hIEYHMOYRhOjvsXdUGqi2OC18RtIcFLstTYJRJ3v9WJQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR10MB2486
X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9883 signatures=668683
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 mlxlogscore=999 phishscore=0
 spamscore=0 suspectscore=0 malwarescore=0 adultscore=0 bulkscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000
 definitions=main-2102020138
X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9883 signatures=668683
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 adultscore=0
 priorityscore=1501 impostorscore=0 malwarescore=0 clxscore=1015
 spamscore=0 lowpriorityscore=0 phishscore=0 mlxlogscore=999 mlxscore=0
 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2009150000 definitions=main-2102020139
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org


> On Feb 2, 2021, at 4:50 PM, Tom Talpey <tom@talpey.com> wrote:
>=20
> On 2/2/2021 2:20 PM, Chuck Lever wrote:
>>> On Feb 2, 2021, at 2:18 PM, Tom Talpey <tom@talpey.com> wrote:
>>>=20
>>> What's not to like about a log that uses the words "with aplomb"? :)
>>>=20
>>> Minor related comment/question below.
>>>=20
>>> On 2/2/2021 9:42 AM, Chuck Lever wrote:
>>>> Clean up.
>>>> Support for FMR was removed by commit ba69cd122ece ("xprtrdma:
>>>> Remove support for FMR memory registration") [Dec 2018]. That means
>>>> the buffer-splitting behavior of rpcrdma_convert_kvec(), added by
>>>> commit 821c791a0bde ("xprtrdma: Segment head and tail XDR buffers
>>>> on page boundaries") [Mar 2016], is no longer necessary. FRWR
>>>> memory registration handles this case with aplomb.
>>>> A related simplification removes an extra conditional branch from
>>>> the SGL set-up loop in frwr_map(): Instead of using either
>>>> sg_set_page() or sg_set_buf(), initialize the mr_page field properly
>>>> when rpcrdma_convert_kvec() converts the kvec to an SGL entry.
>>>> frwr_map() can then invoke sg_set_page() unconditionally.
>>>> Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
>>>> ---
>>>>  net/sunrpc/xprtrdma/frwr_ops.c  |   10 ++--------
>>>>  net/sunrpc/xprtrdma/rpc_rdma.c  |   21 +++++----------------
>>>>  net/sunrpc/xprtrdma/xprt_rdma.h |    2 +-
>>>>  3 files changed, 8 insertions(+), 25 deletions(-)
>>>> diff --git a/net/sunrpc/xprtrdma/frwr_ops.c b/net/sunrpc/xprtrdma/frwr=
_ops.c
>>>> index baca49fe83af..5eb044a5f0be 100644
>>>> --- a/net/sunrpc/xprtrdma/frwr_ops.c
>>>> +++ b/net/sunrpc/xprtrdma/frwr_ops.c
>>>> @@ -306,14 +306,8 @@ struct rpcrdma_mr_seg *frwr_map(struct rpcrdma_xp=
rt *r_xprt,
>>>>  	if (nsegs > ep->re_max_fr_depth)
>>>>  		nsegs =3D ep->re_max_fr_depth;
>>>>  	for (i =3D 0; i < nsegs;) {
>>>> -		if (seg->mr_page)
>>>> -			sg_set_page(&mr->mr_sg[i],
>>>> -				    seg->mr_page,
>>>> -				    seg->mr_len,
>>>> -				    offset_in_page(seg->mr_offset));
>>>> -		else
>>>> -			sg_set_buf(&mr->mr_sg[i], seg->mr_offset,
>>>> -				   seg->mr_len);
>>>> +		sg_set_page(&mr->mr_sg[i], seg->mr_page,
>>>> +			    seg->mr_len, offset_in_page(seg->mr_offset));
>>>>    		++seg;
>>>>  		++i;
>>>> diff --git a/net/sunrpc/xprtrdma/rpc_rdma.c b/net/sunrpc/xprtrdma/rpc_=
rdma.c
>>>> index 8f5d0cb68360..529adb6ad4db 100644
>>>> --- a/net/sunrpc/xprtrdma/rpc_rdma.c
>>>> +++ b/net/sunrpc/xprtrdma/rpc_rdma.c
>>>> @@ -204,9 +204,7 @@ rpcrdma_alloc_sparse_pages(struct xdr_buf *buf)
>>>>  	return 0;
>>>>  }
>>>>  -/* Split @vec on page boundaries into SGEs. FMR registers pages, not
>>>> - * a byte range. Other modes coalesce these SGEs into a single MR
>>>> - * when they can.
>>>> +/* Convert @vec to a single SGL element.
>>>>   *
>>>>   * Returns pointer to next available SGE, and bumps the total number
>>>>   * of SGEs consumed.
>>>> @@ -215,21 +213,12 @@ static struct rpcrdma_mr_seg *
>>>>  rpcrdma_convert_kvec(struct kvec *vec, struct rpcrdma_mr_seg *seg,
>>>>  		     unsigned int *n)
>>>>  {
>>>> -	u32 remaining, page_offset;
>>>> -	char *base;
>>>> -
>>>> -	base =3D vec->iov_base;
>>>> -	page_offset =3D offset_in_page(base);
>>>> -	remaining =3D vec->iov_len;
>>>> -	while (remaining) {
>>>> -		seg->mr_page =3D NULL;
>>>> -		seg->mr_offset =3D base;
>>>> -		seg->mr_len =3D min_t(u32, PAGE_SIZE - page_offset, remaining);
>>>> -		remaining -=3D seg->mr_len;
>>>> -		base +=3D seg->mr_len;
>>>> +	if (vec->iov_len) {
>>>> +		seg->mr_page =3D virt_to_page(vec->iov_base);
>>>> +		seg->mr_offset =3D vec->iov_base;
>>>> +		seg->mr_len =3D vec->iov_len;
>>>>  		++seg;
>>>>  		++(*n);
>>>> -		page_offset =3D 0;
>>>>  	}
>>>>  	return seg;
>>>>  }
>>>> diff --git a/net/sunrpc/xprtrdma/xprt_rdma.h b/net/sunrpc/xprtrdma/xpr=
t_rdma.h
>>>> index 94b28657aeeb..4a9fe6592795 100644
>>>> --- a/net/sunrpc/xprtrdma/xprt_rdma.h
>>>> +++ b/net/sunrpc/xprtrdma/xprt_rdma.h
>>>> @@ -285,7 +285,7 @@ enum {
>>>>    struct rpcrdma_mr_seg {		/* chunk descriptors */
>>>>  	u32		mr_len;		/* length of chunk or segment */
>>>> -	struct page	*mr_page;	/* owning page, if any */
>>>> +	struct page	*mr_page;	/* underlying struct page */
>>>>  	char		*mr_offset;	/* kva if no page, else offset */
>>>=20
>>> Is this comment ("kva if no page") actually correct? The hunk just
>>> above is an example of a case where mr_page is set, yet mr_offset
>>> is an iov_base. Is iov_base not a kva?
>> Ah, well the "if no page" part is now obsolete.
>> I suppose it should be set to "offset_in_page(vec->iov_base)" ?
>=20
> Seems like it, yes. Assuming that only the first element in the sgl
> has a possibly non-zero offset ("FBO"). All others must be zero for
> the FRMR.
>=20
> Is it guaranteed that each kvec is at most one physical page? If not,
> then the length may span into a random physical page, that was not
> necessarily contiguous in the original KVA-addressed buffer.

IIUC kmalloc'd buffers are backed by physically contiguous pages.


--
Chuck Lever