Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
From:   Chuck Lever III <chuck.lever@oracle.com>
To:     Bruce Fields <bfields@fieldses.org>
CC:     Linux NFS Mailing List <linux-nfs@vger.kernel.org>,
        "linux-rdma@vger.kernel.org" <linux-rdma@vger.kernel.org>
Subject: Re: [PATCH RFC 0/6] NFSD: Pull Read chunks in XDR decoders
Thread-Topic: [PATCH RFC 0/6] NFSD: Pull Read chunks in XDR decoders
Thread-Index: AQHXnpsmcuWuedcxvkuGPTniugh266uOFCwAgAALe4A=
Date:   Tue, 31 Aug 2021 21:23:25 +0000
Message-ID: <A56E33CA-C87A-41E7-BBF6-8623795D4607@oracle.com>
References: <163043485613.1415.4979286233971984855.stgit@klimt.1015granger.net>
 <20210831204220.GB7585@fieldses.org>
In-Reply-To: <20210831204220.GB7585@fieldses.org>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?8347pOYcFNXA7EqnuFZA9sd8FJlFispgk9pKeXXgImGXP9JH5zicEsGV3H+W?=
 =?us-ascii?Q?W+pLjEJ1uUEs7B2mmsltPTrlTjEGrhrKHGgkSJRviHRxd3rJ4LSDSvAymrB+?=
 =?us-ascii?Q?JXCVjGUO9Tsnyuy2F8Io8ubMH1dRJ3LV7MoXG/S75FCNu1MuJzEK0vEKGVy5?=
 =?us-ascii?Q?T9II9BaWMrRbboTyPPeuNtpBc5vcfE9Kh8/jWTXQli5ZGdh6teBFl6pFwS3x?=
 =?us-ascii?Q?m3WOjcIi8GPBuF62JqGL42aqw/Ge2mZjrj565TnmaU+vSZHcpb7jOp5kHsQg?=
 =?us-ascii?Q?Hede3Y/iDxMgbdcMvYATqUVkQ/eiOLE9ybngf5f2D9mu5HdE0DWgAkr8zjFg?=
 =?us-ascii?Q?oitgPGZiXQwov653OB2pGzOjKo16AdkXi3AHrqnKik+a/EpJxPrVEkl3TzPk?=
 =?us-ascii?Q?l1VHPB/7GtpKmoPU4PEdYpGY5JNJ2pGlj4Qsnv0TV9UrRJqMo3hPcz8E8s/m?=
 =?us-ascii?Q?Pt3oaavtku6Z6gSmd1i7L0i71XJ0FfCy9GbuBnktgcCKt/rTpwXnVHQ6v6zs?=
 =?us-ascii?Q?+hqrEJ7Ozs36qASYA0d36KniMeWvDW5w/RkR+QU5zXsKa6QYd2Yz/wdCeFfi?=
 =?us-ascii?Q?XvH3agWWZjYAhtrszeFzwlXx7QA1RUhWCti4Ihb7qU9tI7Md2xTFK8idpGwD?=
 =?us-ascii?Q?s7ZTSbEx+1fhOIQAW6MTif+KDrOV2sJLOydhyFRpHT4R4aSLUknLZJ8IE3/B?=
 =?us-ascii?Q?XyhNcfQBwCRTRzPCH72ZCsw4AjPdIpLC36M85CTwepNKLtnt1I0UcJLmV4GC?=
 =?us-ascii?Q?ELFTN9NZxXNdU44S1FkKE9v6kuNRhbkplmVKgdud5pHREgNf5kK3GNZISJU6?=
 =?us-ascii?Q?yIyx9caAPVhjctxzTTQK0CPPSBAJB9uOkdpg7MdNSBSBFToJFkbBebzMqARY?=
 =?us-ascii?Q?lmymhY4CP0OGFgRG37VMDGYHbFZkbXAOSsdQjwSvd392jxwjW2tZWoI78lzq?=
 =?us-ascii?Q?+iVzuch+uo5mB6ghMIVWdWWgJTFZ/OgxwPj0vQWd88UsHA1CAZPjodLBB+9t?=
 =?us-ascii?Q?8HVdeehSJUDPFnhGPXYWsKv1EJT0N8DXOGE+Bk+baNOsqMzeV53gW91yqMUW?=
 =?us-ascii?Q?OQPsK33DnJgxae1X58jKeamm/oq52H/X1Nu6ON5aGr/PLasn4wpP3tlGVSSI?=
 =?us-ascii?Q?W9RRUu38T0rNsnpiGEks4n1x0UWAZfn+mkI6eCBXY/r4Ej6YzgL10/+Sui8y?=
 =?us-ascii?Q?PO1gETv/NWa4ne2L5/noWuTHg5mSuem85PoSAperZj5UGE6bTyVSy+x7GSPG?=
 =?us-ascii?Q?3Nrp7Yxhqg4S86q9KqWSV17uAAZuH9TPgWRKuODn8pKfhVa7OL/gSUdn4ZOy?=
 =?us-ascii?Q?HkFio74TmhGQ9WbbIpmbg/PZ?=
x-ms-exchange-transport-forked: True
Content-Type: text/plain; charset="us-ascii"
Content-ID: <5F3D01E85A79104D8456351E5AB7C8B9@namprd10.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: SJ0PR10MB4688.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 9de33395-2c30-4e08-9eaa-08d96cc591c2
X-MS-Exchange-CrossTenant-originalarrivaltime: 31 Aug 2021 21:23:25.7970
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: 4ObRYcDW0v2XhrgVisZZQzob/R0AIgzCO6OnldwJbzgdaFOj/wQNxRVxi50DXG/VGH+nwy9u7jI+XwDhqx/i1w==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR10MB4637
X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10093 signatures=668682
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 mlxlogscore=999 mlxscore=0
 malwarescore=0 suspectscore=0 spamscore=0 adultscore=0 phishscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2107140000
 definitions=main-2108310116
X-Proofpoint-ORIG-GUID: iCltGm2QgpaaSuHhaK8uLzpoOjnl5Uuw
X-Proofpoint-GUID: iCltGm2QgpaaSuHhaK8uLzpoOjnl5Uuw
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org


> On Aug 31, 2021, at 4:42 PM, J. Bruce Fields <bfields@fieldses.org> wrote=
:
>=20
> On Tue, Aug 31, 2021 at 03:05:09PM -0400, Chuck Lever wrote:
>> Hi Bruce-
>>=20
>> Here is part of what we discussed recently about trying to align
>> pages in NFS WRITE requests so that splice can be used. This
>> series updates server-side RPC and svcrdma to ensure that an
>> aligned xdr_buf::pages array is presented to NFSD, which is then
>> converted into an aligned rq_vec for the VFS layer.
>=20
> Seems sensible to me.
>=20
> Do you have a git tree?

I don't yet, but can set up a topic branch somewhere where we're
a little further along.


> It didn't apply cleanly to 5.14 when I tried,
> but I didn't stop to figure out why.

It might apply to 5.15-pre now that nfsd-5.15 has been merged.


>> The next step would be to look at how to make the best use of the
>> aligned rq_vec.
>=20
> Have you done any performance comparison just with this?

Not yet. I just got it behaving correctly with the usual tests.
Baby steps, sir.


> Doesn't seem like it should make a significant difference, but it might
> be interesting to check anyway.

I expect it to add a small amount of latency to NFS WRITEs, since
RDMA Reads are done just a little later than before.


> --b.
>=20
>> My naive thought is that where there is a PAGE_SIZE
>> entry in rq_vec and there is no page in the file's page cache at
>> that offset, the transport-provided page can be flipped into place.
>> Might work for replacing whole pages as well, but baby steps first.
>>=20
>> This series has been exercised a bit with both TCP and RDMA, but no
>> guarantees that it is completely bug-free. NFSv4 compounds with
>> multiple WRITE payloads on RDMA are treated like TCP: the RPC
>> message is contained in an unstructured stream of unaligned pages.
>>=20
>> Comments encouraged.
>>=20
>> ---
>>=20
>> Chuck Lever (6):
>>      SUNRPC: Capture value of xdr_buf::page_base
>>      SUNRPC: xdr_stream_subsegment() must handle non-zero page_bases
>>      NFSD: Have legacy NFSD WRITE decoders use xdr_stream_subsegment()
>>      SUNRPC: svc_fill_write_vector() must handle non-zero page_bases
>>      NFSD: Add a transport hook for pulling argument payloads
>>      svcrdma: Pull Read chunks in ->xpo_argument_payload
>>=20
>>=20
>> fs/nfsd/nfs3proc.c                       |   3 +-
>> fs/nfsd/nfs3xdr.c                        |  16 +--
>> fs/nfsd/nfs4proc.c                       |   3 +-
>> fs/nfsd/nfs4xdr.c                        |   6 +
>> fs/nfsd/nfsproc.c                        |   3 +-
>> fs/nfsd/nfsxdr.c                         |  13 +--
>> fs/nfsd/xdr.h                            |   2 +-
>> fs/nfsd/xdr3.h                           |   2 +-
>> include/linux/sunrpc/svc.h               |   6 +-
>> include/linux/sunrpc/svc_rdma.h          |   8 ++
>> include/linux/sunrpc/svc_xprt.h          |   3 +
>> include/trace/events/rpcrdma.h           |  26 +++++
>> include/trace/events/sunrpc.h            |  20 +++-
>> net/sunrpc/svc.c                         |  38 +++++--
>> net/sunrpc/svcsock.c                     |   8 ++
>> net/sunrpc/xdr.c                         |  32 +++---
>> net/sunrpc/xprtrdma/svc_rdma_recvfrom.c  |  37 +++++-
>> net/sunrpc/xprtrdma/svc_rdma_rw.c        | 139 ++++++++++++++++++++---
>> net/sunrpc/xprtrdma/svc_rdma_transport.c |   1 +
>> 19 files changed, 292 insertions(+), 74 deletions(-)
>>=20
>> --
>> Chuck Lever

--
Chuck Lever