Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
From:   Chuck Lever III <chuck.lever@oracle.com>
To:     Olga Kornievskaia <olga.kornievskaia@gmail.com>
CC:     Trond Myklebust <trondmy@hammerspace.com>,
        Linux NFS Mailing List <linux-nfs@vger.kernel.org>,
        Anna Schumaker <anna.schumaker@netapp.com>
Subject: Re: [RFC 1/2] xprtrdma: xdr pad optimization revisted again
Thread-Topic: [RFC 1/2] xprtrdma: xdr pad optimization revisted again
Thread-Index: AQHXnb+EPQ6bfsCh/km34j6ENph6pKuMRqKAgAAFyYCAAALCgIAAB7UAgAAEWwCAACcYgIABQxIAgAACPAA=
Date:   Tue, 31 Aug 2021 16:02:17 +0000
Message-ID: <CAF2348E-72EC-447F-A94D-C98D97D79EED@oracle.com>
References: <20210830165302.60225-1-olga.kornievskaia@gmail.com>
 <20210830165302.60225-2-olga.kornievskaia@gmail.com>
 <F7F9A947-B282-416F-BC65-796BF325054F@oracle.com>
 <CAN-5tyEB9nv576uJijBtyhv2pqAHGNB4yeKo=F21btOkVQuaDQ@mail.gmail.com>
 <04f975f95126921f3d239a7a9d80ced2d88b05ff.camel@hammerspace.com>
 <C73DE4AF-9C1D-4694-839B-D88EABAA6DAC@oracle.com>
 <9448f294a39775734212083cbe329642b9e15d09.camel@hammerspace.com>
 <B5C7A8A1-E810-4616-9E1A-265BADEC5432@oracle.com>
 <CAN-5tyFwxC3BtU7xQiaUKdCnBQg1hfKv6QJ-dnnBrLnP0P9kfg@mail.gmail.com>
In-Reply-To: <CAN-5tyFwxC3BtU7xQiaUKdCnBQg1hfKv6QJ-dnnBrLnP0P9kfg@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?VmxPhEoGnzl/XWbRdRl9dtH9uo4xPLd2pFrCEAyOxxr/wHFzFjoaqeAr9qJM?=
 =?us-ascii?Q?8T7Pj8nLn/NVxy3NnDBXzlifrR0pB0QP8S5nVFS17dCqwCz9k55GNjG88pz9?=
 =?us-ascii?Q?zkGGK0T8A8PaR41VA/2U1kyptAViOLfquFaIyR33McA1ZoWvQyMsPcAwjI0B?=
 =?us-ascii?Q?eX5omybi6WAYae0b9X7Q/SWfyujVgUVltJSe9uUpLZ83H47a0wWE9ilztBnw?=
 =?us-ascii?Q?bQMgcHhcfeWKuswt5LZu/aPM8AVu1XSZ06AgJtfPMCX60dNDndblVgA6MMW8?=
 =?us-ascii?Q?SCARh/poT2BU7pWgQV27rj/afeBXChq6rEoq9y+gxD4kYyCwPRwG5tKGe2Hp?=
 =?us-ascii?Q?1KDntxlM90jwvdc1WzqwCCHuwit2+9y1C9Gm455Z1rCLENqssVmyfxQ2Ejo0?=
 =?us-ascii?Q?0OrElIp7aWUvzy7XJGdExmBmFoVxp77Ye+9S387If2vEOidZc9Js1a6+GaCu?=
 =?us-ascii?Q?Ss2AGOaePgkL1BQJp4D19GYdDDF1hmBADQAfzOru8wSONxlyaIBsL3JDZ5bb?=
 =?us-ascii?Q?kSeZk/01VYZ9jUNGm3TgzmOrqrGTBVmpweIKBPy2/rZGwG1U5kxkt3NB+pcu?=
 =?us-ascii?Q?gEF5uBOB8v0/U2ZJFhAPxyXWyjRkbgUvWzwcEum1j7nBfKzJqnWjs1M8Qhct?=
 =?us-ascii?Q?vjzHT7ZTfgoro3vifjrzb/Y6ewa7BO9gJp7kULZ6YeQBtdJOZv1Dqt7GtfHd?=
 =?us-ascii?Q?PIoEVpnlD3B9xQXRvChtVzWWDj1Xn6tIvQlxrGKg+h1guu0G/65IAKM1azq2?=
 =?us-ascii?Q?XKXstDqweTqHHxJme+HV4qy4c5L7UrULdVhJEcanQ4J7rKqnzJMKlB1t06/P?=
 =?us-ascii?Q?l9+p9kZgpBanFYpy8xhQ3wkh1Vzlb5nKrs7tm2A1lxXT/84fQVPrUziOqKOn?=
 =?us-ascii?Q?xn7bVuWhbdlL3TRAyzb+j/JaQOV1AmSZZZJ9bfdZa8xaVRQq880LyKIw59cU?=
 =?us-ascii?Q?A8oIEWC5M6ClzglkdoEynY+nWODlk1fP0UZEI/taBXM3w250H13PGtYW8frc?=
 =?us-ascii?Q?8d72auVoc3u4JbbyxYjZY6/akmIAa+Ls16tUfkauErFVfC+1aSk2HhxZxJZs?=
 =?us-ascii?Q?dPlefjI2X1Nq1tCm3KTl2h73q1dg2nI35aVHptOedqicIPYFC8p5h/QLjfBw?=
 =?us-ascii?Q?TC936WCb41nKlGheBR5CMPsuJK2YKkrDmyJVn66bouvZ+qX1tkvEQiTPIOLw?=
 =?us-ascii?Q?I07OjvsPRSOYbOjDmBTcacNCson719nFJGz7jUzjdbxtqzddbcGkJUt2Jv8u?=
 =?us-ascii?Q?BRBEH7ylMaQpPnP3D6sn13C3g1RewuSmC/VOc2Bp3VnBF69Q0CvegQU+a5xK?=
 =?us-ascii?Q?YW05WApcTRqjf2f74qM6rxPm?=
x-ms-exchange-transport-forked: True
Content-Type: text/plain; charset="us-ascii"
Content-ID: <0043D4307A6E1643A921BF0BD47A8B59@namprd10.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: SJ0PR10MB4688.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 9645382e-2752-4189-9b3e-08d96c98b4be
X-MS-Exchange-CrossTenant-originalarrivaltime: 31 Aug 2021 16:02:17.2277
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: ujKiUEBJREzLVi63L2YGpPwNTPZq3pZWwGSAW/b0jU54f8uZouCYncqUGEr6liQ21KOpZ0+CIR/Z9RnahrmX5Q==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR10MB3859
X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10093 signatures=668682
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 mlxlogscore=999 mlxscore=0
 malwarescore=0 suspectscore=0 spamscore=0 adultscore=0 phishscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2107140000
 definitions=main-2108310086
X-Proofpoint-GUID: mnm3HKsDWdnD7v3zbYnJ7i4LhMRFNX4N
X-Proofpoint-ORIG-GUID: mnm3HKsDWdnD7v3zbYnJ7i4LhMRFNX4N
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org


> On Aug 31, 2021, at 11:54 AM, Olga Kornievskaia <olga.kornievskaia@gmail.=
com> wrote:
>=20
>> However, if there is a problem, it would be simple to create a
>> persistently-registered memory region that is not part of any RPC
>> buffer that can be used to catch unused Write chunk XDR padding.
>=20
> When I think of XDR padding, I'm thinking of a number of bytes less
> than 4. Perhaps I'm confused in my understanding.  Tail.iov_len is
> never a value less than 4. Tail.iov_len can contain bytes after an
> opaque element for which a read or write chunk was used (eg
> READ+VERIFY or WRITE+GETATTR). Thus when RDMA looks at tail.iov_len
> and allocates that much memory, that allocation does not represent
> just XDR padding. Since RDMA can't distinguish between
> padding+operation vs padding, can this even be solved?

Yes, I think it can be solved. xprtrdma was relying on the upper
layers to tell it how big the XDR pad was. Instead, it can either
provide a 4-byte pad segment unconditionally, or it can look at
the value of in the rq_rcv_buf::page_len.

The reason for all the current cleverness in there is because
adding the pad segment has a cost. If we make it no-cost, it can
be added unconditionally, and all the cleverness can be tossed
out.

--
Chuck Lever