Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 2620:137:e000::1:18 as permitted sender) client-ip=2620:137:e000::1:18;
From:   Chuck Lever III <chuck.lever@oracle.com>
To:     Trond Myklebust <trondmy@hammerspace.com>
CC:     Linux NFS Mailing List <linux-nfs@vger.kernel.org>
Subject: Re: write_space and software kTLS
Thread-Topic: write_space and software kTLS
Thread-Index: AQHYN+t2c9NNn2vvyUmVc3p8QXQTC6y/kR4AgADpe4A=
Date:   Tue, 15 Mar 2022 14:01:52 +0000
Message-ID: <624F8B1A-7233-4DE6-A383-A17D88D3BB2F@oracle.com>
References: <B7F08FCD-0EC1-4D7E-A560-6ADA281605AE@oracle.com>
 <378c5635b97c237f869234e24030ac0018b684b3.camel@hammerspace.com>
In-Reply-To: <378c5635b97c237f869234e24030ac0018b684b3.camel@hammerspace.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?GPBoX8DbYzwOLoSoNZGTYpUQSxTlwTo0NK2yVUaLx+QUQCXcKw0HhEgfkJKg?=
 =?us-ascii?Q?C9fbFu9mQ+1EmSPpePjd1naftgeWC5Ty8h6fxtgtLSd570Cje2iJK7r8vdJo?=
 =?us-ascii?Q?w6omJVYfUSzgsykQCWX1xcYJnTjbNkA9Ro/KM20jTWt3O6Iox4AjilCbTGvt?=
 =?us-ascii?Q?TzFql14R/B7DC4pZ/wmd27+aWO9YLXDLg/0z9fxeHdeHYUp2ocbGlp2vBlRT?=
 =?us-ascii?Q?pTDQfqC7NHYhS9Q6kR18keoucWTwMSUF0D2uu9WWFuunbDCCtcZHdfsfFqDl?=
 =?us-ascii?Q?KUc9rbp5xZ7tUu5R/ZzqLTzz/QsrW6zAeGvvrogXf+ssl1ldJpqlMVs+5e5F?=
 =?us-ascii?Q?pPn09CmnevlcGC5kd4mDgsCYTH6qJWISrwJge5ZIl4Gr2CPjNiwu44l9M0sI?=
 =?us-ascii?Q?GKf6BKmgEm+mUKtG+twas6R7WWMXMjrlAqWMk7gpCZlv6P7X9FeC6EDjeWYd?=
 =?us-ascii?Q?hGafTFZOOKXgIqdBlcLrBhdSvUncJp5k1m27NlVJpngvC2Q9hYpQBQ/ILZiy?=
 =?us-ascii?Q?peCeAdeGtynbERhNgkyM/CaUKcD+dVVLMqnwbJQbLwdiG9Lsp3e6odtc9GC2?=
 =?us-ascii?Q?gu75LpOEHCh8jEzqo3CIDHxOLOH7nMzoR5kFjuPbKYVYPJFBZlqQQDYnYb95?=
 =?us-ascii?Q?ApJML6lS4cUjAe7PfPfZkjAyTwLBr59Jhy2bxs1h5zhN4SKjy0B2G3kpgtA5?=
 =?us-ascii?Q?cZmRTz5S0KfEe339r2Es32/ptUkbehYjW43SSrzKldVAgWcw02ppLpbcLANZ?=
 =?us-ascii?Q?Woa1eH2UOWMKzGnGcDSXlACi3kBMSuQo8PQtrpIyqrTXWbsCWqwoJLqyrMah?=
 =?us-ascii?Q?RKo3ZU5o4OM8DwGgy0Ksq0Ncgf8CBopmnzT4txsxqARGo/nE8hjc+K5hVcwm?=
 =?us-ascii?Q?0SD5SqS3P57r6dKMyF0n3LtfSeEGrkaN9aprzq6qgaQwKUi0G3L9/oNDjp4D?=
 =?us-ascii?Q?IZ4jgZIIwvgNSdcRrU8BgXffx/IZh169hK+iCN2Mp4+D8X3nSq2G9TjmH0EH?=
 =?us-ascii?Q?iLF7P9Fx6MXvVJe/ty3CcokWxU9nxrL4AzAIqwXx8B4y97aH8AOGZrvNzTl9?=
 =?us-ascii?Q?mlm83EVOdTElEQ17GD7dcfaj4lTniy40+qx6gT3NQGUHX+Dsr13DkgpMKa3g?=
 =?us-ascii?Q?pyeKXMa11AIkYZ+Wq36iAI8DWSByc0roCCqXn46/vhQq/V+sb1Yn9w5l4/gZ?=
 =?us-ascii?Q?PmP4FqOck4iKgBPEK6VaH3JDoJQxWq2qs7oa1/tzak10zMnOdlAGEeL8F2Sm?=
 =?us-ascii?Q?uSIxbET+rO9jGNOoNqhRpaBJI9s/zfH5WSthU2fR2ozJ86EJK/UJLU3UJwVa?=
 =?us-ascii?Q?m8X0x6j8iE1jxLsMuJIOAl0QDeKsUl2mHr/qJ95EvMI4GG/92M/lFrn3X74c?=
 =?us-ascii?Q?QrJnv9ZCl6RxRkif7qBWZ27OjlnxUj89nkdMQpVnLrwvp5bWDE3nL2CFNMj/?=
 =?us-ascii?Q?lcKUfOr2Z0paqCcunxTJpQCi8KbVj7fdXDcRGyNjU2FiuyaMUiKkOQ=3D=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-ID: <E1593DAF8BCB9B43AEE5AEA5E0C82F06@namprd10.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN0PR10MB5128.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 2650f7cc-105f-4f9a-b52b-08da068c5bbd
X-MS-Exchange-CrossTenant-originalarrivaltime: 15 Mar 2022 14:01:52.9912
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: 8wFrAZIAafyXnaMLve7Hf+d5SNbS1YCfvIRqE6Tr/JbbO3QFSnFkQo4B0BgnCK5bwiW6q1zLOj/gwxe9OUwXRw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS7PR10MB5168
X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10286 signatures=693139
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 phishscore=0
 malwarescore=0 mlxscore=0 suspectscore=0 spamscore=0 bulkscore=0
 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2202240000 definitions=main-2203150092
X-Proofpoint-GUID: oXqDfPOubJ_ClRohfIFCJIKUFq54abV1
X-Proofpoint-ORIG-GUID: oXqDfPOubJ_ClRohfIFCJIKUFq54abV1
X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,RDNS_NONE,SPF_HELO_NONE,T_SCC_BODY_TEXT_LINE
	autolearn=no autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
	lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org


> On Mar 14, 2022, at 8:06 PM, Trond Myklebust <trondmy@hammerspace.com> wr=
ote:
>=20
> On Mon, 2022-03-14 at 21:35 +0000, Chuck Lever III wrote:
>> Hey Trond-
>>=20
>> I've made some progress getting RPC-with-TLS working in
>> the Linux NFS client, but I recently hit an interesting
>> snag and could use a little advice.
>>=20
>> The software kTLS infrastructure uses do_tcp_sendpages()
>> under the covers, and that function is clearing
>> SOCKWQ_ASYNC_NOSPACE from under xs_nospace(). That
>> prevents xs_run_error_worker() from waking up xprt->sending,
>> stalling an RPC transport waiting for more socket write
>> space. I'm not sure how to address this, and I'm interested
>> in your opinion.
>>=20
>=20
> How is it achieving this? We only set SOCKWQ_ASYNC_NOSPACE after the
> call to xprt_sock_sendmsg().

A kworker is clearing NOSPACE between the time xs_tcp_send_request()
sets it and the time xs_write_space() runs.

  kworker/u128:2-33    [003]   155.723869: rpc_socket_nospace:   task:00000=
6cb@00000003 total=3D262380 remaining=3D131308
  kworker/u128:2-33    [003]   155.723870: bprint:               xs_nospace=
: sk=3D0xffff88810a8f0a00 setting SOCKWQ_ASYNC_NOSPACE
  kworker/u128:2-33    [003]   155.723879: xprt_transmit:        task:00000=
6cb@00000003 xid=3D0x8ab69e2e seqno=3D0 status=3D-11
  kworker/u128:2-33    [003]   155.723881: xprt_release_xprt:    task:00000=
6cc@00000003 snd_task:ffffffff
     kworker/3:2-116   [003]   155.723885: bprint:               do_tcp_sen=
dpages: sk=3D0xffff88810a8f0a00 clearing SOCKWQ_ASYNC_NOSPACE
  kworker/u128:2-33    [003]   155.723888: rpc_task_run_action:  task:00000=
6cc@00000003 flags=3DASYNC|MOVEABLE|NORTO|CRED_NOREF runstate=3DRUNNING|ACT=
IVE|NEED_XMIT|NEED_RECV status=3D-11 action=3Dcall_transmit_status
  kworker/u128:2-33    [003]   155.723889: rpc_task_run_action:  task:00000=
6cc@00000003 flags=3DASYNC|MOVEABLE|NORTO|CRED_NOREF runstate=3DRUNNING|ACT=
IVE|NEED_XMIT|NEED_RECV status=3D0 action=3Dcall_transmit
  kworker/u128:2-33    [003]   155.723890: rpc_task_sleep:       task:00000=
6cc@00000003 flags=3DASYNC|MOVEABLE|NORTO|CRED_NOREF runstate=3DRUNNING|ACT=
IVE|NEED_XMIT|NEED_RECV status=3D-11 timeout=3D0 queue=3Dxprt_sending
     kworker/1:2-115   [001]   155.733398: bprint:               do_tcp_sen=
dpages: sk=3D0xffff88810a8f0a00 clearing SOCKWQ_ASYNC_NOSPACE
     kworker/1:2-115   [001]   155.733418: bprint:               do_tcp_sen=
dpages: sk=3D0xffff88810a8f0a00 clearing SOCKWQ_ASYNC_NOSPACE
         openvpn-914   [001]   155.750263: bprint:               xs_write_s=
pace: sk=3D0xffff88810a8f0a00 SOCKWQ_ASYNC_NOSPACE was clear


>> For example, why check that flag rather than just waking
>> up xprt->sending unconditionally?
>=20
> The socket code calls ->write_space() in all sorts of situations, so we
> need to distinguish between the cases where we are actually waiting for
> buffer memory, and the situations where we are not. Otherwise, we'd be
> calling xs_run_error_worker() all the time.

On my (admittedly limited) workloads, sk_stream_is_writeable()
does a good job of avoiding spurious wake-ups. However, to be
absolutely certain of our wake-up accounting, using a flag that
is local to the rpc_xprt and not overloaded might be wise?


--
Chuck Lever