Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
From:   Chuck Lever <chuck.lever@oracle.com>
To:     Daire Byrne <daire@dneg.com>
CC:     Trond Myklebust <trondmy@hammerspace.com>,
        Linux NFS Mailing List <linux-nfs@vger.kernel.org>,
        Bruce Fields <bfields@redhat.com>
Subject: Re: [PATCH] SUNRPC: Use TCP_CORK to optimise send performance on the
 server
Thread-Topic: [PATCH] SUNRPC: Use TCP_CORK to optimise send performance on the
 server
Thread-Index: AQHXAkZmunoZe40LXUmxO15yQAmlIKpWoMoAgAAEs4CAABZCgMgY7/PBt+g1IgC0y9QTNss11YoA
Date:   Mon, 15 Feb 2021 18:22:21 +0000
Message-ID: <86F287AC-04CD-4160-889A-8B2B25A4546C@oracle.com>
References: <20210213202532.23146-1-trondmy@kernel.org>
 <952C605B-C072-4C6B-B9C0-88C25A3B891E@oracle.com>
 <f025fa709f923255b9cb8e76a9b5ad4cca9355c4.camel@hammerspace.com>
 <4CD2739A-D39B-48C9-BCCF-A9DF1047D507@oracle.com>
 <285652682.9476664.1613312016960.JavaMail.zimbra@dneg.com>
 <E39E6630-91A9-48DA-A6CF-6AE5BF6CEDD1@oracle.com>
 <608881118.10473767.1613398098205.JavaMail.zimbra@dneg.com>
In-Reply-To: <608881118.10473767.1613398098205.JavaMail.zimbra@dneg.com>
Accept-Language: en-US
Content-Language: en-US
Content-Type: text/plain; charset="us-ascii"
Content-ID: <5D785FD83AA16641AD5474A9E293915E@namprd10.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-Network-Message-Id: b71fd4f7-cf8e-4169-6785-08d8d1dea2ba
X-MS-Exchange-CrossTenant-originalarrivaltime: 15 Feb 2021 18:22:21.4471
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: eI2+S9meWqPnqWo/yBZZEYiMuwH8ldBcGqXOPiuQsUrojxfs7N3slIDfVf0NDTF+HJNtAgzPwbldMzjSG8nHJg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR10MB4527
X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9896 signatures=668683
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=999 adultscore=0 mlxscore=0
 bulkscore=0 suspectscore=0 malwarescore=0 spamscore=0 phishscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000
 definitions=main-2102150143
X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9896 signatures=668683
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 priorityscore=1501
 lowpriorityscore=0 bulkscore=0 impostorscore=0 mlxlogscore=999
 adultscore=0 malwarescore=0 phishscore=0 clxscore=1015 mlxscore=0
 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2009150000 definitions=main-2102150142
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org


> On Feb 15, 2021, at 9:08 AM, Daire Byrne <daire@dneg.com> wrote:
>=20
>=20
> ----- On 14 Feb, 2021, at 16:59, Chuck Lever chuck.lever@oracle.com wrote=
:
>=20
>>>>> I don't have a performance system to measure the improvement
>>>>> accurately.
>>>>=20
>>>> Then let's have Daire try it out, if possible.
>>>=20
>>> I'm happy to test it out on one of our 2 x 40G NFS servers with 100 x 1=
G clients
>>> (but it's trickier to patch the clients too atm).
>>=20
>> Yes, that's exactly what we need. Thank you!
>>=20
>>> Just so I'm clear, this is in addition to Chuck's "Handle TCP socket se=
nds with
>>> kernel_sendpage() again" patch from bz #209439 (which I think is now in=
 5.11
>>> rc)? Or you want to see what this patch looks like on it's own without =
that
>>> (e.g. v5.10)?
>>=20
>> Please include the "Handle TCP socket sends with kernel_sendpage() again=
" fix.
>> Or, you can pull a recent stable kernel, I think that fix is already in =
there.
>=20
> I took v5.10.16 and used a ~100Gbit capable server with ~150 x 1 gbit cli=
ents all reading the same file from the server's pagecache as the test. Bot=
h with and without the patch, I consistently see around 90gbit worth of sen=
ds from the server for sustained periods. Any differences between them are =
well within margins of error for repeat runs of the benchmark.
>=20
> The only noticeable difference is in the output of perf top where svc_xpr=
t_do_enqueue goes from ~0.9% without the patch to ~3% with the patch. It no=
w takes up second place (up from 17th place) behind native_queued_spin_lock=
_slowpath:
>=20
>   3.57%  [kernel]                     [k] native_queued_spin_lock_slowpat=
h
>   3.07%  [kernel]                     [k] svc_xprt_do_enqueue
>=20
> I also don't really see much difference in the softirq cpu usage.
>=20
> So there doesn't seem to be any negative impacts of the patch but because=
 I'm already pushing the server to it's network hardware limit (without the=
 patch), it's also not clear if it is improving performance for this benchm=
ark either.
>=20
> I also tried with 50 clients and sustained the expected 50gbit sends from=
 the server in both with and without the patch.

Thank you Daire. That is a comforting result. The increase in do_enqueue
is curious, though.

--
Chuck Lever