Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   Chuck Lever III <chuck.lever@oracle.com>
To:     Greg KH <gregkh@linuxfoundation.org>,
        Wolfgang Walter <linux@stwm.de>
CC:     linux-stable <stable@vger.kernel.org>,
        Trond Myklebust <trondmy@gmail.com>,
        Linux NFS Mailing List <linux-nfs@vger.kernel.org>,
        LKML <linux-kernel@vger.kernel.org>
Subject: Re: 5.4.188 and later: massive performance regression with nfsd
Thread-Topic: 5.4.188 and later: massive performance regression with nfsd
Thread-Index: AQHYZR5cM0piVXjf4kWw2Fv1L4O8aq0Zna6AgAAbcgA=
Date:   Wed, 11 May 2022 14:16:19 +0000
Message-ID: <6A15DEE1-CAC9-4C64-8643-AD28EA423046@oracle.com>
References: <f8d9b9112607df4807fba8948ac6e145@stwm.de>
 <YnuuLZe6h80KCNhd@kroah.com>
In-Reply-To: <YnuuLZe6h80KCNhd@kroah.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?Nq1/jZvL9VQdGyyuRSt5Zxd04EBPsBA3HENhZYXKfzdSxoi+Bg2XOzXUp9Z+?=
 =?us-ascii?Q?vyz5WLPhAD4SvsW2TE+usi/W+ClZpadl0X6IqtiJ2jMySQXmXAZ8XGjrefeP?=
 =?us-ascii?Q?c+AS9qJ+kbafFRzIERZ9sxySyM2VqnAJ08N3ug2/IjpzyJNAlUZfo0laoe/e?=
 =?us-ascii?Q?Z/4Tc3rg2T/t6w93Igrleyy9wh+F/AjRQZT6FymCm+HVP/VRur35v2DVle8f?=
 =?us-ascii?Q?PKO/Gi3SA8b6rkhA1JYvBVjxxswJAHItgHDz3OnNNWbD82U/+0EPI5PxJu9Y?=
 =?us-ascii?Q?BNTkYMUeluez5GKy+vBHLq2xiL38pCNVUtipVSfIYboudV7H2OeWt8exB/1d?=
 =?us-ascii?Q?/ZWJjZXwkWyszeN6G6dZ+pnf8jqZrxpyhFeoKGU5tPpUaoMA34YVhl4EUwsD?=
 =?us-ascii?Q?wgN+TlXRMVE388Xz8sNIBRKbFxC/A7n3G/VHgudQikDkKN4b7bfpVJBmDmtt?=
 =?us-ascii?Q?weU1j91Vgl1O1asQjzikDW8LWPCShN21zlCmAMbUUBdwY3MbdGWzxY07AcFF?=
 =?us-ascii?Q?nndJNStEtl1ef7JiA8JjwyfKyH9EaD1LHjiKXitTUTaJnJh14th4nxmosDg7?=
 =?us-ascii?Q?fYZ5ARnSfNxKLQXT4g/UEbSCmhCJrGcjSfbhQiwwXksjh56HytpGmyvG2+AU?=
 =?us-ascii?Q?0nT96FEAIaAOtIinR7o4dDYGKc6cs7w0mo+pcDi0BI8pxXSDut/iGM52vbP/?=
 =?us-ascii?Q?WnkFsZu9hXEp0vvMuvHC3sKS8VhKon1U/rFQhsdfd0zNIvSmO/NEzbER3bq8?=
 =?us-ascii?Q?GXJScgNHDLJKQCsIzBQ2Wodc/t2ICLWWW7c+CiX0pnj/se3L5WzhPVWVc1jM?=
 =?us-ascii?Q?k1xXMvf9CI1jqaejVuWcSZK10NYgpkNcU6ZXw+rpn9Vf7L258JI5Q+oabTGz?=
 =?us-ascii?Q?BXOC4Uo8IEMNzSt/ZGxF/3of46BjhxKoBAMUqER4ealTmwVSoXjmGpXtcdP9?=
 =?us-ascii?Q?MX/Qno+TIaELHYxoe+0Sp7oQPewCt6Mz17bLbjVkIxRb50NsDKCdmFZpJJRT?=
 =?us-ascii?Q?/cWr4Ragd3k20oTgh5EUIMCEeRFVZA/hzLSU04E/9fB5Z8sCOJ6rGPWvwzAI?=
 =?us-ascii?Q?miOFUShq5jiGc13Rz+rlhdTYJY3HYTpPrB3kVNm8ToRBQ9kpBHOyTtln42Vu?=
 =?us-ascii?Q?uTEK1hwSS75D6wYKyPxaWKqKdgAPZLSnROYb4nroWVk4O3+iLo7M7AHb2plf?=
 =?us-ascii?Q?Y9GOKimaO7pavT49kfJwQOSJaCPuZRAVW7t2e7dNY+ffGDZMbSqrNARICPFs?=
 =?us-ascii?Q?9K1GIevA0ghuY3w0vyyRB/BKyWKqBDnBa7UmiQhRCO3GkZb5/xXFhaEPjy47?=
 =?us-ascii?Q?WC2btR6SMFg3UcNgETG3/QBMst5FTdIBxLqKY/acc1SdB7o0raEHKHzNeGlV?=
 =?us-ascii?Q?N1s+Uade5HFDUh3eV6l4WHxln7Epyv53LZ65duicWBCUuWehj0EwqHyFvIKB?=
 =?us-ascii?Q?Ek5JkjmRXJvt9CaNr+T5z0vGXY0WWqR0N+l8Inm92h/75WbjTdIyw3MMxFjY?=
 =?us-ascii?Q?V+XMZZ88c6bITzNHrQWu8q72wpgIcM4Rm2itNCtoX9cRUqQGaiVo9IxL+7vS?=
 =?us-ascii?Q?30JnpT+jQVuyzshaUJatSD4+Y+duZ+K4XlsmWrsGVpjYt9OeX8kMy9EmORdc?=
 =?us-ascii?Q?y88OPJp0UheTd4i9nCpdg27PwjFgTtW93EYRx08+dDpaCk2+wQZ/HcqGw5Ho?=
 =?us-ascii?Q?qysOsL3cBAxJJlu7NFS6v/uL7wo3QABl5u/hrXKlx9SQuA/VOSIQ4PHFPrKg?=
 =?us-ascii?Q?ERPbVKnTC4hJKDvlTROeKQLgz5VQoB8=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-ID: <29E075AC0631A342BAF2EE8E624F4A01@namprd10.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN0PR10MB5128.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 2b4ba5c9-999c-4afc-6d3d-08da3358d1ce
X-MS-Exchange-CrossTenant-originalarrivaltime: 11 May 2022 14:16:19.5826
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: Ow88Jn3e4GFNMk4/gsFJY8sfl9FR3JcWycaThvIth4IQxh9DUXla+dzAA27ISkZQ0xjAAhsqDzpRqEHvKo71aA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CO6PR10MB5556
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.486,18.0.858
 definitions=2022-05-11_05:2022-05-11,2022-05-11 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=999 malwarescore=0
 suspectscore=0 phishscore=0 spamscore=0 bulkscore=0 mlxscore=0
 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2202240000 definitions=main-2205110068
X-Proofpoint-GUID: ATeX09zZ_bUvbhKqyJQikrcElrRP2Cez
X-Proofpoint-ORIG-GUID: ATeX09zZ_bUvbhKqyJQikrcElrRP2Cez
X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_LOW,SPF_HELO_NONE,
        SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no
        version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org


> On May 11, 2022, at 8:38 AM, Greg KH <gregkh@linuxfoundation.org> wrote:
>=20
> On Wed, May 11, 2022 at 12:03:13PM +0200, Wolfgang Walter wrote:
>> Hi,
>>=20
>> starting with 5.4.188 wie see a massive performance regression on our
>> nfs-server. It basically is serving requests very very slowly with cpu
>> utilization of 100% (with 5.4.187 and earlier it is 10%) so that it is
>> unusable as a fileserver.
>>=20
>> The culprit are commits (or one of it):
>>=20
>> c32f1041382a88b17da5736886da4a492353a1bb "nfsd: cleanup
>> nfsd_file_lru_dispose()"
>> 628adfa21815f74c04724abc85847f24b5dd1645 "nfsd: Containerise filecache
>> laundrette"
>>=20
>> (upstream 36ebbdb96b694dd9c6b25ad98f2bbd263d022b63 and
>> 9542e6a643fc69d528dfb3303f145719c61d3050)
>>=20
>> If I revert them in v5.4.192 the kernel works as before and performance =
is
>> ok again.
>>=20
>> I did not try to revert them one by one as any disruption of our nfs-ser=
ver
>> is a severe problem for us and I'm not sure if they are related.
>>=20
>> 5.10 and 5.15 both always performed very badly on our nfs-server in a
>> similar way so we were stuck with 5.4.
>>=20
>> I now think this is because of 36ebbdb96b694dd9c6b25ad98f2bbd263d022b63
>> and/or 9542e6a643fc69d528dfb3303f145719c61d3050 though I didn't tried to
>> revert them in 5.15 yet.
>=20
> Odds are 5.18-rc6 is also a problem?

We believe that

6b8a94332ee4 ("nfsd: Fix a write performance regression")

addresses the performance regression. It was merged into 5.18-rc.


> If so, I'll just wait for the fix to get into Linus's tree as this does
> not seem to be a stable-tree-only issue.

Unfortunately I've received a recent report that the fix introduces
a "sleep while spinlock is held" for NFSv4.0 in rare cases.


--
Chuck Lever