Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   Chuck Lever III <chuck.lever@oracle.com>
To:     Greg KH <gregkh@linuxfoundation.org>
CC:     Wolfgang Walter <linux@stwm.de>,
        linux-stable <stable@vger.kernel.org>,
        Trond Myklebust <trondmy@gmail.com>,
        Linux NFS Mailing List <linux-nfs@vger.kernel.org>,
        LKML <linux-kernel@vger.kernel.org>
Subject: Re: 5.4.188 and later: massive performance regression with nfsd
Thread-Topic: 5.4.188 and later: massive performance regression with nfsd
Thread-Index: AQHYZR5cM0piVXjf4kWw2Fv1L4O8aq0Zna6AgAAbcgCAAAISgIAAA5uA
Date:   Wed, 11 May 2022 14:36:37 +0000
Message-ID: <0F5EA15D-08A0-4D3E-B5A9-D62C49BF2B56@oracle.com>
References: <f8d9b9112607df4807fba8948ac6e145@stwm.de>
 <YnuuLZe6h80KCNhd@kroah.com>
 <6A15DEE1-CAC9-4C64-8643-AD28EA423046@oracle.com>
 <YnvG71v4Ay560D+X@kroah.com>
In-Reply-To: <YnvG71v4Ay560D+X@kroah.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?dX4xffypR7mKyOaKzTLo1Qr9EpRAseL4/e4dpwDDwLPZpyzgcLhtL5zcLLIX?=
 =?us-ascii?Q?m7P2/bpZaSwlgDXG8yOGpS0vjlHa73iY8HstvgkY2vfui3gQfySQHcw1aw9R?=
 =?us-ascii?Q?otwfGrKjk9PRcb9Afspx9i5zoRv8MnRr19p4rEerPjZTxoag3pA6w8GcV+kk?=
 =?us-ascii?Q?GhVNm3SuxSwV+PXkixb9DnV4zrkb22IUA7ifOFVFWL9kNKhbtIxyhYn2LXC0?=
 =?us-ascii?Q?DQXH1OhnMOeXfnPakPLpSUBEtyuDumhZ7kCq15/gk2/RoVyFBxB/5K3ZT/p1?=
 =?us-ascii?Q?De0D99HnuTAHpfGMDbD4roZVD6I1Gx+hP5XI9uT94+jlTFcKpi2kR27WeCPF?=
 =?us-ascii?Q?cwO8PvxJbRptfQfV3PPIeYqmqII1ism85+1W2jqpj33+em7iI74cldoloB2L?=
 =?us-ascii?Q?H1MNiug6NTl9YGaipTsKN2lkLtQnUoo14s2eDUNJ8DAYMdKAgH5StkJYu2D5?=
 =?us-ascii?Q?0FAX62YjXaPlUGCYq/d9yackGSeEjvAB6qYfWl+lWvy7D9Y5kJ2bwAtKImTz?=
 =?us-ascii?Q?NpvsP7oiGPhRwmkZT1rrZIKLmvYJzATCOf2v0Gl+XuSQGp/bix7mwuaXirph?=
 =?us-ascii?Q?L5Fk4p4OjeeJI3WJidlxxCxsOG8adYq6Bg5ltJxYZtMwzfs9rAlxuKlMuwku?=
 =?us-ascii?Q?/Ups8xaGJcWCqkaUDq+V4wn4sQklSMHERjsLE9QgpJNnTiLEor7B25cKVchO?=
 =?us-ascii?Q?mQ2oi9ekL/NQ3uySBPtrM9/eGzvdHwwUQ9Oa4iBHHrbjYxujU4RGv3orkDf3?=
 =?us-ascii?Q?CZEUL8bwXTi1kwdLAK/7yxCtGE8f+UgZGV8GQvWvcB+VhpEOty3D3tjsErmA?=
 =?us-ascii?Q?UYLUVCa5RPh7TDtccswndnycNxuOItucewlvS/e3T69s8vp7IW/AyiqY1S1k?=
 =?us-ascii?Q?QjdJhyc2+cMQZ9TJRA+ZrpZZyV55AAsVc37+yxeO//OzKUIU/6jD10U8i+dR?=
 =?us-ascii?Q?EyKC1Wuui10/y8mrvzrpHZ8+spzkgjyGmM4m8cyv5fu83BLBDia0TAFJE9N8?=
 =?us-ascii?Q?l+IjM8z3TfDNhbJJn/QrwREeWzj95c5CilfHLauJ/7V1cIMlBsKON2OSLKV2?=
 =?us-ascii?Q?yq4fznWsGnf+pkfJwy/FElwkhNbZveYOAkumVsp67pNTVPP7ws9k3QDhaCFM?=
 =?us-ascii?Q?vII8EUEbxTQ8REqdXI3JwDPqYIv8dBjHXIyH7v9aqTWDF1NC9xJSmK9n+AY2?=
 =?us-ascii?Q?9OB4FelRTJg5Ym3SWSDo1+TV/XRF7RiHrF6mVeask5iJqT9wUZCSTFEruSth?=
 =?us-ascii?Q?9rfHf7BwNiaBU2AqxERU1x2NOfh36LTLiF68vTwdXU7+VNc5lzHcb69IVgV1?=
 =?us-ascii?Q?EedjVKw0GmLA9m0ysPjWFp/+1hHZKo9ab/uL3v3Gjwh/qfh7/cUrMG3l6jHw?=
 =?us-ascii?Q?vOhh4RZG5+oxBt72KrgAyIlhl4iA2gQgZbJHCP87zpn6cNio/nCqfQ3rPKHF?=
 =?us-ascii?Q?kyQr+idjqU3ePC1Yjdb+D1N+BeigXvcueIy70pMvkiAFYrvD4jNf2Q9Ktvp7?=
 =?us-ascii?Q?Xh12sltIpwN4I9hwZDyDAkEvT3cZNGeM2FTzoRDf//uRK6q+jrvVFVwoRpk/?=
 =?us-ascii?Q?ftdbT8q5ClQU4vamfTtimokJtO5ZlSm1ZOCLRB+JVeeGuAjbHJ9Gh+kkDVdA?=
 =?us-ascii?Q?e1B4deNpVoUjpX08xdhm1XLkv8F+Gtj11nX02Fuoebg7OTtTdkTAUdUU93cf?=
 =?us-ascii?Q?HGiRwbCNtqiusrabrnyLULjuL3CstNswBZF2wPgjeCdWOLL0a5IA1ZZQKOJO?=
 =?us-ascii?Q?AYyTU2Phah9PSBJNhzedBMBtL3F1Bu0=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-ID: <93440AC397C1D948B352E8A2FEC2142C@namprd10.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN0PR10MB5128.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 2be9359c-6039-49e4-5d0f-08da335ba7db
X-MS-Exchange-CrossTenant-originalarrivaltime: 11 May 2022 14:36:37.7089
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: 8tJ/B+HZ+KSz9q4qf6NptG8kXfe7LCjRPhFPsu7v4+28F49uPRW0ikzfAbZhX3HCrt6klAxaz+vVMrDchXW7KA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL0PR10MB2817
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.486,18.0.858
 definitions=2022-05-11_07:2022-05-11,2022-05-11 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 mlxlogscore=999
 spamscore=0 bulkscore=0 adultscore=0 suspectscore=0 mlxscore=0
 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2202240000 definitions=main-2205110068
X-Proofpoint-GUID: dWP2pE2Tlh_B-OUnLuUJgD8LqK8AWo3r
X-Proofpoint-ORIG-GUID: dWP2pE2Tlh_B-OUnLuUJgD8LqK8AWo3r
X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_LOW,SPF_HELO_NONE,
        SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no
        version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org


> On May 11, 2022, at 10:23 AM, Greg KH <gregkh@linuxfoundation.org> wrote:
>=20
> On Wed, May 11, 2022 at 02:16:19PM +0000, Chuck Lever III wrote:
>>=20
>>=20
>>> On May 11, 2022, at 8:38 AM, Greg KH <gregkh@linuxfoundation.org> wrote=
:
>>>=20
>>> On Wed, May 11, 2022 at 12:03:13PM +0200, Wolfgang Walter wrote:
>>>> Hi,
>>>>=20
>>>> starting with 5.4.188 wie see a massive performance regression on our
>>>> nfs-server. It basically is serving requests very very slowly with cpu
>>>> utilization of 100% (with 5.4.187 and earlier it is 10%) so that it is
>>>> unusable as a fileserver.
>>>>=20
>>>> The culprit are commits (or one of it):
>>>>=20
>>>> c32f1041382a88b17da5736886da4a492353a1bb "nfsd: cleanup
>>>> nfsd_file_lru_dispose()"
>>>> 628adfa21815f74c04724abc85847f24b5dd1645 "nfsd: Containerise filecache
>>>> laundrette"
>>>>=20
>>>> (upstream 36ebbdb96b694dd9c6b25ad98f2bbd263d022b63 and
>>>> 9542e6a643fc69d528dfb3303f145719c61d3050)
>>>>=20
>>>> If I revert them in v5.4.192 the kernel works as before and performanc=
e is
>>>> ok again.
>>>>=20
>>>> I did not try to revert them one by one as any disruption of our nfs-s=
erver
>>>> is a severe problem for us and I'm not sure if they are related.
>>>>=20
>>>> 5.10 and 5.15 both always performed very badly on our nfs-server in a
>>>> similar way so we were stuck with 5.4.
>>>>=20
>>>> I now think this is because of 36ebbdb96b694dd9c6b25ad98f2bbd263d022b6=
3
>>>> and/or 9542e6a643fc69d528dfb3303f145719c61d3050 though I didn't tried =
to
>>>> revert them in 5.15 yet.
>>>=20
>>> Odds are 5.18-rc6 is also a problem?
>>=20
>> We believe that
>>=20
>> 6b8a94332ee4 ("nfsd: Fix a write performance regression")
>>=20
>> addresses the performance regression. It was merged into 5.18-rc.
>=20
> And into 5.17.4 if someone wants to try that release.

I don't have a lot of time to backport this one myself, so
I welcome anyone who wants to apply that commit to their
favorite LTS kernel and test it for us.


>>> If so, I'll just wait for the fix to get into Linus's tree as this does
>>> not seem to be a stable-tree-only issue.
>>=20
>> Unfortunately I've received a recent report that the fix introduces
>> a "sleep while spinlock is held" for NFSv4.0 in rare cases.
>=20
> Ick, not good, any potential fixes for that?

Not yet. I was at LSF last week, so I've just started digging
into this one. I've confirmed that the report is a real bug,
but we still don't know how hard it is to hit it with real
workloads.


--
Chuck Lever