Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <a6a0b2cc-1bc0-3884-8675-f90051454f94@oracle.com>
Date:   Tue, 10 Jan 2023 10:46:52 -0800
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:102.0)
 Gecko/20100101 Thunderbird/102.6.1
Subject: Re: [PATCH 1/1] NFSD: fix WARN_ON_ONCE in __queue_delayed_work
Content-Language: en-US
To:     Chuck Lever III <chuck.lever@oracle.com>
Cc:     Jeff Layton <jlayton@kernel.org>, Mike Galbraith <efault@gmx.de>,
        Linux NFS Mailing List <linux-nfs@vger.kernel.org>
References: <1673333310-24837-1-git-send-email-dai.ngo@oracle.com>
 <57dc06d57b4b643b4bf04daf28acca202c9f7a85.camel@kernel.org>
 <71672c07-5e53-31e6-14b1-e067fd56df57@oracle.com>
 <8C3345FB-6EDF-411A-B942-5AFA03A89BA2@oracle.com>
From:   dai.ngo@oracle.com
In-Reply-To: <8C3345FB-6EDF-411A-B942-5AFA03A89BA2@oracle.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?bUMydnd3b21VMkJlNTRiOGxvb2NPN1hGYzh0OEZXdCtYQTdTdGFLdS9EL3NV?=
 =?utf-8?B?Y0x6VDRQYnc2bUlxMU5IcTkrTjhEenJRdnZFMjQxaXhUd3lpZ2FsTmpVQkxJ?=
 =?utf-8?B?T0pmWHY2WkV5QUcyMVNyYnErdVhRemdEamVPcUdsbkxZazFTQzB4d0VZSE5R?=
 =?utf-8?B?R0dIdytYOW5RSnRaQWtYU2ZpVlY1b2VtWVBzNWg0WkJ1RUZhSnRleXF3TGUw?=
 =?utf-8?B?cEN0UFNDc1ZCS29iR3paem8zMWVoKzhLeXo3a1RoeXE1UVQ0WHFSVDV4eldN?=
 =?utf-8?B?QTlJQmU2SkpRek1tVWRWTTdMK3dheUV6TjBqSFIxTXJSMy85SHBTM3ZUWmd2?=
 =?utf-8?B?emdkVXhrM2FOeVpIVTYvR2xoOHNvWFhSMTJFL3E0YzNkeHVDY2NxYXREUVVP?=
 =?utf-8?B?dHhrMjkyOEJRdzBtMHZkZGEvVkM4L2krZXhRdHRBa0JTeDJ2QTZGd1F2bHcw?=
 =?utf-8?B?TXNxckxncEg0Z3B6TUdaUXFrRVR6emF4VWdDVmxJaEwwbGp2eFRwdGdSUnlP?=
 =?utf-8?B?VThpT2RjdWpkRFBDeXdZUkVHTXg2S0VONlhKc2xlRk52emtkTEl2Z21HeEQr?=
 =?utf-8?B?cHZrZVBjWW1VUE8wUlBZVnhmQkpsbmVFamNOc1hvVHB5YUcrdS9XTU1UaGlt?=
 =?utf-8?B?b0Z3OC9JVmc3ZkY4VFJWY1RjODhCMUFkL2lTcjlRWGFZZjBuZ21TdlRwMTdX?=
 =?utf-8?B?bzg1cGhpNkprWk9GVUw3Y0VzR05kWnY4N1JmVHBlanh0cEY0Y3dlWVh2KytU?=
 =?utf-8?B?aUV3NmJUaGpMcStWNFJpSFgrdm1FWjJVcHczdFlaSnByeVIzQ3VPak4wcWlt?=
 =?utf-8?B?ZjYyWkRVQ1pneXRkWUtWZW91WVZCR2M1RDBiUUt5dG0vVmt2MDRzTHJ4Vk1r?=
 =?utf-8?B?Z0FIT2JjN0JKZTljMjZwN2RORmdBMDgwMHllSWp6YXNkS0NnSDNMNVl5Um1X?=
 =?utf-8?B?bHF3RjREa1IvRnJIS3ZXbE9qZDBsOEVEWDFtcEhEVm9KL09KU01oR2dZWlpX?=
 =?utf-8?B?YnI3WHZKTjZEYjd4bXB2WFRGNjlhMjdUWW1BRzZjb3ZvT0lkL2tHZVdCWmhn?=
 =?utf-8?B?YUJxbEIvSGVKRVdldWNJdzUwUTRPYkZPWjZQR0NjcncxOWw0WnlTb1JVRlBC?=
 =?utf-8?B?UjVaMGI4L2xlWlZ6Q1dWcUlVOVFkOU9YeDViR1BsN3NrZjQ1UkU4a2R6ZmFt?=
 =?utf-8?B?UGptR0NnbCtTbVFNK3JYbFZlL01nQW9rWWNoMWgxcmFUV3BJSTNJUGdsRFBF?=
 =?utf-8?B?WHBUMFdTV0pVMzJ4SkpCYlhJNFFSVmQvMXRIT0c3WW9RT2FjUVRNRlZZZEFK?=
 =?utf-8?B?MzFTUk1JUGtCeE9xWEJMN284YS9nd0luUGdqUGVZU1J3eGtXWVJQNEIwS3R4?=
 =?utf-8?B?MENGZXRWbW1OY2UvT0ZMVFc5NzRrZ0hpb2tnazVaZUx0dCtGTCt5ck5TTUU2?=
 =?utf-8?B?L0g5WDhDR3hYQWJJS1VGaFhVdlRwbnFLZkpKSS9EcjZaNjJFck9KSWRHTjB1?=
 =?utf-8?B?NUp5UXlxK0pjdVh1eWdxQUtGdnRQVWVBZUpnbHZQbGQvMG1DYVZqTnVEb0Fv?=
 =?utf-8?B?TmUxV0VyWmtsNENjcUUwRkkrb0JKbCsrTVdRTnBNYTRObm1IdmdMSDdydUhL?=
 =?utf-8?B?K2ZLWUZXTFBwYXhOeW1uNlJWZ2dEamRtR29sUkhLamxoTTh2NFErQXA4OHlO?=
 =?utf-8?B?SnRaeEtGTG1yTk8vVHYvZVpnTG41T2Jlcmt4QjFxaHBsVlpHbUdobWhkbnlo?=
 =?utf-8?B?NlZLcXJXYUNWYjRBdndLZGhjWWh5L2JzcUsvSm04aEJKd0Q0NjBIazd3NVo1?=
 =?utf-8?B?OUo4MDg2YU5DMms2RGpTaUtwd1hYcWVYa3BzSlVabnRld0tudklYdk1rVVdz?=
 =?utf-8?B?MnkvS0w2Nk9sY0dpSCtBSlBRR3Brc2RrQjdTSFNhSmNDWHFhbGpuUUdKK28v?=
 =?utf-8?B?Yzk4bkNKNVArQW1aODhZTThBZlFRVHVRblZyRVpnQnpuSXJpdVdTUWc4WEVL?=
 =?utf-8?B?UGdzZFlQWEpVNU1sbmlIZHFrSW53Nk5xNGs2WEE3MHdqYXhnQlBoNCsxSDJa?=
 =?utf-8?B?VFBuSnFoejVGcnM2aXFjTnp0TUV3dUJydGZ6dDJQb3luUjN5bDA4ZmM1b1hw?=
 =?utf-8?Q?FtHkEu9HuraGeBM428IlSQUHh?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: UE4SYgwI0/VFcu6birloDIfeT7+JJilTeTMf3gsuZBKBVF3gXtKyUI3G84zAf78PQO3wf8hCSunVFBkPDbzpGEOYMfTQRcGK6JyhKFe3lFuzFRWFPUz/MaOhSlUX2UrWu/iNzXC9jGYFHaKYKBO9MnFuhFOU3XNYNN5XbUbEkSkcT67zEfc9un5SEzaPZxqoOQPTLbe28eQttnyq84OKEQzkDjUZ6pHWe0j8XDARRAHe9m078TtFthU7ycxSebHretcpj/pp0Lk2mPDwvBkmJXuNC7Ooos/xHo/LKlYoCm/Vg1WZ+y7JzgqVch82+TnR0voSKKVT/W82KkwW0X6ZmdHlmJF4hDkzIKA2279V4ivumOWv7wlpvHweSKOHcZZBsqr4QMf3ZmlHwrZmTMjIZblmJKWeAW+xfrW37rtDFfF2p/wEhEl3CHbitX0Zs3nAdTM7iA1fHHzuI+Kzj3/FmktgoEB3JpjGnrflJ925Szv2UfYdGCuSMDar1yF6Es0BOlT70eyM9CdwJ44/KdDcuRv5HW7MpFolk4I+J9CihH8Bb2S854cxV8xve8B6FgdiiK370TwhBdBRVrENmu8qNd+7EbXTldS8r9h7ZnY2CriX7L8nO3+T5hrYhHOR5nusm2atZ1HkY5dIoz+x5v6vXMhVNwWwp6NikZ1HhlcCOXkcpUqCfFPHMO+JVOfpOs4uBPXq/W5+yy6n8NhSa1CiF4LGnGvenerPRE1aYNc0nQmNMSJiR2S3Z3hKlubYUV9puZmJqBkWbNu89T28JC3mFHOfxo43G8v1zFDFWx6PedfA8XgI+oF/aSO53wfjhfk1
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 93825a95-16fa-4b85-32ce-08daf33b0c01
X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4257.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 Jan 2023 18:46:55.8558
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 1fFQa0XDX6JB73wXQx/LRoJoxoXIhevyfMd9IM/ytxF9QAYLCLSs3NPRaTTDrfncgjpQs/fwh+g3UpKBajxw1g==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN0PR10MB5383
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.205,Aquarius:18.0.923,Hydra:6.0.545,FMLib:17.11.122.1
 definitions=2023-01-10_07,2023-01-10_03,2022-06-22_01
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 spamscore=0 mlxscore=0
 mlxlogscore=999 suspectscore=0 bulkscore=0 adultscore=0 phishscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2212070000
 definitions=main-2301100122
X-Proofpoint-ORIG-GUID: b-d7UnLq4WFoZPubTUJUd64mVG_k51YM
X-Proofpoint-GUID: b-d7UnLq4WFoZPubTUJUd64mVG_k51YM
X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,RCVD_IN_DNSWL_LOW,
        RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org


On 1/10/23 10:17 AM, Chuck Lever III wrote:
>
>> On Jan 10, 2023, at 12:33 PM, Dai Ngo <dai.ngo@oracle.com> wrote:
>>
>>
>> On 1/10/23 2:30 AM, Jeff Layton wrote:
>>> On Mon, 2023-01-09 at 22:48 -0800, Dai Ngo wrote:
>>>> Currently nfsd4_state_shrinker_worker can be schduled multiple times
>>>> from nfsd4_state_shrinker_count when memory is low. This causes
>>>> the WARN_ON_ONCE in __queue_delayed_work to trigger.
>>>>
>>>> This patch allows only one instance of nfsd4_state_shrinker_worker
>>>> at a time using the nfsd_shrinker_active flag, protected by the
>>>> client_lock.
>>>>
>>>> Replace mod_delayed_work with queue_delayed_work since we
>>>> don't expect to modify the delay of any pending work.
>>>>
>>>> Fixes: 44df6f439a17 ("NFSD: add delegation reaper to react to low memory condition")
>>>> Reported-by: Mike Galbraith <efault@gmx.de>
>>>> Signed-off-by: Dai Ngo <dai.ngo@oracle.com>
>>>> ---
>>>>   fs/nfsd/netns.h     |  1 +
>>>>   fs/nfsd/nfs4state.c | 16 ++++++++++++++--
>>>>   2 files changed, 15 insertions(+), 2 deletions(-)
>>>>
>>>> diff --git a/fs/nfsd/netns.h b/fs/nfsd/netns.h
>>>> index 8c854ba3285b..801d70926442 100644
>>>> --- a/fs/nfsd/netns.h
>>>> +++ b/fs/nfsd/netns.h
>>>> @@ -196,6 +196,7 @@ struct nfsd_net {
>>>>   	atomic_t		nfsd_courtesy_clients;
>>>>   	struct shrinker		nfsd_client_shrinker;
>>>>   	struct delayed_work	nfsd_shrinker_work;
>>>> +	bool			nfsd_shrinker_active;
>>>>   };
>>>>     /* Simple check to find out if a given net was properly initialized */
>>>> diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
>>>> index ee56c9466304..e00551af6a11 100644
>>>> --- a/fs/nfsd/nfs4state.c
>>>> +++ b/fs/nfsd/nfs4state.c
>>>> @@ -4407,11 +4407,20 @@ nfsd4_state_shrinker_count(struct shrinker *shrink, struct shrink_control *sc)
>>>>   	struct nfsd_net *nn = container_of(shrink,
>>>>   			struct nfsd_net, nfsd_client_shrinker);
>>>>   +	spin_lock(&nn->client_lock);
>>>> +	if (nn->nfsd_shrinker_active) {
>>>> +		spin_unlock(&nn->client_lock);
>>>> +		return 0;
>>>> +	}
>>> Is this extra machinery really necessary? The bool and spinlock don't
>>> seem to be needed. Typically there is no issue with calling
>>> queued_delayed_work when the work is already queued. It just returns
>>> false in that case without doing anything.
>> When there are multiple calls to mod_delayed_work/queue_delayed_work
>> we hit the WARN_ON_ONCE's in __queue_delayed_work and __queue_work if
>> the work is queued but not execute yet.
> The delay argument of zero is interesting. If it's set to a value
> greater than zero, do you still see a problem?

I tried and tried but could not reproduce the problem that Mike
reported. I guess my VMs don't have fast enough cpus to make it
happen.

As Jeff mentioned, delay 0 should be safe and we want to run
the shrinker as soon as possible when memory is low.

-Dai

>
>
>> This problem was reported by Mike. I initially tried with only the
>> bool but that was not enough that was why the spinlock was added.
>> Mike verified that the patch fixed the problem.
>>
>> -Dai
>>
>>>>   	count = atomic_read(&nn->nfsd_courtesy_clients);
>>>>   	if (!count)
>>>>   		count = atomic_long_read(&num_delegations);
>>>> -	if (count)
>>>> -		mod_delayed_work(laundry_wq, &nn->nfsd_shrinker_work, 0);
>>>> +	if (count) {
>>>> +		nn->nfsd_shrinker_active = true;
>>>> +		spin_unlock(&nn->client_lock);
>>>> +		queue_delayed_work(laundry_wq, &nn->nfsd_shrinker_work, 0);
>>>> +	} else
>>>> +		spin_unlock(&nn->client_lock);
>>>>   	return (unsigned long)count;
>>>>   }
>>>>   @@ -6239,6 +6248,9 @@ nfsd4_state_shrinker_worker(struct work_struct *work)
>>>>     	courtesy_client_reaper(nn);
>>>>   	deleg_reaper(nn);
>>>> +	spin_lock(&nn->client_lock);
>>>> +	nn->nfsd_shrinker_active = 0;
>>>> +	spin_unlock(&nn->client_lock);
>>>>   }
>>>>     static inline __be32 nfs4_check_fh(struct svc_fh *fhp, struct nfs4_stid *stp)
> --
> Chuck Lever
>
>
>