Received-SPF: pass (google.com: domain of linux-nfs+bounces-169-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1;
Date: Wed, 29 Nov 2023 09:04:47 -0500
From: Chuck Lever <chuck.lever@oracle.com>
To: NeilBrown <neilb@suse.de>
Cc: Christian Brauner <brauner@kernel.org>, Al Viro <viro@zeniv.linux.org.uk>,
        Jeff Layton <jlayton@kernel.org>, linux-fsdevel@vger.kernel.org,
        linux-kernel@vger.kernel.org, linux-nfs@vger.kernel.org
Subject: Re: [PATCH/RFC] core/nfsd: allow kernel threads to use task_work.
Message-ID: <ZWdE/7bNvxcsY3ae@tissot.1015granger.net>
References: <170112272125.7109.6245462722883333440@noble.neil.brown.name>
 <ZWUfNyO6OG/+aFuo@tissot.1015granger.net>
 <170113056683.7109.13851405274459689039@noble.neil.brown.name>
 <20231128-blumig-anreichern-b9d8d1dc49b3@brauner>
 <170121362397.7109.17858114692838122621@noble.neil.brown.name>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <170121362397.7109.17858114692838122621@noble.neil.brown.name>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: 
	=?us-ascii?Q?IZwjB88xynGRwjtnFjx673sUT+r16iugIpwMD5x68yKUcu8dFFjrTSapWjug?=
 =?us-ascii?Q?K1uBRsMBwOaLNqJ056x76Sakf9/9GppLDckGh620Uaj+6w+Zfzzd0fpe3fYj?=
 =?us-ascii?Q?iGPnWbmWRXtqzZHLnTP6Gv4WR7dYwRpbM25xDkHSkaLRpCQIXjVxRm6DtOCi?=
 =?us-ascii?Q?4WK0Sc+48o3c/xLf5+DzeSwEF+F53Hf4c1Io8fwRdxcpWBrvqxrR3wqpo5EN?=
 =?us-ascii?Q?cNgeZH/A6BtgM+0lCYE4dVSkN0wntqDerXNlLWOzs2HEsFDS152zekqvACpX?=
 =?us-ascii?Q?p5dUt+aielRBZmPBmsFpeNUkRsyPSefpiYwSu87nZtBDL5Rh52a/GDyvhY4s?=
 =?us-ascii?Q?xVivuFAiSqP6TSZMwVgMvj14z/cFCiKCFp54S+XDFGhcjoNAS6+iJajw584N?=
 =?us-ascii?Q?oyz7a0L1IQHirwRaZeygpxjKpHJXZ0N+7p+tGjoCXRTKqLjc9JPiJsoItzvO?=
 =?us-ascii?Q?a8LCMf6tv8STlsCP9H+03KM0YGmsgXHxY63b6Ce++Zs1og3eE0D8kznVvnUI?=
 =?us-ascii?Q?KEjLzpH+lwHSx+drim5U0plu+1kILVEYq0HqjiHwnjY2tuSC7XllU7cLmSmp?=
 =?us-ascii?Q?NrK5roCquwxF2d9d6Xx+ZLpIjjOoffUwxl5y44sDxUz9or5syAZDR5ywRXmH?=
 =?us-ascii?Q?NwkxDOzn9sUbyGvMgL0Xa+rxlEVzbHHAstsLwIBxC2S5lrh/Ho6YOcKdCsLB?=
 =?us-ascii?Q?jOM/4xsfRE68bMAwf0dYaA72Y8mcdI0/BenEs+YDbk9CPQWNG3sdj0Vg9yao?=
 =?us-ascii?Q?HXI0APvbhMD3GVf99V7RDk8/rt9dyQ4DszefmGUG4oaSNKPRv+f0yOlt+ebd?=
 =?us-ascii?Q?MMW/k7ovY+dgPUvZt9eGq01tgDgKOBEBvRUsbEJzknFfRP5W8SK2JBrFRbIo?=
 =?us-ascii?Q?P1yggNw2sxOGO6RIlXoZRVSmsk7iRIoAw4d9sJX+A1a0EQ2ZgFUYQ91Qek5A?=
 =?us-ascii?Q?RqBCem7mAH0fnsbtPXNMrHeX+CYusreVr35MQ26E9TXAs2PO3NQgeE8IVjRj?=
 =?us-ascii?Q?jmLtuU4TlkbBEN+V9n/WsriWUzGLs+sVYaeMM1FIrW8pjtGQoJBacI4OLaK8?=
 =?us-ascii?Q?Msz7Y2mS/xW/kSkZQJCVU19SpvKy2rXuX3e09n9JNikiAevWwapUsSTtjGcz?=
 =?us-ascii?Q?elhbDT/tQEZq16z+S7KXiSp2sZ5jmlSZ49hC9GaBhDHGOa0TmT6ZJz3/yukz?=
 =?us-ascii?Q?2H2kGnRjMlQ+Wg2g6LIaR9d+qteV88UAbfj2o2J8iDiWDPgKCTovW3VVlWsD?=
 =?us-ascii?Q?IWluPnlpNpPJT19knjFH87j+jIfIAZVh3E312cPJIYdQ5b/NEutDpWd0EZXj?=
 =?us-ascii?Q?lFuAm1S/wPqskIied9pjFhKkBeN7AEQQOq171kOHlZp9wQdP7tk+M1lIq/32?=
 =?us-ascii?Q?jE2vJz4TQmkuoDxeLXb0vRBUYVHCX4HYDpjP41wOoLUBAd6U1Hv2jx1FCUcs?=
 =?us-ascii?Q?K/iAd2uUDx08eygoK+bHKF/+o3ItHazS+W47OqKAkp6cgb8kboy8S3iZkVoE?=
 =?us-ascii?Q?H+Trc4ltEWIDmB2DEtFozfQY4rdSYdOwHnEX9BsKy3kYe+KEGAixKo95GMJa?=
 =?us-ascii?Q?NTKIJKodBh0u01wbw7zkMgnjgtGIaL/WQogfnyPPPGFh8T0F3LVjAjbYDvaS?=
 =?us-ascii?Q?vg=3D=3D?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 
	NnvZxhgPomjoihP8BHYUD/sbl2wdOwLCqwuNCSntX43ktTnxMrp9MZvypd7GZzRC5mUFsZAH6+2jeo9RdCqIQMrsqqDbpJkWtA8iXiduz13tpmb2zaxrWGphKM6K5n5WPQ7gWYiBa9OPMTeE5LkVo1GtsEMGXLt0Ugx6HfMujYQTG2SehuUw0ha4W/Jdd9VhV6F5MYYyHeCyeR9lSRK/1RA6Jk4aA61sk3raQghLBHu6PNYU++0QYYNrvmeXdzqbdzICy/ltlxjcBJVtcajgcW3GCVW3G0uPdgYkjE+co3P8gfz2wUqaeNB4v7ky6hz8xa8ClMiyHyYDVbLqz61H+iDFatSr2gBYtqVb8c2D90ROCFPv6oC+8g1lX7D3ZzvCzy2G1rNIPD3DxwxKNu/tdY/Ro0Qgp+OZvWIRiS0CYWU43qMsJNqkV35AfZKZPkVaal4NpH5kyROG7q/lqhTQY3vXzQiNrggOfEypmilvy/tECj2Eii191Dqynf0+52M3Ssbphp8eLtQ4Nib26SWxd26jCevbDalsdL6EPA1gqsksRqwEYB08NuNzSQd5bP+e4IsNBFLLRksgx8BWhU7NNsJ+uioTq0kLbNwF7BrRLbnUJO9xMRjN2E85EcKIVtFAkJEOD55PRM78Lchg/mIjSgRLOmaKhMeGdJipqC6zFzYMhVlbJ5IwDNqzq54CH3i32WapA+w0hxHJYztsZ8VR9WKqR0Vmbj2D10U9JEOTKn3YhzNJQN3EaKFe1ZM/oA3RkCFqkxvAxNiYYvoYiJ453/V9Q06kh68Sy3um3/aeE+OLPIqHanSj6BFahBHf1T5QPn4HsvLKv5nMKC6dv6P+J92QAYb4Z39xjsBQJpI+OQ4=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 2a004e39-46f8-402e-0245-08dbf0e42713
X-MS-Exchange-CrossTenant-AuthSource: BN0PR10MB5128.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 29 Nov 2023 14:04:50.3123
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: soq+mCS5k/ZF/taMRR76d5mq2+DEoovH/uyik/UjbYOixRThAVONJNtLHioM7Rd3bZHXCkzsYmrEHpbf+6r4ig==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR10MB6309
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26
 definitions=2023-11-29_11,2023-11-29_01,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 malwarescore=0
 suspectscore=0 spamscore=0 mlxlogscore=907 mlxscore=0 phishscore=0
 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2311060000 definitions=main-2311290106
X-Proofpoint-GUID: ur_Nf2njMmzub_d6eMIXyC_itP-9m2N2
X-Proofpoint-ORIG-GUID: ur_Nf2njMmzub_d6eMIXyC_itP-9m2N2

On Wed, Nov 29, 2023 at 10:20:23AM +1100, NeilBrown wrote:
> On Wed, 29 Nov 2023, Christian Brauner wrote:
> > [Reusing the trimmed Cc]
> > 
> > On Tue, Nov 28, 2023 at 11:16:06AM +1100, NeilBrown wrote:
> > > On Tue, 28 Nov 2023, Chuck Lever wrote:
> > > > On Tue, Nov 28, 2023 at 09:05:21AM +1100, NeilBrown wrote:
> > > > > 
> > > > > I have evidence from a customer site of 256 nfsd threads adding files to
> > > > > delayed_fput_lists nearly twice as fast they are retired by a single
> > > > > work-queue thread running delayed_fput().  As you might imagine this
> > > > > does not end well (20 million files in the queue at the time a snapshot
> > > > > was taken for analysis).
> > > > > 
> > > > > While this might point to a problem with the filesystem not handling the
> > > > > final close efficiently, such problems should only hurt throughput, not
> > > > > lead to memory exhaustion.
> > > > 
> > > > I have this patch queued for v6.8:
> > > > 
> > > > https://git.kernel.org/pub/scm/linux/kernel/git/cel/linux.git/commit/?h=nfsd-next&id=c42661ffa58acfeaf73b932dec1e6f04ce8a98c0
> > > > 
> > > 
> > > Thanks....
> > > I think that change is good, but I don't think it addresses the problem
> > > mentioned in the description, and it is not directly relevant to the
> > > problem I saw ... though it is complicated.
> > > 
> > > The problem "workqueue ...  hogged cpu..." probably means that
> > > nfsd_file_dispose_list() needs a cond_resched() call in the loop.
> > > That will stop it from hogging the CPU whether it is tied to one CPU or
> > > free to roam.
> > > 
> > > Also that work is calling filp_close() which primarily calls
> > > filp_flush().
> > > It also calls fput() but that does minimal work.  If there is much work
> > > to do then that is offloaded to another work-item.  *That* is the
> > > workitem that I had problems with.
> > > 
> > > The problem I saw was with an older kernel which didn't have the nfsd
> > > file cache and so probably is calling filp_close more often.  So maybe
> > > my patch isn't so important now.  Particularly as nfsd now isn't closing
> > > most files in-task but instead offloads that to another task.  So the
> > > final fput will not be handled by the nfsd task either.
> > > 
> > > But I think there is room for improvement.  Gathering lots of files
> > > together into a list and closing them sequentially is not going to be as
> > > efficient as closing them in parallel.
> > > 
> > > > 
> > > > > For normal threads, the thread that closes the file also calls the
> > > > > final fput so there is natural rate limiting preventing excessive growth
> > > > > in the list of delayed fputs.  For kernel threads, and particularly for
> > > > > nfsd, delayed in the final fput do not impose any throttling to prevent
> > > > > the thread from closing more files.
> > > > 
> > > > I don't think we want to block nfsd threads waiting for files to
> > > > close. Won't that be a potential denial of service?
> > > 
> > > Not as much as the denial of service caused by memory exhaustion due to
> > > an indefinitely growing list of files waiting to be closed by a single
> > > thread of workqueue.
> > 
> > It seems less likely that you run into memory exhausting than a DOS
> > because nfsd() is busy closing fds. Especially because you default to
> > single nfsd thread afaict.
> 
> An nfsd thread would not end up being busy closing fds any more than it
> can already be busy reading data or busy syncing out changes or busying
> renaming a file.
> Which it is say: of course it can be busy doing this, but doing this sort
> of thing is its whole purpose in life.
> 
> If an nfsd thread only completes the close that it initiated the close
> on (which is what I am currently proposing) then there would be at most
> one, or maybe 2, fds to close after handling each request.

Closing files more aggressively would seem to entirely defeat the
purpose of the file cache, which is to avoid the overhead of opens
and closes on frequently-used files.

And usually Linux prefers to let the workload consume as many free
resources as possible before it applies back pressure or cache
eviction.

IMO the first step should be removing head-of-queue blocking from
the file cache's background closing mechanism. That might be enough
to avoid forming a backlog in most cases.


> > > For NFSv3 it is more complex.  On the kernel where I saw a problem the
> > > filp_close happen after each READ or WRITE (though I think the customer
> > > was using NFSv4...).  With the file cache there is no thread that is
> > > obviously responsible for the close.
> > > To get the sort of throttling that I think is need, we could possibly
> > > have each "nfsd_open" check if there are pending closes, and to wait for
> > > some small amount of progress.
> > > 
> > > But don't think it is reasonable for the nfsd threads to take none of
> > > the burden of closing files as that can result in imbalance.
> > 
> > It feels that this really needs to be tested under a similar workload in
> > question to see whether this is a viable solution.
> 
> Creating that workload might be a challenge.  I know it involved
> accessing 10s of millions of files with a server that was somewhat
> memory constrained.  I don't know anything about the access pattern.
> 
> Certainly I'll try to reproduce something similar by inserting delays in
> suitable places.  This will help exercise the code, but won't really
> replicate the actual workload.

It's likely that the fundamental bottleneck is writeback during
close.


-- 
Chuck Lever