From: Linus Torvalds <[email protected]>
commit e386dfc56f837da66d00a078e5314bc8382fab83 upstream.
Commit 054aa8d439b9 ("fget: check that the fd still exists after getting
a ref to it") fixed a race with getting a reference to a file just as it
was being closed. It was a fairly minimal patch, and I didn't think
re-checking the file pointer lookup would be a measurable overhead,
since it was all right there and cached.
But I was wrong, as pointed out by the kernel test robot.
The 'poll2' case of the will-it-scale.per_thread_ops benchmark regressed
quite noticeably. Admittedly it seems to be a very artificial test:
doing "poll()" system calls on regular files in a very tight loop in
multiple threads.
That means that basically all the time is spent just looking up file
descriptors without ever doing anything useful with them (not that doing
'poll()' on a regular file is useful to begin with). And as a result it
shows the extra "re-check fd" cost as a sore thumb.
Happily, the regression is fixable by just writing the code to loook up
the fd to be better and clearer. There's still a cost to verify the
file pointer, but now it's basically in the noise even for that
benchmark that does nothing else - and the code is more understandable
and has better comments too.
[ Side note: this patch is also a classic case of one that looks very
messy with the default greedy Myers diff - it's much more legible with
either the patience of histogram diff algorithm ]
Link: https://lore.kernel.org/lkml/20211210053743.GA36420@xsang-OptiPlex-9020/
Link: https://lore.kernel.org/lkml/[email protected]/
Reported-by: kernel test robot <[email protected]>
Tested-by: Carel Si <[email protected]>
Cc: Jann Horn <[email protected]>
Cc: Miklos Szeredi <[email protected]>
Signed-off-by: Linus Torvalds <[email protected]>
Signed-off-by: Baokun Li <[email protected]>
---
fs/file.c | 72 ++++++++++++++++++++++++++++++++++++++++++-------------
1 file changed, 56 insertions(+), 16 deletions(-)
diff --git a/fs/file.c b/fs/file.c
index 9d02352fa18c..79a76d04c7c3 100644
--- a/fs/file.c
+++ b/fs/file.c
@@ -817,28 +817,68 @@ void do_close_on_exec(struct files_struct *files)
spin_unlock(&files->file_lock);
}
-static struct file *__fget_files(struct files_struct *files, unsigned int fd,
- fmode_t mask, unsigned int refs)
+static inline struct file *__fget_files_rcu(struct files_struct *files,
+ unsigned int fd, fmode_t mask, unsigned int refs)
{
- struct file *file;
+ for (;;) {
+ struct file *file;
+ struct fdtable *fdt = rcu_dereference_raw(files->fdt);
+ struct file __rcu **fdentry;
- rcu_read_lock();
-loop:
- file = fcheck_files(files, fd);
- if (file) {
- /* File object ref couldn't be taken.
- * dup2() atomicity guarantee is the reason
- * we loop to catch the new file (or NULL pointer)
+ if (unlikely(fd >= fdt->max_fds))
+ return NULL;
+
+ fdentry = fdt->fd + array_index_nospec(fd, fdt->max_fds);
+ file = rcu_dereference_raw(*fdentry);
+ if (unlikely(!file))
+ return NULL;
+
+ if (unlikely(file->f_mode & mask))
+ return NULL;
+
+ /*
+ * Ok, we have a file pointer. However, because we do
+ * this all locklessly under RCU, we may be racing with
+ * that file being closed.
+ *
+ * Such a race can take two forms:
+ *
+ * (a) the file ref already went down to zero,
+ * and get_file_rcu_many() fails. Just try
+ * again:
*/
- if (file->f_mode & mask)
- file = NULL;
- else if (!get_file_rcu_many(file, refs))
- goto loop;
- else if (__fcheck_files(files, fd) != file) {
+ if (unlikely(!get_file_rcu_many(file, refs)))
+ continue;
+
+ /*
+ * (b) the file table entry has changed under us.
+ * Note that we don't need to re-check the 'fdt->fd'
+ * pointer having changed, because it always goes
+ * hand-in-hand with 'fdt'.
+ *
+ * If so, we need to put our refs and try again.
+ */
+ if (unlikely(rcu_dereference_raw(files->fdt) != fdt) ||
+ unlikely(rcu_dereference_raw(*fdentry) != file)) {
fput_many(file, refs);
- goto loop;
+ continue;
}
+
+ /*
+ * Ok, we have a ref to the file, and checked that it
+ * still exists.
+ */
+ return file;
}
+}
+
+static struct file *__fget_files(struct files_struct *files, unsigned int fd,
+ fmode_t mask, unsigned int refs)
+{
+ struct file *file;
+
+ rcu_read_lock();
+ file = __fget_files_rcu(files, fd, mask, refs);
rcu_read_unlock();
return file;
--
2.31.1
On Tue, Feb 15, 2022 at 02:51:07PM +0800, Baokun Li wrote:
> From: Linus Torvalds <[email protected]>
>
> commit e386dfc56f837da66d00a078e5314bc8382fab83 upstream.
>
> Commit 054aa8d439b9 ("fget: check that the fd still exists after getting
> a ref to it") fixed a race with getting a reference to a file just as it
> was being closed. It was a fairly minimal patch, and I didn't think
> re-checking the file pointer lookup would be a measurable overhead,
> since it was all right there and cached.
>
> But I was wrong, as pointed out by the kernel test robot.
>
> The 'poll2' case of the will-it-scale.per_thread_ops benchmark regressed
> quite noticeably. Admittedly it seems to be a very artificial test:
> doing "poll()" system calls on regular files in a very tight loop in
> multiple threads.
>
> That means that basically all the time is spent just looking up file
> descriptors without ever doing anything useful with them (not that doing
> 'poll()' on a regular file is useful to begin with). And as a result it
> shows the extra "re-check fd" cost as a sore thumb.
>
> Happily, the regression is fixable by just writing the code to loook up
> the fd to be better and clearer. There's still a cost to verify the
> file pointer, but now it's basically in the noise even for that
> benchmark that does nothing else - and the code is more understandable
> and has better comments too.
>
> [ Side note: this patch is also a classic case of one that looks very
> messy with the default greedy Myers diff - it's much more legible with
> either the patience of histogram diff algorithm ]
>
> Link: https://lore.kernel.org/lkml/20211210053743.GA36420@xsang-OptiPlex-9020/
> Link: https://lore.kernel.org/lkml/[email protected]/
> Reported-by: kernel test robot <[email protected]>
> Tested-by: Carel Si <[email protected]>
> Cc: Jann Horn <[email protected]>
> Cc: Miklos Szeredi <[email protected]>
> Signed-off-by: Linus Torvalds <[email protected]>
> Signed-off-by: Baokun Li <[email protected]>
> ---
> fs/file.c | 72 ++++++++++++++++++++++++++++++++++++++++++-------------
> 1 file changed, 56 insertions(+), 16 deletions(-)
Now queued up, thanks.
Any chance you can do this for 5.4 and older kernels too?
greg k-h
在 2022/2/18 2:55, Greg KH 写道:
> On Tue, Feb 15, 2022 at 02:51:07PM +0800, Baokun Li wrote:
>> From: Linus Torvalds <[email protected]>
>>
>> commit e386dfc56f837da66d00a078e5314bc8382fab83 upstream.
>>
>> Commit 054aa8d439b9 ("fget: check that the fd still exists after getting
>> a ref to it") fixed a race with getting a reference to a file just as it
>> was being closed. It was a fairly minimal patch, and I didn't think
>> re-checking the file pointer lookup would be a measurable overhead,
>> since it was all right there and cached.
>>
>> But I was wrong, as pointed out by the kernel test robot.
>>
>> The 'poll2' case of the will-it-scale.per_thread_ops benchmark regressed
>> quite noticeably. Admittedly it seems to be a very artificial test:
>> doing "poll()" system calls on regular files in a very tight loop in
>> multiple threads.
>>
>> That means that basically all the time is spent just looking up file
>> descriptors without ever doing anything useful with them (not that doing
>> 'poll()' on a regular file is useful to begin with). And as a result it
>> shows the extra "re-check fd" cost as a sore thumb.
>>
>> Happily, the regression is fixable by just writing the code to loook up
>> the fd to be better and clearer. There's still a cost to verify the
>> file pointer, but now it's basically in the noise even for that
>> benchmark that does nothing else - and the code is more understandable
>> and has better comments too.
>>
>> [ Side note: this patch is also a classic case of one that looks very
>> messy with the default greedy Myers diff - it's much more legible with
>> either the patience of histogram diff algorithm ]
>>
>> Link: https://lore.kernel.org/lkml/20211210053743.GA36420@xsang-OptiPlex-9020/
>> Link: https://lore.kernel.org/lkml/[email protected]/
>> Reported-by: kernel test robot <[email protected]>
>> Tested-by: Carel Si <[email protected]>
>> Cc: Jann Horn <[email protected]>
>> Cc: Miklos Szeredi <[email protected]>
>> Signed-off-by: Linus Torvalds <[email protected]>
>> Signed-off-by: Baokun Li <[email protected]>
>> ---
>> fs/file.c | 72 ++++++++++++++++++++++++++++++++++++++++++-------------
>> 1 file changed, 56 insertions(+), 16 deletions(-)
> Now queued up, thanks.
Thanks!
>
> Any chance you can do this for 5.4 and older kernels too?
It's my pleasure. I'll sync this patch to 5.4, 4.19, 4.14, 4.9, 4.4.
>
> greg k-h
> .
--
With Best Regards,
Baokun Li
.