The graph from my v3 patch series [0] which tries to resolve the virtual
memory lost bytes due to duplicates says it all:
+----------------------------------------------------------------------------+
14GB |-+ + + + + *+ +-|
| **** |
| *** |
| ** |
12GB |-+ ** +-|
| ** |
| ** |
| ** |
| ** |
10GB |-+ ** +-|
| ** |
| ** |
| ** |
8GB |-+ ** +-|
waste | ** ### |
| ** #### |
| ** ####### |
6GB |-+ **** #### +-|
| * #### |
| * #### |
| ***** #### |
4GB |-+ ** #### +-|
| ** #### |
| ** #### |
| ** #### |
2GB |-+ ** ##### +-|
| * #### |
| * #### Before ******* |
| **## + + + + After ####### |
+----------------------------------------------------------------------------+
0 50 100 150 200 250 300
CPUs count
So we really need to debug to see WTF, because really, WTF. The first
patch tries to answer the question if the issue is module auto-loading
being abused and that causing the issues. The patch proves that the
answer is no, but it does also help us find *a few* requests which can
get a bit of love to avoid duplicates. My system at least found one. So
it adds a debugging facility to let you do that.
As I was writing the commit log for my first patch series [0] I was noting
that this is it... and the obvious conclusion is that the culprit is udev
issuing requests per CPU for tons of modules. I didn't feel comfortable in
writing that this is it and we can't really do anything before really
trying hard. So I gave it a good 'ol college try. At first I wondered if
we could use file descriptor hints to just exlude users early on boot
before SYSTEM_RUNNING. I couldn't find much, but if there are some ways
to do that -- then the last patch can be simplified to do just that.
The second patch proves essentially that we can just send -EBUSY to
duplicate requests, at least for duplicate module loads and the world
doesn't fall apart. It *would* solve the issue. The patch however
borrows tons of the code from the first, and if we're realy going to
rely on something like that we may as well share. But I'm hopeful that
perhaps there are some jucier file descriptor tricks we can use to
just make a file mutually exlusivive and introduce a new kread which
lets finit_module() use that. The saving grace is that at least all
finit_module() calls *wait*, contray to request_module() calls and so
the solution can be much simpler.
The end result is 0 wasted virtual memory bytes.
Any ideas how not to make patch 2 suck as-is ?
Yes -- we can also go fix udev, or libkmod, and that's what should be
done. However, it seems silly to not fix if the fix is as trivial as
patch 2 demonstrates.
If you want to test / muck with all this you can use my branch
20230413-module-alloc-opts [1]:
[0] https://lkml.kernel.org/r/[email protected]
[1] https://git.kernel.org/pub/scm/linux/kernel/git/mcgrof/linux.git/log/?h=20230413-module-alloc-opts
Luis Chamberlain (2):
module: add debugging auto-load duplicate module support
kread: avoid duplicates
fs/kernel_read_file.c | 150 +++++++++++++++++++++++++
kernel/module/Kconfig | 40 +++++++
kernel/module/Makefile | 1 +
kernel/module/dups.c | 234 +++++++++++++++++++++++++++++++++++++++
kernel/module/internal.h | 15 +++
kernel/module/kmod.c | 23 +++-
kernel/module/main.c | 6 +-
7 files changed, 463 insertions(+), 6 deletions(-)
create mode 100644 kernel/module/dups.c
--
2.39.2
On Thu, Apr 13, 2023 at 10:28:38PM -0700, Luis Chamberlain wrote:
> At first I wondered if
> we could use file descriptor hints to just exlude users early on boot
> before SYSTEM_RUNNING. I couldn't find much, but if there are some ways
> to do that -- then the last patch can be simplified to do just that.
> The second patch proves essentially that we can just send -EBUSY to
> duplicate requests, at least for duplicate module loads and the world
> doesn't fall apart. It *would* solve the issue. The patch however
> borrows tons of the code from the first, and if we're realy going to
> rely on something like that we may as well share. But I'm hopeful that
> perhaps there are some jucier file descriptor tricks we can use to
> just make a file mutually exlusivive and introduce a new kread which
> lets finit_module() use that. The saving grace is that at least all
> finit_module() calls *wait*, contray to request_module() calls and so
> the solution can be much simpler.
>
> The end result is 0 wasted virtual memory bytes.
>
> Any ideas how not to make patch 2 suck as-is ?
The more I think about it, a file descriptor based approach would
have to loop over all tasks as we're not sure if userspace would
use the same thread and just fork requests or what. One would
then have to loop over all tasks and look for files that match the
same name. This approach is domain specific to internal kernel reads
and I am thinking now it is more appropriate.
Since this is a bootup issue for races with module loading what we
could do is just have say kernel_read_file_from_fd_excl() which
would just call a shared __kernel_read_file_from_fd(..., bool excl, ...)
and the kernel_read_file_from_fd() could just set that excl to false
while the new one sets it to true. Then finit_module() could just use
that.
Luis