kmod 30 is out:
https://www.kernel.org/pub/linux/utils/kernel/kmod/kmod-30.tar.xz
https://www.kernel.org/pub/linux/utils/kernel/kmod/kmod-30.tar.sign
- Improvements
- Stop adding duplicate information on modules.builtin.alias.bin, just use
the modules.builtin.bin index
- Speedup depmod, particularly under qemu with emulated arch, by
avoiding a lot of open/read/close of modules.alias.bin. On an
emulated ARM rootfs, depmod with only 2 modules was taking ~32s
vs ~0.07s now.
- Add kmod_module_new_from_name_lookup() which allows doing a lookup by
module name, without considering the aliases. Other than that search
order is similar to kmod_module_new_from_lookup().
- modinfo learned the --modname option to explicitely show information
about the module, even if there is an alias with the same name. This
allows showing information about e.g. kernel/lib/crc32.ko, even if
kernel also exports a crc32 alias in modules.alias:
alias crc32 crc32_pclmul
alias crc32 crc32_generic
Same behavior will be used to other modules and to aliases provided
by user/distro.
- depmod.conf learned a new "excludedir" directive so distro/user can
configure more directories to be excluded from its search, besides
the hardcoded values "build" and "source".
- Better group modprobe options on help output under "Management, Query and General".
- modprobe learned a --wait <MSEC> option to be used together with -r
when removing a module. This allows modprobe to keep trying the
removal if it fails because the module is still in use. An exponential backoff
time is used for further retries.
The wait behavior provided by the kernel when not passing O_NONBLOCK
to delete_module() was removed in v3.13 due to not be used and the
consequences of having to support it in the kernel. However there may
be some users, particularly on testsuites for individual susbsystems, that
would want that. So provide a userspace implementation inside modprobe for
such users. "rmmod" doesn't have a --wait as it remains a bare minimal over
the API provided by the kernel. In future the --wait behavior can be added
to libkmod for testsuites not exec'ing modprobe for module removal.
- kmod_module_remove_module() learned a new flag to silence output when
caller wants to handle them - this is particularly important for the
--wait flag to modprobe, as it's not desired to keep seeing error messages
while waiting for the module to be unused.
- Add SM3 hash algo support to modinfo output, as already available in the kernel.
- Bug Fixes
- Fix modinfo output when showing information for a .ko module when running
on a kernel that has that module as builtin.
- Fix kmod_module_new_from_lookup() returning > 0 rather than 0
when it matches an alias.
- Fix modinfo segfault when module doesn't exist.
- Add missing function in the html documentation: kmod_get_dirname().
- Fix modprobe incorrectly handling number of arguments when prepending values from
MODPROBE_OPTIONS environment variable.
- Fix modprobe -r --remove-dependencies and since "dependencies" was a
misnomer, add the preferred argument option: "--remove-holders". This
is the same name used by the kernel. It allows users to also remove
other modules holding the one that is being removed.
- Fix off-by-one in max module name length in depmod.
- Infra/internal
- Start some changes in the out-of-tree test modules in kmod so they are useful
for being really inserted in the kernel rather than relying on kmod's mock
interface. This helps manual testing and may be used to exercise to test
changes in the kernel.
Shortlog is below:
HuaxinLu (1):
libkmod: Support SM3 hash algorithm
Lucas De Marchi (30):
libkmod: Prefer builtin index over builtin.alias
depmod: Do not duplicate builtin index
depmod: Stop opening modules.modinfo once per module
test-initstate: Check for negative value on error
libkmod-module: Fix return code for kmod_module_new_from_lookup()
gitignore: ignore gtk-doc.make
libkmod: Add helper function to iterate lookup options
libkmod: Update docs about indexes order
libkmod: Add lookup from module name
modinfo: Update help message with "modulename"
modinfo: Allow to force arg as module name
libkmod: Fix use of sizeof instead of ARRAY_SIZE
docs: Add missing functions to documentation
modprobe: Rename rmmod_do_deps_list
modprobe: Fix holders removal
modprobe: move check for remove_holders to caller
modprobe: Make rmmod_do_module() contain all the removal sequence
depmod: Fix writing over array length
modprobe: re-use modname variable
modprobe: Move -R to "Query options"
libkmod: Allow to ignore log message on module removal
module-playground: Add debugfs entry in mod-simple
util: Add time-related functions from testsuite
util: Add msec variants for time-related functions
util: Add exponential backoff sleep
testsuite: Add tests for sleep calculation
modprobe: Add --wait
Keep only one readme
README: Update optional dependencies
kmod 30
Luis Chamberlain (1):
libkmod: add a library notice log level print
Masahiro Yamada (2):
modprobe: fix the NULL-termination of new_argv
modprobe: remove unneeded variable str_start
Michal Suchanek (1):
libkmod: Set builtin to no when module is created from path.
Saul Wold (1):
depmod: Add support for excluding a directory
Yauheni Kaliuta (1):
libkmod-module: check "new_from_name" return value in get_builtin
Thank you all for the contributions.
Lucas De Marchi
On Thu, Jun 30, 2022 at 08:36:21AM -0700, Lucas De Marchi wrote:
> - modprobe learned a --wait <MSEC> option to be used together with -r
> when removing a module. This allows modprobe to keep trying the
> removal if it fails because the module is still in use. An exponential backoff
> time is used for further retries.
>
> The wait behavior provided by the kernel when not passing O_NONBLOCK
> to delete_module() was removed in v3.13 due to not be used and the
> consequences of having to support it in the kernel. However there may
> be some users, particularly on testsuites for individual susbsystems, that
> would want that. So provide a userspace implementation inside modprobe for
> such users. "rmmod" doesn't have a --wait as it remains a bare minimal over
> the API provided by the kernel. In future the --wait behavior can be added
> to libkmod for testsuites not exec'ing modprobe for module removal.
Sorry for the super late review, I was swamped. OK so the only issue
I can think of is that rmmod *used* to support the kernel wait support
with $(rmmod --wait) so wouldn't this be odd?
It is why I had gone with:
-p | --remove-patiently patiently removes the module
-t | --timeout timeout in ms to remove the module
You would know better though.
Also just curious, is it really terrible to just support waiting
forever?
Luis
On Thu, Jun 30, 2022 at 03:09:32PM -0700, Luis Chamberlain wrote:
>On Thu, Jun 30, 2022 at 08:36:21AM -0700, Lucas De Marchi wrote:
>> - modprobe learned a --wait <MSEC> option to be used together with -r
>> when removing a module. This allows modprobe to keep trying the
>> removal if it fails because the module is still in use. An exponential backoff
>> time is used for further retries.
>>
>> The wait behavior provided by the kernel when not passing O_NONBLOCK
>> to delete_module() was removed in v3.13 due to not be used and the
>> consequences of having to support it in the kernel. However there may
>> be some users, particularly on testsuites for individual susbsystems, that
>> would want that. So provide a userspace implementation inside modprobe for
>> such users. "rmmod" doesn't have a --wait as it remains a bare minimal over
>> the API provided by the kernel. In future the --wait behavior can be added
>> to libkmod for testsuites not exec'ing modprobe for module removal.
>
>Sorry for the super late review, I was swamped. OK so the only issue
>I can think of is that rmmod *used* to support the kernel wait support
>with $(rmmod --wait) so wouldn't this be odd?
any reason not to use modprobe -r? Argument for rmmod supporting it in
the past is that the wait was implemented on the kernel side and rmmod
is the minimum wrapper around what the kernel provides.
On the other side, user shouldn't need to know where that is
implemented.
Over time libkmod grew much more to support loading/querying modules
rather than removing. I think for next version I will move some of the
module-removal support to libkmod rather than modprobe/rmmod. Then we
can think again on supporting that flag there.
>
>It is why I had gone with:
>
> -p | --remove-patiently patiently removes the module
> -t | --timeout timeout in ms to remove the module
>
>You would know better though.
>
>Also just curious, is it really terrible to just support waiting
>forever?
is there a use case for that? If we are trying to cover some races, I
imagine a small timeout would be sufficient. Also notice that if the
timeout is too big, so will be the interval between the retries. On
your v2 I had suggested polling the refcnt so we would get notificed
on changes, but as you also noticed, that didn't work very well. So I
went back to a time-based retry solution.
if there is a use-case, should we cap the interval between retries?
thanks
Lucas De Marchi
>
> Luis
On Thu, Jun 30, 2022 at 03:33:23PM -0700, Lucas De Marchi wrote:
> On Thu, Jun 30, 2022 at 03:09:32PM -0700, Luis Chamberlain wrote:
> > Sorry for the super late review, I was swamped. OK so the only issue
> > I can think of is that rmmod *used* to support the kernel wait support
> > with $(rmmod --wait) so wouldn't this be odd?
>
> any reason not to use modprobe -r?
I was referring to old scripts which may have used $(rmmod --wait) before.
But since support for that was ripped, then yeah I can see that should
not be an issue.
However I can think of *one* issue, did we ever support `modprobe--wait`?
Because the way fstests / blktests would implement this feature
detection is with something like this now:
_has_modprobe_patient()
{
modprobe --help >& /dev/null || return 1
modprobe --help | grep -q -1 "remove-patiently" || return 1
return 0
}
> > It is why I had gone with:
> >
> > -p | --remove-patiently patiently removes the module
> > -t | --timeout timeout in ms to remove the module
> >
> > You would know better though.
> >
> > Also just curious, is it really terrible to just support waiting
> > forever?
>
> is there a use case for that? If we are trying to cover some races, I
> imagine a small timeout would be sufficient. Also notice that if the
> timeout is too big, so will be the interval between the retries. On
> your v2 I had suggested polling the refcnt so we would get notificed
> on changes, but as you also noticed, that didn't work very well. So I
> went back to a time-based retry solution.
>
> if there is a use-case, should we cap the interval between retries?
I really can't think of a use case except for catching glaring
unexpected bugs in test suites where the kernel developer would
really like to know something really bad happened, but even then
a timeout is likely desirable.
So just a heads up the timeout I'll use for fstests / blktests will be
of 100 seconds.
Thanks for this work!
Luis
On Fri, Jul 01, 2022 at 10:49:10AM -0700, Luis Chamberlain wrote:
>On Thu, Jun 30, 2022 at 03:33:23PM -0700, Lucas De Marchi wrote:
>> On Thu, Jun 30, 2022 at 03:09:32PM -0700, Luis Chamberlain wrote:
>> > Sorry for the super late review, I was swamped. OK so the only issue
>> > I can think of is that rmmod *used* to support the kernel wait support
>> > with $(rmmod --wait) so wouldn't this be odd?
>>
>> any reason not to use modprobe -r?
>
>I was referring to old scripts which may have used $(rmmod --wait) before.
>But since support for that was ripped, then yeah I can see that should
>not be an issue.
>
>However I can think of *one* issue, did we ever support `modprobe--wait`?
no
>
>Because the way fstests / blktests would implement this feature
>detection is with something like this now:
>
>_has_modprobe_patient()
>{
> modprobe --help >& /dev/null || return 1
> modprobe --help | grep -q -1 "remove-patiently" || return 1
> return 0
>}
the grep would need to be changed to something like "-w, --wait"
>
>> > It is why I had gone with:
>> >
>> > -p | --remove-patiently patiently removes the module
>> > -t | --timeout timeout in ms to remove the module
>> >
>> > You would know better though.
>> >
>> > Also just curious, is it really terrible to just support waiting
>> > forever?
>>
>> is there a use case for that? If we are trying to cover some races, I
>> imagine a small timeout would be sufficient. Also notice that if the
>> timeout is too big, so will be the interval between the retries. On
>> your v2 I had suggested polling the refcnt so we would get notificed
>> on changes, but as you also noticed, that didn't work very well. So I
>> went back to a time-based retry solution.
>>
>> if there is a use-case, should we cap the interval between retries?
>
>I really can't think of a use case except for catching glaring
>unexpected bugs in test suites where the kernel developer would
>really like to know something really bad happened, but even then
>a timeout is likely desirable.
yeah... if developer wants to possibly wait for a long time, `--wait -1`
is available to wait for years.
>
>So just a heads up the timeout I'll use for fstests / blktests will be
>of 100 seconds.
yeah... 100 seconds still make more sense than -1 IMO
Lucas De Marchi
>
>Thanks for this work!
>
> Luis
On Fri, Jul 01, 2022 at 11:13:21AM -0700, Lucas De Marchi wrote:
> On Fri, Jul 01, 2022 at 10:49:10AM -0700, Luis Chamberlain wrote:
> > On Thu, Jun 30, 2022 at 03:33:23PM -0700, Lucas De Marchi wrote:
> > > On Thu, Jun 30, 2022 at 03:09:32PM -0700, Luis Chamberlain wrote:
> > > > Sorry for the super late review, I was swamped. OK so the only issue
> > > > I can think of is that rmmod *used* to support the kernel wait support
> > > > with $(rmmod --wait) so wouldn't this be odd?
> > >
> > > any reason not to use modprobe -r?
> >
> > I was referring to old scripts which may have used $(rmmod --wait) before.
> > But since support for that was ripped, then yeah I can see that should
> > not be an issue.
> >
> > However I can think of *one* issue, did we ever support `modprobe--wait`?
>
> no
>
> >
> > Because the way fstests / blktests would implement this feature
> > detection is with something like this now:
> >
> > _has_modprobe_patient()
> > {
> > modprobe --help >& /dev/null || return 1
> > modprobe --help | grep -q -1 "remove-patiently" || return 1
> > return 0
> > }
>
> the grep would need to be changed to something like "-w, --wait"
Yes of course, that's easy, but we modprobe ever supportedl "--wait"
this would be a problem. Since it did not, we are good then!
> > > > It is why I had gone with:
> > > >
> > > > -p | --remove-patiently patiently removes the module
> > > > -t | --timeout timeout in ms to remove the module
> > > >
> > > > You would know better though.
> > > >
> > > > Also just curious, is it really terrible to just support waiting
> > > > forever?
> > >
> > > is there a use case for that? If we are trying to cover some races, I
> > > imagine a small timeout would be sufficient. Also notice that if the
> > > timeout is too big, so will be the interval between the retries. On
> > > your v2 I had suggested polling the refcnt so we would get notificed
> > > on changes, but as you also noticed, that didn't work very well. So I
> > > went back to a time-based retry solution.
> > >
> > > if there is a use-case, should we cap the interval between retries?
> >
> > I really can't think of a use case except for catching glaring
> > unexpected bugs in test suites where the kernel developer would
> > really like to know something really bad happened, but even then
> > a timeout is likely desirable.
>
> yeah... if developer wants to possibly wait for a long time, `--wait -1`
> is available to wait for years.
:) good call
> > So just a heads up the timeout I'll use for fstests / blktests will be
> > of 100 seconds.
>
> yeah... 100 seconds still make more sense than -1 IMO
Great we'll go with that. Hopefully this will put a nail on the flaky
modules issue for good!
Luis