Any suggestions on how to debug why rmmod fails with EBUSY?
I was trying to debug a problem where rmmod fails during an xfstest
run but couldn't find useful information on how to get ideas 'why'
rmmod is failing (in this case with EBUSY) and I could only find two
places where EBUSY would plausibly be returned but don't see the
pr_debug message in the log (e.g. "already dying") that I would expect
to get in those cases.
It also fails in this test scenario (which takes a while to reproduce
so isn't trivial to repro) with EBUSY when doing "rmmod --verbose
--force" and the --verbose didn't display any additional info.
I also tried "trace-cmd record -e *module*" which showed it (one call)
returning 0xFFFFFFF0 but nothing useful that I could see.
Any ideas on how to debug *why* an rmmod fails?
--
Thanks,
Steve
On 9/16/22 13:13, Steve French wrote:
> Any suggestions on how to debug why rmmod fails with EBUSY?
>
> I was trying to debug a problem where rmmod fails during an xfstest
> run but couldn't find useful information on how to get ideas 'why'
> rmmod is failing (in this case with EBUSY) and I could only find two
> places where EBUSY would plausibly be returned but don't see the
> pr_debug message in the log (e.g. "already dying") that I would expect
> to get in those cases.
>
> It also fails in this test scenario (which takes a while to reproduce
> so isn't trivial to repro) with EBUSY when doing "rmmod --verbose
> --force" and the --verbose didn't display any additional info.
>
> I also tried "trace-cmd record -e *module*" which showed it (one call)
> returning 0xFFFFFFF0 but nothing useful that I could see.
In situations like this I sometimes find it useful to use the ftrace
function_graph tracer. It can show you every single function call,
starting from some root function. Of course it's a bit of a footgun, so
be careful with it :)
I used it to trace the delete_module system call like so:
# trace-cmd record -p function_graph -g __x64_sys_delete_module \
-F rmmod $MODULE_TO_REMOVE
...
# trace-cmd report
rmmod-41656 [007] 5506.963846: funcgraph_entry: | __x64_sys_delete_module() {
rmmod-41656 [007] 5506.963846: funcgraph_entry: | capable() {
rmmod-41656 [007] 5506.963846: funcgraph_entry: | security_capable() {
... ...
rmmod-41656 [007] 5506.965521: funcgraph_entry: | free_module() {
rmmod-41656 [007] 5506.965521: funcgraph_entry: | mod_sysfs_teardown() {
rmmod-41656 [007] 5506.965522: funcgraph_entry: | mutex_lock() {
Of course, my module teardown succeeded, but you should be able to
cross reference your function calls with the source code to see where
an error may have occurred.
Regards,
Stephen
>
> Any ideas on how to debug *why* an rmmod fails?