Note that the immediate pressure for this patch should be relieved by the
NAPI patch series [1], but this sort of problem could easily arise again.
So would this change make sense?
When running heavy test workloads with KASAN enabled, RCU Tasks grace
periods can extend for many tens of seconds, significantly slowing
trace registration. Therefore, make the registration-side RCU Tasks
grace period be asynchronous via call_rcu_tasks().
Reported-by: Jakub Kicinski <[email protected]>
Reported-by: Alexei Starovoitov <[email protected]>
Reported-by: Chris Mason <[email protected]>
Signed-off-by: Paul E. McKenney <[email protected]>
Cc: Steven Rostedt <[email protected]>
Cc: Masami Hiramatsu <[email protected]>
Cc: Mark Rutland <[email protected]>
Cc: Mathieu Desnoyers <[email protected]>
Cc: <[email protected]>
diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c
index da1710499698b..6d21e4e97ed48 100644
--- a/kernel/trace/ftrace.c
+++ b/kernel/trace/ftrace.c
@@ -5366,6 +5366,13 @@ static void remove_direct_functions_hash(struct ftrace_hash *hash, unsigned long
}
}
+static void register_ftrace_direct_cb(struct rcu_head *rhp)
+{
+ struct ftrace_hash *fhp = container_of(rhp, struct ftrace_hash, rcu);
+
+ free_ftrace_hash(fhp);
+}
+
/**
* register_ftrace_direct - Call a custom trampoline directly
* for multiple functions registered in @ops
@@ -5464,10 +5471,8 @@ int register_ftrace_direct(struct ftrace_ops *ops, unsigned long addr)
out_unlock:
mutex_unlock(&direct_mutex);
- if (free_hash && free_hash != EMPTY_HASH) {
- synchronize_rcu_tasks();
- free_ftrace_hash(free_hash);
- }
+ if (free_hash && free_hash != EMPTY_HASH)
+ call_rcu_tasks(&free_hash->rcu, register_ftrace_direct_cb);
if (new_hash)
free_ftrace_hash(new_hash);
On Wed, 3 Apr 2024 11:53:14 -0700
"Paul E. McKenney" <[email protected]> wrote:
> @@ -5366,6 +5366,13 @@ static void remove_direct_functions_hash(struct ftrace_hash *hash, unsigned long
> }
> }
>
> +static void register_ftrace_direct_cb(struct rcu_head *rhp)
> +{
> + struct ftrace_hash *fhp = container_of(rhp, struct ftrace_hash, rcu);
> +
> + free_ftrace_hash(fhp);
> +}
> +
> /**
> * register_ftrace_direct - Call a custom trampoline directly
> * for multiple functions registered in @ops
> @@ -5464,10 +5471,8 @@ int register_ftrace_direct(struct ftrace_ops *ops, unsigned long addr)
> out_unlock:
> mutex_unlock(&direct_mutex);
>
> - if (free_hash && free_hash != EMPTY_HASH) {
> - synchronize_rcu_tasks();
> - free_ftrace_hash(free_hash);
> - }
> + if (free_hash && free_hash != EMPTY_HASH)
> + call_rcu_tasks(&free_hash->rcu, register_ftrace_direct_cb);
>
> if (new_hash)
> free_ftrace_hash(new_hash);
I'm getting ready to go on PTO, but a quick glance doesn't look like this
would cause any harm.
-- Steve
On Wed, Apr 03, 2024 at 03:29:12PM -0400, Steven Rostedt wrote:
> On Wed, 3 Apr 2024 11:53:14 -0700
> "Paul E. McKenney" <[email protected]> wrote:
>
> > @@ -5366,6 +5366,13 @@ static void remove_direct_functions_hash(struct ftrace_hash *hash, unsigned long
> > }
> > }
> >
> > +static void register_ftrace_direct_cb(struct rcu_head *rhp)
> > +{
> > + struct ftrace_hash *fhp = container_of(rhp, struct ftrace_hash, rcu);
> > +
> > + free_ftrace_hash(fhp);
> > +}
> > +
> > /**
> > * register_ftrace_direct - Call a custom trampoline directly
> > * for multiple functions registered in @ops
> > @@ -5464,10 +5471,8 @@ int register_ftrace_direct(struct ftrace_ops *ops, unsigned long addr)
> > out_unlock:
> > mutex_unlock(&direct_mutex);
> >
> > - if (free_hash && free_hash != EMPTY_HASH) {
> > - synchronize_rcu_tasks();
> > - free_ftrace_hash(free_hash);
> > - }
> > + if (free_hash && free_hash != EMPTY_HASH)
> > + call_rcu_tasks(&free_hash->rcu, register_ftrace_direct_cb);
> >
> > if (new_hash)
> > free_ftrace_hash(new_hash);
>
> I'm getting ready to go on PTO, but a quick glance doesn't look like this
> would cause any harm.
It is not urgent, as in v6.10 merge window at the earliest. So if I
don't hear from you in a few weeks, I will re-send. ;-)
Thanx, Paul