I'm debugging crash of our product which does live kernel patching together
with ISV security scanner which uses ftrace kprobes to do whatever it does.
What happens is that is ftrace ever detects .text change, refuses to patch
and prints a warning with FTRACE_WARN_ON_ONCE() then there is reliable way
to cause UAF on kmalloc-512 cache by trying to register kprobe with
perf_event_open() and then unregistering it by exiting the process.
1) live kernel patching happens, first instruction of some function changes
2) kprobe on that function is registered with perf_event_open()
WARNING: CPU: 5 PID: 2109 at kernel/trace/ftrace.c:1853 ftrace_bug+0x25d/0x270
[<ffffffff811638ed>] ftrace_bug+0x25d/0x270
[<ffffffff81065571>] ftrace_replace_code+0x2b1/0x420
[<ffffffff81163f9a>] ftrace_modify_all_code+0x6a/0xb0
[<ffffffff810656f0>] arch_ftrace_update_code+0x10/0x20
[<ffffffff81164077>] ftrace_run_update_code+0x17/0x70
[<ffffffff81165512>] ftrace_set_hash+0x1c2/0x1f0
[<ffffffff8126fee0>] ? SyS_dup2+0x60/0x60
[<ffffffff8126fee0>] ? SyS_dup2+0x60/0x60
[<ffffffff811655a0>] ftrace_set_filter_ip+0x60/0x70
[<ffffffff8179624c>] arm_kprobe+0x9c/0x140
[<ffffffff81796368>] enable_kprobe+0x78/0xa0
[<ffffffff81187bab>] enable_trace_kprobe+0x7b/0x120
[<ffffffff81797e5f>] kprobe_register+0x2f/0x60
[<ffffffff8118348a>] perf_trace_event_init+0x1aa/0x230
[<ffffffff811836b7>] perf_kprobe_init+0xa7/0xf0
[<ffffffff811a8919>] perf_kprobe_event_init+0x49/0x70
[<ffffffff811aa569>] perf_try_init_event+0x99/0xc0
[<ffffffff811b29f2>] perf_init_event+0x92/0x150
[<ffffffff811b2fa1>] perf_event_alloc+0x4f1/0x910
[<ffffffff811b3789>] SYSC_perf_event_open+0x3c9/0xe50
[<ffffffff811b4679>] SyS_perf_event_open+0x9/0x10
[<ffffffff81799f92>] system_call_fastpath+0x25/0x2a
ftrace failed to modify [<ffffffff8126fee0>] SyS_dup+0x0/0x120
actual: e9:4b:50:2e:3f
3) FTRACE_WARN_ON_ONCE() calls ftrace_kill() which sets ftrace_disabled:
ftrace_disabled = 1;
4) process exits, all kprobes are unregistered but "ftrace_disabled" is 1 now
so disarming fails with -ENODEV:
[ 253.042821] WARNING: CPU: 2 PID: 1682 at kernel/kprobes.c:1006 disarm_kprobe+0x102/0x180
[ 253.044424] Failed to disarm kprobe-ftrace at do_exit+0x0/0xa30 (-19)
[ 253.086658] [<ffffffff811655a0>] ? ftrace_set_filter_ip+0x60/0x70
[ 253.089599] [<ffffffff810a1810>] ? mm_update_next_owner+0x230/0x230
[ 253.090878] [<ffffffff81796492>] disarm_kprobe+0x102/0x180
[ 253.092084] [<ffffffff817965dd>] __disable_kprobe+0xcd/0xf0
[ 253.093242] [<ffffffff81796c13>] disable_kprobe+0x23/0x40
[ 253.094390] [<ffffffff81187d14>] disable_trace_kprobe+0xc4/0x100
[ 253.095751] [<ffffffff81797e53>] kprobe_register+0x23/0x60
[ 253.097420] [<ffffffff8118327c>] perf_trace_event_unreg.isra.1+0x3c/0xa0
[ 253.098866] [<ffffffff81183730>] perf_kprobe_destroy+0x30/0x40
[ 253.100085] [<ffffffff811af50a>] _free_event+0xfa/0x2f0
[ 253.101147] [<ffffffff811af769>] put_event+0x19/0x20
[ 253.102154] [<ffffffff811af993>] perf_event_release_kernel+0x223/0x300
[ 253.103457] [<ffffffff811afa80>] perf_release+0x10/0x20
[ 253.106188] [<ffffffff8125063c>] __fput+0xec/0x230
and more importantly underlying kprobe is not removed from "kprobe_table" hashtable:
unregister_kprobe
unregister_kprobes
__unregister_kprobe_top
__disable_kprobe
ret = disarm_kprobe(orig_p, true);
disarm_kprobe_ftrace
__disarm_kprobe_ftrace
ftrace_set_filter_ip
ftrace_set_addr
ftrace_set_hash
if (unlikely(ftrace_disabled))
return -ENODEV;
// BOOM, function does nothing!!!
if (ret) {
p->flags &= ~KPROBE_FLAG_DISABLED;
return ERR_PTR(ret);
}
perf_kprobe_destroy() will free the containing kprobe with inner kprobe
still in hashtable manifesting the bug as regular oopses, mystical oopses
in unrelated processes and doublefaults even.
void perf_kprobe_destroy(struct perf_event *p_event)
{
perf_trace_event_close(p_event);
// does more or less nothing
perf_trace_event_unreg(p_event);
// does kfree
destroy_local_trace_kprobe(p_event->tp_event);
}
crash> p kprobe_table
kprobe_table = $1 =
{{
}, {
first = 0xffff880135e9ddd8
}, {
...
crash> struct kprobe 0xffff880135e9ddd8
struct kprobe {
hlist = {
next = 0x6b6b6b6b6b6b6b6b,
pprev = 0x6b6b6b6b6b6b6b6b
},
list = {
next = 0x6b6b6b6b6b6b6b6b,
prev = 0x6b6b6b6b6b6b6b6b
},
nmissed = 7740398493674204011,
addr = 0x6b6b6b6b6b6b6b6b <Address 0x6b6b6b6b6b6b6b6b out of bounds>,
....
This is much easier to reproduce by adding the following BUG_ON:
--- a/kernel/trace/trace_kprobe.c.orig 2022-07-20 14:35:06.760511285 +0300
+++ b/kernel/trace/trace_kprobe.c 2022-07-20 14:41:14.699877744 +0300
@@ -512,8 +512,10 @@ static void __unregister_trace_kprobe(st
if (trace_probe_is_registered(&tk->tp)) {
if (trace_kprobe_is_return(tk))
unregister_kretprobe(&tk->rp);
- else
+ else {
unregister_kprobe(&tk->rp.kp);
+ BUG_ON(!hlist_unhashed(&tk->rp.kp.hlist));
+ }
tk->tp.flags &= ~TP_FLAG_REGISTERED;
/* Cleanup kprobe for reuse */
if (tk->rp.kp.symbol_name)
Basically, if ftrace_kill() is ever called ever there is a ticking UAFbomb.
Alexey (CloudLinux)
On Wed, 20 Jul 2022 18:22:25 +0300
Alexey Dobriyan <[email protected]> wrote:
> Basically, if ftrace_kill() is ever called ever there is a ticking UAFbomb.
That's because ftrace_kill() means something horribly went wrong, and
because ftrace modifies the kernel text, it shuts everything down and this
needs to be fixed.
The fix is not to have something handle ftrace_kill(), it's to fix the
situation so that ftrace_kill() is not called.
Hmm, perhaps we should add an option to make ftrace_kill() call BUG().
I'll go back and take a look at your analysis.
Thanks,
-- Steve
[ Adding live kernel patching folks ]
On Wed, 20 Jul 2022 18:22:25 +0300
Alexey Dobriyan <[email protected]> wrote:
> I'm debugging crash of our product which does live kernel patching together
> with ISV security scanner which uses ftrace kprobes to do whatever it does.
>
> What happens is that is ftrace ever detects .text change, refuses to patch
> and prints a warning with FTRACE_WARN_ON_ONCE() then there is reliable way
> to cause UAF on kmalloc-512 cache by trying to register kprobe with
> perf_event_open() and then unregistering it by exiting the process.
>
> 1) live kernel patching happens, first instruction of some function changes
But live kernel patching uses ftrace to do this, so the question remains,
why doesn't ftrace know about this change?
>
> 2) kprobe on that function is registered with perf_event_open()
>
> WARNING: CPU: 5 PID: 2109 at kernel/trace/ftrace.c:1853 ftrace_bug+0x25d/0x270
> [<ffffffff811638ed>] ftrace_bug+0x25d/0x270
> [<ffffffff81065571>] ftrace_replace_code+0x2b1/0x420
> [<ffffffff81163f9a>] ftrace_modify_all_code+0x6a/0xb0
> [<ffffffff810656f0>] arch_ftrace_update_code+0x10/0x20
> [<ffffffff81164077>] ftrace_run_update_code+0x17/0x70
> [<ffffffff81165512>] ftrace_set_hash+0x1c2/0x1f0
> [<ffffffff8126fee0>] ? SyS_dup2+0x60/0x60
> [<ffffffff8126fee0>] ? SyS_dup2+0x60/0x60
> [<ffffffff811655a0>] ftrace_set_filter_ip+0x60/0x70
> [<ffffffff8179624c>] arm_kprobe+0x9c/0x140
> [<ffffffff81796368>] enable_kprobe+0x78/0xa0
> [<ffffffff81187bab>] enable_trace_kprobe+0x7b/0x120
> [<ffffffff81797e5f>] kprobe_register+0x2f/0x60
> [<ffffffff8118348a>] perf_trace_event_init+0x1aa/0x230
> [<ffffffff811836b7>] perf_kprobe_init+0xa7/0xf0
> [<ffffffff811a8919>] perf_kprobe_event_init+0x49/0x70
> [<ffffffff811aa569>] perf_try_init_event+0x99/0xc0
> [<ffffffff811b29f2>] perf_init_event+0x92/0x150
> [<ffffffff811b2fa1>] perf_event_alloc+0x4f1/0x910
> [<ffffffff811b3789>] SYSC_perf_event_open+0x3c9/0xe50
> [<ffffffff811b4679>] SyS_perf_event_open+0x9/0x10
> [<ffffffff81799f92>] system_call_fastpath+0x25/0x2a
> ftrace failed to modify [<ffffffff8126fee0>] SyS_dup+0x0/0x120
> actual: e9:4b:50:2e:3f
>
Again, why did the above happen. This is a kernel bug that needs to be
fixed. Everything else after this is unimportant, because it's just fall
out to the above bug.
-- Steve
> 3) FTRACE_WARN_ON_ONCE() calls ftrace_kill() which sets ftrace_disabled:
>
> ftrace_disabled = 1;
>
> 4) process exits, all kprobes are unregistered but "ftrace_disabled" is 1 now
> so disarming fails with -ENODEV:
>
> [ 253.042821] WARNING: CPU: 2 PID: 1682 at kernel/kprobes.c:1006 disarm_kprobe+0x102/0x180
> [ 253.044424] Failed to disarm kprobe-ftrace at do_exit+0x0/0xa30 (-19)
> [ 253.086658] [<ffffffff811655a0>] ? ftrace_set_filter_ip+0x60/0x70
> [ 253.089599] [<ffffffff810a1810>] ? mm_update_next_owner+0x230/0x230
> [ 253.090878] [<ffffffff81796492>] disarm_kprobe+0x102/0x180
> [ 253.092084] [<ffffffff817965dd>] __disable_kprobe+0xcd/0xf0
> [ 253.093242] [<ffffffff81796c13>] disable_kprobe+0x23/0x40
> [ 253.094390] [<ffffffff81187d14>] disable_trace_kprobe+0xc4/0x100
> [ 253.095751] [<ffffffff81797e53>] kprobe_register+0x23/0x60
> [ 253.097420] [<ffffffff8118327c>] perf_trace_event_unreg.isra.1+0x3c/0xa0
> [ 253.098866] [<ffffffff81183730>] perf_kprobe_destroy+0x30/0x40
> [ 253.100085] [<ffffffff811af50a>] _free_event+0xfa/0x2f0
> [ 253.101147] [<ffffffff811af769>] put_event+0x19/0x20
> [ 253.102154] [<ffffffff811af993>] perf_event_release_kernel+0x223/0x300
> [ 253.103457] [<ffffffff811afa80>] perf_release+0x10/0x20
> [ 253.106188] [<ffffffff8125063c>] __fput+0xec/0x230
>
> and more importantly underlying kprobe is not removed from "kprobe_table" hashtable:
>
> unregister_kprobe
> unregister_kprobes
> __unregister_kprobe_top
> __disable_kprobe
> ret = disarm_kprobe(orig_p, true);
> disarm_kprobe_ftrace
> __disarm_kprobe_ftrace
> ftrace_set_filter_ip
> ftrace_set_addr
> ftrace_set_hash
> if (unlikely(ftrace_disabled))
> return -ENODEV;
>
> // BOOM, function does nothing!!!
>
> if (ret) {
> p->flags &= ~KPROBE_FLAG_DISABLED;
> return ERR_PTR(ret);
> }
>
>
> perf_kprobe_destroy() will free the containing kprobe with inner kprobe
> still in hashtable manifesting the bug as regular oopses, mystical oopses
> in unrelated processes and doublefaults even.
>
> void perf_kprobe_destroy(struct perf_event *p_event)
> {
> perf_trace_event_close(p_event);
>
> // does more or less nothing
> perf_trace_event_unreg(p_event);
> // does kfree
> destroy_local_trace_kprobe(p_event->tp_event);
> }
>
>
> crash> p kprobe_table
> kprobe_table = $1 =
> {{
> }, {
> first = 0xffff880135e9ddd8
> }, {
> ...
>
> crash> struct kprobe 0xffff880135e9ddd8
> struct kprobe {
> hlist = {
> next = 0x6b6b6b6b6b6b6b6b,
> pprev = 0x6b6b6b6b6b6b6b6b
> },
> list = {
> next = 0x6b6b6b6b6b6b6b6b,
> prev = 0x6b6b6b6b6b6b6b6b
> },
> nmissed = 7740398493674204011,
> addr = 0x6b6b6b6b6b6b6b6b <Address 0x6b6b6b6b6b6b6b6b out of bounds>,
> ....
>
> This is much easier to reproduce by adding the following BUG_ON:
>
> --- a/kernel/trace/trace_kprobe.c.orig 2022-07-20 14:35:06.760511285 +0300
> +++ b/kernel/trace/trace_kprobe.c 2022-07-20 14:41:14.699877744 +0300
> @@ -512,8 +512,10 @@ static void __unregister_trace_kprobe(st
> if (trace_probe_is_registered(&tk->tp)) {
> if (trace_kprobe_is_return(tk))
> unregister_kretprobe(&tk->rp);
> - else
> + else {
> unregister_kprobe(&tk->rp.kp);
> + BUG_ON(!hlist_unhashed(&tk->rp.kp.hlist));
> + }
> tk->tp.flags &= ~TP_FLAG_REGISTERED;
> /* Cleanup kprobe for reuse */
> if (tk->rp.kp.symbol_name)
>
> Basically, if ftrace_kill() is ever called ever there is a ticking UAFbomb.
>
> Alexey (CloudLinux)
On Wed, Jul 20, 2022 at 12:11:02PM -0400, Steven Rostedt wrote:
>
> [ Adding live kernel patching folks ]
>
> On Wed, 20 Jul 2022 18:22:25 +0300
> Alexey Dobriyan <[email protected]> wrote:
>
> > I'm debugging crash of our product which does live kernel patching together
> > with ISV security scanner which uses ftrace kprobes to do whatever it does.
> >
> > What happens is that is ftrace ever detects .text change, refuses to patch
> > and prints a warning with FTRACE_WARN_ON_ONCE() then there is reliable way
> > to cause UAF on kmalloc-512 cache by trying to register kprobe with
> > perf_event_open() and then unregistering it by exiting the process.
> >
> > 1) live kernel patching happens, first instruction of some function changes
>
> But live kernel patching uses ftrace to do this, so the question remains,
> why doesn't ftrace know about this change?
That a line from our product which changes first instruction of the function.
> > 2) kprobe on that function is registered with perf_event_open()
> >
> > WARNING: CPU: 5 PID: 2109 at kernel/trace/ftrace.c:1853 ftrace_bug+0x25d/0x270
> > [<ffffffff811638ed>] ftrace_bug+0x25d/0x270
> > [<ffffffff81065571>] ftrace_replace_code+0x2b1/0x420
> > [<ffffffff81163f9a>] ftrace_modify_all_code+0x6a/0xb0
> > [<ffffffff810656f0>] arch_ftrace_update_code+0x10/0x20
> > [<ffffffff81164077>] ftrace_run_update_code+0x17/0x70
> > [<ffffffff81165512>] ftrace_set_hash+0x1c2/0x1f0
> > [<ffffffff8126fee0>] ? SyS_dup2+0x60/0x60
> > [<ffffffff8126fee0>] ? SyS_dup2+0x60/0x60
> > [<ffffffff811655a0>] ftrace_set_filter_ip+0x60/0x70
> > [<ffffffff8179624c>] arm_kprobe+0x9c/0x140
> > [<ffffffff81796368>] enable_kprobe+0x78/0xa0
> > [<ffffffff81187bab>] enable_trace_kprobe+0x7b/0x120
> > [<ffffffff81797e5f>] kprobe_register+0x2f/0x60
> > [<ffffffff8118348a>] perf_trace_event_init+0x1aa/0x230
> > [<ffffffff811836b7>] perf_kprobe_init+0xa7/0xf0
> > [<ffffffff811a8919>] perf_kprobe_event_init+0x49/0x70
> > [<ffffffff811aa569>] perf_try_init_event+0x99/0xc0
> > [<ffffffff811b29f2>] perf_init_event+0x92/0x150
> > [<ffffffff811b2fa1>] perf_event_alloc+0x4f1/0x910
> > [<ffffffff811b3789>] SYSC_perf_event_open+0x3c9/0xe50
> > [<ffffffff811b4679>] SyS_perf_event_open+0x9/0x10
> > [<ffffffff81799f92>] system_call_fastpath+0x25/0x2a
> > ftrace failed to modify [<ffffffff8126fee0>] SyS_dup+0x0/0x120
> > actual: e9:4b:50:2e:3f
> >
>
> Again, why did the above happen. This is a kernel bug that needs to be
> fixed. Everything else after this is unimportant, because it's just fall
> out to the above bug.
I'll double check what we (not KLP) do and maybe even what KLP does!
On Wed, 20 Jul 2022 19:46:02 +0300
Alexey Dobriyan <[email protected]> wrote:
> On Wed, Jul 20, 2022 at 12:11:02PM -0400, Steven Rostedt wrote:
> >
> > [ Adding live kernel patching folks ]
> >
> > On Wed, 20 Jul 2022 18:22:25 +0300
> > Alexey Dobriyan <[email protected]> wrote:
> >
> > > I'm debugging crash of our product which does live kernel patching together
> > > with ISV security scanner which uses ftrace kprobes to do whatever it does.
> > >
> > > What happens is that is ftrace ever detects .text change, refuses to patch
> > > and prints a warning with FTRACE_WARN_ON_ONCE() then there is reliable way
> > > to cause UAF on kmalloc-512 cache by trying to register kprobe with
> > > perf_event_open() and then unregistering it by exiting the process.
> > >
> > > 1) live kernel patching happens, first instruction of some function changes
> >
> > But live kernel patching uses ftrace to do this, so the question remains,
> > why doesn't ftrace know about this change?
>
> That a line from our product which changes first instruction of the function.
>
So this is an out-of-tree change?
> > > 2) kprobe on that function is registered with perf_event_open()
> > >
> > > WARNING: CPU: 5 PID: 2109 at kernel/trace/ftrace.c:1853 ftrace_bug+0x25d/0x270
> > > [<ffffffff811638ed>] ftrace_bug+0x25d/0x270
> > > [<ffffffff81065571>] ftrace_replace_code+0x2b1/0x420
> > > [<ffffffff81163f9a>] ftrace_modify_all_code+0x6a/0xb0
> > > [<ffffffff810656f0>] arch_ftrace_update_code+0x10/0x20
> > > [<ffffffff81164077>] ftrace_run_update_code+0x17/0x70
> > > [<ffffffff81165512>] ftrace_set_hash+0x1c2/0x1f0
> > > [<ffffffff8126fee0>] ? SyS_dup2+0x60/0x60
> > > [<ffffffff8126fee0>] ? SyS_dup2+0x60/0x60
> > > [<ffffffff811655a0>] ftrace_set_filter_ip+0x60/0x70
> > > [<ffffffff8179624c>] arm_kprobe+0x9c/0x140
> > > [<ffffffff81796368>] enable_kprobe+0x78/0xa0
> > > [<ffffffff81187bab>] enable_trace_kprobe+0x7b/0x120
> > > [<ffffffff81797e5f>] kprobe_register+0x2f/0x60
> > > [<ffffffff8118348a>] perf_trace_event_init+0x1aa/0x230
> > > [<ffffffff811836b7>] perf_kprobe_init+0xa7/0xf0
> > > [<ffffffff811a8919>] perf_kprobe_event_init+0x49/0x70
> > > [<ffffffff811aa569>] perf_try_init_event+0x99/0xc0
> > > [<ffffffff811b29f2>] perf_init_event+0x92/0x150
> > > [<ffffffff811b2fa1>] perf_event_alloc+0x4f1/0x910
> > > [<ffffffff811b3789>] SYSC_perf_event_open+0x3c9/0xe50
> > > [<ffffffff811b4679>] SyS_perf_event_open+0x9/0x10
> > > [<ffffffff81799f92>] system_call_fastpath+0x25/0x2a
> > > ftrace failed to modify [<ffffffff8126fee0>] SyS_dup+0x0/0x120
> > > actual: e9:4b:50:2e:3f
> > >
> >
> > Again, why did the above happen. This is a kernel bug that needs to be
> > fixed. Everything else after this is unimportant, because it's just fall
> > out to the above bug.
>
> I'll double check what we (not KLP) do and maybe even what KLP does!
I would guess the fix is to have your stuff not monkey around with ftrace
locations or to modify your own kernel to have ftrace know about this.
-- Steve