> On May 6, 2020, at 1:19 AM, Walter Wu <[email protected]> wrote:
>
> This patchset improves KASAN reports by making them to have
> call_rcu() call stack information. It is helpful for programmers
> to solve use-after-free or double-free memory issue.
>
> The KASAN report was as follows(cleaned up slightly):
>
> BUG: KASAN: use-after-free in kasan_rcu_reclaim+0x58/0x60
>
> Freed by task 0:
> save_stack+0x24/0x50
> __kasan_slab_free+0x110/0x178
> kasan_slab_free+0x10/0x18
> kfree+0x98/0x270
> kasan_rcu_reclaim+0x1c/0x60
> rcu_core+0x8b4/0x10f8
> rcu_core_si+0xc/0x18
> efi_header_end+0x238/0xa6c
>
> First call_rcu() call stack:
> save_stack+0x24/0x50
> kasan_record_callrcu+0xc8/0xd8
> call_rcu+0x190/0x580
> kasan_rcu_uaf+0x1d8/0x278
>
> Last call_rcu() call stack:
> (stack is not available)
>
>
> Add new CONFIG option to record first and last call_rcu() call stack
> and KASAN report prints two call_rcu() call stack.
>
> This option doesn't increase the cost of memory consumption. It is
> only suitable for generic KASAN.
I don’t understand why this needs to be a Kconfig option at all. If call_rcu() stacks are useful in general, then just always gather those information. How do developers judge if they need to select this option or not?
On Wed, 2020-05-06 at 01:53 -0400, Qian Cai wrote:
>
> > On May 6, 2020, at 1:19 AM, Walter Wu <[email protected]> wrote:
> >
> > This patchset improves KASAN reports by making them to have
> > call_rcu() call stack information. It is helpful for programmers
> > to solve use-after-free or double-free memory issue.
> >
> > The KASAN report was as follows(cleaned up slightly):
> >
> > BUG: KASAN: use-after-free in kasan_rcu_reclaim+0x58/0x60
> >
> > Freed by task 0:
> > save_stack+0x24/0x50
> > __kasan_slab_free+0x110/0x178
> > kasan_slab_free+0x10/0x18
> > kfree+0x98/0x270
> > kasan_rcu_reclaim+0x1c/0x60
> > rcu_core+0x8b4/0x10f8
> > rcu_core_si+0xc/0x18
> > efi_header_end+0x238/0xa6c
> >
> > First call_rcu() call stack:
> > save_stack+0x24/0x50
> > kasan_record_callrcu+0xc8/0xd8
> > call_rcu+0x190/0x580
> > kasan_rcu_uaf+0x1d8/0x278
> >
> > Last call_rcu() call stack:
> > (stack is not available)
> >
> >
> > Add new CONFIG option to record first and last call_rcu() call stack
> > and KASAN report prints two call_rcu() call stack.
> >
> > This option doesn't increase the cost of memory consumption. It is
> > only suitable for generic KASAN.
>
> I don’t understand why this needs to be a Kconfig option at all. If call_rcu() stacks are useful in general, then just always gather those information. How do developers judge if they need to select this option or not?
Because we don't want to increase slub meta-data size, so enabling this
option can print call_rcu() stacks, but the in-use slub object doesn't
print free stack. So if have out-of-bound issue, then it will not print
free stack. It is a trade-off, see [1].
[1] https://bugzilla.kernel.org/show_bug.cgi?id=198437
Thanks
On Wed, May 6, 2020 at 8:23 AM Walter Wu <[email protected]> wrote:
> > > This patchset improves KASAN reports by making them to have
> > > call_rcu() call stack information. It is helpful for programmers
> > > to solve use-after-free or double-free memory issue.
> > >
> > > The KASAN report was as follows(cleaned up slightly):
> > >
> > > BUG: KASAN: use-after-free in kasan_rcu_reclaim+0x58/0x60
> > >
> > > Freed by task 0:
> > > save_stack+0x24/0x50
> > > __kasan_slab_free+0x110/0x178
> > > kasan_slab_free+0x10/0x18
> > > kfree+0x98/0x270
> > > kasan_rcu_reclaim+0x1c/0x60
> > > rcu_core+0x8b4/0x10f8
> > > rcu_core_si+0xc/0x18
> > > efi_header_end+0x238/0xa6c
> > >
> > > First call_rcu() call stack:
> > > save_stack+0x24/0x50
> > > kasan_record_callrcu+0xc8/0xd8
> > > call_rcu+0x190/0x580
> > > kasan_rcu_uaf+0x1d8/0x278
> > >
> > > Last call_rcu() call stack:
> > > (stack is not available)
> > >
> > >
> > > Add new CONFIG option to record first and last call_rcu() call stack
> > > and KASAN report prints two call_rcu() call stack.
> > >
> > > This option doesn't increase the cost of memory consumption. It is
> > > only suitable for generic KASAN.
> >
> > I don’t understand why this needs to be a Kconfig option at all. If call_rcu() stacks are useful in general, then just always gather those information. How do developers judge if they need to select this option or not?
>
> Because we don't want to increase slub meta-data size, so enabling this
> option can print call_rcu() stacks, but the in-use slub object doesn't
> print free stack. So if have out-of-bound issue, then it will not print
> free stack. It is a trade-off, see [1].
>
> [1] https://bugzilla.kernel.org/show_bug.cgi?id=198437
Hi Walter,
Great you are tackling this!
I have the same general sentiment as Qian. I would enable this
unconditionally because:
1. We still can't get both rcu stack and free stack. I would assume
most kernel testing systems need to enable this (we definitely enable
on syzbot). This means we do not have free stack for allocation
objects in any reports coming from testing systems. Which greatly
diminishes the value of the other mode.
2. Kernel is undertested. Introducing any additional configuration
options is a problem in such context. Chances are that some of the
modes are not working or will break in future.
3. That free stack actually causes lots of confusion and I never found
it useful:
https://bugzilla.kernel.org/show_bug.cgi?id=198425
If it's a very delayed UAF, either one may get another report for the
same bug with not so delayed UAF, or if it's way too delayed, then the
previous free stack is wrong as well.
4. Most users don't care that much about debugging tools to learn
every bit of every debugging tool and spend time fine-tuning it for
their context. Most KASAN users won't even be aware of this choice,
and they will just use whatever is the default.
5. Each configuration option increases implementation complexity.
What would have value is if we figure out how to make both of them
work at the same time without increasing memory consumption. But I
don't see any way to do this.
I propose to make this the only mode. I am sure lots of users will
find this additional stack useful, whereas the free stack is even
frequently confusing.
On Wed, 2020-05-06 at 11:37 +0200, 'Dmitry Vyukov' via kasan-dev wrote:
> On Wed, May 6, 2020 at 8:23 AM Walter Wu <[email protected]> wrote:
> > > > This patchset improves KASAN reports by making them to have
> > > > call_rcu() call stack information. It is helpful for programmers
> > > > to solve use-after-free or double-free memory issue.
> > > >
> > > > The KASAN report was as follows(cleaned up slightly):
> > > >
> > > > BUG: KASAN: use-after-free in kasan_rcu_reclaim+0x58/0x60
> > > >
> > > > Freed by task 0:
> > > > save_stack+0x24/0x50
> > > > __kasan_slab_free+0x110/0x178
> > > > kasan_slab_free+0x10/0x18
> > > > kfree+0x98/0x270
> > > > kasan_rcu_reclaim+0x1c/0x60
> > > > rcu_core+0x8b4/0x10f8
> > > > rcu_core_si+0xc/0x18
> > > > efi_header_end+0x238/0xa6c
> > > >
> > > > First call_rcu() call stack:
> > > > save_stack+0x24/0x50
> > > > kasan_record_callrcu+0xc8/0xd8
> > > > call_rcu+0x190/0x580
> > > > kasan_rcu_uaf+0x1d8/0x278
> > > >
> > > > Last call_rcu() call stack:
> > > > (stack is not available)
> > > >
> > > >
> > > > Add new CONFIG option to record first and last call_rcu() call stack
> > > > and KASAN report prints two call_rcu() call stack.
> > > >
> > > > This option doesn't increase the cost of memory consumption. It is
> > > > only suitable for generic KASAN.
> > >
> > > I don’t understand why this needs to be a Kconfig option at all. If call_rcu() stacks are useful in general, then just always gather those information. How do developers judge if they need to select this option or not?
> >
> > Because we don't want to increase slub meta-data size, so enabling this
> > option can print call_rcu() stacks, but the in-use slub object doesn't
> > print free stack. So if have out-of-bound issue, then it will not print
> > free stack. It is a trade-off, see [1].
> >
> > [1] https://bugzilla.kernel.org/show_bug.cgi?id=198437
>
> Hi Walter,
>
> Great you are tackling this!
>
> I have the same general sentiment as Qian. I would enable this
> unconditionally because:
>
> 1. We still can't get both rcu stack and free stack. I would assume
> most kernel testing systems need to enable this (we definitely enable
> on syzbot). This means we do not have free stack for allocation
> objects in any reports coming from testing systems. Which greatly
> diminishes the value of the other mode.
>
> 2. Kernel is undertested. Introducing any additional configuration
> options is a problem in such context. Chances are that some of the
> modes are not working or will break in future.
>
> 3. That free stack actually causes lots of confusion and I never found
> it useful:
> https://bugzilla.kernel.org/show_bug.cgi?id=198425
> If it's a very delayed UAF, either one may get another report for the
> same bug with not so delayed UAF, or if it's way too delayed, then the
> previous free stack is wrong as well.
>
> 4. Most users don't care that much about debugging tools to learn
> every bit of every debugging tool and spend time fine-tuning it for
> their context. Most KASAN users won't even be aware of this choice,
> and they will just use whatever is the default.
>
> 5. Each configuration option increases implementation complexity.
>
> What would have value is if we figure out how to make both of them
> work at the same time without increasing memory consumption. But I
> don't see any way to do this.
>
> I propose to make this the only mode. I am sure lots of users will
> find this additional stack useful, whereas the free stack is even
> frequently confusing.
>
Ok.
If we want to have a default enabling it, but it should only work in
generic KASAN, because we need to get object status(allocation or
freeing) from shadow memory, tag-based KASAN can't do it. So we should
have a default enabling it in generic KASAN?
On Wed, May 6, 2020 at 2:01 PM Walter Wu <[email protected]> wrote:
>
> On Wed, 2020-05-06 at 11:37 +0200, 'Dmitry Vyukov' via kasan-dev wrote:
> > On Wed, May 6, 2020 at 8:23 AM Walter Wu <[email protected]> wrote:
> > > > > This patchset improves KASAN reports by making them to have
> > > > > call_rcu() call stack information. It is helpful for programmers
> > > > > to solve use-after-free or double-free memory issue.
> > > > >
> > > > > The KASAN report was as follows(cleaned up slightly):
> > > > >
> > > > > BUG: KASAN: use-after-free in kasan_rcu_reclaim+0x58/0x60
> > > > >
> > > > > Freed by task 0:
> > > > > save_stack+0x24/0x50
> > > > > __kasan_slab_free+0x110/0x178
> > > > > kasan_slab_free+0x10/0x18
> > > > > kfree+0x98/0x270
> > > > > kasan_rcu_reclaim+0x1c/0x60
> > > > > rcu_core+0x8b4/0x10f8
> > > > > rcu_core_si+0xc/0x18
> > > > > efi_header_end+0x238/0xa6c
> > > > >
> > > > > First call_rcu() call stack:
> > > > > save_stack+0x24/0x50
> > > > > kasan_record_callrcu+0xc8/0xd8
> > > > > call_rcu+0x190/0x580
> > > > > kasan_rcu_uaf+0x1d8/0x278
> > > > >
> > > > > Last call_rcu() call stack:
> > > > > (stack is not available)
> > > > >
> > > > >
> > > > > Add new CONFIG option to record first and last call_rcu() call stack
> > > > > and KASAN report prints two call_rcu() call stack.
> > > > >
> > > > > This option doesn't increase the cost of memory consumption. It is
> > > > > only suitable for generic KASAN.
> > > >
> > > > I don’t understand why this needs to be a Kconfig option at all. If call_rcu() stacks are useful in general, then just always gather those information. How do developers judge if they need to select this option or not?
> > >
> > > Because we don't want to increase slub meta-data size, so enabling this
> > > option can print call_rcu() stacks, but the in-use slub object doesn't
> > > print free stack. So if have out-of-bound issue, then it will not print
> > > free stack. It is a trade-off, see [1].
> > >
> > > [1] https://bugzilla.kernel.org/show_bug.cgi?id=198437
> >
> > Hi Walter,
> >
> > Great you are tackling this!
> >
> > I have the same general sentiment as Qian. I would enable this
> > unconditionally because:
> >
> > 1. We still can't get both rcu stack and free stack. I would assume
> > most kernel testing systems need to enable this (we definitely enable
> > on syzbot). This means we do not have free stack for allocation
> > objects in any reports coming from testing systems. Which greatly
> > diminishes the value of the other mode.
> >
> > 2. Kernel is undertested. Introducing any additional configuration
> > options is a problem in such context. Chances are that some of the
> > modes are not working or will break in future.
> >
> > 3. That free stack actually causes lots of confusion and I never found
> > it useful:
> > https://bugzilla.kernel.org/show_bug.cgi?id=198425
> > If it's a very delayed UAF, either one may get another report for the
> > same bug with not so delayed UAF, or if it's way too delayed, then the
> > previous free stack is wrong as well.
> >
> > 4. Most users don't care that much about debugging tools to learn
> > every bit of every debugging tool and spend time fine-tuning it for
> > their context. Most KASAN users won't even be aware of this choice,
> > and they will just use whatever is the default.
> >
> > 5. Each configuration option increases implementation complexity.
> >
> > What would have value is if we figure out how to make both of them
> > work at the same time without increasing memory consumption. But I
> > don't see any way to do this.
> >
> > I propose to make this the only mode. I am sure lots of users will
> > find this additional stack useful, whereas the free stack is even
> > frequently confusing.
> >
>
> Ok.
> If we want to have a default enabling it, but it should only work in
> generic KASAN, because we need to get object status(allocation or
> freeing) from shadow memory, tag-based KASAN can't do it. So we should
> have a default enabling it in generic KASAN?
Yes, let's do generic KASAN always memorizes rcu stack; tags KASAN
never memorizes rcu stacks. No new configurations.