On 6/13/19 11:13 AM, Walter Wu wrote:
> This patch adds memory corruption identification at bug report for
> software tag-based mode, the report show whether it is "use-after-free"
> or "out-of-bound" error instead of "invalid-access" error.This will make
> it easier for programmers to see the memory corruption problem.
>
> Now we extend the quarantine to support both generic and tag-based kasan.
> For tag-based kasan, the quarantine stores only freed object information
> to check if an object is freed recently. When tag-based kasan reports an
> error, we can check if the tagged addr is in the quarantine and make a
> good guess if the object is more like "use-after-free" or "out-of-bound".
>
We already have all the information and don't need the quarantine to make such guess.
Basically if shadow of the first byte of object has the same tag as tag in pointer than it's out-of-bounds,
otherwise it's use-after-free.
In pseudo-code it's something like this:
u8 object_tag = *(u8 *)kasan_mem_to_shadow(nearest_object(cacche, page, access_addr));
if (access_addr_tag == object_tag && object_tag != KASAN_TAG_INVALID)
// out-of-bounds
else
// use-after-free
On Thu, 2019-06-13 at 15:27 +0300, Andrey Ryabinin wrote:
>
> On 6/13/19 11:13 AM, Walter Wu wrote:
> > This patch adds memory corruption identification at bug report for
> > software tag-based mode, the report show whether it is "use-after-free"
> > or "out-of-bound" error instead of "invalid-access" error.This will make
> > it easier for programmers to see the memory corruption problem.
> >
> > Now we extend the quarantine to support both generic and tag-based kasan.
> > For tag-based kasan, the quarantine stores only freed object information
> > to check if an object is freed recently. When tag-based kasan reports an
> > error, we can check if the tagged addr is in the quarantine and make a
> > good guess if the object is more like "use-after-free" or "out-of-bound".
> >
>
>
> We already have all the information and don't need the quarantine to make such guess.
> Basically if shadow of the first byte of object has the same tag as tag in pointer than it's out-of-bounds,
> otherwise it's use-after-free.
>
> In pseudo-code it's something like this:
>
> u8 object_tag = *(u8 *)kasan_mem_to_shadow(nearest_object(cacche, page, access_addr));
>
> if (access_addr_tag == object_tag && object_tag != KASAN_TAG_INVALID)
> // out-of-bounds
> else
> // use-after-free
Thanks your explanation.
I see, we can use it to decide corruption type.
But some use-after-free issues, it may not have accurate free-backtrace.
Unfortunately in that situation, free-backtrace is the most important.
please see below example
In generic KASAN, it gets accurate free-backrace(ptr1).
In tag-based KASAN, it gets wrong free-backtrace(ptr2). It will make
programmer misjudge, so they may not believe tag-based KASAN.
So We provide this patch, we hope tag-based KASAN bug report is the same
accurate with generic KASAN.
---
ptr1 = kmalloc(size, GFP_KERNEL);
ptr1_free(ptr1);
ptr2 = kmalloc(size, GFP_KERNEL);
ptr2_free(ptr2);
ptr1[size] = 'x'; //corruption here
static noinline void ptr1_free(char* ptr)
{
kfree(ptr);
}
static noinline void ptr2_free(char* ptr)
{
kfree(ptr);
}
---
On Fri, 2019-06-14 at 01:46 +0800, Walter Wu wrote:
> On Thu, 2019-06-13 at 15:27 +0300, Andrey Ryabinin wrote:
> >
> > On 6/13/19 11:13 AM, Walter Wu wrote:
> > > This patch adds memory corruption identification at bug report for
> > > software tag-based mode, the report show whether it is "use-after-free"
> > > or "out-of-bound" error instead of "invalid-access" error.This will make
> > > it easier for programmers to see the memory corruption problem.
> > >
> > > Now we extend the quarantine to support both generic and tag-based kasan.
> > > For tag-based kasan, the quarantine stores only freed object information
> > > to check if an object is freed recently. When tag-based kasan reports an
> > > error, we can check if the tagged addr is in the quarantine and make a
> > > good guess if the object is more like "use-after-free" or "out-of-bound".
> > >
> >
> >
> > We already have all the information and don't need the quarantine to make such guess.
> > Basically if shadow of the first byte of object has the same tag as tag in pointer than it's out-of-bounds,
> > otherwise it's use-after-free.
> >
> > In pseudo-code it's something like this:
> >
> > u8 object_tag = *(u8 *)kasan_mem_to_shadow(nearest_object(cacche, page, access_addr));
> >
> > if (access_addr_tag == object_tag && object_tag != KASAN_TAG_INVALID)
> > // out-of-bounds
> > else
> > // use-after-free
>
> Thanks your explanation.
> I see, we can use it to decide corruption type.
> But some use-after-free issues, it may not have accurate free-backtrace.
> Unfortunately in that situation, free-backtrace is the most important.
> please see below example
>
> In generic KASAN, it gets accurate free-backrace(ptr1).
> In tag-based KASAN, it gets wrong free-backtrace(ptr2). It will make
> programmer misjudge, so they may not believe tag-based KASAN.
> So We provide this patch, we hope tag-based KASAN bug report is the same
> accurate with generic KASAN.
>
> ---
> ptr1 = kmalloc(size, GFP_KERNEL);
> ptr1_free(ptr1);
>
> ptr2 = kmalloc(size, GFP_KERNEL);
> ptr2_free(ptr2);
>
> ptr1[size] = 'x'; //corruption here
>
>
> static noinline void ptr1_free(char* ptr)
> {
> kfree(ptr);
> }
> static noinline void ptr2_free(char* ptr)
> {
> kfree(ptr);
> }
> ---
>
We think of another question about deciding by that shadow of the first
byte.
In tag-based KASAN, it is immediately released after calling kfree(), so
the slub is easy to be used by another pointer, then it will change
shadow memory to the tag of new pointer, it will not be the
KASAN_TAG_INVALID, so there are many false negative cases, especially in
small size allocation.
Our patch is to solve those problems. so please consider it, thanks.
On Fri, 2019-06-14 at 10:32 +0800, Walter Wu wrote:
> On Fri, 2019-06-14 at 01:46 +0800, Walter Wu wrote:
> > On Thu, 2019-06-13 at 15:27 +0300, Andrey Ryabinin wrote:
> > >
> > > On 6/13/19 11:13 AM, Walter Wu wrote:
> > > > This patch adds memory corruption identification at bug report for
> > > > software tag-based mode, the report show whether it is "use-after-free"
> > > > or "out-of-bound" error instead of "invalid-access" error.This will make
> > > > it easier for programmers to see the memory corruption problem.
> > > >
> > > > Now we extend the quarantine to support both generic and tag-based kasan.
> > > > For tag-based kasan, the quarantine stores only freed object information
> > > > to check if an object is freed recently. When tag-based kasan reports an
> > > > error, we can check if the tagged addr is in the quarantine and make a
> > > > good guess if the object is more like "use-after-free" or "out-of-bound".
> > > >
> > >
> > >
> > > We already have all the information and don't need the quarantine to make such guess.
> > > Basically if shadow of the first byte of object has the same tag as tag in pointer than it's out-of-bounds,
> > > otherwise it's use-after-free.
> > >
> > > In pseudo-code it's something like this:
> > >
> > > u8 object_tag = *(u8 *)kasan_mem_to_shadow(nearest_object(cacche, page, access_addr));
> > >
> > > if (access_addr_tag == object_tag && object_tag != KASAN_TAG_INVALID)
> > > // out-of-bounds
> > > else
> > > // use-after-free
> >
> > Thanks your explanation.
> > I see, we can use it to decide corruption type.
> > But some use-after-free issues, it may not have accurate free-backtrace.
> > Unfortunately in that situation, free-backtrace is the most important.
> > please see below example
> >
> > In generic KASAN, it gets accurate free-backrace(ptr1).
> > In tag-based KASAN, it gets wrong free-backtrace(ptr2). It will make
> > programmer misjudge, so they may not believe tag-based KASAN.
> > So We provide this patch, we hope tag-based KASAN bug report is the same
> > accurate with generic KASAN.
> >
> > ---
> > ptr1 = kmalloc(size, GFP_KERNEL);
> > ptr1_free(ptr1);
> >
> > ptr2 = kmalloc(size, GFP_KERNEL);
> > ptr2_free(ptr2);
> >
> > ptr1[size] = 'x'; //corruption here
> >
> >
> > static noinline void ptr1_free(char* ptr)
> > {
> > kfree(ptr);
> > }
> > static noinline void ptr2_free(char* ptr)
> > {
> > kfree(ptr);
> > }
> > ---
> >
> We think of another question about deciding by that shadow of the first
> byte.
> In tag-based KASAN, it is immediately released after calling kfree(), so
> the slub is easy to be used by another pointer, then it will change
> shadow memory to the tag of new pointer, it will not be the
> KASAN_TAG_INVALID, so there are many false negative cases, especially in
> small size allocation.
>
> Our patch is to solve those problems. so please consider it, thanks.
>
Hi, Andrey and Dmitry,
I am sorry to bother you.
Would you tell me what you think about this patch?
We want to use tag-based KASAN, so we hope its bug report is clear and
correct as generic KASAN.
Thanks your review.
Walter
On Mon, Jun 17, 2019 at 6:00 AM Walter Wu <[email protected]> wrote:
>
> On Fri, 2019-06-14 at 10:32 +0800, Walter Wu wrote:
> > On Fri, 2019-06-14 at 01:46 +0800, Walter Wu wrote:
> > > On Thu, 2019-06-13 at 15:27 +0300, Andrey Ryabinin wrote:
> > > >
> > > > On 6/13/19 11:13 AM, Walter Wu wrote:
> > > > > This patch adds memory corruption identification at bug report for
> > > > > software tag-based mode, the report show whether it is "use-after-free"
> > > > > or "out-of-bound" error instead of "invalid-access" error.This will make
> > > > > it easier for programmers to see the memory corruption problem.
> > > > >
> > > > > Now we extend the quarantine to support both generic and tag-based kasan.
> > > > > For tag-based kasan, the quarantine stores only freed object information
> > > > > to check if an object is freed recently. When tag-based kasan reports an
> > > > > error, we can check if the tagged addr is in the quarantine and make a
> > > > > good guess if the object is more like "use-after-free" or "out-of-bound".
> > > > >
> > > >
> > > >
> > > > We already have all the information and don't need the quarantine to make such guess.
> > > > Basically if shadow of the first byte of object has the same tag as tag in pointer than it's out-of-bounds,
> > > > otherwise it's use-after-free.
> > > >
> > > > In pseudo-code it's something like this:
> > > >
> > > > u8 object_tag = *(u8 *)kasan_mem_to_shadow(nearest_object(cacche, page, access_addr));
> > > >
> > > > if (access_addr_tag == object_tag && object_tag != KASAN_TAG_INVALID)
> > > > // out-of-bounds
> > > > else
> > > > // use-after-free
> > >
> > > Thanks your explanation.
> > > I see, we can use it to decide corruption type.
> > > But some use-after-free issues, it may not have accurate free-backtrace.
> > > Unfortunately in that situation, free-backtrace is the most important.
> > > please see below example
> > >
> > > In generic KASAN, it gets accurate free-backrace(ptr1).
> > > In tag-based KASAN, it gets wrong free-backtrace(ptr2). It will make
> > > programmer misjudge, so they may not believe tag-based KASAN.
> > > So We provide this patch, we hope tag-based KASAN bug report is the same
> > > accurate with generic KASAN.
> > >
> > > ---
> > > ptr1 = kmalloc(size, GFP_KERNEL);
> > > ptr1_free(ptr1);
> > >
> > > ptr2 = kmalloc(size, GFP_KERNEL);
> > > ptr2_free(ptr2);
> > >
> > > ptr1[size] = 'x'; //corruption here
> > >
> > >
> > > static noinline void ptr1_free(char* ptr)
> > > {
> > > kfree(ptr);
> > > }
> > > static noinline void ptr2_free(char* ptr)
> > > {
> > > kfree(ptr);
> > > }
> > > ---
> > >
> > We think of another question about deciding by that shadow of the first
> > byte.
> > In tag-based KASAN, it is immediately released after calling kfree(), so
> > the slub is easy to be used by another pointer, then it will change
> > shadow memory to the tag of new pointer, it will not be the
> > KASAN_TAG_INVALID, so there are many false negative cases, especially in
> > small size allocation.
> >
> > Our patch is to solve those problems. so please consider it, thanks.
> >
> Hi, Andrey and Dmitry,
>
> I am sorry to bother you.
> Would you tell me what you think about this patch?
> We want to use tag-based KASAN, so we hope its bug report is clear and
> correct as generic KASAN.
>
> Thanks your review.
> Walter
Hi Walter,
I will probably be busy till the next week. Sorry for delays.
On Mon, 2019-06-17 at 13:57 +0200, Dmitry Vyukov wrote:
> On Mon, Jun 17, 2019 at 6:00 AM Walter Wu <[email protected]> wrote:
> >
> > On Fri, 2019-06-14 at 10:32 +0800, Walter Wu wrote:
> > > On Fri, 2019-06-14 at 01:46 +0800, Walter Wu wrote:
> > > > On Thu, 2019-06-13 at 15:27 +0300, Andrey Ryabinin wrote:
> > > > >
> > > > > On 6/13/19 11:13 AM, Walter Wu wrote:
> > > > > > This patch adds memory corruption identification at bug report for
> > > > > > software tag-based mode, the report show whether it is "use-after-free"
> > > > > > or "out-of-bound" error instead of "invalid-access" error.This will make
> > > > > > it easier for programmers to see the memory corruption problem.
> > > > > >
> > > > > > Now we extend the quarantine to support both generic and tag-based kasan.
> > > > > > For tag-based kasan, the quarantine stores only freed object information
> > > > > > to check if an object is freed recently. When tag-based kasan reports an
> > > > > > error, we can check if the tagged addr is in the quarantine and make a
> > > > > > good guess if the object is more like "use-after-free" or "out-of-bound".
> > > > > >
> > > > >
> > > > >
> > > > > We already have all the information and don't need the quarantine to make such guess.
> > > > > Basically if shadow of the first byte of object has the same tag as tag in pointer than it's out-of-bounds,
> > > > > otherwise it's use-after-free.
> > > > >
> > > > > In pseudo-code it's something like this:
> > > > >
> > > > > u8 object_tag = *(u8 *)kasan_mem_to_shadow(nearest_object(cacche, page, access_addr));
> > > > >
> > > > > if (access_addr_tag == object_tag && object_tag != KASAN_TAG_INVALID)
> > > > > // out-of-bounds
> > > > > else
> > > > > // use-after-free
> > > >
> > > > Thanks your explanation.
> > > > I see, we can use it to decide corruption type.
> > > > But some use-after-free issues, it may not have accurate free-backtrace.
> > > > Unfortunately in that situation, free-backtrace is the most important.
> > > > please see below example
> > > >
> > > > In generic KASAN, it gets accurate free-backrace(ptr1).
> > > > In tag-based KASAN, it gets wrong free-backtrace(ptr2). It will make
> > > > programmer misjudge, so they may not believe tag-based KASAN.
> > > > So We provide this patch, we hope tag-based KASAN bug report is the same
> > > > accurate with generic KASAN.
> > > >
> > > > ---
> > > > ptr1 = kmalloc(size, GFP_KERNEL);
> > > > ptr1_free(ptr1);
> > > >
> > > > ptr2 = kmalloc(size, GFP_KERNEL);
> > > > ptr2_free(ptr2);
> > > >
> > > > ptr1[size] = 'x'; //corruption here
> > > >
> > > >
> > > > static noinline void ptr1_free(char* ptr)
> > > > {
> > > > kfree(ptr);
> > > > }
> > > > static noinline void ptr2_free(char* ptr)
> > > > {
> > > > kfree(ptr);
> > > > }
> > > > ---
> > > >
> > > We think of another question about deciding by that shadow of the first
> > > byte.
> > > In tag-based KASAN, it is immediately released after calling kfree(), so
> > > the slub is easy to be used by another pointer, then it will change
> > > shadow memory to the tag of new pointer, it will not be the
> > > KASAN_TAG_INVALID, so there are many false negative cases, especially in
> > > small size allocation.
> > >
> > > Our patch is to solve those problems. so please consider it, thanks.
> > >
> > Hi, Andrey and Dmitry,
> >
> > I am sorry to bother you.
> > Would you tell me what you think about this patch?
> > We want to use tag-based KASAN, so we hope its bug report is clear and
> > correct as generic KASAN.
> >
> > Thanks your review.
> > Walter
>
> Hi Walter,
>
> I will probably be busy till the next week. Sorry for delays.
It's ok. Thanks your kindly help.
I hope I can contribute to tag-based KASAN. It is a very important tool
for us.
On Mon, 2019-06-17 at 20:32 +0800, Walter Wu wrote:
> On Mon, 2019-06-17 at 13:57 +0200, Dmitry Vyukov wrote:
> > On Mon, Jun 17, 2019 at 6:00 AM Walter Wu <[email protected]> wrote:
> > >
> > > On Fri, 2019-06-14 at 10:32 +0800, Walter Wu wrote:
> > > > On Fri, 2019-06-14 at 01:46 +0800, Walter Wu wrote:
> > > > > On Thu, 2019-06-13 at 15:27 +0300, Andrey Ryabinin wrote:
> > > > > >
> > > > > > On 6/13/19 11:13 AM, Walter Wu wrote:
> > > > > > > This patch adds memory corruption identification at bug report for
> > > > > > > software tag-based mode, the report show whether it is "use-after-free"
> > > > > > > or "out-of-bound" error instead of "invalid-access" error.This will make
> > > > > > > it easier for programmers to see the memory corruption problem.
> > > > > > >
> > > > > > > Now we extend the quarantine to support both generic and tag-based kasan.
> > > > > > > For tag-based kasan, the quarantine stores only freed object information
> > > > > > > to check if an object is freed recently. When tag-based kasan reports an
> > > > > > > error, we can check if the tagged addr is in the quarantine and make a
> > > > > > > good guess if the object is more like "use-after-free" or "out-of-bound".
> > > > > > >
> > > > > >
> > > > > >
> > > > > > We already have all the information and don't need the quarantine to make such guess.
> > > > > > Basically if shadow of the first byte of object has the same tag as tag in pointer than it's out-of-bounds,
> > > > > > otherwise it's use-after-free.
> > > > > >
> > > > > > In pseudo-code it's something like this:
> > > > > >
> > > > > > u8 object_tag = *(u8 *)kasan_mem_to_shadow(nearest_object(cacche, page, access_addr));
> > > > > >
> > > > > > if (access_addr_tag == object_tag && object_tag != KASAN_TAG_INVALID)
> > > > > > // out-of-bounds
> > > > > > else
> > > > > > // use-after-free
> > > > >
> > > > > Thanks your explanation.
> > > > > I see, we can use it to decide corruption type.
> > > > > But some use-after-free issues, it may not have accurate free-backtrace.
> > > > > Unfortunately in that situation, free-backtrace is the most important.
> > > > > please see below example
> > > > >
> > > > > In generic KASAN, it gets accurate free-backrace(ptr1).
> > > > > In tag-based KASAN, it gets wrong free-backtrace(ptr2). It will make
> > > > > programmer misjudge, so they may not believe tag-based KASAN.
> > > > > So We provide this patch, we hope tag-based KASAN bug report is the same
> > > > > accurate with generic KASAN.
> > > > >
> > > > > ---
> > > > > ptr1 = kmalloc(size, GFP_KERNEL);
> > > > > ptr1_free(ptr1);
> > > > >
> > > > > ptr2 = kmalloc(size, GFP_KERNEL);
> > > > > ptr2_free(ptr2);
> > > > >
> > > > > ptr1[size] = 'x'; //corruption here
> > > > >
> > > > >
> > > > > static noinline void ptr1_free(char* ptr)
> > > > > {
> > > > > kfree(ptr);
> > > > > }
> > > > > static noinline void ptr2_free(char* ptr)
> > > > > {
> > > > > kfree(ptr);
> > > > > }
> > > > > ---
> > > > >
> > > > We think of another question about deciding by that shadow of the first
> > > > byte.
> > > > In tag-based KASAN, it is immediately released after calling kfree(), so
> > > > the slub is easy to be used by another pointer, then it will change
> > > > shadow memory to the tag of new pointer, it will not be the
> > > > KASAN_TAG_INVALID, so there are many false negative cases, especially in
> > > > small size allocation.
> > > >
> > > > Our patch is to solve those problems. so please consider it, thanks.
> > > >
> > > Hi, Andrey and Dmitry,
> > >
> > > I am sorry to bother you.
> > > Would you tell me what you think about this patch?
> > > We want to use tag-based KASAN, so we hope its bug report is clear and
> > > correct as generic KASAN.
> > >
> > > Thanks your review.
> > > Walter
> >
> > Hi Walter,
> >
> > I will probably be busy till the next week. Sorry for delays.
>
> It's ok. Thanks your kindly help.
> I hope I can contribute to tag-based KASAN. It is a very important tool
> for us.
Hi, Dmitry,
Would you have free time to discuss this patch together?
Thanks.
Walter
On Mon, Jul 1, 2019 at 11:56 AM Walter Wu <[email protected]> wrote:
> > > > > > > > This patch adds memory corruption identification at bug report for
> > > > > > > > software tag-based mode, the report show whether it is "use-after-free"
> > > > > > > > or "out-of-bound" error instead of "invalid-access" error.This will make
> > > > > > > > it easier for programmers to see the memory corruption problem.
> > > > > > > >
> > > > > > > > Now we extend the quarantine to support both generic and tag-based kasan.
> > > > > > > > For tag-based kasan, the quarantine stores only freed object information
> > > > > > > > to check if an object is freed recently. When tag-based kasan reports an
> > > > > > > > error, we can check if the tagged addr is in the quarantine and make a
> > > > > > > > good guess if the object is more like "use-after-free" or "out-of-bound".
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > We already have all the information and don't need the quarantine to make such guess.
> > > > > > > Basically if shadow of the first byte of object has the same tag as tag in pointer than it's out-of-bounds,
> > > > > > > otherwise it's use-after-free.
> > > > > > >
> > > > > > > In pseudo-code it's something like this:
> > > > > > >
> > > > > > > u8 object_tag = *(u8 *)kasan_mem_to_shadow(nearest_object(cacche, page, access_addr));
> > > > > > >
> > > > > > > if (access_addr_tag == object_tag && object_tag != KASAN_TAG_INVALID)
> > > > > > > // out-of-bounds
> > > > > > > else
> > > > > > > // use-after-free
> > > > > >
> > > > > > Thanks your explanation.
> > > > > > I see, we can use it to decide corruption type.
> > > > > > But some use-after-free issues, it may not have accurate free-backtrace.
> > > > > > Unfortunately in that situation, free-backtrace is the most important.
> > > > > > please see below example
> > > > > >
> > > > > > In generic KASAN, it gets accurate free-backrace(ptr1).
> > > > > > In tag-based KASAN, it gets wrong free-backtrace(ptr2). It will make
> > > > > > programmer misjudge, so they may not believe tag-based KASAN.
> > > > > > So We provide this patch, we hope tag-based KASAN bug report is the same
> > > > > > accurate with generic KASAN.
> > > > > >
> > > > > > ---
> > > > > > ptr1 = kmalloc(size, GFP_KERNEL);
> > > > > > ptr1_free(ptr1);
> > > > > >
> > > > > > ptr2 = kmalloc(size, GFP_KERNEL);
> > > > > > ptr2_free(ptr2);
> > > > > >
> > > > > > ptr1[size] = 'x'; //corruption here
> > > > > >
> > > > > >
> > > > > > static noinline void ptr1_free(char* ptr)
> > > > > > {
> > > > > > kfree(ptr);
> > > > > > }
> > > > > > static noinline void ptr2_free(char* ptr)
> > > > > > {
> > > > > > kfree(ptr);
> > > > > > }
> > > > > > ---
> > > > > >
> > > > > We think of another question about deciding by that shadow of the first
> > > > > byte.
> > > > > In tag-based KASAN, it is immediately released after calling kfree(), so
> > > > > the slub is easy to be used by another pointer, then it will change
> > > > > shadow memory to the tag of new pointer, it will not be the
> > > > > KASAN_TAG_INVALID, so there are many false negative cases, especially in
> > > > > small size allocation.
> > > > >
> > > > > Our patch is to solve those problems. so please consider it, thanks.
> > > > >
> > > > Hi, Andrey and Dmitry,
> > > >
> > > > I am sorry to bother you.
> > > > Would you tell me what you think about this patch?
> > > > We want to use tag-based KASAN, so we hope its bug report is clear and
> > > > correct as generic KASAN.
> > > >
> > > > Thanks your review.
> > > > Walter
> > >
> > > Hi Walter,
> > >
> > > I will probably be busy till the next week. Sorry for delays.
> >
> > It's ok. Thanks your kindly help.
> > I hope I can contribute to tag-based KASAN. It is a very important tool
> > for us.
>
> Hi, Dmitry,
>
> Would you have free time to discuss this patch together?
> Thanks.
Sorry for delays. I am overwhelm by some urgent work. I afraid to
promise any dates because the next week I am on a conference, then
again a backlog and an intern starting...
Andrey, do you still have concerns re this patch? This change allows
to print the free stack.
We also have a quarantine for hwasan in user-space. Though it works a
bit differently then the normal asan quarantine. We keep a per-thread
fixed-size ring-buffer of recent allocations:
https://github.com/llvm-mirror/compiler-rt/blob/master/lib/hwasan/hwasan_report.cpp#L274-L284
and scan these ring buffers during reports.
On 7/5/19 4:34 PM, Dmitry Vyukov wrote:
> On Mon, Jul 1, 2019 at 11:56 AM Walter Wu <[email protected]> wrote:
>>>>>>>>> This patch adds memory corruption identification at bug report for
>>>>>>>>> software tag-based mode, the report show whether it is "use-after-free"
>>>>>>>>> or "out-of-bound" error instead of "invalid-access" error.This will make
>>>>>>>>> it easier for programmers to see the memory corruption problem.
>>>>>>>>>
>>>>>>>>> Now we extend the quarantine to support both generic and tag-based kasan.
>>>>>>>>> For tag-based kasan, the quarantine stores only freed object information
>>>>>>>>> to check if an object is freed recently. When tag-based kasan reports an
>>>>>>>>> error, we can check if the tagged addr is in the quarantine and make a
>>>>>>>>> good guess if the object is more like "use-after-free" or "out-of-bound".
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> We already have all the information and don't need the quarantine to make such guess.
>>>>>>>> Basically if shadow of the first byte of object has the same tag as tag in pointer than it's out-of-bounds,
>>>>>>>> otherwise it's use-after-free.
>>>>>>>>
>>>>>>>> In pseudo-code it's something like this:
>>>>>>>>
>>>>>>>> u8 object_tag = *(u8 *)kasan_mem_to_shadow(nearest_object(cacche, page, access_addr));
>>>>>>>>
>>>>>>>> if (access_addr_tag == object_tag && object_tag != KASAN_TAG_INVALID)
>>>>>>>> // out-of-bounds
>>>>>>>> else
>>>>>>>> // use-after-free
>>>>>>>
>>>>>>> Thanks your explanation.
>>>>>>> I see, we can use it to decide corruption type.
>>>>>>> But some use-after-free issues, it may not have accurate free-backtrace.
>>>>>>> Unfortunately in that situation, free-backtrace is the most important.
>>>>>>> please see below example
>>>>>>>
>>>>>>> In generic KASAN, it gets accurate free-backrace(ptr1).
>>>>>>> In tag-based KASAN, it gets wrong free-backtrace(ptr2). It will make
>>>>>>> programmer misjudge, so they may not believe tag-based KASAN.
>>>>>>> So We provide this patch, we hope tag-based KASAN bug report is the same
>>>>>>> accurate with generic KASAN.
>>>>>>>
>>>>>>> ---
>>>>>>> ptr1 = kmalloc(size, GFP_KERNEL);
>>>>>>> ptr1_free(ptr1);
>>>>>>>
>>>>>>> ptr2 = kmalloc(size, GFP_KERNEL);
>>>>>>> ptr2_free(ptr2);
>>>>>>>
>>>>>>> ptr1[size] = 'x'; //corruption here
>>>>>>>
>>>>>>>
>>>>>>> static noinline void ptr1_free(char* ptr)
>>>>>>> {
>>>>>>> kfree(ptr);
>>>>>>> }
>>>>>>> static noinline void ptr2_free(char* ptr)
>>>>>>> {
>>>>>>> kfree(ptr);
>>>>>>> }
>>>>>>> ---
>>>>>>>
>>>>>> We think of another question about deciding by that shadow of the first
>>>>>> byte.
>>>>>> In tag-based KASAN, it is immediately released after calling kfree(), so
>>>>>> the slub is easy to be used by another pointer, then it will change
>>>>>> shadow memory to the tag of new pointer, it will not be the
>>>>>> KASAN_TAG_INVALID, so there are many false negative cases, especially in
>>>>>> small size allocation.
>>>>>>
>>>>>> Our patch is to solve those problems. so please consider it, thanks.
>>>>>>
>>>>> Hi, Andrey and Dmitry,
>>>>>
>>>>> I am sorry to bother you.
>>>>> Would you tell me what you think about this patch?
>>>>> We want to use tag-based KASAN, so we hope its bug report is clear and
>>>>> correct as generic KASAN.
>>>>>
>>>>> Thanks your review.
>>>>> Walter
>>>>
>>>> Hi Walter,
>>>>
>>>> I will probably be busy till the next week. Sorry for delays.
>>>
>>> It's ok. Thanks your kindly help.
>>> I hope I can contribute to tag-based KASAN. It is a very important tool
>>> for us.
>>
>> Hi, Dmitry,
>>
>> Would you have free time to discuss this patch together?
>> Thanks.
>
> Sorry for delays. I am overwhelm by some urgent work. I afraid to
> promise any dates because the next week I am on a conference, then
> again a backlog and an intern starting...
>
> Andrey, do you still have concerns re this patch? This change allows
> to print the free stack.
I 'm not sure that quarantine is a best way to do that. Quarantine is made to delay freeing, but we don't that here.
If we want to remember more free stacks wouldn't be easier simply to remember more stacks in object itself?
Same for previously used tags for better use-after-free identification.
> We also have a quarantine for hwasan in user-space. Though it works a
> bit differently then the normal asan quarantine. We keep a per-thread
> fixed-size ring-buffer of recent allocations:
> https://github.com/llvm-mirror/compiler-rt/blob/master/lib/hwasan/hwasan_report.cpp#L274-L284
> and scan these ring buffers during reports.
>
On Mon, 2019-07-08 at 19:33 +0300, Andrey Ryabinin wrote:
>
> On 7/5/19 4:34 PM, Dmitry Vyukov wrote:
> > On Mon, Jul 1, 2019 at 11:56 AM Walter Wu <[email protected]> wrote:
> >>>>>>>>> This patch adds memory corruption identification at bug report for
> >>>>>>>>> software tag-based mode, the report show whether it is "use-after-free"
> >>>>>>>>> or "out-of-bound" error instead of "invalid-access" error.This will make
> >>>>>>>>> it easier for programmers to see the memory corruption problem.
> >>>>>>>>>
> >>>>>>>>> Now we extend the quarantine to support both generic and tag-based kasan.
> >>>>>>>>> For tag-based kasan, the quarantine stores only freed object information
> >>>>>>>>> to check if an object is freed recently. When tag-based kasan reports an
> >>>>>>>>> error, we can check if the tagged addr is in the quarantine and make a
> >>>>>>>>> good guess if the object is more like "use-after-free" or "out-of-bound".
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> We already have all the information and don't need the quarantine to make such guess.
> >>>>>>>> Basically if shadow of the first byte of object has the same tag as tag in pointer than it's out-of-bounds,
> >>>>>>>> otherwise it's use-after-free.
> >>>>>>>>
> >>>>>>>> In pseudo-code it's something like this:
> >>>>>>>>
> >>>>>>>> u8 object_tag = *(u8 *)kasan_mem_to_shadow(nearest_object(cacche, page, access_addr));
> >>>>>>>>
> >>>>>>>> if (access_addr_tag == object_tag && object_tag != KASAN_TAG_INVALID)
> >>>>>>>> // out-of-bounds
> >>>>>>>> else
> >>>>>>>> // use-after-free
> >>>>>>>
> >>>>>>> Thanks your explanation.
> >>>>>>> I see, we can use it to decide corruption type.
> >>>>>>> But some use-after-free issues, it may not have accurate free-backtrace.
> >>>>>>> Unfortunately in that situation, free-backtrace is the most important.
> >>>>>>> please see below example
> >>>>>>>
> >>>>>>> In generic KASAN, it gets accurate free-backrace(ptr1).
> >>>>>>> In tag-based KASAN, it gets wrong free-backtrace(ptr2). It will make
> >>>>>>> programmer misjudge, so they may not believe tag-based KASAN.
> >>>>>>> So We provide this patch, we hope tag-based KASAN bug report is the same
> >>>>>>> accurate with generic KASAN.
> >>>>>>>
> >>>>>>> ---
> >>>>>>> ptr1 = kmalloc(size, GFP_KERNEL);
> >>>>>>> ptr1_free(ptr1);
> >>>>>>>
> >>>>>>> ptr2 = kmalloc(size, GFP_KERNEL);
> >>>>>>> ptr2_free(ptr2);
> >>>>>>>
> >>>>>>> ptr1[size] = 'x'; //corruption here
> >>>>>>>
> >>>>>>>
> >>>>>>> static noinline void ptr1_free(char* ptr)
> >>>>>>> {
> >>>>>>> kfree(ptr);
> >>>>>>> }
> >>>>>>> static noinline void ptr2_free(char* ptr)
> >>>>>>> {
> >>>>>>> kfree(ptr);
> >>>>>>> }
> >>>>>>> ---
> >>>>>>>
> >>>>>> We think of another question about deciding by that shadow of the first
> >>>>>> byte.
> >>>>>> In tag-based KASAN, it is immediately released after calling kfree(), so
> >>>>>> the slub is easy to be used by another pointer, then it will change
> >>>>>> shadow memory to the tag of new pointer, it will not be the
> >>>>>> KASAN_TAG_INVALID, so there are many false negative cases, especially in
> >>>>>> small size allocation.
> >>>>>>
> >>>>>> Our patch is to solve those problems. so please consider it, thanks.
> >>>>>>
> >>>>> Hi, Andrey and Dmitry,
> >>>>>
> >>>>> I am sorry to bother you.
> >>>>> Would you tell me what you think about this patch?
> >>>>> We want to use tag-based KASAN, so we hope its bug report is clear and
> >>>>> correct as generic KASAN.
> >>>>>
> >>>>> Thanks your review.
> >>>>> Walter
> >>>>
> >>>> Hi Walter,
> >>>>
> >>>> I will probably be busy till the next week. Sorry for delays.
> >>>
> >>> It's ok. Thanks your kindly help.
> >>> I hope I can contribute to tag-based KASAN. It is a very important tool
> >>> for us.
> >>
> >> Hi, Dmitry,
> >>
> >> Would you have free time to discuss this patch together?
> >> Thanks.
> >
> > Sorry for delays. I am overwhelm by some urgent work. I afraid to
> > promise any dates because the next week I am on a conference, then
> > again a backlog and an intern starting...
> >
> > Andrey, do you still have concerns re this patch? This change allows
> > to print the free stack.
>
> I 'm not sure that quarantine is a best way to do that. Quarantine is made to delay freeing, but we don't that here.
> If we want to remember more free stacks wouldn't be easier simply to remember more stacks in object itself?
> Same for previously used tags for better use-after-free identification.
>
Hi Andrey,
We ever tried to use object itself to determine use-after-free
identification, but tag-based KASAN immediately released the pointer
after call kfree(), the original object will be used by another
pointer, if we use object itself to determine use-after-free issue, then
it has many false negative cases. so we create a lite quarantine(ring
buffers) to record recent free stacks in order to avoid those false
negative situations.
We hope to have one solution to cover all cases and be accurate. Our
patch is configurable feature option, it can provide some programmers to
easy see the tag-based KASAN report.
> > We also have a quarantine for hwasan in user-space. Though it works a
> > bit differently then the normal asan quarantine. We keep a per-thread
> > fixed-size ring-buffer of recent allocations:
> > https://github.com/llvm-mirror/compiler-rt/blob/master/lib/hwasan/hwasan_report.cpp#L274-L284
> > and scan these ring buffers during reports.
> >
Thanks your information, it looks like the same idea with our patch.
On 7/9/19 5:53 AM, Walter Wu wrote:
> On Mon, 2019-07-08 at 19:33 +0300, Andrey Ryabinin wrote:
>>
>> On 7/5/19 4:34 PM, Dmitry Vyukov wrote:
>>> On Mon, Jul 1, 2019 at 11:56 AM Walter Wu <[email protected]> wrote:
>>>
>>> Sorry for delays. I am overwhelm by some urgent work. I afraid to
>>> promise any dates because the next week I am on a conference, then
>>> again a backlog and an intern starting...
>>>
>>> Andrey, do you still have concerns re this patch? This change allows
>>> to print the free stack.
>>
>> I 'm not sure that quarantine is a best way to do that. Quarantine is made to delay freeing, but we don't that here.
>> If we want to remember more free stacks wouldn't be easier simply to remember more stacks in object itself?
>> Same for previously used tags for better use-after-free identification.
>>
>
> Hi Andrey,
>
> We ever tried to use object itself to determine use-after-free
> identification, but tag-based KASAN immediately released the pointer
> after call kfree(), the original object will be used by another
> pointer, if we use object itself to determine use-after-free issue, then
> it has many false negative cases. so we create a lite quarantine(ring
> buffers) to record recent free stacks in order to avoid those false
> negative situations.
I'm telling that *more* than one free stack and also tags per object can be stored.
If object reused we would still have information about n-last usages of the object.
It seems like much easier and more efficient solution than patch you proposing.
As for other concern about this particular patch
- It wasn't tested. There is deadlock (sleep in atomic) on the report path which would have been noticed it tested.
Also GFP_NOWAIT allocation which fails very noisy and very often, especially in memory constraint enviromnent where tag-based KASAN supposed to be used.
- Inefficient usage of memory:
48 bytes (sizeof (qlist_object) + sizeof(kasan_alloc_meta)) per kfree() call seems like a lot. It could be less.
The same 'struct kasan_track' stored twice in two different places (in object and in quarantine).
Basically, at least some part of the quarantine always duplicates information that we already know about
recently freed object.
Since now we call kmalloc() from kfree() path, every unique kfree() stacktrace now generates additional unique stacktrace that
takes space in stackdepot.
On Wed, 2019-07-10 at 21:24 +0300, Andrey Ryabinin wrote:
>
> On 7/9/19 5:53 AM, Walter Wu wrote:
> > On Mon, 2019-07-08 at 19:33 +0300, Andrey Ryabinin wrote:
> >>
> >> On 7/5/19 4:34 PM, Dmitry Vyukov wrote:
> >>> On Mon, Jul 1, 2019 at 11:56 AM Walter Wu <[email protected]> wrote:
>
> >>>
> >>> Sorry for delays. I am overwhelm by some urgent work. I afraid to
> >>> promise any dates because the next week I am on a conference, then
> >>> again a backlog and an intern starting...
> >>>
> >>> Andrey, do you still have concerns re this patch? This change allows
> >>> to print the free stack.
> >>
> >> I 'm not sure that quarantine is a best way to do that. Quarantine is made to delay freeing, but we don't that here.
> >> If we want to remember more free stacks wouldn't be easier simply to remember more stacks in object itself?
> >> Same for previously used tags for better use-after-free identification.
> >>
> >
> > Hi Andrey,
> >
> > We ever tried to use object itself to determine use-after-free
> > identification, but tag-based KASAN immediately released the pointer
> > after call kfree(), the original object will be used by another
> > pointer, if we use object itself to determine use-after-free issue, then
> > it has many false negative cases. so we create a lite quarantine(ring
> > buffers) to record recent free stacks in order to avoid those false
> > negative situations.
>
> I'm telling that *more* than one free stack and also tags per object can be stored.
> If object reused we would still have information about n-last usages of the object.
> It seems like much easier and more efficient solution than patch you proposing.
>
To make the object reused, we must ensure that no other pointers uses it
after kfree() release the pointer.
Scenario:
1). The object reused information is valid when no another pointer uses
it.
2). The object reused information is invalid when another pointer uses
it.
Do you mean that the object reused is scenario 1) ?
If yes, maybe we can change the calling quarantine_put() location. It
will be fully use that quarantine, but at scenario 2) it looks like to
need this patch.
If no, maybe i miss your meaning, would you tell me how to use invalid
object information? or?
> As for other concern about this particular patch
> - It wasn't tested. There is deadlock (sleep in atomic) on the report path which would have been noticed it tested.
we already used it on qemu and ran kasan UT. It look like ok.
> Also GFP_NOWAIT allocation which fails very noisy and very often, especially in memory constraint enviromnent where tag-based KASAN supposed to be used.
>
Maybe, we can change it into GFP_KERNEL.
> - Inefficient usage of memory:
> 48 bytes (sizeof (qlist_object) + sizeof(kasan_alloc_meta)) per kfree() call seems like a lot. It could be less.
>
We will think it.
> The same 'struct kasan_track' stored twice in two different places (in object and in quarantine).
> Basically, at least some part of the quarantine always duplicates information that we already know about
> recently freed object.
>
> Since now we call kmalloc() from kfree() path, every unique kfree() stacktrace now generates additional unique stacktrace that
> takes space in stackdepot.
>
Duplicate information is solved after change the calling
quarantine_put() location.
On 7/11/19 1:06 PM, Walter Wu wrote:
> On Wed, 2019-07-10 at 21:24 +0300, Andrey Ryabinin wrote:
>>
>> On 7/9/19 5:53 AM, Walter Wu wrote:
>>> On Mon, 2019-07-08 at 19:33 +0300, Andrey Ryabinin wrote:
>>>>
>>>> On 7/5/19 4:34 PM, Dmitry Vyukov wrote:
>>>>> On Mon, Jul 1, 2019 at 11:56 AM Walter Wu <[email protected]> wrote:
>>
>>>>>
>>>>> Sorry for delays. I am overwhelm by some urgent work. I afraid to
>>>>> promise any dates because the next week I am on a conference, then
>>>>> again a backlog and an intern starting...
>>>>>
>>>>> Andrey, do you still have concerns re this patch? This change allows
>>>>> to print the free stack.
>>>>
>>>> I 'm not sure that quarantine is a best way to do that. Quarantine is made to delay freeing, but we don't that here.
>>>> If we want to remember more free stacks wouldn't be easier simply to remember more stacks in object itself?
>>>> Same for previously used tags for better use-after-free identification.
>>>>
>>>
>>> Hi Andrey,
>>>
>>> We ever tried to use object itself to determine use-after-free
>>> identification, but tag-based KASAN immediately released the pointer
>>> after call kfree(), the original object will be used by another
>>> pointer, if we use object itself to determine use-after-free issue, then
>>> it has many false negative cases. so we create a lite quarantine(ring
>>> buffers) to record recent free stacks in order to avoid those false
>>> negative situations.
>>
>> I'm telling that *more* than one free stack and also tags per object can be stored.
>> If object reused we would still have information about n-last usages of the object.
>> It seems like much easier and more efficient solution than patch you proposing.
>>
> To make the object reused, we must ensure that no other pointers uses it
> after kfree() release the pointer.
> Scenario:
> 1). The object reused information is valid when no another pointer uses
> it.
> 2). The object reused information is invalid when another pointer uses
> it.
> Do you mean that the object reused is scenario 1) ?
> If yes, maybe we can change the calling quarantine_put() location. It
> will be fully use that quarantine, but at scenario 2) it looks like to
> need this patch.
> If no, maybe i miss your meaning, would you tell me how to use invalid
> object information? or?
>
KASAN keeps information about object with the object, right after payload in the kasan_alloc_meta struct.
This information is always valid as long as slab page allocated. Currently it keeps only one last free stacktrace.
It could be extended to record more free stacktraces and also record previously used tags which will allow you
to identify use-after-free and extract right free stacktrace.
On Fri, 2019-07-12 at 13:52 +0300, Andrey Ryabinin wrote:
>
> On 7/11/19 1:06 PM, Walter Wu wrote:
> > On Wed, 2019-07-10 at 21:24 +0300, Andrey Ryabinin wrote:
> >>
> >> On 7/9/19 5:53 AM, Walter Wu wrote:
> >>> On Mon, 2019-07-08 at 19:33 +0300, Andrey Ryabinin wrote:
> >>>>
> >>>> On 7/5/19 4:34 PM, Dmitry Vyukov wrote:
> >>>>> On Mon, Jul 1, 2019 at 11:56 AM Walter Wu <[email protected]> wrote:
> >>
> >>>>>
> >>>>> Sorry for delays. I am overwhelm by some urgent work. I afraid to
> >>>>> promise any dates because the next week I am on a conference, then
> >>>>> again a backlog and an intern starting...
> >>>>>
> >>>>> Andrey, do you still have concerns re this patch? This change allows
> >>>>> to print the free stack.
> >>>>
> >>>> I 'm not sure that quarantine is a best way to do that. Quarantine is made to delay freeing, but we don't that here.
> >>>> If we want to remember more free stacks wouldn't be easier simply to remember more stacks in object itself?
> >>>> Same for previously used tags for better use-after-free identification.
> >>>>
> >>>
> >>> Hi Andrey,
> >>>
> >>> We ever tried to use object itself to determine use-after-free
> >>> identification, but tag-based KASAN immediately released the pointer
> >>> after call kfree(), the original object will be used by another
> >>> pointer, if we use object itself to determine use-after-free issue, then
> >>> it has many false negative cases. so we create a lite quarantine(ring
> >>> buffers) to record recent free stacks in order to avoid those false
> >>> negative situations.
> >>
> >> I'm telling that *more* than one free stack and also tags per object can be stored.
> >> If object reused we would still have information about n-last usages of the object.
> >> It seems like much easier and more efficient solution than patch you proposing.
> >>
> > To make the object reused, we must ensure that no other pointers uses it
> > after kfree() release the pointer.
> > Scenario:
> > 1). The object reused information is valid when no another pointer uses
> > it.
> > 2). The object reused information is invalid when another pointer uses
> > it.
> > Do you mean that the object reused is scenario 1) ?
> > If yes, maybe we can change the calling quarantine_put() location. It
> > will be fully use that quarantine, but at scenario 2) it looks like to
> > need this patch.
> > If no, maybe i miss your meaning, would you tell me how to use invalid
> > object information? or?
> >
>
>
> KASAN keeps information about object with the object, right after payload in the kasan_alloc_meta struct.
> This information is always valid as long as slab page allocated. Currently it keeps only one last free stacktrace.
> It could be extended to record more free stacktraces and also record previously used tags which will allow you
> to identify use-after-free and extract right free stacktrace.
Thanks for your explanation.
For extend slub object, if one record is 9B (sizeof(u8)+ sizeof(struct
kasan_track)) and add five records into slub object, every slub object
may add 45B usage after the system runs longer.
Slub object number is easy more than 1,000,000(maybe it may be more
bigger), then the extending object memory usage should be 45MB, and
unfortunately it is no limit. The memory usage is more bigger than our
patch.
We hope tag-based KASAN advantage is smaller memory usage. If it’s
possible, we should spend less memory in order to identify
use-after-free. Would you accept our patch after fine tune it?
On 7/15/19 6:06 AM, Walter Wu wrote:
> On Fri, 2019-07-12 at 13:52 +0300, Andrey Ryabinin wrote:
>>
>> On 7/11/19 1:06 PM, Walter Wu wrote:
>>> On Wed, 2019-07-10 at 21:24 +0300, Andrey Ryabinin wrote:
>>>>
>>>> On 7/9/19 5:53 AM, Walter Wu wrote:
>>>>> On Mon, 2019-07-08 at 19:33 +0300, Andrey Ryabinin wrote:
>>>>>>
>>>>>> On 7/5/19 4:34 PM, Dmitry Vyukov wrote:
>>>>>>> On Mon, Jul 1, 2019 at 11:56 AM Walter Wu <[email protected]> wrote:
>>>>
>>>>>>>
>>>>>>> Sorry for delays. I am overwhelm by some urgent work. I afraid to
>>>>>>> promise any dates because the next week I am on a conference, then
>>>>>>> again a backlog and an intern starting...
>>>>>>>
>>>>>>> Andrey, do you still have concerns re this patch? This change allows
>>>>>>> to print the free stack.
>>>>>>
>>>>>> I 'm not sure that quarantine is a best way to do that. Quarantine is made to delay freeing, but we don't that here.
>>>>>> If we want to remember more free stacks wouldn't be easier simply to remember more stacks in object itself?
>>>>>> Same for previously used tags for better use-after-free identification.
>>>>>>
>>>>>
>>>>> Hi Andrey,
>>>>>
>>>>> We ever tried to use object itself to determine use-after-free
>>>>> identification, but tag-based KASAN immediately released the pointer
>>>>> after call kfree(), the original object will be used by another
>>>>> pointer, if we use object itself to determine use-after-free issue, then
>>>>> it has many false negative cases. so we create a lite quarantine(ring
>>>>> buffers) to record recent free stacks in order to avoid those false
>>>>> negative situations.
>>>>
>>>> I'm telling that *more* than one free stack and also tags per object can be stored.
>>>> If object reused we would still have information about n-last usages of the object.
>>>> It seems like much easier and more efficient solution than patch you proposing.
>>>>
>>> To make the object reused, we must ensure that no other pointers uses it
>>> after kfree() release the pointer.
>>> Scenario:
>>> 1). The object reused information is valid when no another pointer uses
>>> it.
>>> 2). The object reused information is invalid when another pointer uses
>>> it.
>>> Do you mean that the object reused is scenario 1) ?
>>> If yes, maybe we can change the calling quarantine_put() location. It
>>> will be fully use that quarantine, but at scenario 2) it looks like to
>>> need this patch.
>>> If no, maybe i miss your meaning, would you tell me how to use invalid
>>> object information? or?
>>>
>>
>>
>> KASAN keeps information about object with the object, right after payload in the kasan_alloc_meta struct.
>> This information is always valid as long as slab page allocated. Currently it keeps only one last free stacktrace.
>> It could be extended to record more free stacktraces and also record previously used tags which will allow you
>> to identify use-after-free and extract right free stacktrace.
>
> Thanks for your explanation.
>
> For extend slub object, if one record is 9B (sizeof(u8)+ sizeof(struct
> kasan_track)) and add five records into slub object, every slub object
> may add 45B usage after the system runs longer.
> Slub object number is easy more than 1,000,000(maybe it may be more
> bigger), then the extending object memory usage should be 45MB, and
> unfortunately it is no limit. The memory usage is more bigger than our
> patch.
No, it's not necessarily more.
And there are other aspects to consider such as performance, how simple reliable the code is.
>
> We hope tag-based KASAN advantage is smaller memory usage. If it’s
> possible, we should spend less memory in order to identify
> use-after-free. Would you accept our patch after fine tune it?
Sure, if you manage to fix issues and demonstrate that performance penalty of your
patch is close to zero.
On Thu, 2019-07-18 at 19:11 +0300, Andrey Ryabinin wrote:
>
> On 7/15/19 6:06 AM, Walter Wu wrote:
> > On Fri, 2019-07-12 at 13:52 +0300, Andrey Ryabinin wrote:
> >>
> >> On 7/11/19 1:06 PM, Walter Wu wrote:
> >>> On Wed, 2019-07-10 at 21:24 +0300, Andrey Ryabinin wrote:
> >>>>
> >>>> On 7/9/19 5:53 AM, Walter Wu wrote:
> >>>>> On Mon, 2019-07-08 at 19:33 +0300, Andrey Ryabinin wrote:
> >>>>>>
> >>>>>> On 7/5/19 4:34 PM, Dmitry Vyukov wrote:
> >>>>>>> On Mon, Jul 1, 2019 at 11:56 AM Walter Wu <[email protected]> wrote:
> >>>>
> >>>>>>>
> >>>>>>> Sorry for delays. I am overwhelm by some urgent work. I afraid to
> >>>>>>> promise any dates because the next week I am on a conference, then
> >>>>>>> again a backlog and an intern starting...
> >>>>>>>
> >>>>>>> Andrey, do you still have concerns re this patch? This change allows
> >>>>>>> to print the free stack.
> >>>>>>
> >>>>>> I 'm not sure that quarantine is a best way to do that. Quarantine is made to delay freeing, but we don't that here.
> >>>>>> If we want to remember more free stacks wouldn't be easier simply to remember more stacks in object itself?
> >>>>>> Same for previously used tags for better use-after-free identification.
> >>>>>>
> >>>>>
> >>>>> Hi Andrey,
> >>>>>
> >>>>> We ever tried to use object itself to determine use-after-free
> >>>>> identification, but tag-based KASAN immediately released the pointer
> >>>>> after call kfree(), the original object will be used by another
> >>>>> pointer, if we use object itself to determine use-after-free issue, then
> >>>>> it has many false negative cases. so we create a lite quarantine(ring
> >>>>> buffers) to record recent free stacks in order to avoid those false
> >>>>> negative situations.
> >>>>
> >>>> I'm telling that *more* than one free stack and also tags per object can be stored.
> >>>> If object reused we would still have information about n-last usages of the object.
> >>>> It seems like much easier and more efficient solution than patch you proposing.
> >>>>
> >>> To make the object reused, we must ensure that no other pointers uses it
> >>> after kfree() release the pointer.
> >>> Scenario:
> >>> 1). The object reused information is valid when no another pointer uses
> >>> it.
> >>> 2). The object reused information is invalid when another pointer uses
> >>> it.
> >>> Do you mean that the object reused is scenario 1) ?
> >>> If yes, maybe we can change the calling quarantine_put() location. It
> >>> will be fully use that quarantine, but at scenario 2) it looks like to
> >>> need this patch.
> >>> If no, maybe i miss your meaning, would you tell me how to use invalid
> >>> object information? or?
> >>>
> >>
> >>
> >> KASAN keeps information about object with the object, right after payload in the kasan_alloc_meta struct.
> >> This information is always valid as long as slab page allocated. Currently it keeps only one last free stacktrace.
> >> It could be extended to record more free stacktraces and also record previously used tags which will allow you
> >> to identify use-after-free and extract right free stacktrace.
> >
> > Thanks for your explanation.
> >
> > For extend slub object, if one record is 9B (sizeof(u8)+ sizeof(struct
> > kasan_track)) and add five records into slub object, every slub object
> > may add 45B usage after the system runs longer.
> > Slub object number is easy more than 1,000,000(maybe it may be more
> > bigger), then the extending object memory usage should be 45MB, and
> > unfortunately it is no limit. The memory usage is more bigger than our
> > patch.
>
> No, it's not necessarily more.
> And there are other aspects to consider such as performance, how simple reliable the code is.
>
> >
> > We hope tag-based KASAN advantage is smaller memory usage. If it’s
> > possible, we should spend less memory in order to identify
> > use-after-free. Would you accept our patch after fine tune it?
>
> Sure, if you manage to fix issues and demonstrate that performance penalty of your
> patch is close to zero.
I remember that there are already the lists which you concern. Maybe we
can try to solve those problems one by one.
1. deadlock issue? cause by kmalloc() after kfree()?
2. decrease allocation fail, to modify GFP_NOWAIT flag to GFP_KERNEL?
3. check whether slim 48 bytes (sizeof (qlist_object) +
sizeof(kasan_alloc_meta)) and additional unique stacktrace in
stackdepot?
4. duplicate struct 'kasan_track' information in two different places
Would you have any other concern? or?
On 7/22/19 12:52 PM, Walter Wu wrote:
> On Thu, 2019-07-18 at 19:11 +0300, Andrey Ryabinin wrote:
>>
>> On 7/15/19 6:06 AM, Walter Wu wrote:
>>> On Fri, 2019-07-12 at 13:52 +0300, Andrey Ryabinin wrote:
>>>>
>>>> On 7/11/19 1:06 PM, Walter Wu wrote:
>>>>> On Wed, 2019-07-10 at 21:24 +0300, Andrey Ryabinin wrote:
>>>>>>
>>>>>> On 7/9/19 5:53 AM, Walter Wu wrote:
>>>>>>> On Mon, 2019-07-08 at 19:33 +0300, Andrey Ryabinin wrote:
>>>>>>>>
>>>>>>>> On 7/5/19 4:34 PM, Dmitry Vyukov wrote:
>>>>>>>>> On Mon, Jul 1, 2019 at 11:56 AM Walter Wu <[email protected]> wrote:
>>>>>>
>>>>>>>>>
>>>>>>>>> Sorry for delays. I am overwhelm by some urgent work. I afraid to
>>>>>>>>> promise any dates because the next week I am on a conference, then
>>>>>>>>> again a backlog and an intern starting...
>>>>>>>>>
>>>>>>>>> Andrey, do you still have concerns re this patch? This change allows
>>>>>>>>> to print the free stack.
>>>>>>>>
>>>>>>>> I 'm not sure that quarantine is a best way to do that. Quarantine is made to delay freeing, but we don't that here.
>>>>>>>> If we want to remember more free stacks wouldn't be easier simply to remember more stacks in object itself?
>>>>>>>> Same for previously used tags for better use-after-free identification.
>>>>>>>>
>>>>>>>
>>>>>>> Hi Andrey,
>>>>>>>
>>>>>>> We ever tried to use object itself to determine use-after-free
>>>>>>> identification, but tag-based KASAN immediately released the pointer
>>>>>>> after call kfree(), the original object will be used by another
>>>>>>> pointer, if we use object itself to determine use-after-free issue, then
>>>>>>> it has many false negative cases. so we create a lite quarantine(ring
>>>>>>> buffers) to record recent free stacks in order to avoid those false
>>>>>>> negative situations.
>>>>>>
>>>>>> I'm telling that *more* than one free stack and also tags per object can be stored.
>>>>>> If object reused we would still have information about n-last usages of the object.
>>>>>> It seems like much easier and more efficient solution than patch you proposing.
>>>>>>
>>>>> To make the object reused, we must ensure that no other pointers uses it
>>>>> after kfree() release the pointer.
>>>>> Scenario:
>>>>> 1). The object reused information is valid when no another pointer uses
>>>>> it.
>>>>> 2). The object reused information is invalid when another pointer uses
>>>>> it.
>>>>> Do you mean that the object reused is scenario 1) ?
>>>>> If yes, maybe we can change the calling quarantine_put() location. It
>>>>> will be fully use that quarantine, but at scenario 2) it looks like to
>>>>> need this patch.
>>>>> If no, maybe i miss your meaning, would you tell me how to use invalid
>>>>> object information? or?
>>>>>
>>>>
>>>>
>>>> KASAN keeps information about object with the object, right after payload in the kasan_alloc_meta struct.
>>>> This information is always valid as long as slab page allocated. Currently it keeps only one last free stacktrace.
>>>> It could be extended to record more free stacktraces and also record previously used tags which will allow you
>>>> to identify use-after-free and extract right free stacktrace.
>>>
>>> Thanks for your explanation.
>>>
>>> For extend slub object, if one record is 9B (sizeof(u8)+ sizeof(struct
>>> kasan_track)) and add five records into slub object, every slub object
>>> may add 45B usage after the system runs longer.
>>> Slub object number is easy more than 1,000,000(maybe it may be more
>>> bigger), then the extending object memory usage should be 45MB, and
>>> unfortunately it is no limit. The memory usage is more bigger than our
>>> patch.
>>
>> No, it's not necessarily more.
>> And there are other aspects to consider such as performance, how simple reliable the code is.
>>
>>>
>>> We hope tag-based KASAN advantage is smaller memory usage. If it’s
>>> possible, we should spend less memory in order to identify
>>> use-after-free. Would you accept our patch after fine tune it?
>>
>> Sure, if you manage to fix issues and demonstrate that performance penalty of your
>> patch is close to zero.
>
>
> I remember that there are already the lists which you concern. Maybe we
> can try to solve those problems one by one.
>
> 1. deadlock issue? cause by kmalloc() after kfree()?
smp_call_on_cpu()
> 2. decrease allocation fail, to modify GFP_NOWAIT flag to GFP_KERNEL?
No, this is not gonna work. Ideally we shouldn't have any allocations there.
It's not reliable and it hurts performance.
> 3. check whether slim 48 bytes (sizeof (qlist_object) +
> sizeof(kasan_alloc_meta)) and additional unique stacktrace in
> stackdepot?
> 4. duplicate struct 'kasan_track' information in two different places
>
Yup.
> Would you have any other concern? or?
>
It would be nice to see some performance numbers. Something that uses slab allocations a lot, e.g. netperf STREAM_STREAM test.
On Fri, 2019-07-26 at 15:00 +0300, Andrey Ryabinin wrote:
>
> On 7/22/19 12:52 PM, Walter Wu wrote:
> > On Thu, 2019-07-18 at 19:11 +0300, Andrey Ryabinin wrote:
> >>
> >> On 7/15/19 6:06 AM, Walter Wu wrote:
> >>> On Fri, 2019-07-12 at 13:52 +0300, Andrey Ryabinin wrote:
> >>>>
> >>>> On 7/11/19 1:06 PM, Walter Wu wrote:
> >>>>> On Wed, 2019-07-10 at 21:24 +0300, Andrey Ryabinin wrote:
> >>>>>>
> >>>>>> On 7/9/19 5:53 AM, Walter Wu wrote:
> >>>>>>> On Mon, 2019-07-08 at 19:33 +0300, Andrey Ryabinin wrote:
> >>>>>>>>
> >>>>>>>> On 7/5/19 4:34 PM, Dmitry Vyukov wrote:
> >>>>>>>>> On Mon, Jul 1, 2019 at 11:56 AM Walter Wu <[email protected]> wrote:
> >>>>>>
> >>>>>>>>>
> >>>>>>>>> Sorry for delays. I am overwhelm by some urgent work. I afraid to
> >>>>>>>>> promise any dates because the next week I am on a conference, then
> >>>>>>>>> again a backlog and an intern starting...
> >>>>>>>>>
> >>>>>>>>> Andrey, do you still have concerns re this patch? This change allows
> >>>>>>>>> to print the free stack.
> >>>>>>>>
> >>>>>>>> I 'm not sure that quarantine is a best way to do that. Quarantine is made to delay freeing, but we don't that here.
> >>>>>>>> If we want to remember more free stacks wouldn't be easier simply to remember more stacks in object itself?
> >>>>>>>> Same for previously used tags for better use-after-free identification.
> >>>>>>>>
> >>>>>>>
> >>>>>>> Hi Andrey,
> >>>>>>>
> >>>>>>> We ever tried to use object itself to determine use-after-free
> >>>>>>> identification, but tag-based KASAN immediately released the pointer
> >>>>>>> after call kfree(), the original object will be used by another
> >>>>>>> pointer, if we use object itself to determine use-after-free issue, then
> >>>>>>> it has many false negative cases. so we create a lite quarantine(ring
> >>>>>>> buffers) to record recent free stacks in order to avoid those false
> >>>>>>> negative situations.
> >>>>>>
> >>>>>> I'm telling that *more* than one free stack and also tags per object can be stored.
> >>>>>> If object reused we would still have information about n-last usages of the object.
> >>>>>> It seems like much easier and more efficient solution than patch you proposing.
> >>>>>>
> >>>>> To make the object reused, we must ensure that no other pointers uses it
> >>>>> after kfree() release the pointer.
> >>>>> Scenario:
> >>>>> 1). The object reused information is valid when no another pointer uses
> >>>>> it.
> >>>>> 2). The object reused information is invalid when another pointer uses
> >>>>> it.
> >>>>> Do you mean that the object reused is scenario 1) ?
> >>>>> If yes, maybe we can change the calling quarantine_put() location. It
> >>>>> will be fully use that quarantine, but at scenario 2) it looks like to
> >>>>> need this patch.
> >>>>> If no, maybe i miss your meaning, would you tell me how to use invalid
> >>>>> object information? or?
> >>>>>
> >>>>
> >>>>
> >>>> KASAN keeps information about object with the object, right after payload in the kasan_alloc_meta struct.
> >>>> This information is always valid as long as slab page allocated. Currently it keeps only one last free stacktrace.
> >>>> It could be extended to record more free stacktraces and also record previously used tags which will allow you
> >>>> to identify use-after-free and extract right free stacktrace.
> >>>
> >>> Thanks for your explanation.
> >>>
> >>> For extend slub object, if one record is 9B (sizeof(u8)+ sizeof(struct
> >>> kasan_track)) and add five records into slub object, every slub object
> >>> may add 45B usage after the system runs longer.
> >>> Slub object number is easy more than 1,000,000(maybe it may be more
> >>> bigger), then the extending object memory usage should be 45MB, and
> >>> unfortunately it is no limit. The memory usage is more bigger than our
> >>> patch.
> >>
> >> No, it's not necessarily more.
> >> And there are other aspects to consider such as performance, how simple reliable the code is.
> >>
> >>>
> >>> We hope tag-based KASAN advantage is smaller memory usage. If it’s
> >>> possible, we should spend less memory in order to identify
> >>> use-after-free. Would you accept our patch after fine tune it?
> >>
> >> Sure, if you manage to fix issues and demonstrate that performance penalty of your
> >> patch is close to zero.
> >
> >
> > I remember that there are already the lists which you concern. Maybe we
> > can try to solve those problems one by one.
> >
> > 1. deadlock issue? cause by kmalloc() after kfree()?
>
> smp_call_on_cpu()
> > 2. decrease allocation fail, to modify GFP_NOWAIT flag to GFP_KERNEL?
>
> No, this is not gonna work. Ideally we shouldn't have any allocations there.
> It's not reliable and it hurts performance.
>
I dont know this meaning, we need create a qobject and put into
quarantine, so may need to call kmem_cache_alloc(), would you agree this
action?
>
> > 3. check whether slim 48 bytes (sizeof (qlist_object) +
> > sizeof(kasan_alloc_meta)) and additional unique stacktrace in
> > stackdepot?
> > 4. duplicate struct 'kasan_track' information in two different places
> >
>
> Yup.
>
> > Would you have any other concern? or?
> >
>
> It would be nice to see some performance numbers. Something that uses slab allocations a lot, e.g. netperf STREAM_STREAM test.
>
ok, we will do it.
On 7/26/19 3:28 PM, Walter Wu wrote:
> On Fri, 2019-07-26 at 15:00 +0300, Andrey Ryabinin wrote:
>>
>
>>>
>>>
>>> I remember that there are already the lists which you concern. Maybe we
>>> can try to solve those problems one by one.
>>>
>>> 1. deadlock issue? cause by kmalloc() after kfree()?
>>
>> smp_call_on_cpu()
>
>>> 2. decrease allocation fail, to modify GFP_NOWAIT flag to GFP_KERNEL?
>>
>> No, this is not gonna work. Ideally we shouldn't have any allocations there.
>> It's not reliable and it hurts performance.
>>
> I dont know this meaning, we need create a qobject and put into
> quarantine, so may need to call kmem_cache_alloc(), would you agree this
> action?
>
How is this any different from what you have now?