Hi all:
When I investigate the mm/memory-failure.c code again, I found there's a possible race window
between memory_failure and alloc_fresh_huge_page/free_huge_page. Thank about the below scene:
CPU 1 CPU 2
alloc_fresh_huge_page -- page refcnt > 0 memory_failure
prep_new_huge_page get_huge_page_for_hwpoison
!PageHeadHuge -- so 2(not a hugepage) is returned
hugetlb_vmemmap_optimize -- subpages is read-only
set_compound_page_dtor -- PageHuge is true now, but too late!!!
TestSetPageHWPoison(p)
-- We might write to read-only subpages here!!!
Another similar scene:
CPU 1 CPU 2
free_huge_page -- page refcnt == 0 and not PageHuge memory_failure
get_huge_page_for_hwpoison
!PageHeadHuge -- so 2(not a hugepage) is returned
TestSetPageHWPoison(p)
-- We might write to read-only subpages here!!!
hugetlb_vmemmap_restore -- subpages can be written to now, but too late!!!
I think the above scenes are possible. But I can't found a stable solution to fix it. Any suggestions?
Or is it not worth to fix it as it's too rare? Or am I miss something?
Any response would be appreciated!
Thanks!
On Tue, Aug 02, 2022 at 10:00:50AM +0800, Miaohe Lin wrote:
> Hi all:
> When I investigate the mm/memory-failure.c code again, I found there's a possible race window
> between memory_failure and alloc_fresh_huge_page/free_huge_page. Thank about the below scene:
>
> CPU 1 CPU 2
> alloc_fresh_huge_page -- page refcnt > 0 memory_failure
> prep_new_huge_page get_huge_page_for_hwpoison
> !PageHeadHuge -- so 2(not a hugepage) is returned
> hugetlb_vmemmap_optimize -- subpages is read-only
> set_compound_page_dtor -- PageHuge is true now, but too late!!!
> TestSetPageHWPoison(p)
> -- We might write to read-only subpages here!!!
>
> Another similar scene:
>
> CPU 1 CPU 2
> free_huge_page -- page refcnt == 0 and not PageHuge memory_failure
> get_huge_page_for_hwpoison
> !PageHeadHuge -- so 2(not a hugepage) is returned
> TestSetPageHWPoison(p)
> -- We might write to read-only subpages here!!!
> hugetlb_vmemmap_restore -- subpages can be written to now, but too late!!!
>
I agree this race is possible, I have proposed this race in thread [1].
But I didn't think more how to solve it.
[1] https://lore.kernel.org/linux-mm/[email protected]/T/#ma094a7cea7df8fd9a77a91551bf39077d89e23bd
> I think the above scenes are possible. But I can't found a stable solution to fix it. Any suggestions?
> Or is it not worth to fix it as it's too rare? Or am I miss something?
>
Luckily, the system will panic at once, which encountering this race. However,
we don't see any bug report. If we have an easy way to fix it, I think it is worth.
Just a quick reply, no suggestion/solutions from me.
Thanks.
> Any response would be appreciated!
>
> Thanks!
>
On 2022/8/2 12:07, Muchun Song wrote:
> On Tue, Aug 02, 2022 at 10:00:50AM +0800, Miaohe Lin wrote:
>> Hi all:
>> When I investigate the mm/memory-failure.c code again, I found there's a possible race window
>> between memory_failure and alloc_fresh_huge_page/free_huge_page. Thank about the below scene:
>>
>> CPU 1 CPU 2
>> alloc_fresh_huge_page -- page refcnt > 0 memory_failure
>> prep_new_huge_page get_huge_page_for_hwpoison
>> !PageHeadHuge -- so 2(not a hugepage) is returned
>> hugetlb_vmemmap_optimize -- subpages is read-only
>> set_compound_page_dtor -- PageHuge is true now, but too late!!!
>> TestSetPageHWPoison(p)
>> -- We might write to read-only subpages here!!!
>>
>> Another similar scene:
>>
>> CPU 1 CPU 2
>> free_huge_page -- page refcnt == 0 and not PageHuge memory_failure
>> get_huge_page_for_hwpoison
>> !PageHeadHuge -- so 2(not a hugepage) is returned
>> TestSetPageHWPoison(p)
>> -- We might write to read-only subpages here!!!
>> hugetlb_vmemmap_restore -- subpages can be written to now, but too late!!!
>>
>
> I agree this race is possible, I have proposed this race in thread [1].
Oh, I remember I see the race proposed in [1] but I did not look into that carefully at that time. Sorry.
> But I didn't think more how to solve it.
I hope this thread can find a good solution. :)
>
> [1] https://lore.kernel.org/linux-mm/[email protected]/T/#ma094a7cea7df8fd9a77a91551bf39077d89e23bd
>
>> I think the above scenes are possible. But I can't found a stable solution to fix it. Any suggestions?
>> Or is it not worth to fix it as it's too rare? Or am I miss something?
>>
>
> Luckily, the system will panic at once, which encountering this race. However,
> we don't see any bug report. If we have an easy way to fix it, I think it is worth.
Agree. But I can't find a easy way to fix it yet.
> Just a quick reply, no suggestion/solutions from me.
Many thanks for your quick reply.
>
> Thanks.
>
>> Any response would be appreciated!
>>
>> Thanks!
>>
> .
>
On Thu, Aug 04, 2022 at 04:40:25PM +0900, Naoya Horiguchi wrote:
> On Tue, Aug 02, 2022 at 02:27:36PM +0800, Miaohe Lin wrote:
> > On 2022/8/2 12:07, Muchun Song wrote:
> > > On Tue, Aug 02, 2022 at 10:00:50AM +0800, Miaohe Lin wrote:
> > >> Hi all:
> > >> When I investigate the mm/memory-failure.c code again, I found there's a possible race window
> > >> between memory_failure and alloc_fresh_huge_page/free_huge_page. Thank about the below scene:
> > >>
> > >> CPU 1 CPU 2
> > >> alloc_fresh_huge_page -- page refcnt > 0 memory_failure
> > >> prep_new_huge_page get_huge_page_for_hwpoison
> > >> !PageHeadHuge -- so 2(not a hugepage) is returned
> > >> hugetlb_vmemmap_optimize -- subpages is read-only
> > >> set_compound_page_dtor -- PageHuge is true now, but too late!!!
> > >> TestSetPageHWPoison(p)
> > >> -- We might write to read-only subpages here!!!
> > >>
> > >> Another similar scene:
> > >>
> > >> CPU 1 CPU 2
> > >> free_huge_page -- page refcnt == 0 and not PageHuge memory_failure
> > >> get_huge_page_for_hwpoison
> > >> !PageHeadHuge -- so 2(not a hugepage) is returned
> > >> TestSetPageHWPoison(p)
> > >> -- We might write to read-only subpages here!!!
> > >> hugetlb_vmemmap_restore -- subpages can be written to now, but too late!!!
> > >>
> > >
> > > I agree this race is possible, I have proposed this race in thread [1].
>
> Thank you for reminding this, and I agree that we need some solution.
>
> >
> > Oh, I remember I see the race proposed in [1] but I did not look into that carefully at that time. Sorry.
> >
> > > But I didn't think more how to solve it.
> > I hope this thread can find a good solution. :)
>
> Both of the races show that __get_huge_page_for_hwpoison() fails to
> capture the case of generic compound page during turning into hugetlb,
I noticed my wrong description just after sending this out. The second race
is about generic compound page turning into "buddy", sorry.
- Naoya Horiguchi
> What makes things complicated is that no one knows to which state
> such a compound page finally turns into. So I think that if the page
> to be handled is an unknown compound page, we need to wait until it
> becomes some known page state to avoid misjudging.
>
> If we need a quick small fix, we may replace the check "!PageHeadHuge()"
> in __get_huge_page_for_hwpoison() with "!PageCompound()", and add another
> retry path in get_huge_page_for_hwpoison() for non-hugetlb compound pages.
>
> Thanks,
> Naoya Horiguchi
On Tue, Aug 02, 2022 at 02:27:36PM +0800, Miaohe Lin wrote:
> On 2022/8/2 12:07, Muchun Song wrote:
> > On Tue, Aug 02, 2022 at 10:00:50AM +0800, Miaohe Lin wrote:
> >> Hi all:
> >> When I investigate the mm/memory-failure.c code again, I found there's a possible race window
> >> between memory_failure and alloc_fresh_huge_page/free_huge_page. Thank about the below scene:
> >>
> >> CPU 1 CPU 2
> >> alloc_fresh_huge_page -- page refcnt > 0 memory_failure
> >> prep_new_huge_page get_huge_page_for_hwpoison
> >> !PageHeadHuge -- so 2(not a hugepage) is returned
> >> hugetlb_vmemmap_optimize -- subpages is read-only
> >> set_compound_page_dtor -- PageHuge is true now, but too late!!!
> >> TestSetPageHWPoison(p)
> >> -- We might write to read-only subpages here!!!
> >>
> >> Another similar scene:
> >>
> >> CPU 1 CPU 2
> >> free_huge_page -- page refcnt == 0 and not PageHuge memory_failure
> >> get_huge_page_for_hwpoison
> >> !PageHeadHuge -- so 2(not a hugepage) is returned
> >> TestSetPageHWPoison(p)
> >> -- We might write to read-only subpages here!!!
> >> hugetlb_vmemmap_restore -- subpages can be written to now, but too late!!!
> >>
> >
> > I agree this race is possible, I have proposed this race in thread [1].
Thank you for reminding this, and I agree that we need some solution.
>
> Oh, I remember I see the race proposed in [1] but I did not look into that carefully at that time. Sorry.
>
> > But I didn't think more how to solve it.
> I hope this thread can find a good solution. :)
Both of the races show that __get_huge_page_for_hwpoison() fails to
capture the case of generic compound page during turning into hugetlb,
What makes things complicated is that no one knows to which state
such a compound page finally turns into. So I think that if the page
to be handled is an unknown compound page, we need to wait until it
becomes some known page state to avoid misjudging.
If we need a quick small fix, we may replace the check "!PageHeadHuge()"
in __get_huge_page_for_hwpoison() with "!PageCompound()", and add another
retry path in get_huge_page_for_hwpoison() for non-hugetlb compound pages.
Thanks,
Naoya Horiguchi
>
> >
> > [1] https://lore.kernel.org/linux-mm/[email protected]/T/#ma094a7cea7df8fd9a77a91551bf39077d89e23bd
> >
> >> I think the above scenes are possible. But I can't found a stable solution to fix it. Any suggestions?
> >> Or is it not worth to fix it as it's too rare? Or am I miss something?
> >>
> >
> > Luckily, the system will panic at once, which encountering this race. However,
> > we don't see any bug report. If we have an easy way to fix it, I think it is worth.
>
> Agree. But I can't find a easy way to fix it yet.
>
> > Just a quick reply, no suggestion/solutions from me.
>
> Many thanks for your quick reply.
>
> >
> > Thanks.
> >
> >> Any response would be appreciated!
> >>
> >> Thanks!
> >>
> > .
> >
>
On 2022/8/4 15:44, Naoya Horiguchi wrote:
> On Thu, Aug 04, 2022 at 04:40:25PM +0900, Naoya Horiguchi wrote:
>> On Tue, Aug 02, 2022 at 02:27:36PM +0800, Miaohe Lin wrote:
>>> On 2022/8/2 12:07, Muchun Song wrote:
>>>> On Tue, Aug 02, 2022 at 10:00:50AM +0800, Miaohe Lin wrote:
>>>>> Hi all:
>>>>> When I investigate the mm/memory-failure.c code again, I found there's a possible race window
>>>>> between memory_failure and alloc_fresh_huge_page/free_huge_page. Thank about the below scene:
>>>>>
>>>>> CPU 1 CPU 2
>>>>> alloc_fresh_huge_page -- page refcnt > 0 memory_failure
>>>>> prep_new_huge_page get_huge_page_for_hwpoison
>>>>> !PageHeadHuge -- so 2(not a hugepage) is returned
>>>>> hugetlb_vmemmap_optimize -- subpages is read-only
>>>>> set_compound_page_dtor -- PageHuge is true now, but too late!!!
>>>>> TestSetPageHWPoison(p)
>>>>> -- We might write to read-only subpages here!!!
>>>>>
>>>>> Another similar scene:
>>>>>
>>>>> CPU 1 CPU 2
>>>>> free_huge_page -- page refcnt == 0 and not PageHuge memory_failure
>>>>> get_huge_page_for_hwpoison
>>>>> !PageHeadHuge -- so 2(not a hugepage) is returned
>>>>> TestSetPageHWPoison(p)
>>>>> -- We might write to read-only subpages here!!!
>>>>> hugetlb_vmemmap_restore -- subpages can be written to now, but too late!!!
>>>>>
>>>>
>>>> I agree this race is possible, I have proposed this race in thread [1].
>>
>> Thank you for reminding this, and I agree that we need some solution.
>>
>>>
>>> Oh, I remember I see the race proposed in [1] but I did not look into that carefully at that time. Sorry.
>>>
>>>> But I didn't think more how to solve it.
>>> I hope this thread can find a good solution. :)
>>
>> Both of the races show that __get_huge_page_for_hwpoison() fails to
>> capture the case of generic compound page during turning into hugetlb,
>
> I noticed my wrong description just after sending this out. The second race
> is about generic compound page turning into "buddy", sorry.
Never mind. Thanks for your thought.
>
> - Naoya Horiguchi
>
>> What makes things complicated is that no one knows to which state
>> such a compound page finally turns into. So I think that if the page
>> to be handled is an unknown compound page, we need to wait until it
>> becomes some known page state to avoid misjudging.
>>
>> If we need a quick small fix, we may replace the check "!PageHeadHuge()"
>> in __get_huge_page_for_hwpoison() with "!PageCompound()", and add another
>> retry path in get_huge_page_for_hwpoison() for non-hugetlb compound pages.
Sounds like a good idea. If we can figure out a stable way to tell it from other
compound page, it will be worth fixing it. Let's have a try. :)
>>
>> Thanks,
>> Naoya Horiguchi
Many thanks for your suggestion, Naoya.
> .
>