Received-SPF: pass (google.com: domain of linux-kernel+bounces-112644-linux.lists.archive=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99;
Precedence: bulk
MIME-Version: 1.0
References: <be75859d7e4245939d825da5e881902c@BJMBX01.spreadtrum.com>
 <Zfg0WLrcOmCtdn_M@casper.infradead.org> <CAGWkznGhiuzkqfeewNq-ykKehvFTBjH2v_==xAS2_7iFqsFk5A@mail.gmail.com>
 <Zfj_-LbiIsAVWcPX@casper.infradead.org> <CAGWkznHOWVPpKHOuvvDF2zppkTT0_x4WyJ5S75j+EghhZmEDdg@mail.gmail.com>
 <ZfwpuRjAZV07_lc3@casper.infradead.org> <CAGWkznFtez1fj2L2CtFnA5k-Tn4WtxmDOw=fjOWPg-ZGJX=VWw@mail.gmail.com>
 <Zfz4_GJAHRInB8ul@casper.infradead.org>
In-Reply-To: <Zfz4_GJAHRInB8ul@casper.infradead.org>
From: Zhaoyang Huang <huangzhaoyang@gmail.com>
Date: Sun, 24 Mar 2024 19:14:27 +0800
Message-ID: <CAGWkznEzpcWi4oXVn_WFahnQj3dHcwc_4VW6m1Ss-KJ8mD3F3Q@mail.gmail.com>
Subject: Re: summarize all information again at bottom//reply: reply: [PATCH]
 mm: fix a race scenario in folio_isolate_lru
To: Matthew Wilcox <willy@infradead.org>
Cc: =?UTF-8?B?6buE5pyd6ZizIChaaGFveWFuZyBIdWFuZyk=?= <zhaoyang.huang@unisoc.com>, 
	Andrew Morton <akpm@linux-foundation.org>, "linux-mm@kvack.org" <linux-mm@kvack.org>, 
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>, 
	=?UTF-8?B?5bq357qq5ruoIChTdGV2ZSBLYW5nKQ==?= <Steve.Kang@unisoc.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

On Fri, Mar 22, 2024 at 11:20=E2=80=AFAM Matthew Wilcox <willy@infradead.or=
g> wrote:
>
> On Fri, Mar 22, 2024 at 09:52:36AM +0800, Zhaoyang Huang wrote:
> > Thanks for the comments. fix the typo and update the timing sequence
> > by amending possible preempt points to have the refcnt make sense.
> >
> > 0. Thread_bad gets the folio by find_get_entry and preempted before
> > take refcnt(could be the second round scan of
> > truncate_inode_pages_range)
> >     refcnt =3D=3D 1(page_cache), PG_lru =3D=3D true, PG_lock =3D=3D fal=
se
> >     find_get_entry
> >         folio =3D xas_find
> >         <preempted>
> >         folio_try_get_rcu
> >
> > 1. Thread_filemap get the folio via
> > filemap_map_pages->next_uptodate_folio->xas_next_entry and gets preempt=
ed
> >     refcnt =3D=3D 1(page_cache), PG_lru =3D=3D true, PG_lock =3D=3D fal=
se
> >     filemap_map_pages
> >         next_uptodate_folio
> >            xas_next_entry
> >            <preempted>
> >            folio_try_get_rcu
> >
> > 2. Thread_truncate get the folio via
> > truncate_inode_pages_range->find_lock_entries
> >     refcnt =3D=3D 2(page_cache, fbatch_truncate), PG_lru =3D=3D true, P=
G_lock =3D=3D true
> >
> > 3. Thread_truncate proceed to truncate_cleanup_folio
> >     refcnt =3D=3D 2(page_cache, fbatch_truncate), PG_lru =3D=3D true, P=
G_lock =3D=3D true
> >
> > 4. Thread_truncate proceed to delete_from_page_cache_batch
> >     refcnt =3D=3D 1(fbatch_truncate), PG_lru =3D=3D true, PG_lock =3D=
=3D true
> >
> > 4.1 folio_unlock
> >     refcnt =3D=3D 1(fbatch_truncate), PG_lru =3D=3D true, PG_lock =3D=
=3D false
>
> OK, so by the time we get to folio_unlock(), the folio has been removed
> from the i_pages xarray.
>
> > 5. Thread_filemap schedule back from '1' and proceed to setup a pte
> > and have folio->_mapcnt =3D 0 & folio->refcnt +=3D 1
> >     refcnt =3D=3D 1->2(+fbatch_filemap)->3->2(pte, fbatch_truncate),
> > PG_lru =3D=3D true, PG_lock =3D=3D true->false
>
> This line succeeds (in next_uptodate_folio):
>                 if (!folio_try_get_rcu(folio))
>                         continue;
> but then this fails:
>
>                 if (unlikely(folio !=3D xas_reload(xas)))
>                         goto skip;
> skip:
>                 folio_put(folio);
>
> because xas_reload() will return NULL due to the folio being deleted
> in step 4.  So we never get to the point where we set up a PTE.
>
> There should be no way to create a new PTE for a folio which has been
> removed from the page cache.  Bugs happen, of course, but I don't see
> one yet.
>
> > 6. Thread_madv clear folio's PG_lru by
> > madvise_xxx_pte_range->folio_isolate_lru->folio_test_clear_lru
> >     refcnt =3D=3D 2(pte,fbatch_truncate), PG_lru =3D=3D false, PG_lock =
=3D=3D false
> >
> > 7. Thread_truncate call folio_fbatch_release and failed in freeing
> > folio as refcnt not reach 0
> >     refcnt =3D=3D 1(pte), PG_lru =3D=3D false, PG_lock =3D=3D false
> > ********folio becomes an orphan here which is not on the page cache
> > but on the task's VM**********
> >
> > 8. Thread_bad scheduled back from '0' to be collected in fbatch_bad
> >     refcnt =3D=3D 2(pte, fbatch_bad), PG_lru =3D=3D false, PG_lock =3D=
=3D true
> >
> > 9. Thread_bad clear one refcnt wrongly when doing filemap_remove_folio
> > as it take this refcnt as the page cache one
> >     refcnt =3D=3D 1(fbatch_bad), PG_lru =3D=3D false, PG_lock =3D=3D tr=
ue->false
> >     truncate_inode_folio
> >         filemap_remove_folio
> >              filemap_free_folio
> > ******refcnt decreased wrongly here by being taken as the page cache on=
e ******
> >
> > 10. Thread_bad calls release_pages(fbatch_bad) and has the folio
> > introduce the bug.
> >     release_pages
> >         folio_put_testzero =3D=3D true
> >         folio_test_lru =3D=3D false
> >         list_add(folio->lru, pages_to_free)

ok. It seems like madvise is robust enough to leave no BUGs. I presume
another two scenarios which call folio_isloate_lru by any other ways
but PTE. Besides, scenario 2 reminds me of a previous bug reported by
me as find_get_entry entered in a livelock where the folio's refcnt =3D=3D
0 but remains at xarray which causes the reset->retry loops forever. I
would like to reply in that thread for more details.

Scenario 1:
0. Thread_bad gets the folio by find_get_entry and preempted before
folio_lock (could be the second round scan of
truncate_inode_pages_range)
    refcnt =3D=3D 2(page_cache, fbatch_bad), PG_lru =3D=3D true, PG_lock =
=3D=3D false
        folio =3D find_get_entry
        folio_try_get_rcu
        <preempted>
        folio_try_lock

1. Thread_truncate get the folio via
truncate_inode_pages_range->find_lock_entries
    refcnt =3D=3D 3(page_cache, fbatch_bad, fbatch_truncate), PG_lru =3D=3D
true, PG_lock =3D=3D true

2. Thread_truncate proceed to truncate_cleanup_folio
    refcnt =3D=3D 3(page_cache, fbatch_bad, fbatch_truncate), PG_lru =3D=3D
true, PG_lock =3D=3D true

3. Thread_truncate proceed to delete_from_page_cache_batch
    refcnt =3D=3D 2(fbatch_bad, fbatch_truncate), PG_lru =3D=3D true, PG_lo=
ck =3D=3D true

4  folio_unlock
    refcnt =3D=3D 2(fbatch_bad, fbatch_truncate), PG_lru =3D=3D true, PG_lo=
ck =3D=3D false

5. Thread_bad schedule back from step 0 and clear one refcnt wrongly
when doing truncate_inode_folio->filemap_remove_folio as it take this
refcnt as the page cache one
    refcnt =3D=3D 1'(fbatch_truncate), PG_lru =3D=3D false, PG_lock =3D=3D =
true
        folio =3D find_get_entry
        folio_try_get_rcu
        folio_try_lock
        truncate_inode_folio
            filemap_remove_folio
            <preempted>

6. Thread_isolate get one refcnt and call folio_isolate_lru(could be
any process)
    refcnt =3D=3D 2'(fbatch_truncate, thread_isolate), PG_lru =3D=3D true,
PG_lock =3D=3D true

7. Thread_isolate proceed to clear PG_lru and get preempted before folio_ge=
t
    refcnt =3D=3D 2'(fbatch_truncate, thread_isolate), PG_lru =3D=3D false,
PG_lock =3D=3D true
        folio_test_clear_folio
        <preempted>
        folio_get

8. Thread_bad scheduling back from step 5 and proceed to drop one refcnt
    refcnt =3D=3D 1'(thread_isolate), PG_lru =3D=3D false, PG_lock =3D=3D t=
rue
        folio =3D find_get_entry
        folio_try_get_rcu
        folio_try_lock
        truncate_inode_folio
            filemap_remove_folio
        folio_unlock
            <preempted>

9. Thread_truncate schedule back from step 3 and proceed to drop one
refcnt by release_pages and hit the BUG
    refcnt =3D=3D 0'(thread_isolate), PG_lru =3D=3D false, PG_lock =3D=3D f=
alse

Scenario 2:
0. Thread_bad gets the folio by find_get_entry and preempted before
folio_lock (could be the second round scan of
truncate_inode_pages_range)
    refcnt =3D=3D 2(page_cache, fbatch_bad), PG_lru =3D=3D true, PG_lock =
=3D=3D false
        folio =3D find_get_entry
        folio_try_get_rcu
        <preempted>
        folio_try_lock

1. Thread_readahead remove the folio from page cache and drop one
refcnt by filemap_remove_folio(get rid of the folios which failed to
launch IO during readahead)
    refcnt =3D=3D 1(fbatch_bad), PG_lru =3D=3D true, PG_lock =3D=3D true

2.  folio_unlock
    refcnt =3D=3D 1(fbatch_bad), PG_lru =3D=3D true, PG_lock =3D=3D false

3. Thread_isolate get one refcnt and call folio_isolate_lru(could be
any process)
    refcnt =3D=3D 2(fbatch_bad, thread_isolate), PG_lru =3D=3D true, PG_loc=
k =3D=3D false

4. Thread_bad schedule back from step 0 and clear one refcnt wrongly
when doing truncate_inode_folio->filemap_remove_folio as it take this
refcnt as the page cache one
    refcnt =3D=3D 1'(thread_isolate), PG_lru =3D=3D false, PG_lock =3D=3D f=
alse
        find_get_entries
            folio =3D find_get_entry
            folio_try_get_rcu
        folio_lock
        <no check as folio->mapping !=3D mapping as folio_lock_entries does=
>
        truncate_inode_folio
            filemap_remove_folio
            <preempted>

5. Thread_isolate proceed to clear PG_lru and get preempted before folio_ge=
t
    refcnt =3D=3D 1'(fbatch_truncate, thread_isolate), PG_lru =3D=3D false,
PG_lock =3D=3D false
        folio_test_clear_folio
        <preempted>
        folio_get

6. Thread_bad schedule back from step 4 and proceed to drop one refcnt
by release_pages and hit the BUG
    refcnt =3D=3D 0'(thread_isolate), PG_lru =3D=3D false, PG_lock =3D=3D f=
alse