Received-SPF: pass (google.com: domain of linux-kernel+bounces-117722-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) client-ip=147.75.48.161;
Precedence: bulk
MIME-Version: 1.0
References: <20240324210447.956973-1-hannes@cmpxchg.org> <CAJD7tkaWQAV=X1pzYG=VkWe7Ue9ZFbjt9uQ5m1NJujtLspWJTA@mail.gmail.com>
 <CAGsJ_4yeBmNsMGXEWwC+1Hs5zJUP+becq4wG+6CpU7V1=EOvhg@mail.gmail.com>
 <CAJD7tka5K69q20bxTsBk38JC7mdPr3UsxXpsnggDO_iQA=qxug@mail.gmail.com>
 <1e7ce417-b9dd-4d62-9f54-0adf1ccdae35@linux.dev> <CAJD7tkYc3oFho5eEkS1zmr_+CC-Ag1HucUTyAy2RJbEb4SqRoQ@mail.gmail.com>
 <d556ec84-da6a-4486-a68e-5982baf46879@linux.dev> <CAJD7tkbNWeoNnkB7Q1z9-XdrE1L4_2okdjFGvLjnuwoXT9XmwA@mail.gmail.com>
 <339639f3-4334-44a8-a811-d20b3c578f74@linux.dev>
In-Reply-To: <339639f3-4334-44a8-a811-d20b3c578f74@linux.dev>
From: Yosry Ahmed <yosryahmed@google.com>
Date: Mon, 25 Mar 2024 11:35:06 -0700
Message-ID: <CAJD7tkZ=fsBi_t5Hcauty_3YKraJUmGA24tSvz8LjRd=Ao--2w@mail.gmail.com>
Subject: Re: [PATCH] mm: zswap: fix data loss on SWP_SYNCHRONOUS_IO devices
To: Chengming Zhou <chengming.zhou@linux.dev>
Cc: Barry Song <21cnbao@gmail.com>, Johannes Weiner <hannes@cmpxchg.org>, 
	Andrew Morton <akpm@linux-foundation.org>, Zhongkun He <hezhongkun.hzk@bytedance.com>, 
	Chengming Zhou <zhouchengming@bytedance.com>, Chris Li <chrisl@kernel.org>, 
	Nhat Pham <nphamcs@gmail.com>, linux-mm@kvack.org, linux-kernel@vger.kernel.org, 
	Kairui Song <kasong@tencent.com>, Minchan Kim <minchan@kernel.org>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

On Mon, Mar 25, 2024 at 2:46=E2=80=AFAM Chengming Zhou <chengming.zhou@linu=
x.dev> wrote:
>
> On 2024/3/25 17:40, Yosry Ahmed wrote:
> > On Mon, Mar 25, 2024 at 2:22=E2=80=AFAM Chengming Zhou <chengming.zhou@=
linux.dev> wrote:
> >>
> >> On 2024/3/25 16:38, Yosry Ahmed wrote:
> >>> On Mon, Mar 25, 2024 at 12:33=E2=80=AFAM Chengming Zhou
> >>> <chengming.zhou@linux.dev> wrote:
> >>>>
> >>>> On 2024/3/25 15:06, Yosry Ahmed wrote:
> >>>>> On Sun, Mar 24, 2024 at 9:54=E2=80=AFPM Barry Song <21cnbao@gmail.c=
om> wrote:
> >>>>>>
> >>>>>> On Mon, Mar 25, 2024 at 10:23=E2=80=AFAM Yosry Ahmed <yosryahmed@g=
oogle.com> wrote:
> >>>>>>>
> >>>>>>> On Sun, Mar 24, 2024 at 2:04=E2=80=AFPM Johannes Weiner <hannes@c=
mpxchg.org> wrote:
> >>>>>>>>
> >>>>>>>> Zhongkun He reports data corruption when combining zswap with zr=
am.
> >>>>>>>>
> >>>>>>>> The issue is the exclusive loads we're doing in zswap. They assu=
me
> >>>>>>>> that all reads are going into the swapcache, which can assume
> >>>>>>>> authoritative ownership of the data and so the zswap copy can go=
.
> >>>>>>>>
> >>>>>>>> However, zram files are marked SWP_SYNCHRONOUS_IO, and faults wi=
ll try
> >>>>>>>> to bypass the swapcache. This results in an optimistic read of t=
he
> >>>>>>>> swap data into a page that will be dismissed if the fault fails =
due to
> >>>>>>>> races. In this case, zswap mustn't drop its authoritative copy.
> >>>>>>>>
> >>>>>>>> Link: https://lore.kernel.org/all/CACSyD1N+dUvsu8=3DzV9P691B9bVq=
33erwOXNTmEaUbi9DrDeJzw@mail.gmail.com/
> >>>>>>>> Reported-by: Zhongkun He <hezhongkun.hzk@bytedance.com>
> >>>>>>>> Fixes: b9c91c43412f ("mm: zswap: support exclusive loads")
> >>>>>>>> Cc: stable@vger.kernel.org      [6.5+]
> >>>>>>>> Signed-off-by: Johannes Weiner <hannes@cmpxchg.org>
> >>>>>>>> Tested-by: Zhongkun He <hezhongkun.hzk@bytedance.com>
> >>>>>>
> >>>>>> Acked-by: Barry Song <baohua@kernel.org>
> >>>>>>
> >>>>>>>
> >>>>>>> Do we also want to mention somewhere (commit log or comment) that
> >>>>>>> keeping the entry in the tree is fine because we are still protec=
ted
> >>>>>>> from concurrent loads/invalidations/writeback by swapcache_prepar=
e()
> >>>>>>> setting SWAP_HAS_CACHE or so?
> >>>>>>
> >>>>>> It seems that Kairui's patch comprehensively addresses the issue a=
t hand.
> >>>>>> Johannes's solution, on the other hand, appears to align zswap beh=
avior
> >>>>>> more closely with that of a traditional swap device, only releasin=
g an entry
> >>>>>> when the corresponding swap slot is freed, particularly in the syn=
c-io case.
> >>>>>
> >>>>> It actually worked out quite well that Kairui's fix landed shortly
> >>>>> before this bug was reported, as this fix wouldn't have been possib=
le
> >>>>> without it as far as I can tell.
> >>>>>
> >>>>>>
> >>>>>> Johannes' patch has inspired me to consider whether zRAM could ach=
ieve
> >>>>>> a comparable outcome by immediately releasing objects in swap cach=
e
> >>>>>> scenarios.  When I have the opportunity, I plan to experiment with=
 zRAM.
> >>>>>
> >>>>> That would be interesting. I am curious if it would be as
> >>>>> straightforward in zram to just mark the folio as dirty in this cas=
e
> >>>>> like zswap does, given its implementation as a block device.
> >>>>>
> >>>>
> >>>> This makes me wonder who is responsible for marking folio dirty in t=
his swapcache
> >>>> bypass case? Should we call folio_mark_dirty() after the swap_read_f=
olio()?
> >>>
> >>> In shrink_folio_list(), we try to add anonymous folios to the
> >>> swapcache if they are not there before checking if they are dirty.
> >>> add_to_swap() calls folio_mark_dirty(), so this should take care of
> >>
> >> Right, thanks for your clarification, so should be no problem here.
> >> Although it was a fix just for MADV_FREE case.
> >>
> >>> it. There is an interesting comment there though. It says that PTE
> >>> should be dirty, so unmapping the folio should have already marked it
> >>> as dirty by the time we are adding it to the swapcache, except for th=
e
> >>> MADV_FREE case.
> >>
> >> It seems to say the folio will be dirtied when unmap later, supposing =
the
> >> PTE is dirty.
> >
> > Oh yeah it could mean that the folio will be dirted later.
> >
> >>
> >>>
> >>> However, I think we actually unmap the folio after we add it to the
> >>> swapcache in shrink_folio_list(). Also, I don't immediately see why
> >>> the PTE would be dirty. In do_swap_page(), making the PTE dirty seems
> >>
> >> If all anon pages on LRU list are faulted by write, it should be true.
> >> We could just use the zero page if faulted by read, right?
> >
> > This applies for the initial fault that creates the folio, but this is
> > a swap fault. It could be a read fault and in that case we still need
> > to make the folio dirty because it's not in the swapcache and we need
> > to write it out if it's reclaimed, right?
>
> Yes, IMHO I think it should be marked as dirty here.
>
> But it should be no problem with that unconditional folio_mark_dirty()
> in add_to_swap(). Not sure if there are other issues.

I don't believe there are any issues now. Dirtying the folio in
add_to_swap() was introduced before SWP_SYNCHRONOUS_IO, so I guess
things have always worked.

I think we should update the comment there though to mention that
dirtying the folio is also needed for this case (not just MADV_FREE),
or dirty the PTE during the fault. Otherwise, if someone is making
MADV_FREE changes they could end up breaking SWP_SYNCHRONOUS_IO
faults.

Adding Minchan here in case he can confirm that we in fact rely on
add_to_swap()->folio_mark_dirty() for SWP_SYNCHRONOUS_IO to work as
intended.