Received-SPF: pass (google.com: domain of linux-kernel+bounces-116768-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1;
Message-ID: <339639f3-4334-44a8-a811-d20b3c578f74@linux.dev>
Date: Mon, 25 Mar 2024 17:46:21 +0800
Precedence: bulk
MIME-Version: 1.0
Subject: Re: [PATCH] mm: zswap: fix data loss on SWP_SYNCHRONOUS_IO devices
Content-Language: en-US
To: Yosry Ahmed <yosryahmed@google.com>
Cc: Barry Song <21cnbao@gmail.com>, Johannes Weiner <hannes@cmpxchg.org>,
 Andrew Morton <akpm@linux-foundation.org>,
 Zhongkun He <hezhongkun.hzk@bytedance.com>,
 Chengming Zhou <zhouchengming@bytedance.com>, Chris Li <chrisl@kernel.org>,
 Nhat Pham <nphamcs@gmail.com>, linux-mm@kvack.org,
 linux-kernel@vger.kernel.org, Kairui Song <kasong@tencent.com>
References: <20240324210447.956973-1-hannes@cmpxchg.org>
 <CAJD7tkaWQAV=X1pzYG=VkWe7Ue9ZFbjt9uQ5m1NJujtLspWJTA@mail.gmail.com>
 <CAGsJ_4yeBmNsMGXEWwC+1Hs5zJUP+becq4wG+6CpU7V1=EOvhg@mail.gmail.com>
 <CAJD7tka5K69q20bxTsBk38JC7mdPr3UsxXpsnggDO_iQA=qxug@mail.gmail.com>
 <1e7ce417-b9dd-4d62-9f54-0adf1ccdae35@linux.dev>
 <CAJD7tkYc3oFho5eEkS1zmr_+CC-Ag1HucUTyAy2RJbEb4SqRoQ@mail.gmail.com>
 <d556ec84-da6a-4486-a68e-5982baf46879@linux.dev>
 <CAJD7tkbNWeoNnkB7Q1z9-XdrE1L4_2okdjFGvLjnuwoXT9XmwA@mail.gmail.com>
From: Chengming Zhou <chengming.zhou@linux.dev>
In-Reply-To: <CAJD7tkbNWeoNnkB7Q1z9-XdrE1L4_2okdjFGvLjnuwoXT9XmwA@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

On 2024/3/25 17:40, Yosry Ahmed wrote:
> On Mon, Mar 25, 2024 at 2:22 AM Chengming Zhou <chengming.zhou@linux.dev> wrote:
>>
>> On 2024/3/25 16:38, Yosry Ahmed wrote:
>>> On Mon, Mar 25, 2024 at 12:33 AM Chengming Zhou
>>> <chengming.zhou@linux.dev> wrote:
>>>>
>>>> On 2024/3/25 15:06, Yosry Ahmed wrote:
>>>>> On Sun, Mar 24, 2024 at 9:54 PM Barry Song <21cnbao@gmail.com> wrote:
>>>>>>
>>>>>> On Mon, Mar 25, 2024 at 10:23 AM Yosry Ahmed <yosryahmed@google.com> wrote:
>>>>>>>
>>>>>>> On Sun, Mar 24, 2024 at 2:04 PM Johannes Weiner <hannes@cmpxchg.org> wrote:
>>>>>>>>
>>>>>>>> Zhongkun He reports data corruption when combining zswap with zram.
>>>>>>>>
>>>>>>>> The issue is the exclusive loads we're doing in zswap. They assume
>>>>>>>> that all reads are going into the swapcache, which can assume
>>>>>>>> authoritative ownership of the data and so the zswap copy can go.
>>>>>>>>
>>>>>>>> However, zram files are marked SWP_SYNCHRONOUS_IO, and faults will try
>>>>>>>> to bypass the swapcache. This results in an optimistic read of the
>>>>>>>> swap data into a page that will be dismissed if the fault fails due to
>>>>>>>> races. In this case, zswap mustn't drop its authoritative copy.
>>>>>>>>
>>>>>>>> Link: https://lore.kernel.org/all/CACSyD1N+dUvsu8=zV9P691B9bVq33erwOXNTmEaUbi9DrDeJzw@mail.gmail.com/
>>>>>>>> Reported-by: Zhongkun He <hezhongkun.hzk@bytedance.com>
>>>>>>>> Fixes: b9c91c43412f ("mm: zswap: support exclusive loads")
>>>>>>>> Cc: stable@vger.kernel.org      [6.5+]
>>>>>>>> Signed-off-by: Johannes Weiner <hannes@cmpxchg.org>
>>>>>>>> Tested-by: Zhongkun He <hezhongkun.hzk@bytedance.com>
>>>>>>
>>>>>> Acked-by: Barry Song <baohua@kernel.org>
>>>>>>
>>>>>>>
>>>>>>> Do we also want to mention somewhere (commit log or comment) that
>>>>>>> keeping the entry in the tree is fine because we are still protected
>>>>>>> from concurrent loads/invalidations/writeback by swapcache_prepare()
>>>>>>> setting SWAP_HAS_CACHE or so?
>>>>>>
>>>>>> It seems that Kairui's patch comprehensively addresses the issue at hand.
>>>>>> Johannes's solution, on the other hand, appears to align zswap behavior
>>>>>> more closely with that of a traditional swap device, only releasing an entry
>>>>>> when the corresponding swap slot is freed, particularly in the sync-io case.
>>>>>
>>>>> It actually worked out quite well that Kairui's fix landed shortly
>>>>> before this bug was reported, as this fix wouldn't have been possible
>>>>> without it as far as I can tell.
>>>>>
>>>>>>
>>>>>> Johannes' patch has inspired me to consider whether zRAM could achieve
>>>>>> a comparable outcome by immediately releasing objects in swap cache
>>>>>> scenarios.  When I have the opportunity, I plan to experiment with zRAM.
>>>>>
>>>>> That would be interesting. I am curious if it would be as
>>>>> straightforward in zram to just mark the folio as dirty in this case
>>>>> like zswap does, given its implementation as a block device.
>>>>>
>>>>
>>>> This makes me wonder who is responsible for marking folio dirty in this swapcache
>>>> bypass case? Should we call folio_mark_dirty() after the swap_read_folio()?
>>>
>>> In shrink_folio_list(), we try to add anonymous folios to the
>>> swapcache if they are not there before checking if they are dirty.
>>> add_to_swap() calls folio_mark_dirty(), so this should take care of
>>
>> Right, thanks for your clarification, so should be no problem here.
>> Although it was a fix just for MADV_FREE case.
>>
>>> it. There is an interesting comment there though. It says that PTE
>>> should be dirty, so unmapping the folio should have already marked it
>>> as dirty by the time we are adding it to the swapcache, except for the
>>> MADV_FREE case.
>>
>> It seems to say the folio will be dirtied when unmap later, supposing the
>> PTE is dirty.
> 
> Oh yeah it could mean that the folio will be dirted later.
> 
>>
>>>
>>> However, I think we actually unmap the folio after we add it to the
>>> swapcache in shrink_folio_list(). Also, I don't immediately see why
>>> the PTE would be dirty. In do_swap_page(), making the PTE dirty seems
>>
>> If all anon pages on LRU list are faulted by write, it should be true.
>> We could just use the zero page if faulted by read, right?
> 
> This applies for the initial fault that creates the folio, but this is
> a swap fault. It could be a read fault and in that case we still need
> to make the folio dirty because it's not in the swapcache and we need
> to write it out if it's reclaimed, right?

Yes, IMHO I think it should be marked as dirty here.

But it should be no problem with that unconditional folio_mark_dirty()
in add_to_swap(). Not sure if there are other issues.

> 
>>
>>> to be conditional on the fault being a write fault, but I didn't look
>>> thoroughly, maybe I missed it. It is also possible that the comment is
>>> just outdated.
>>
>> Yeah, dirty is only marked on write fault.
>>
>> Thanks.