The zswap writeback mechanism can cause a race condition resulting in
memory corruption, where a swapped out page gets swapped in with data
that was written to a different page.
The race unfolds like this:
1. a page with data A and swap offset X is stored in zswap
2. page A is removed off the LRU by zpool driver for writeback in
zswap-shrink work, data for A is mapped by zpool driver
3. user space program faults and invalidates page entry A, offset X is
considered free
4. kswapd stores page B at offset X in zswap (zswap could also be full,
if so, page B would then be IOed to X, then skip step 5.)
5. entry A is replaced by B in tree->rbroot, this doesn't affect the
local reference held by zswap-shrink work
6. zswap-shrink work writes back A at X, and frees zswap entry A
7. swapin of slot X brings A in memory instead of B
The fix:
Once the swap page cache has been allocated (case ZSWAP_SWAPCACHE_NEW),
zswap-shrink work just checks that the local zswap_entry reference is
still the same as the one in the tree. If it's not the same it means
that it's either been invalidated or replaced, in both cases the
writeback is aborted because the local entry contains stale data.
Reproducer:
I originally found this by running `stress` overnight to validate my
work on the zswap writeback mechanism, it manifested after hours on my
test machine. The key to make it happen is having zswap writebacks, so
whatever setup pumps /sys/kernel/debug/zswap/written_back_pages should
do the trick.
In order to reproduce this faster on a vm, I setup a system with ~100M
of available memory and a 500M swap file, then running
`stress --vm 1 --vm-bytes 300000000 --vm-stride 4000` makes it happen
in matter of tens of minutes. One can speed things up even more by
swinging /sys/module/zswap/parameters/max_pool_percent up and down
between, say, 20 and 1; this makes it reproduce in tens of seconds.
It's crucial to set `--vm-stride` to something other than 4096 otherwise
`stress` won't realize that memory has been corrupted because all pages
would have the same data.
V2:
- updated comment with better explaination of the situation being
addressed in the check
Signed-off-by: Domenico Cerasuolo <[email protected]>
Acked-by: Johannes Weiner <[email protected]>
---
mm/zswap.c | 16 ++++++++++++++++
1 file changed, 16 insertions(+)
diff --git a/mm/zswap.c b/mm/zswap.c
index f6c89049cf70..5d5977c9ea45 100644
--- a/mm/zswap.c
+++ b/mm/zswap.c
@@ -995,6 +995,22 @@ static int zswap_writeback_entry(struct zpool *pool, unsigned long handle)
goto fail;
case ZSWAP_SWAPCACHE_NEW: /* page is locked */
+ /*
+ * Having a local reference to the zswap entry doesn't exclude
+ * swapping from invalidating and recycling the swap slot. Once
+ * the swapcache is secured against concurrent swapping to and
+ * from the slot, recheck that the entry is still current before
+ * writing.
+ */
+ spin_lock(&tree->lock);
+ if (zswap_rb_search(&tree->rbroot, entry->offset) != entry) {
+ spin_unlock(&tree->lock);
+ delete_from_swap_cache(page_folio(page));
+ ret = -ENOMEM;
+ goto fail;
+ }
+ spin_unlock(&tree->lock);
+
/* decompress */
acomp_ctx = raw_cpu_ptr(entry->pool->acomp_ctx);
dlen = PAGE_SIZE;
--
2.34.1
Hi Domenico,
On Wed, May 03, 2023 at 05:12:00PM +0200, Domenico Cerasuolo wrote:
> 1. a page with data A and swap offset X is stored in zswap
> 2. page A is removed off the LRU by zpool driver for writeback in
> zswap-shrink work, data for A is mapped by zpool driver
> 3. user space program faults and invalidates page entry A, offset X is
> considered free
> 4. kswapd stores page B at offset X in zswap (zswap could also be full,
> if so, page B would then be IOed to X, then skip step 5.)
> 5. entry A is replaced by B in tree->rbroot, this doesn't affect the
> local reference held by zswap-shrink work
> 6. zswap-shrink work writes back A at X, and frees zswap entry A
> 7. swapin of slot X brings A in memory instead of B
Thanks for the interesting discovery.
> V2:
> - updated comment with better explaination of the situation being
> addressed in the check
>
> Signed-off-by: Domenico Cerasuolo <[email protected]>
> Acked-by: Johannes Weiner <[email protected]>
> ---
> mm/zswap.c | 16 ++++++++++++++++
> 1 file changed, 16 insertions(+)
>
> diff --git a/mm/zswap.c b/mm/zswap.c
> index f6c89049cf70..5d5977c9ea45 100644
> --- a/mm/zswap.c
> +++ b/mm/zswap.c
> @@ -995,6 +995,22 @@ static int zswap_writeback_entry(struct zpool *pool, unsigned long handle)
> goto fail;
>
> case ZSWAP_SWAPCACHE_NEW: /* page is locked */
> + /*
> + * Having a local reference to the zswap entry doesn't exclude
> + * swapping from invalidating and recycling the swap slot. Once
> + * the swapcache is secured against concurrent swapping to and
> + * from the slot, recheck that the entry is still current before
> + * writing.
> + */
> + spin_lock(&tree->lock);
> + if (zswap_rb_search(&tree->rbroot, entry->offset) != entry) {
> + spin_unlock(&tree->lock);
> + delete_from_swap_cache(page_folio(page));
> + ret = -ENOMEM;
> + goto fail;
> + }
> + spin_unlock(&tree->lock);
> +
The race condition is still there, just making it much harder to hit.
What happens after you perform the rb tree search, release tree lock.
Then the entry gets invalid and recycled right here before the decompress
step?
> /* decompress */
> acomp_ctx = raw_cpu_ptr(entry->pool->acomp_ctx);
> dlen = PAGE_SIZE;
>
Chris
On Wed, May 03, 2023 at 02:59:31PM -0700, Chris Li wrote:
> Hi Domenico,
>
> On Wed, May 03, 2023 at 05:12:00PM +0200, Domenico Cerasuolo wrote:
> > 1. a page with data A and swap offset X is stored in zswap
> > 2. page A is removed off the LRU by zpool driver for writeback in
> > zswap-shrink work, data for A is mapped by zpool driver
> > 3. user space program faults and invalidates page entry A, offset X is
> > considered free
> > 4. kswapd stores page B at offset X in zswap (zswap could also be full,
> > if so, page B would then be IOed to X, then skip step 5.)
> > 5. entry A is replaced by B in tree->rbroot, this doesn't affect the
> > local reference held by zswap-shrink work
> > 6. zswap-shrink work writes back A at X, and frees zswap entry A
> > 7. swapin of slot X brings A in memory instead of B
>
> Thanks for the interesting discovery.
>
> > V2:
> > - updated comment with better explaination of the situation being
> > addressed in the check
> >
> > Signed-off-by: Domenico Cerasuolo <[email protected]>
> > Acked-by: Johannes Weiner <[email protected]>
> > ---
> > mm/zswap.c | 16 ++++++++++++++++
> > 1 file changed, 16 insertions(+)
> >
> > diff --git a/mm/zswap.c b/mm/zswap.c
> > index f6c89049cf70..5d5977c9ea45 100644
> > --- a/mm/zswap.c
> > +++ b/mm/zswap.c
> > @@ -995,6 +995,22 @@ static int zswap_writeback_entry(struct zpool *pool, unsigned long handle)
> > goto fail;
> >
> > case ZSWAP_SWAPCACHE_NEW: /* page is locked */
^^^^^^^^^^^^^^^^^^^^
> > + /*
> > + * Having a local reference to the zswap entry doesn't exclude
> > + * swapping from invalidating and recycling the swap slot. Once
> > + * the swapcache is secured against concurrent swapping to and
> > + * from the slot, recheck that the entry is still current before
> > + * writing.
> > + */
> > + spin_lock(&tree->lock);
> > + if (zswap_rb_search(&tree->rbroot, entry->offset) != entry) {
> > + spin_unlock(&tree->lock);
> > + delete_from_swap_cache(page_folio(page));
> > + ret = -ENOMEM;
> > + goto fail;
> > + }
> > + spin_unlock(&tree->lock);
> > +
>
> The race condition is still there, just making it much harder to hit.
> What happens after you perform the rb tree search, release tree lock.
> Then the entry gets invalid and recycled right here before the decompress
> step?
Recyling can only happen up until we see ZSWAP_SWAPCACHE_NEW.
Once we see it, we're holding the page lock* on a new swapcache page
for a valid, in-use** swp_entry_t.
The lock of the swapcache page prevents swapin, which would be
required for the count to drop and the entry to be recycled.
__read_swap_cache_async() checked that the entry is valid, so the slot
cannot be allocated to someone else.
Now we just have to check if that entry is the right one, iow the slot
wasn't recycled.
If the slot wasn't recycled, we know we have the right data and we can
start the IO and unlock the page. (After that swapins can continue and
the data can change, but regular writeback vs redirtying rules apply.)
If the slot was indeed recycled before we get ZSWAP_SWAPCACHE_NEW, we
see the mismatch, delete the page from the swapcache and unlock it. A
racing do_swap_page() may have found and reffed the page in swapcache,
and acquire the page lock after us; but it'll see it's no longer in
the swapcache, drop the reference (free the page) and retry the fault.
On Wed, May 03, 2023 at 10:29:04PM -0400, Johannes Weiner wrote:
> > >
> > > case ZSWAP_SWAPCACHE_NEW: /* page is locked */
>
> ^^^^^^^^^^^^^^^^^^^^
>
> > > + /*
> > > + * Having a local reference to the zswap entry doesn't exclude
> > > + * swapping from invalidating and recycling the swap slot. Once
> > > + * the swapcache is secured against concurrent swapping to and
> > > + * from the slot, recheck that the entry is still current before
> > > + * writing.
> > > + */
> > > + spin_lock(&tree->lock);
> > > + if (zswap_rb_search(&tree->rbroot, entry->offset) != entry) {
> > > + spin_unlock(&tree->lock);
> > > + delete_from_swap_cache(page_folio(page));
> > > + ret = -ENOMEM;
> > > + goto fail;
> > > + }
> > > + spin_unlock(&tree->lock);
> > > +
> >
> > The race condition is still there, just making it much harder to hit.
> > What happens after you perform the rb tree search, release tree lock.
> > Then the entry gets invalid and recycled right here before the decompress
> > step?
>
> Recyling can only happen up until we see ZSWAP_SWAPCACHE_NEW.
>
> Once we see it, we're holding the page lock* on a new swapcache page
> for a valid, in-use** swp_entry_t.
>
> The lock of the swapcache page prevents swapin, which would be
> required for the count to drop and the entry to be recycled.
Thanks for the explain. I miss the locked page will prevent swapin part.
> __read_swap_cache_async() checked that the entry is valid, so the slot
> cannot be allocated to someone else.
>
> Now we just have to check if that entry is the right one, iow the slot
> wasn't recycled.
>
> If the slot wasn't recycled, we know we have the right data and we can
> start the IO and unlock the page. (After that swapins can continue and
> the data can change, but regular writeback vs redirtying rules apply.)
>
> If the slot was indeed recycled before we get ZSWAP_SWAPCACHE_NEW, we
> see the mismatch, delete the page from the swapcache and unlock it. A
> racing do_swap_page() may have found and reffed the page in swapcache,
> and acquire the page lock after us; but it'll see it's no longer in
> the swapcache, drop the reference (free the page) and retry the fault.
LGTM then. Please feel free to add:
Reviewed-by: Chris Li (Google) <[email protected]>
Chris