Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Thu,  1 Dec 2022 11:26:44 -0800
Mime-Version: 1.0
Message-ID: <20221201192644.1941049-1-axelrasmussen@google.com>
Subject: [PATCH] mm: multi-gen LRU: fix LRU size accounting on folio removal
From:   Axel Rasmussen <axelrasmussen@google.com>
To:     Andrew Morton <akpm@linux-foundation.org>,
        Suleiman Souhlal <suleiman@google.com>,
        Steven Barrett <steven@liquorix.net>,
        Yu Zhao <yuzhao@google.com>,
        Oleksandr Natalenko <oleksandr@natalenko.name>,
        Suren Baghdasaryan <surenb@google.com>,
        Arnd Bergmann <arnd@arndb.de>, Peter Xu <peterx@redhat.com>,
        Gaosheng Cui <cuigaosheng1@huawei.com>,
        Hugh Dickins <hughd@google.com>,
        Brian Geffon <bgeffon@google.com>,
        "Jan Alexander Steffens (heftig)" <heftig@archlinux.org>
Cc:     linux-kernel@vger.kernel.org, Yosry Ahmed <yosryahmed@google.com>,
        Axel Rasmussen <axelrasmussen@google.com>
Content-Type: text/plain; charset="UTF-8"
Precedence: bulk

When removing a folio from MGLRU, we want to update the LRU size
accordingly based on the generation it belonged to previously -
lru_gen_update_size() does this.

The bug here is, set_mask_bits effectively clears the generation bits.
Ignoring the complexity set_mask_bits is meant to handle, the code being
changed here is in effect:

    flags = !reclaiming && lru_gen_is_active(lruvec, gen) ? BIT(PG_active) : 0;
    flags = *folio->flags = (*folio->flags & ~LRU_GEN_MASK) | flags;
    gen = ((flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF) - 1;

In other words, the bug is we clear all of the `LGU_GEN_MASK` bits, and
then we recalculate `gen` - but of course after clearing the bits
`flags & LRU_GEN_MASK` is always zero, and so now `gen` is always -1.

So we effectively always call:

    lru_gen_update_size(lruvec, folio, -1, -1);

This leads `lru_gen_update_size` to incorrectly conclude that we're
**adding**, not removing, a folio. We take this path:

    /* addition */
    if (old_gen < 0) {
        /* always false, new_gen is -1 too */
        if (lru_gen_is_active(lruvec, new_gen))
            /* ... */
	__update_lru_size(lruvec, lru, zone, delta);
	return;
    }

In other words, when removing, we incorrectly *add* the delta to the
inactive LRU instead of subtracting.

The fix is simple. We already have the generation number the folio
belonged to: we set `int gen = folio_lru_gen(folio);` at the top of
`lru_gen_del_folio`. So, just delete the line incorrectly recalculating
the generation number.

Fixes: ec1c86b25f4b ("mm: multi-gen LRU: groundwork")
Signed-off-by: Axel Rasmussen <axelrasmussen@google.com>
---
 include/linux/mm_inline.h | 1 -
 1 file changed, 1 deletion(-)

diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h
index e8ed225d8f7c..5bba6e0b0840 100644
--- a/include/linux/mm_inline.h
+++ b/include/linux/mm_inline.h
@@ -277,7 +277,6 @@ static inline bool lru_gen_del_folio(struct lruvec *lruvec, struct folio *folio,
 	/* for folio_migrate_flags() */
 	flags = !reclaiming && lru_gen_is_active(lruvec, gen) ? BIT(PG_active) : 0;
 	flags = set_mask_bits(&folio->flags, LRU_GEN_MASK, flags);
-	gen = ((flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF) - 1;
 
 	lru_gen_update_size(lruvec, folio, gen, -1);
 	list_del(&folio->lru);
-- 
2.39.0.rc0.267.gcb52ba06e7-goog