When removing a folio from MGLRU, we want to update the LRU size
accordingly based on the generation it belonged to previously -
lru_gen_update_size() does this.
The bug here is, set_mask_bits effectively clears the generation bits.
Ignoring the complexity set_mask_bits is meant to handle, the code being
changed here is in effect:
flags = !reclaiming && lru_gen_is_active(lruvec, gen) ? BIT(PG_active) : 0;
flags = *folio->flags = (*folio->flags & ~LRU_GEN_MASK) | flags;
gen = ((flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF) - 1;
In other words, the bug is we clear all of the `LGU_GEN_MASK` bits, and
then we recalculate `gen` - but of course after clearing the bits
`flags & LRU_GEN_MASK` is always zero, and so now `gen` is always -1.
So we effectively always call:
lru_gen_update_size(lruvec, folio, -1, -1);
This leads `lru_gen_update_size` to incorrectly conclude that we're
**adding**, not removing, a folio. We take this path:
/* addition */
if (old_gen < 0) {
/* always false, new_gen is -1 too */
if (lru_gen_is_active(lruvec, new_gen))
/* ... */
__update_lru_size(lruvec, lru, zone, delta);
return;
}
In other words, when removing, we incorrectly *add* the delta to the
inactive LRU instead of subtracting.
The fix is simple. We already have the generation number the folio
belonged to: we set `int gen = folio_lru_gen(folio);` at the top of
`lru_gen_del_folio`. So, just delete the line incorrectly recalculating
the generation number.
Fixes: ec1c86b25f4b ("mm: multi-gen LRU: groundwork")
Signed-off-by: Axel Rasmussen <[email protected]>
---
include/linux/mm_inline.h | 1 -
1 file changed, 1 deletion(-)
diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h
index e8ed225d8f7c..5bba6e0b0840 100644
--- a/include/linux/mm_inline.h
+++ b/include/linux/mm_inline.h
@@ -277,7 +277,6 @@ static inline bool lru_gen_del_folio(struct lruvec *lruvec, struct folio *folio,
/* for folio_migrate_flags() */
flags = !reclaiming && lru_gen_is_active(lruvec, gen) ? BIT(PG_active) : 0;
flags = set_mask_bits(&folio->flags, LRU_GEN_MASK, flags);
- gen = ((flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF) - 1;
lru_gen_update_size(lruvec, folio, gen, -1);
list_del(&folio->lru);
--
2.39.0.rc0.267.gcb52ba06e7-goog
On Thu, Dec 1, 2022 at 12:26 PM Axel Rasmussen <[email protected]> wrote:
>
> When removing a folio from MGLRU, we want to update the LRU size
> accordingly based on the generation it belonged to previously -
> lru_gen_update_size() does this.
>
> The bug here is, set_mask_bits effectively clears the generation bits.
> Ignoring the complexity set_mask_bits is meant to handle, the code being
> changed here is in effect:
>
> flags = !reclaiming && lru_gen_is_active(lruvec, gen) ? BIT(PG_active) : 0;
> flags = *folio->flags = (*folio->flags & ~LRU_GEN_MASK) | flags;
> gen = ((flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF) - 1;
>
> In other words, the bug is we clear all of the `LGU_GEN_MASK` bits, and
> then we recalculate `gen` - but of course after clearing the bits
> `flags & LRU_GEN_MASK` is always zero, and so now `gen` is always -1.
>
> So we effectively always call:
>
> lru_gen_update_size(lruvec, folio, -1, -1);
>
> This leads `lru_gen_update_size` to incorrectly conclude that we're
> **adding**, not removing, a folio. We take this path:
>
> /* addition */
> if (old_gen < 0) {
> /* always false, new_gen is -1 too */
> if (lru_gen_is_active(lruvec, new_gen))
> /* ... */
> __update_lru_size(lruvec, lru, zone, delta);
> return;
> }
>
> In other words, when removing, we incorrectly *add* the delta to the
> inactive LRU instead of subtracting.
>
> The fix is simple. We already have the generation number the folio
> belonged to: we set `int gen = folio_lru_gen(folio);` at the top of
> `lru_gen_del_folio`. So, just delete the line incorrectly recalculating
> the generation number.
>
> Fixes: ec1c86b25f4b ("mm: multi-gen LRU: groundwork")
> Signed-off-by: Axel Rasmussen <[email protected]>
NAK.
You are referencing our old (9xx) set_mask_bits(), which returns "new"
(a bad behavior). Its latest version returns "old".
Even if it was a bug:
1. lru_gen_update_size(lruvec, folio, -1, -1) would have been caught by
VM_WARN_ON_ONCE(old_gen == -1 && new_gen == -1).
2. The fix is still wrong, because "gen" read from
folio_lru_gen(folio) is non atomic and can change before
set_mask_bits() finishes.