Received: by 2002:a25:e74b:0:0:0:0:0 with SMTP id e72csp1130890ybh; Thu, 23 Jul 2020 00:51:33 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwyY9muI+JHN7gfQet8ZuRWUMn1EjRQdXxySPRjRguYsyEPZ/4xyRjlehCR/xw6ri0HcDHQ X-Received: by 2002:a05:6402:1507:: with SMTP id f7mr2015911edw.218.1595490693110; Thu, 23 Jul 2020 00:51:33 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1595490693; cv=none; d=google.com; s=arc-20160816; b=yM4U6uSPFT9hgQ2b1U80a/IEzX2YDwSP3nt01oLnWWtfdhN/ZEfqDip3H/AAPYfmt7 qrDCNQ3gx0OsqNx1uS5KmxIlF3YAGgJFw3fWQEgWItVlG4J5ojyXuSEVArx9hIwNqjWX 0ZB+i9AbT2TkKzzGChd/BW02PCzIPvB2dCs2E12QvhPnDVdFt3+JNUaFB3L1a+u7zNH8 hjAWz630FC7Q1/q3WkaPPdy0q7CihbpYJORXp16YwsR9PQVZgKJPR2m5RnUormapnFyq Btd3Y5Yag/hJOAV0VddlR9UNmKB3XXh8ItioggxsQcM6lOxslS0XyDboBMDH//loXHQM op0g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature; bh=nMJaxT009VYfWfSDnZgpATklA3yQDr7DBKWPTY196gs=; b=c4RiaK6vFbrnP8zq3H3c+h6ptB8vN4D+E40SJ/weTatVfEHAPONru4SpE1JL0QsOgD BFck0xeuf0Y7aMGtaQReY2jDaHR+W9xuJCAoFD7r8S1JmGkWwabU7vewA0xOl14xpZwD IJvd+2KY2T3xGYz/ALbse6yqMyGfc3/0UjVmF/3VivnE18hnH+SclOzAxZwJCNZqgjID 69imqW+drjrnV5EdFkBtun3xAx+Yvr64W7RqOARUwlYtTuQLB9WoAQSsgKyd1haMOb+8 jEnN1hEQMJ/gnY72d6vYAJsJSpKPbTFQN2a9SkhFXqnbMPNLlWjZNxexeMTZ2fEiWZ9i btsA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=R7YCPWss; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id j20si1518353ejx.721.2020.07.23.00.51.10; Thu, 23 Jul 2020 00:51:33 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=R7YCPWss; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727808AbgGWHt5 (ORCPT + 99 others); Thu, 23 Jul 2020 03:49:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43770 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726141AbgGWHt5 (ORCPT ); Thu, 23 Jul 2020 03:49:57 -0400 Received: from mail-pg1-x544.google.com (mail-pg1-x544.google.com [IPv6:2607:f8b0:4864:20::544]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EFF96C0619DC for ; Thu, 23 Jul 2020 00:49:56 -0700 (PDT) Received: by mail-pg1-x544.google.com with SMTP id z5so2670936pgb.6 for ; Thu, 23 Jul 2020 00:49:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=nMJaxT009VYfWfSDnZgpATklA3yQDr7DBKWPTY196gs=; b=R7YCPWssLYkFwfeNv++E63VyhosRbAta7GEpT410A48DocTsjm+U9spvC+FGC4GbG1 DMmF5sa8+mV3XtZCLkSejYScLOj6ech5jTf5LXYtzkrACQAh0JROkFNHRUa9HK5d5WGi cu0ypSLgfpTlaS3wtHx429qp+D1GC65BhLjYrfJkQ7y7tN9PZ6uwzxode7IqjFTuHpxn 0GeJKzopBE7D87FXts2ARy7PrWnjM+TdaumuJYXHzVYGYEi3BejRzKX/A2Ynh3JvgqrC a95dS4VNz/V4eksltGQ4VXYUQF0DqqPtIgawpQw70cxzRBsiUtIBKhETHenNMCWNXGAt mMUA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=nMJaxT009VYfWfSDnZgpATklA3yQDr7DBKWPTY196gs=; b=BJ5QrmzLrBastmyH2GM//NcWez5G6exlG+YsJTGhaWEiW0+Ae6Fx15K0GXuFzDXEnV c5bSGmlA9t/qUsfTCW88umgDZljKjQlHkEW8nJff1UpWIzBOS8x4s8ameL/1Bfl8+KHu eS/oFVHjyWFT2SxHIw/BissB7e3nIJlkXrNCLPyPPn7pVlyGKbbzPpWul8Um/ZfyqWzE 81az6yIlikIy4GLqWukOJ4rVbVsZYJjk+Vkf1Ru8+W0Svg3V6RRfjTbPc/l6cudOS1Me oRksFM6iIPdEPa/oxyUnD1Zt04hwHF2jvDs4BFHzQ5ClEIft7Rycl3dDnNU8zOawF5mK Dkaw== X-Gm-Message-State: AOAM532jAND3ISNYGHZ8sxtfAWzh6hEofLu7fPGD2fIxLqJ/RyO0DIOv SKQ8Ht6cMHBt0SYh7vA+2tg= X-Received: by 2002:a63:e045:: with SMTP id n5mr3323394pgj.274.1595490596480; Thu, 23 Jul 2020 00:49:56 -0700 (PDT) Received: from localhost.localdomain ([114.206.198.176]) by smtp.gmail.com with ESMTPSA id f71sm9164879pje.0.2020.07.23.00.49.53 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 23 Jul 2020 00:49:56 -0700 (PDT) From: js1304@gmail.com X-Google-Original-From: iamjoonsoo.kim@lge.com To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Johannes Weiner , Michal Hocko , Hugh Dickins , Minchan Kim , Vlastimil Babka , Mel Gorman , Matthew Wilcox , kernel-team@lge.com, Joonsoo Kim Subject: [PATCH v7 5/6] mm/swap: implement workingset detection for anonymous LRU Date: Thu, 23 Jul 2020 16:49:19 +0900 Message-Id: <1595490560-15117-6-git-send-email-iamjoonsoo.kim@lge.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1595490560-15117-1-git-send-email-iamjoonsoo.kim@lge.com> References: <1595490560-15117-1-git-send-email-iamjoonsoo.kim@lge.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Joonsoo Kim This patch implements workingset detection for anonymous LRU. All the infrastructure is implemented by the previous patches so this patch just activates the workingset detection by installing/retrieving the shadow entry and adding refault calculation. Acked-by: Johannes Weiner Acked-by: Vlastimil Babka Signed-off-by: Joonsoo Kim --- include/linux/swap.h | 6 ++++++ mm/memory.c | 11 ++++------- mm/swap_state.c | 23 ++++++++++++++++++----- mm/vmscan.c | 7 ++++--- mm/workingset.c | 15 +++++++++++---- 5 files changed, 43 insertions(+), 19 deletions(-) diff --git a/include/linux/swap.h b/include/linux/swap.h index 8a4c592..6610469 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -414,6 +414,7 @@ extern struct address_space *swapper_spaces[]; extern unsigned long total_swapcache_pages(void); extern void show_swap_cache_info(void); extern int add_to_swap(struct page *page); +extern void *get_shadow_from_swap_cache(swp_entry_t entry); extern int add_to_swap_cache(struct page *page, swp_entry_t entry, gfp_t gfp, void **shadowp); extern void __delete_from_swap_cache(struct page *page, @@ -573,6 +574,11 @@ static inline int add_to_swap(struct page *page) return 0; } +static inline void *get_shadow_from_swap_cache(swp_entry_t entry) +{ + return NULL; +} + static inline int add_to_swap_cache(struct page *page, swp_entry_t entry, gfp_t gfp_mask, void **shadowp) { diff --git a/mm/memory.c b/mm/memory.c index 25769b6..4934dbc 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3100,6 +3100,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) int locked; int exclusive = 0; vm_fault_t ret = 0; + void *shadow = NULL; if (!pte_unmap_same(vma->vm_mm, vmf->pmd, vmf->pte, vmf->orig_pte)) goto out; @@ -3151,13 +3152,9 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) goto out_page; } - /* - * XXX: Move to lru_cache_add() when it - * supports new vs putback - */ - spin_lock_irq(&page_pgdat(page)->lru_lock); - lru_note_cost_page(page); - spin_unlock_irq(&page_pgdat(page)->lru_lock); + shadow = get_shadow_from_swap_cache(entry); + if (shadow) + workingset_refault(page, shadow); lru_cache_add(page); swap_readpage(page, true); diff --git a/mm/swap_state.c b/mm/swap_state.c index 13d8d66..146a86d 100644 --- a/mm/swap_state.c +++ b/mm/swap_state.c @@ -106,6 +106,20 @@ void show_swap_cache_info(void) printk("Total swap = %lukB\n", total_swap_pages << (PAGE_SHIFT - 10)); } +void *get_shadow_from_swap_cache(swp_entry_t entry) +{ + struct address_space *address_space = swap_address_space(entry); + pgoff_t idx = swp_offset(entry); + struct page *page; + + page = find_get_entry(address_space, idx); + if (xa_is_value(page)) + return page; + if (page) + put_page(page); + return NULL; +} + /* * add_to_swap_cache resembles add_to_page_cache_locked on swapper_space, * but sets SwapCache flag and private instead of mapping and index. @@ -406,6 +420,7 @@ struct page *__read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask, { struct swap_info_struct *si; struct page *page; + void *shadow = NULL; *new_page_allocated = false; @@ -474,7 +489,7 @@ struct page *__read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask, __SetPageSwapBacked(page); /* May fail (-ENOMEM) if XArray node allocation failed. */ - if (add_to_swap_cache(page, entry, gfp_mask & GFP_RECLAIM_MASK, NULL)) { + if (add_to_swap_cache(page, entry, gfp_mask & GFP_RECLAIM_MASK, &shadow)) { put_swap_page(page, entry); goto fail_unlock; } @@ -484,10 +499,8 @@ struct page *__read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask, goto fail_unlock; } - /* XXX: Move to lru_cache_add() when it supports new vs putback */ - spin_lock_irq(&page_pgdat(page)->lru_lock); - lru_note_cost_page(page); - spin_unlock_irq(&page_pgdat(page)->lru_lock); + if (shadow) + workingset_refault(page, shadow); /* Caller will initiate read into locked page */ SetPageWorkingset(page); diff --git a/mm/vmscan.c b/mm/vmscan.c index b9b543e..9d4e28c 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -854,6 +854,7 @@ static int __remove_mapping(struct address_space *mapping, struct page *page, { unsigned long flags; int refcount; + void *shadow = NULL; BUG_ON(!PageLocked(page)); BUG_ON(mapping != page_mapping(page)); @@ -896,13 +897,13 @@ static int __remove_mapping(struct address_space *mapping, struct page *page, if (PageSwapCache(page)) { swp_entry_t swap = { .val = page_private(page) }; mem_cgroup_swapout(page, swap); - __delete_from_swap_cache(page, swap, NULL); + if (reclaimed && !mapping_exiting(mapping)) + shadow = workingset_eviction(page, target_memcg); + __delete_from_swap_cache(page, swap, shadow); xa_unlock_irqrestore(&mapping->i_pages, flags); put_swap_page(page, swap); - workingset_eviction(page, target_memcg); } else { void (*freepage)(struct page *); - void *shadow = NULL; freepage = mapping->a_ops->freepage; /* diff --git a/mm/workingset.c b/mm/workingset.c index 2d77e4d..92e6611 100644 --- a/mm/workingset.c +++ b/mm/workingset.c @@ -353,15 +353,22 @@ void workingset_refault(struct page *page, void *shadow) /* * Compare the distance to the existing workingset size. We * don't activate pages that couldn't stay resident even if - * all the memory was available to the page cache. Whether - * cache can compete with anon or not depends on having swap. + * all the memory was available to the workingset. Whether + * workingset competition needs to consider anon or not depends + * on having swap. */ workingset_size = lruvec_page_state(eviction_lruvec, NR_ACTIVE_FILE); - if (mem_cgroup_get_nr_swap_pages(memcg) > 0) { + if (!file) { workingset_size += lruvec_page_state(eviction_lruvec, - NR_INACTIVE_ANON); + NR_INACTIVE_FILE); + } + if (mem_cgroup_get_nr_swap_pages(memcg) > 0) { workingset_size += lruvec_page_state(eviction_lruvec, NR_ACTIVE_ANON); + if (file) { + workingset_size += lruvec_page_state(eviction_lruvec, + NR_INACTIVE_ANON); + } } if (refault_distance > workingset_size) goto out; -- 2.7.4