From: Huang Ying <[email protected]>
Hi, Andrew, could you help me to check whether the overall design is
reasonable?
Hi, Hugh, Shaohua, Minchan and Rik, could you help me to review the
swap part of the patchset? Especially [1/9], [3/9], [4/9], [5/9],
[6/9], [9/9].
Hi, Andrea could you help me to review the THP part of the patchset?
Especially [2/9], [7/9] and [8/9].
Hi, Johannes, Michal and Vladimir, I am not very confident about the
memory cgroup part, especially [2/9]. Could you help me to review it?
And for all, Any comment is welcome!
Recently, the performance of the storage devices improved so fast that
we cannot saturate the disk bandwidth with single logical CPU when do
page swap out even on a high-end server machine. Because the
performance of the storage device improved faster than that of single
logical CPU. And it seems that the trend will not change in the near
future. On the other hand, the THP becomes more and more popular
because of increased memory size. So it becomes necessary to optimize
THP swap performance.
The advantages of the THP swap support include:
- Batch the swap operations for the THP to reduce lock
acquiring/releasing, including allocating/freeing the swap space,
adding/deleting to/from the swap cache, and writing/reading the swap
space, etc. This will help improve the performance of the THP swap.
- The THP swap space read/write will be 2M sequential IO. It is
particularly helpful for the swap read, which are usually 4k random
IO. This will improve the performance of the THP swap too.
- It will help the memory fragmentation, especially when the THP is
heavily used by the applications. The 2M continuous pages will be
free up after THP swapping out.
- It will improve the THP utilization on the system with the swap
turned on. Because the speed for khugepaged to collapse the normal
pages into the THP is quite slow. After the THP is split during the
swapping out, it will take quite long time for the normal pages to
collapse back into the THP after being swapped in. The high THP
utilization helps the efficiency of the page based memory management
too.
There are some concerns regarding THP swap in, mainly because possible
enlarged read/write IO size (for swap in/out) may put more overhead on
the storage device. To deal with that, the THP swap in should be
turned on only when necessary. For example, it can be selected via
"always/never/madvise" logic, to be turned on globally, turned off
globally, or turned on only for VMA with MADV_HUGEPAGE, etc.
This patchset is based on 03/17 head of mmotm/master.
This patchset is the first step for the THP swap support. The plan is
to delay splitting THP step by step, finally avoid splitting THP
during the THP swapping out and swap out/in the THP as a whole.
As the first step, in this patchset, the splitting huge page is
delayed from almost the first step of swapping out to after allocating
the swap space for the THP and adding the THP into the swap cache.
This will reduce lock acquiring/releasing for the locks used for the
swap cache management.
With the patchset, the swap out throughput improves 14.9% (from about
3.77GB/s to about 4.34GB/s) in the vm-scalability swap-w-seq test case
with 8 processes. The test is done on a Xeon E5 v3 system. The swap
device used is a RAM simulated PMEM (persistent memory) device. To
test the sequential swapping out, the test case creates 8 processes,
which sequentially allocate and write to the anonymous pages until the
RAM and part of the swap device is used up.
The detailed comparison result is as follow,
base base+patchset
---------------- --------------------------
%stddev %change %stddev
\ | \
7043990 ± 0% +21.2% 8536807 ± 0% vm-scalability.throughput
109.94 ± 1% -16.2% 92.09 ± 0% vm-scalability.time.elapsed_time
3957091 ± 0% +14.9% 4547173 ± 0% vmstat.swap.so
31.46 ± 1% -38.3% 19.42 ± 0% perf-stat.cache-miss-rate%
1.04 ± 1% +22.2% 1.27 ± 0% perf-stat.ipc
9.33 ± 2% -60.7% 3.67 ± 1% perf-profile.calltrace.cycles-pp.add_to_swap.shrink_page_list.shrink_inactive_list.shrink_node_memcg.shrink_node
Changelog:
v7:
- Rebased on latest -mm tree
- Revise get_swap_pages() THP support per Tim's comments
v6:
- Rebased on latest -mm tree (cluster lock, etc).
- Fix a potential uninitialized variable bug in __swap_entry_free()
- Revise the swap read-ahead changes to avoid a potential race
condition between swap off and swap out in theory.
v5:
- Per Hillf's comments, fix a locking bug in error path of
__add_to_swap_cache(). And merge the code to calculate extra_pins
into can_split_huge_page().
v4:
- Per Johannes' comments, simplified swap cgroup array accessing code.
- Per Kirill and Dave Hansen's comments, used HPAGE_PMD_NR instead of
HPAGE_SIZE/PAGE_SIZE.
- Per Anshuman's comments, used HPAGE_PMD_NR instead of 512 in patch
description.
v3:
- Per Andrew's suggestion, used a more systematical way to determine
whether to enable THP swap optimization
- Per Andrew's comments, moved as much as possible code into
#ifdef CONFIG_TRANSPARENT_HUGE_PAGE/#endif or "if (PageTransHuge())"
- Fixed some coding style warning.
v2:
- Original [1/11] sent separately and merged
- Use switch in 10/10 per Hiff's suggestion
Best Regards,
Huang, Ying
From: Huang Ying <[email protected]>
In this patch, the size of the swap cluster is changed to that of the
THP (Transparent Huge Page) on x86_64 architecture (512). This is for
the THP swap support on x86_64. Where one swap cluster will be used to
hold the contents of each THP swapped out. And some information of the
swapped out THP (such as compound map count) will be recorded in the
swap_cluster_info data structure.
For other architectures which want THP swap support,
ARCH_USES_THP_SWAP_CLUSTER need to be selected in the Kconfig file for
the architecture.
In effect, this will enlarge swap cluster size by 2 times on x86_64.
Which may make it harder to find a free cluster when the swap space
becomes fragmented. So that, this may reduce the continuous swap space
allocation and sequential write in theory. The performance test in 0day
shows no regressions caused by this.
Cc: Hugh Dickins <[email protected]>
Cc: Shaohua Li <[email protected]>
Cc: Minchan Kim <[email protected]>
Cc: Rik van Riel <[email protected]>
Suggested-by: Andrew Morton <[email protected]>
Signed-off-by: "Huang, Ying" <[email protected]>
---
arch/x86/Kconfig | 1 +
mm/Kconfig | 13 +++++++++++++
mm/swapfile.c | 4 ++++
3 files changed, 18 insertions(+)
diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index abfc31fb0bee..852d13878793 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -178,6 +178,7 @@ config X86
select USER_STACKTRACE_SUPPORT
select VIRT_TO_BUS
select X86_FEATURE_NAMES if PROC_FS
+ select ARCH_USES_THP_SWAP_CLUSTER if X86_64
config INSTRUCTION_DECODER
def_bool y
diff --git a/mm/Kconfig b/mm/Kconfig
index 9b8fccb969dc..7b708e200c29 100644
--- a/mm/Kconfig
+++ b/mm/Kconfig
@@ -499,6 +499,19 @@ config FRONTSWAP
If unsure, say Y to enable frontswap.
+config ARCH_USES_THP_SWAP_CLUSTER
+ bool
+ default n
+
+config THP_SWAP_CLUSTER
+ bool
+ depends on SWAP && TRANSPARENT_HUGEPAGE && ARCH_USES_THP_SWAP_CLUSTER
+ default y
+ help
+ Use one swap cluster to hold the contents of the THP
+ (Transparent Huge Page) swapped out. The size of the swap
+ cluster will be same as that of THP.
+
config CMA
bool "Contiguous Memory Allocator"
depends on HAVE_MEMBLOCK && MMU
diff --git a/mm/swapfile.c b/mm/swapfile.c
index 53b5881ee0d6..abc401f72a0a 100644
--- a/mm/swapfile.c
+++ b/mm/swapfile.c
@@ -199,7 +199,11 @@ static void discard_swap_cluster(struct swap_info_struct *si,
}
}
+#ifdef CONFIG_THP_SWAP_CLUSTER
+#define SWAPFILE_CLUSTER HPAGE_PMD_NR
+#else
#define SWAPFILE_CLUSTER 256
+#endif
#define LATENCY_LIMIT 256
static inline void cluster_set_flag(struct swap_cluster_info *info,
--
2.11.0
From: Huang Ying <[email protected]>
In this patch, splitting huge page is delayed from almost the first step
of swapping out to after allocating the swap space for the
THP (Transparent Huge Page) and adding the THP into the swap cache.
This will reduce lock acquiring/releasing for the locks used for the
swap cache management.
This is the first step for the THP swap support. The plan is to delay
splitting the THP step by step and avoid splitting the THP finally.
The advantages of the THP swap support include:
- Batch the swap operations for the THP to reduce lock
acquiring/releasing, including allocating/freeing the swap space,
adding/deleting to/from the swap cache, and writing/reading the swap
space, etc. This will help to improve the THP swap performance.
- The THP swap space read/write will be 2M sequential IO. It is
particularly helpful for the swap read, which usually are 4k random
IO. This will help to improve the THP swap performance too.
- It will help the memory fragmentation, especially when the THP is
heavily used by the applications. The 2M continuous pages will be
free up after the THP swapping out.
- It will improve the THP utilization on the system with the swap
turned on. Because the speed for khugepaged to collapse the normal
pages into the THP is quite slow. After the THP is split during the
swapping out, it will take quite long time for the normal pages to
collapse back into the THP after being swapped in. The high THP
utilization helps the efficiency of the page based memory management
too.
There are some concerns regarding THP swap in, mainly because possible
enlarged read/write IO size (for swap in/out) may put more overhead on
the storage device. To deal with that, the THP swap in should be
turned on only when necessary. For example, it can be selected via
"always/never/madvise" logic, to be turned on globally, turned off
globally, or turned on only for VMA with MADV_HUGEPAGE, etc.
With the patchset, the swap out throughput improves 14.9% (from about
3.77GB/s to about 4.34GB/s) in the vm-scalability swap-w-seq test case
with 8 processes. The test is done on a Xeon E5 v3 system. The swap
device used is a RAM simulated PMEM (persistent memory) device. To
test the sequential swapping out, the test case creates 8 processes,
which sequentially allocate and write to the anonymous pages until the
RAM and part of the swap device is used up.
The detailed comparison result is as follow,
base base+patchset
---------------- --------------------------
%stddev %change %stddev
\ | \
7043990 ± 0% +21.2% 8536807 ± 0% vm-scalability.throughput
109.94 ± 1% -16.2% 92.09 ± 0% vm-scalability.time.elapsed_time
3957091 ± 0% +14.9% 4547173 ± 0% vmstat.swap.so
31.46 ± 1% -38.3% 19.42 ± 0% perf-stat.cache-miss-rate%
1.04 ± 1% +22.2% 1.27 ± 0% perf-stat.ipc
9.33 ± 2% -60.7% 3.67 ± 1% perf-profile.calltrace.cycles-pp.add_to_swap.shrink_page_list.shrink_inactive_list.shrink_node_memcg.shrink_node
Signed-off-by: "Huang, Ying" <[email protected]>
---
mm/swap_state.c | 60 ++++++++++++++++++++++++++++++++++++++++++++++++++++++---
1 file changed, 57 insertions(+), 3 deletions(-)
diff --git a/mm/swap_state.c b/mm/swap_state.c
index 504f67d73f67..6d63ff703d39 100644
--- a/mm/swap_state.c
+++ b/mm/swap_state.c
@@ -19,6 +19,7 @@
#include <linux/migrate.h>
#include <linux/vmalloc.h>
#include <linux/swap_slots.h>
+#include <linux/huge_mm.h>
#include <asm/pgtable.h>
@@ -183,12 +184,53 @@ void __delete_from_swap_cache(struct page *page)
ADD_CACHE_INFO(del_total, nr);
}
+#ifdef CONFIG_THP_SWAP_CLUSTER
+int add_to_swap_trans_huge(struct page *page, struct list_head *list)
+{
+ swp_entry_t entry;
+ int ret = 0;
+
+ /* cannot split, which may be needed during swap in, skip it */
+ if (!can_split_huge_page(page, NULL))
+ return -EBUSY;
+ /* fallback to split huge page firstly if no PMD map */
+ if (!compound_mapcount(page))
+ return 0;
+ entry = get_huge_swap_page();
+ if (!entry.val)
+ return 0;
+ if (mem_cgroup_try_charge_swap(page, entry, HPAGE_PMD_NR)) {
+ __swapcache_free(entry, true);
+ return -EOVERFLOW;
+ }
+ ret = add_to_swap_cache(page, entry,
+ __GFP_HIGH | __GFP_NOMEMALLOC|__GFP_NOWARN);
+ /* -ENOMEM radix-tree allocation failure */
+ if (ret) {
+ __swapcache_free(entry, true);
+ return 0;
+ }
+ ret = split_huge_page_to_list(page, list);
+ if (ret) {
+ delete_from_swap_cache(page);
+ return -EBUSY;
+ }
+ return 1;
+}
+#else
+static inline int add_to_swap_trans_huge(struct page *page,
+ struct list_head *list)
+{
+ return 0;
+}
+#endif
+
/**
* add_to_swap - allocate swap space for a page
* @page: page we want to move to swap
*
* Allocate swap space for the page and add the page to the
- * swap cache. Caller needs to hold the page lock.
+ * swap cache. Caller needs to hold the page lock.
*/
int add_to_swap(struct page *page, struct list_head *list)
{
@@ -198,6 +240,18 @@ int add_to_swap(struct page *page, struct list_head *list)
VM_BUG_ON_PAGE(!PageLocked(page), page);
VM_BUG_ON_PAGE(!PageUptodate(page), page);
+ if (unlikely(PageTransHuge(page))) {
+ err = add_to_swap_trans_huge(page, list);
+ switch (err) {
+ case 1:
+ return 1;
+ case 0:
+ /* fallback to split firstly if return 0 */
+ break;
+ default:
+ return 0;
+ }
+ }
entry = get_swap_page();
if (!entry.val)
return 0;
@@ -315,7 +369,7 @@ struct page * lookup_swap_cache(swp_entry_t entry)
page = find_get_page(swap_address_space(entry), swp_offset(entry));
- if (page) {
+ if (page && likely(!PageTransCompound(page))) {
INC_CACHE_INFO(find_success);
if (TestClearPageReadahead(page))
atomic_inc(&swapin_readahead_hits);
@@ -526,7 +580,7 @@ struct page *swapin_readahead(swp_entry_t entry, gfp_t gfp_mask,
gfp_mask, vma, addr);
if (!page)
continue;
- if (offset != entry_offset)
+ if (offset != entry_offset && likely(!PageTransCompound(page)))
SetPageReadahead(page);
put_page(page);
}
--
2.11.0
From: Huang Ying <[email protected]>
__swapcache_free() is added to support to clear the SWAP_HAS_CACHE flag
for the huge page. This will free the specified swap cluster now.
Because now this function will be called only in the error path to free
the swap cluster just allocated. So the corresponding swap_map[i] ==
SWAP_HAS_CACHE, that is, the swap count is 0. This makes the
implementation simpler than that of the ordinary swap entry.
This will be used for delaying splitting THP (Transparent Huge Page)
during swapping out. Where for one THP to swap out, we will allocate a
swap cluster, add the THP into the swap cache, then split the THP. If
anything fails after allocating the swap cluster and before splitting
the THP successfully, the swapcache_free_trans_huge() will be used to
free the swap space allocated.
Cc: Andrea Arcangeli <[email protected]>
Cc: Kirill A. Shutemov <[email protected]>
Cc: Hugh Dickins <[email protected]>
Cc: Shaohua Li <[email protected]>
Cc: Minchan Kim <[email protected]>
Cc: Rik van Riel <[email protected]>
Signed-off-by: "Huang, Ying" <[email protected]>
---
include/linux/swap.h | 9 +++++++--
mm/swapfile.c | 34 ++++++++++++++++++++++++++++++++--
2 files changed, 39 insertions(+), 4 deletions(-)
diff --git a/include/linux/swap.h b/include/linux/swap.h
index e3a7609a8989..2f2a6c0363aa 100644
--- a/include/linux/swap.h
+++ b/include/linux/swap.h
@@ -394,7 +394,7 @@ extern void swap_shmem_alloc(swp_entry_t);
extern int swap_duplicate(swp_entry_t);
extern int swapcache_prepare(swp_entry_t);
extern void swap_free(swp_entry_t);
-extern void swapcache_free(swp_entry_t);
+extern void __swapcache_free(swp_entry_t entry, bool huge);
extern void swapcache_free_entries(swp_entry_t *entries, int n);
extern int free_swap_and_cache(swp_entry_t);
extern int swap_type_of(dev_t, sector_t, struct block_device **);
@@ -456,7 +456,7 @@ static inline void swap_free(swp_entry_t swp)
{
}
-static inline void swapcache_free(swp_entry_t swp)
+static inline void __swapcache_free(swp_entry_t swp, bool huge)
{
}
@@ -544,6 +544,11 @@ static inline swp_entry_t get_huge_swap_page(void)
}
#endif
+static inline void swapcache_free(swp_entry_t entry)
+{
+ __swapcache_free(entry, false);
+}
+
#ifdef CONFIG_MEMCG
static inline int mem_cgroup_swappiness(struct mem_cgroup *memcg)
{
diff --git a/mm/swapfile.c b/mm/swapfile.c
index 382e84541e16..055cfc1be057 100644
--- a/mm/swapfile.c
+++ b/mm/swapfile.c
@@ -855,6 +855,29 @@ static void swap_free_huge_cluster(struct swap_info_struct *si,
_swap_entry_free(si, offset, true);
}
+static void swapcache_free_trans_huge(struct swap_info_struct *si,
+ swp_entry_t entry)
+{
+ unsigned long offset = swp_offset(entry);
+ unsigned long idx = offset / SWAPFILE_CLUSTER;
+ struct swap_cluster_info *ci;
+ unsigned char *map;
+ unsigned int i;
+
+ spin_lock(&si->lock);
+ ci = lock_cluster(si, offset);
+ map = si->swap_map + offset;
+ for (i = 0; i < SWAPFILE_CLUSTER; i++) {
+ VM_BUG_ON(map[i] != SWAP_HAS_CACHE);
+ map[i] = 0;
+ }
+ unlock_cluster(ci);
+ /* Cluster size is same as huge pmd size */
+ mem_cgroup_uncharge_swap(entry, HPAGE_PMD_NR);
+ swap_free_huge_cluster(si, idx);
+ spin_unlock(&si->lock);
+}
+
static int swap_alloc_huge_cluster(struct swap_info_struct *si,
swp_entry_t *slot)
{
@@ -887,6 +910,11 @@ static inline int swap_alloc_huge_cluster(struct swap_info_struct *si,
{
return 0;
}
+
+static inline void swapcache_free_trans_huge(struct swap_info_struct *si,
+ swp_entry_t entry)
+{
+}
#endif
static unsigned long scan_swap_map(struct swap_info_struct *si,
@@ -1157,13 +1185,15 @@ void swap_free(swp_entry_t entry)
/*
* Called after dropping swapcache to decrease refcnt to swap entries.
*/
-void swapcache_free(swp_entry_t entry)
+void __swapcache_free(swp_entry_t entry, bool huge)
{
struct swap_info_struct *p;
p = _swap_info_get(entry);
if (p) {
- if (!__swap_entry_free(p, entry, SWAP_HAS_CACHE))
+ if (unlikely(huge))
+ swapcache_free_trans_huge(p, entry);
+ else if (!__swap_entry_free(p, entry, SWAP_HAS_CACHE))
free_swap_slot(entry);
}
}
--
2.11.0
From: Huang Ying <[email protected]>
The swap cluster allocation/free functions are added based on the
existing swap cluster management mechanism for SSD. These functions
don't work for the rotating hard disks because the existing swap cluster
management mechanism doesn't work for them. The hard disks support may
be added if someone really need it. But that needn't be included in
this patchset.
This will be used for the THP (Transparent Huge Page) swap support.
Where one swap cluster will hold the contents of each THP swapped out.
Cc: Andrea Arcangeli <[email protected]>
Cc: Kirill A. Shutemov <[email protected]>
Cc: Hugh Dickins <[email protected]>
Cc: Shaohua Li <[email protected]>
Cc: Minchan Kim <[email protected]>
Cc: Rik van Riel <[email protected]>
Signed-off-by: "Huang, Ying" <[email protected]>
---
mm/swapfile.c | 217 +++++++++++++++++++++++++++++++++++++++++-----------------
1 file changed, 156 insertions(+), 61 deletions(-)
diff --git a/mm/swapfile.c b/mm/swapfile.c
index 1ef4fc82c0fa..54480acbbeef 100644
--- a/mm/swapfile.c
+++ b/mm/swapfile.c
@@ -378,6 +378,14 @@ static void swap_cluster_schedule_discard(struct swap_info_struct *si,
schedule_work(&si->discard_work);
}
+static void __free_cluster(struct swap_info_struct *si, unsigned long idx)
+{
+ struct swap_cluster_info *ci = si->cluster_info;
+
+ cluster_set_flag(ci + idx, CLUSTER_FLAG_FREE);
+ cluster_list_add_tail(&si->free_clusters, ci, idx);
+}
+
/*
* Doing discard actually. After a cluster discard is finished, the cluster
* will be added to free cluster list. caller should hold si->lock.
@@ -398,10 +406,7 @@ static void swap_do_scheduled_discard(struct swap_info_struct *si)
spin_lock(&si->lock);
ci = lock_cluster(si, idx * SWAPFILE_CLUSTER);
- cluster_set_flag(ci, CLUSTER_FLAG_FREE);
- unlock_cluster(ci);
- cluster_list_add_tail(&si->free_clusters, info, idx);
- ci = lock_cluster(si, idx * SWAPFILE_CLUSTER);
+ __free_cluster(si, idx);
memset(si->swap_map + idx * SWAPFILE_CLUSTER,
0, SWAPFILE_CLUSTER);
unlock_cluster(ci);
@@ -419,6 +424,34 @@ static void swap_discard_work(struct work_struct *work)
spin_unlock(&si->lock);
}
+static void alloc_cluster(struct swap_info_struct *si, unsigned long idx)
+{
+ struct swap_cluster_info *ci = si->cluster_info;
+
+ VM_BUG_ON(cluster_list_first(&si->free_clusters) != idx);
+ cluster_list_del_first(&si->free_clusters, ci);
+ cluster_set_count_flag(ci + idx, 0, 0);
+}
+
+static void free_cluster(struct swap_info_struct *si, unsigned long idx)
+{
+ struct swap_cluster_info *ci = si->cluster_info + idx;
+
+ VM_BUG_ON(cluster_count(ci) != 0);
+ /*
+ * If the swap is discardable, prepare discard the cluster
+ * instead of free it immediately. The cluster will be freed
+ * after discard.
+ */
+ if ((si->flags & (SWP_WRITEOK | SWP_PAGE_DISCARD)) ==
+ (SWP_WRITEOK | SWP_PAGE_DISCARD)) {
+ swap_cluster_schedule_discard(si, idx);
+ return;
+ }
+
+ __free_cluster(si, idx);
+}
+
/*
* The cluster corresponding to page_nr will be used. The cluster will be
* removed from free cluster list and its usage counter will be increased.
@@ -430,11 +463,8 @@ static void inc_cluster_info_page(struct swap_info_struct *p,
if (!cluster_info)
return;
- if (cluster_is_free(&cluster_info[idx])) {
- VM_BUG_ON(cluster_list_first(&p->free_clusters) != idx);
- cluster_list_del_first(&p->free_clusters, cluster_info);
- cluster_set_count_flag(&cluster_info[idx], 0, 0);
- }
+ if (cluster_is_free(&cluster_info[idx]))
+ alloc_cluster(p, idx);
VM_BUG_ON(cluster_count(&cluster_info[idx]) >= SWAPFILE_CLUSTER);
cluster_set_count(&cluster_info[idx],
@@ -458,21 +488,8 @@ static void dec_cluster_info_page(struct swap_info_struct *p,
cluster_set_count(&cluster_info[idx],
cluster_count(&cluster_info[idx]) - 1);
- if (cluster_count(&cluster_info[idx]) == 0) {
- /*
- * If the swap is discardable, prepare discard the cluster
- * instead of free it immediately. The cluster will be freed
- * after discard.
- */
- if ((p->flags & (SWP_WRITEOK | SWP_PAGE_DISCARD)) ==
- (SWP_WRITEOK | SWP_PAGE_DISCARD)) {
- swap_cluster_schedule_discard(p, idx);
- return;
- }
-
- cluster_set_flag(&cluster_info[idx], CLUSTER_FLAG_FREE);
- cluster_list_add_tail(&p->free_clusters, cluster_info, idx);
- }
+ if (cluster_count(&cluster_info[idx]) == 0)
+ free_cluster(p, idx);
}
/*
@@ -562,6 +579,71 @@ static bool scan_swap_map_try_ssd_cluster(struct swap_info_struct *si,
return found_free;
}
+#ifdef CONFIG_THP_SWAP_CLUSTER
+static inline unsigned int huge_cluster_nr_entries(bool huge)
+{
+ return huge ? SWAPFILE_CLUSTER : 1;
+}
+#else
+#define huge_cluster_nr_entries(huge) 1
+#endif
+
+static void _swap_entry_alloc(struct swap_info_struct *si,
+ unsigned long offset, bool huge)
+{
+ unsigned int nr_entries = huge_cluster_nr_entries(huge);
+ unsigned int end = offset + nr_entries - 1;
+
+ if (offset == si->lowest_bit)
+ si->lowest_bit += nr_entries;
+ if (end == si->highest_bit)
+ si->highest_bit -= nr_entries;
+ si->inuse_pages += nr_entries;
+ if (si->inuse_pages == si->pages) {
+ si->lowest_bit = si->max;
+ si->highest_bit = 0;
+ spin_lock(&swap_avail_lock);
+ plist_del(&si->avail_list, &swap_avail_head);
+ spin_unlock(&swap_avail_lock);
+ }
+}
+
+static void _swap_entry_free(struct swap_info_struct *si, unsigned long offset,
+ bool huge)
+{
+ unsigned int nr_entries = huge_cluster_nr_entries(huge);
+ unsigned long end = offset + nr_entries - 1;
+ void (*swap_slot_free_notify)(struct block_device *, unsigned long);
+
+ if (offset < si->lowest_bit)
+ si->lowest_bit = offset;
+ if (end > si->highest_bit) {
+ bool was_full = !si->highest_bit;
+
+ si->highest_bit = end;
+ if (was_full && (si->flags & SWP_WRITEOK)) {
+ spin_lock(&swap_avail_lock);
+ WARN_ON(!plist_node_empty(&si->avail_list));
+ if (plist_node_empty(&si->avail_list))
+ plist_add(&si->avail_list, &swap_avail_head);
+ spin_unlock(&swap_avail_lock);
+ }
+ }
+ atomic_long_add(nr_entries, &nr_swap_pages);
+ si->inuse_pages -= nr_entries;
+ if (si->flags & SWP_BLKDEV)
+ swap_slot_free_notify =
+ si->bdev->bd_disk->fops->swap_slot_free_notify;
+ else
+ swap_slot_free_notify = NULL;
+ while (offset <= end) {
+ frontswap_invalidate_page(si->type, offset);
+ if (swap_slot_free_notify)
+ swap_slot_free_notify(si->bdev, offset);
+ offset++;
+ }
+}
+
static int scan_swap_map_slots(struct swap_info_struct *si,
unsigned char usage, int nr,
swp_entry_t slots[])
@@ -680,18 +762,7 @@ static int scan_swap_map_slots(struct swap_info_struct *si,
inc_cluster_info_page(si, si->cluster_info, offset);
unlock_cluster(ci);
- if (offset == si->lowest_bit)
- si->lowest_bit++;
- if (offset == si->highest_bit)
- si->highest_bit--;
- si->inuse_pages++;
- if (si->inuse_pages == si->pages) {
- si->lowest_bit = si->max;
- si->highest_bit = 0;
- spin_lock(&swap_avail_lock);
- plist_del(&si->avail_list, &swap_avail_head);
- spin_unlock(&swap_avail_lock);
- }
+ _swap_entry_alloc(si, offset, false);
si->cluster_next = offset + 1;
slots[n_ret++] = swp_entry(si->type, offset);
@@ -770,6 +841,54 @@ static int scan_swap_map_slots(struct swap_info_struct *si,
return n_ret;
}
+#ifdef CONFIG_THP_SWAP_CLUSTER
+static void swap_free_huge_cluster(struct swap_info_struct *si,
+ unsigned long idx)
+{
+ struct swap_cluster_info *ci;
+ unsigned long offset = idx * SWAPFILE_CLUSTER;
+
+ ci = lock_cluster(si, offset);
+ cluster_set_count_flag(ci, 0, 0);
+ free_cluster(si, idx);
+ unlock_cluster(ci);
+ _swap_entry_free(si, offset, true);
+}
+
+static int swap_alloc_huge_cluster(struct swap_info_struct *si,
+ swp_entry_t *slot)
+{
+ unsigned long idx;
+ struct swap_cluster_info *ci;
+ unsigned long offset, i;
+ unsigned char *map;
+
+ if (cluster_list_empty(&si->free_clusters))
+ return 0;
+
+ idx = cluster_list_first(&si->free_clusters);
+ offset = idx * SWAPFILE_CLUSTER;
+ ci = lock_cluster(si, offset);
+ alloc_cluster(si, idx);
+ cluster_set_count_flag(ci, SWAPFILE_CLUSTER, 0);
+
+ map = si->swap_map + offset;
+ for (i = 0; i < SWAPFILE_CLUSTER; i++)
+ map[i] = SWAP_HAS_CACHE;
+ unlock_cluster(ci);
+ _swap_entry_alloc(si, offset, true);
+ *slot = swp_entry(si->type, offset);
+
+ return 1;
+}
+#else
+static inline int swap_alloc_huge_cluster(struct swap_info_struct *si,
+ swp_entry_t *slot)
+{
+ return 0;
+}
+#endif
+
static unsigned long scan_swap_map(struct swap_info_struct *si,
unsigned char usage)
{
@@ -1013,31 +1132,7 @@ static void swap_entry_free(struct swap_info_struct *p, swp_entry_t entry)
unlock_cluster(ci);
mem_cgroup_uncharge_swap(entry, 1);
- if (offset < p->lowest_bit)
- p->lowest_bit = offset;
- if (offset > p->highest_bit) {
- bool was_full = !p->highest_bit;
-
- p->highest_bit = offset;
- if (was_full && (p->flags & SWP_WRITEOK)) {
- spin_lock(&swap_avail_lock);
- WARN_ON(!plist_node_empty(&p->avail_list));
- if (plist_node_empty(&p->avail_list))
- plist_add(&p->avail_list,
- &swap_avail_head);
- spin_unlock(&swap_avail_lock);
- }
- }
- atomic_long_inc(&nr_swap_pages);
- p->inuse_pages--;
- frontswap_invalidate_page(p->type, offset);
- if (p->flags & SWP_BLKDEV) {
- struct gendisk *disk = p->bdev->bd_disk;
-
- if (disk->fops->swap_slot_free_notify)
- disk->fops->swap_slot_free_notify(p->bdev,
- offset);
- }
+ _swap_entry_free(p, offset, false);
}
/*
--
2.11.0
From: Huang Ying <[email protected]>
With this patch, a THP (Transparent Huge Page) can be added/deleted
to/from the swap cache as a set of (HPAGE_PMD_NR) sub-pages.
This will be used for the THP (Transparent Huge Page) swap support.
Where one THP may be added/delted to/from the swap cache. This will
batch the swap cache operations to reduce the lock acquire/release times
for the THP swap too.
Cc: Hugh Dickins <[email protected]>
Cc: Shaohua Li <[email protected]>
Cc: Minchan Kim <[email protected]>
Cc: Rik van Riel <[email protected]>
Cc: Andrea Arcangeli <[email protected]>
Cc: Kirill A. Shutemov <[email protected]>
Signed-off-by: "Huang, Ying" <[email protected]>
---
include/linux/page-flags.h | 5 ++--
mm/swap_state.c | 64 ++++++++++++++++++++++++++++++----------------
2 files changed, 45 insertions(+), 24 deletions(-)
diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h
index 6b5818d6de32..f4acd6c4f808 100644
--- a/include/linux/page-flags.h
+++ b/include/linux/page-flags.h
@@ -326,11 +326,12 @@ PAGEFLAG_FALSE(HighMem)
#ifdef CONFIG_SWAP
static __always_inline int PageSwapCache(struct page *page)
{
+ page = compound_head(page);
return PageSwapBacked(page) && test_bit(PG_swapcache, &page->flags);
}
-SETPAGEFLAG(SwapCache, swapcache, PF_NO_COMPOUND)
-CLEARPAGEFLAG(SwapCache, swapcache, PF_NO_COMPOUND)
+SETPAGEFLAG(SwapCache, swapcache, PF_NO_TAIL)
+CLEARPAGEFLAG(SwapCache, swapcache, PF_NO_TAIL)
#else
PAGEFLAG_FALSE(SwapCache)
#endif
diff --git a/mm/swap_state.c b/mm/swap_state.c
index 199a07efc44d..504f67d73f67 100644
--- a/mm/swap_state.c
+++ b/mm/swap_state.c
@@ -38,6 +38,7 @@ struct address_space *swapper_spaces[MAX_SWAPFILES];
static unsigned int nr_swapper_spaces[MAX_SWAPFILES];
#define INC_CACHE_INFO(x) do { swap_cache_info.x++; } while (0)
+#define ADD_CACHE_INFO(x, nr) do { swap_cache_info.x += (nr); } while (0)
static struct {
unsigned long add_total;
@@ -90,39 +91,52 @@ void show_swap_cache_info(void)
*/
int __add_to_swap_cache(struct page *page, swp_entry_t entry)
{
- int error;
+ int error, i, nr = hpage_nr_pages(page);
struct address_space *address_space;
+ struct page *cur_page;
+ swp_entry_t cur_entry;
VM_BUG_ON_PAGE(!PageLocked(page), page);
VM_BUG_ON_PAGE(PageSwapCache(page), page);
VM_BUG_ON_PAGE(!PageSwapBacked(page), page);
- get_page(page);
+ page_ref_add(page, nr);
SetPageSwapCache(page);
- set_page_private(page, entry.val);
address_space = swap_address_space(entry);
+ cur_page = page;
+ cur_entry.val = entry.val;
spin_lock_irq(&address_space->tree_lock);
- error = radix_tree_insert(&address_space->page_tree,
- swp_offset(entry), page);
- if (likely(!error)) {
- address_space->nrpages++;
- __inc_node_page_state(page, NR_FILE_PAGES);
- INC_CACHE_INFO(add_total);
+ for (i = 0; i < nr; i++, cur_page++, cur_entry.val++) {
+ set_page_private(cur_page, cur_entry.val);
+ error = radix_tree_insert(&address_space->page_tree,
+ swp_offset(cur_entry), cur_page);
+ if (unlikely(error))
+ break;
}
- spin_unlock_irq(&address_space->tree_lock);
-
- if (unlikely(error)) {
+ if (likely(!error)) {
+ address_space->nrpages += nr;
+ __mod_node_page_state(page_pgdat(page), NR_FILE_PAGES, nr);
+ ADD_CACHE_INFO(add_total, nr);
+ } else {
/*
* Only the context which have set SWAP_HAS_CACHE flag
* would call add_to_swap_cache().
* So add_to_swap_cache() doesn't returns -EEXIST.
*/
VM_BUG_ON(error == -EEXIST);
- set_page_private(page, 0UL);
+ set_page_private(cur_page, 0UL);
+ while (i--) {
+ cur_page--;
+ cur_entry.val--;
+ radix_tree_delete(&address_space->page_tree,
+ swp_offset(cur_entry));
+ set_page_private(cur_page, 0UL);
+ }
ClearPageSwapCache(page);
- put_page(page);
+ page_ref_sub(page, nr);
}
+ spin_unlock_irq(&address_space->tree_lock);
return error;
}
@@ -132,7 +146,7 @@ int add_to_swap_cache(struct page *page, swp_entry_t entry, gfp_t gfp_mask)
{
int error;
- error = radix_tree_maybe_preload(gfp_mask);
+ error = radix_tree_maybe_preload_order(gfp_mask, compound_order(page));
if (!error) {
error = __add_to_swap_cache(page, entry);
radix_tree_preload_end();
@@ -148,6 +162,7 @@ void __delete_from_swap_cache(struct page *page)
{
swp_entry_t entry;
struct address_space *address_space;
+ int i, nr = hpage_nr_pages(page);
VM_BUG_ON_PAGE(!PageLocked(page), page);
VM_BUG_ON_PAGE(!PageSwapCache(page), page);
@@ -155,12 +170,17 @@ void __delete_from_swap_cache(struct page *page)
entry.val = page_private(page);
address_space = swap_address_space(entry);
- radix_tree_delete(&address_space->page_tree, swp_offset(entry));
- set_page_private(page, 0);
+ for (i = 0; i < nr; i++, entry.val++) {
+ struct page *cur_page = page + i;
+
+ radix_tree_delete(&address_space->page_tree,
+ swp_offset(entry));
+ set_page_private(cur_page, 0);
+ }
ClearPageSwapCache(page);
- address_space->nrpages--;
- __dec_node_page_state(page, NR_FILE_PAGES);
- INC_CACHE_INFO(del_total);
+ address_space->nrpages -= nr;
+ __mod_node_page_state(page_pgdat(page), NR_FILE_PAGES, -nr);
+ ADD_CACHE_INFO(del_total, nr);
}
/**
@@ -237,8 +257,8 @@ void delete_from_swap_cache(struct page *page)
__delete_from_swap_cache(page);
spin_unlock_irq(&address_space->tree_lock);
- swapcache_free(entry);
- put_page(page);
+ __swapcache_free(entry, PageTransHuge(page));
+ page_ref_sub(page, hpage_nr_pages(page));
}
/*
--
2.11.0
From: Huang Ying <[email protected]>
This patch make it possible to charge or uncharge a set of continuous
swap entries in the swap cgroup. The number of swap entries is
specified via an added parameter.
This will be used for the THP (Transparent Huge Page) swap support.
Where a swap cluster backing a THP may be allocated and freed as a
whole. So a set of (HPAGE_PMD_NR) continuous swap entries backing one
THP need to be charged or uncharged together. This will batch the
cgroup operations for the THP swap too.
Cc: Andrea Arcangeli <[email protected]>
Cc: Kirill A. Shutemov <[email protected]>
Cc: Vladimir Davydov <[email protected]>
Cc: Johannes Weiner <[email protected]>
Cc: Michal Hocko <[email protected]>
Cc: Tejun Heo <[email protected]>
Cc: [email protected]
Signed-off-by: "Huang, Ying" <[email protected]>
---
include/linux/swap.h | 12 ++++++----
include/linux/swap_cgroup.h | 6 +++--
mm/memcontrol.c | 57 +++++++++++++++++++++++++--------------------
mm/shmem.c | 2 +-
mm/swap_cgroup.c | 40 +++++++++++++++++++++++--------
mm/swap_state.c | 2 +-
mm/swapfile.c | 2 +-
7 files changed, 77 insertions(+), 44 deletions(-)
diff --git a/include/linux/swap.h b/include/linux/swap.h
index 486494e6b2fc..278e1349a424 100644
--- a/include/linux/swap.h
+++ b/include/linux/swap.h
@@ -550,8 +550,10 @@ static inline int mem_cgroup_swappiness(struct mem_cgroup *mem)
#ifdef CONFIG_MEMCG_SWAP
extern void mem_cgroup_swapout(struct page *page, swp_entry_t entry);
-extern int mem_cgroup_try_charge_swap(struct page *page, swp_entry_t entry);
-extern void mem_cgroup_uncharge_swap(swp_entry_t entry);
+extern int mem_cgroup_try_charge_swap(struct page *page, swp_entry_t entry,
+ unsigned int nr_entries);
+extern void mem_cgroup_uncharge_swap(swp_entry_t entry,
+ unsigned int nr_entries);
extern long mem_cgroup_get_nr_swap_pages(struct mem_cgroup *memcg);
extern bool mem_cgroup_swap_full(struct page *page);
#else
@@ -560,12 +562,14 @@ static inline void mem_cgroup_swapout(struct page *page, swp_entry_t entry)
}
static inline int mem_cgroup_try_charge_swap(struct page *page,
- swp_entry_t entry)
+ swp_entry_t entry,
+ unsigned int nr_entries)
{
return 0;
}
-static inline void mem_cgroup_uncharge_swap(swp_entry_t entry)
+static inline void mem_cgroup_uncharge_swap(swp_entry_t entry,
+ unsigned int nr_entries)
{
}
diff --git a/include/linux/swap_cgroup.h b/include/linux/swap_cgroup.h
index 145306bdc92f..b2b8ec7bda3f 100644
--- a/include/linux/swap_cgroup.h
+++ b/include/linux/swap_cgroup.h
@@ -7,7 +7,8 @@
extern unsigned short swap_cgroup_cmpxchg(swp_entry_t ent,
unsigned short old, unsigned short new);
-extern unsigned short swap_cgroup_record(swp_entry_t ent, unsigned short id);
+extern unsigned short swap_cgroup_record(swp_entry_t ent, unsigned short id,
+ unsigned int nr_ents);
extern unsigned short lookup_swap_cgroup_id(swp_entry_t ent);
extern int swap_cgroup_swapon(int type, unsigned long max_pages);
extern void swap_cgroup_swapoff(int type);
@@ -15,7 +16,8 @@ extern void swap_cgroup_swapoff(int type);
#else
static inline
-unsigned short swap_cgroup_record(swp_entry_t ent, unsigned short id)
+unsigned short swap_cgroup_record(swp_entry_t ent, unsigned short id,
+ unsigned int nr_ents)
{
return 0;
}
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 490d5b4676c1..13ee82fe81c8 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -2393,10 +2393,9 @@ void mem_cgroup_split_huge_fixup(struct page *head)
#ifdef CONFIG_MEMCG_SWAP
static void mem_cgroup_swap_statistics(struct mem_cgroup *memcg,
- bool charge)
+ int nr_entries)
{
- int val = (charge) ? 1 : -1;
- this_cpu_add(memcg->stat->count[MEM_CGROUP_STAT_SWAP], val);
+ this_cpu_add(memcg->stat->count[MEM_CGROUP_STAT_SWAP], nr_entries);
}
/**
@@ -2422,8 +2421,8 @@ static int mem_cgroup_move_swap_account(swp_entry_t entry,
new_id = mem_cgroup_id(to);
if (swap_cgroup_cmpxchg(entry, old_id, new_id) == old_id) {
- mem_cgroup_swap_statistics(from, false);
- mem_cgroup_swap_statistics(to, true);
+ mem_cgroup_swap_statistics(from, -1);
+ mem_cgroup_swap_statistics(to, 1);
return 0;
}
return -EINVAL;
@@ -5451,7 +5450,7 @@ void mem_cgroup_commit_charge(struct page *page, struct mem_cgroup *memcg,
* let's not wait for it. The page already received a
* memory+swap charge, drop the swap entry duplicate.
*/
- mem_cgroup_uncharge_swap(entry);
+ mem_cgroup_uncharge_swap(entry, nr_pages);
}
}
@@ -5879,9 +5878,9 @@ void mem_cgroup_swapout(struct page *page, swp_entry_t entry)
* ancestor for the swap instead and transfer the memory+swap charge.
*/
swap_memcg = mem_cgroup_id_get_online(memcg);
- oldid = swap_cgroup_record(entry, mem_cgroup_id(swap_memcg));
+ oldid = swap_cgroup_record(entry, mem_cgroup_id(swap_memcg), 1);
VM_BUG_ON_PAGE(oldid, page);
- mem_cgroup_swap_statistics(swap_memcg, true);
+ mem_cgroup_swap_statistics(swap_memcg, 1);
page->mem_cgroup = NULL;
@@ -5908,16 +5907,19 @@ void mem_cgroup_swapout(struct page *page, swp_entry_t entry)
css_put(&memcg->css);
}
-/*
- * mem_cgroup_try_charge_swap - try charging a swap entry
+/**
+ * mem_cgroup_try_charge_swap - try charging a set of swap entries
* @page: page being added to swap
- * @entry: swap entry to charge
+ * @entry: the first swap entry to charge
+ * @nr_entries: the number of swap entries to charge
*
- * Try to charge @entry to the memcg that @page belongs to.
+ * Try to charge @nr_entries swap entries starting from @entry to the
+ * memcg that @page belongs to.
*
* Returns 0 on success, -ENOMEM on failure.
*/
-int mem_cgroup_try_charge_swap(struct page *page, swp_entry_t entry)
+int mem_cgroup_try_charge_swap(struct page *page, swp_entry_t entry,
+ unsigned int nr_entries)
{
struct mem_cgroup *memcg;
struct page_counter *counter;
@@ -5935,25 +5937,29 @@ int mem_cgroup_try_charge_swap(struct page *page, swp_entry_t entry)
memcg = mem_cgroup_id_get_online(memcg);
if (!mem_cgroup_is_root(memcg) &&
- !page_counter_try_charge(&memcg->swap, 1, &counter)) {
+ !page_counter_try_charge(&memcg->swap, nr_entries, &counter)) {
mem_cgroup_id_put(memcg);
return -ENOMEM;
}
- oldid = swap_cgroup_record(entry, mem_cgroup_id(memcg));
+ if (nr_entries > 1)
+ mem_cgroup_id_get_many(memcg, nr_entries - 1);
+ oldid = swap_cgroup_record(entry, mem_cgroup_id(memcg), nr_entries);
VM_BUG_ON_PAGE(oldid, page);
- mem_cgroup_swap_statistics(memcg, true);
+ mem_cgroup_swap_statistics(memcg, nr_entries);
return 0;
}
/**
- * mem_cgroup_uncharge_swap - uncharge a swap entry
- * @entry: swap entry to uncharge
+ * mem_cgroup_uncharge_swap - uncharge a set of swap entries
+ * @entry: the first swap entry to uncharge
+ * @nr_entries: the number of swap entries to uncharge
*
- * Drop the swap charge associated with @entry.
+ * Drop the swap charge associated with @nr_entries swap entries
+ * starting from @entry.
*/
-void mem_cgroup_uncharge_swap(swp_entry_t entry)
+void mem_cgroup_uncharge_swap(swp_entry_t entry, unsigned int nr_entries)
{
struct mem_cgroup *memcg;
unsigned short id;
@@ -5961,18 +5967,19 @@ void mem_cgroup_uncharge_swap(swp_entry_t entry)
if (!do_swap_account)
return;
- id = swap_cgroup_record(entry, 0);
+ id = swap_cgroup_record(entry, 0, nr_entries);
rcu_read_lock();
memcg = mem_cgroup_from_id(id);
if (memcg) {
if (!mem_cgroup_is_root(memcg)) {
if (cgroup_subsys_on_dfl(memory_cgrp_subsys))
- page_counter_uncharge(&memcg->swap, 1);
+ page_counter_uncharge(&memcg->swap, nr_entries);
else
- page_counter_uncharge(&memcg->memsw, 1);
+ page_counter_uncharge(&memcg->memsw,
+ nr_entries);
}
- mem_cgroup_swap_statistics(memcg, false);
- mem_cgroup_id_put(memcg);
+ mem_cgroup_swap_statistics(memcg, -nr_entries);
+ mem_cgroup_id_put_many(memcg, nr_entries);
}
rcu_read_unlock();
}
diff --git a/mm/shmem.c b/mm/shmem.c
index e67d6ba4e98e..effe07ef5c26 100644
--- a/mm/shmem.c
+++ b/mm/shmem.c
@@ -1294,7 +1294,7 @@ static int shmem_writepage(struct page *page, struct writeback_control *wbc)
if (!swap.val)
goto redirty;
- if (mem_cgroup_try_charge_swap(page, swap))
+ if (mem_cgroup_try_charge_swap(page, swap, 1))
goto free_swap;
/*
diff --git a/mm/swap_cgroup.c b/mm/swap_cgroup.c
index 310ac0b8f974..8cee2d125815 100644
--- a/mm/swap_cgroup.c
+++ b/mm/swap_cgroup.c
@@ -58,21 +58,27 @@ static int swap_cgroup_prepare(int type)
return -ENOMEM;
}
+static struct swap_cgroup *__lookup_swap_cgroup(struct swap_cgroup_ctrl *ctrl,
+ pgoff_t offset)
+{
+ struct page *mappage;
+ struct swap_cgroup *sc;
+
+ mappage = ctrl->map[offset / SC_PER_PAGE];
+ sc = page_address(mappage);
+ return sc + offset % SC_PER_PAGE;
+}
+
static struct swap_cgroup *lookup_swap_cgroup(swp_entry_t ent,
struct swap_cgroup_ctrl **ctrlp)
{
pgoff_t offset = swp_offset(ent);
struct swap_cgroup_ctrl *ctrl;
- struct page *mappage;
- struct swap_cgroup *sc;
ctrl = &swap_cgroup_ctrl[swp_type(ent)];
if (ctrlp)
*ctrlp = ctrl;
-
- mappage = ctrl->map[offset / SC_PER_PAGE];
- sc = page_address(mappage);
- return sc + offset % SC_PER_PAGE;
+ return __lookup_swap_cgroup(ctrl, offset);
}
/**
@@ -105,25 +111,39 @@ unsigned short swap_cgroup_cmpxchg(swp_entry_t ent,
}
/**
- * swap_cgroup_record - record mem_cgroup for this swp_entry.
- * @ent: swap entry to be recorded into
+ * swap_cgroup_record - record mem_cgroup for a set of swap entries
+ * @ent: the first swap entry to be recorded into
* @id: mem_cgroup to be recorded
+ * @nr_ents: number of swap entries to be recorded
*
* Returns old value at success, 0 at failure.
* (Of course, old value can be 0.)
*/
-unsigned short swap_cgroup_record(swp_entry_t ent, unsigned short id)
+unsigned short swap_cgroup_record(swp_entry_t ent, unsigned short id,
+ unsigned int nr_ents)
{
struct swap_cgroup_ctrl *ctrl;
struct swap_cgroup *sc;
unsigned short old;
unsigned long flags;
+ pgoff_t offset = swp_offset(ent);
+ pgoff_t end = offset + nr_ents;
sc = lookup_swap_cgroup(ent, &ctrl);
spin_lock_irqsave(&ctrl->lock, flags);
old = sc->id;
- sc->id = id;
+ for (;;) {
+ VM_BUG_ON(sc->id != old);
+ sc->id = id;
+ offset++;
+ if (offset == end)
+ break;
+ if (offset % SC_PER_PAGE)
+ sc++;
+ else
+ sc = __lookup_swap_cgroup(ctrl, offset);
+ }
spin_unlock_irqrestore(&ctrl->lock, flags);
return old;
diff --git a/mm/swap_state.c b/mm/swap_state.c
index 7bfb9bd1ca21..199a07efc44d 100644
--- a/mm/swap_state.c
+++ b/mm/swap_state.c
@@ -182,7 +182,7 @@ int add_to_swap(struct page *page, struct list_head *list)
if (!entry.val)
return 0;
- if (mem_cgroup_try_charge_swap(page, entry)) {
+ if (mem_cgroup_try_charge_swap(page, entry, 1)) {
swapcache_free(entry);
return 0;
}
diff --git a/mm/swapfile.c b/mm/swapfile.c
index abc401f72a0a..1ef4fc82c0fa 100644
--- a/mm/swapfile.c
+++ b/mm/swapfile.c
@@ -1012,7 +1012,7 @@ static void swap_entry_free(struct swap_info_struct *p, swp_entry_t entry)
dec_cluster_info_page(p, p->cluster_info, offset);
unlock_cluster(ci);
- mem_cgroup_uncharge_swap(entry);
+ mem_cgroup_uncharge_swap(entry, 1);
if (offset < p->lowest_bit)
p->lowest_bit = offset;
if (offset > p->highest_bit) {
--
2.11.0
From: Huang Ying <[email protected]>
Separates checking whether we can split the huge page from
split_huge_page_to_list() into a function. This will help to check that
before splitting the THP (Transparent Huge Page) really.
This will be used for delaying splitting THP during swapping out. Where
for a THP, we will allocate a swap cluster, add the THP into the swap
cache, then split the THP. To avoid the unnecessary operations for the
un-splittable THP, we will check that firstly.
There is no functionality change in this patch.
Cc: Andrea Arcangeli <[email protected]>
Cc: Ebru Akagunduz <[email protected]>
Signed-off-by: "Huang, Ying" <[email protected]>
Acked-by: Kirill A. Shutemov <[email protected]>
---
include/linux/huge_mm.h | 7 +++++++
mm/huge_memory.c | 17 ++++++++++++++---
2 files changed, 21 insertions(+), 3 deletions(-)
diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h
index a3762d49ba39..d3b3e8fcc717 100644
--- a/include/linux/huge_mm.h
+++ b/include/linux/huge_mm.h
@@ -113,6 +113,7 @@ extern unsigned long thp_get_unmapped_area(struct file *filp,
extern void prep_transhuge_page(struct page *page);
extern void free_transhuge_page(struct page *page);
+bool can_split_huge_page(struct page *page, int *pextra_pins);
int split_huge_page_to_list(struct page *page, struct list_head *list);
static inline int split_huge_page(struct page *page)
{
@@ -231,6 +232,12 @@ static inline void prep_transhuge_page(struct page *page) {}
#define thp_get_unmapped_area NULL
+static inline bool
+can_split_huge_page(struct page *page, int *pextra_pins)
+{
+ BUILD_BUG();
+ return false;
+}
static inline int
split_huge_page_to_list(struct page *page, struct list_head *list)
{
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index d14dd961f626..08ccf0cebe8f 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -2367,6 +2367,19 @@ int page_trans_huge_mapcount(struct page *page, int *total_mapcount)
return ret;
}
+/* Racy check whether the huge page can be split */
+bool can_split_huge_page(struct page *page, int *pextra_pins)
+{
+ int extra_pins = 0;
+
+ /* Additional pins from radix tree */
+ if (!PageAnon(page))
+ extra_pins = HPAGE_PMD_NR;
+ if (pextra_pins)
+ *pextra_pins = extra_pins;
+ return total_mapcount(page) == page_count(page) - extra_pins - 1;
+}
+
/*
* This function splits huge page into normal pages. @page can point to any
* subpage of huge page to split. Split doesn't change the position of @page.
@@ -2426,8 +2439,6 @@ int split_huge_page_to_list(struct page *page, struct list_head *list)
goto out;
}
- /* Addidional pins from radix tree */
- extra_pins = HPAGE_PMD_NR;
anon_vma = NULL;
i_mmap_lock_read(mapping);
}
@@ -2436,7 +2447,7 @@ int split_huge_page_to_list(struct page *page, struct list_head *list)
* Racy check if we can split the page, before freeze_page() will
* split PMDs
*/
- if (total_mapcount(head) != page_count(head) - extra_pins - 1) {
+ if (!can_split_huge_page(head, &extra_pins)) {
ret = -EBUSY;
goto out_unlock;
}
--
2.11.0
From: Huang Ying <[email protected]>
This patch enhanced the split_huge_page_to_list() to work properly for
the THP (Transparent Huge Page) in the swap cache during swapping out.
This is used for delaying splitting the THP during swapping out. Where
for a THP to be swapped out, we will allocate a swap cluster, add the
THP into the swap cache, then split the THP. The page lock will be held
during this process. So in the code path other than swapping out, if
the THP need to be split, the PageSwapCache(THP) will be always false.
Cc: Andrea Arcangeli <[email protected]>
Cc: Ebru Akagunduz <[email protected]>
Signed-off-by: "Huang, Ying" <[email protected]>
Acked-by: Kirill A. Shutemov <[email protected]>
---
mm/huge_memory.c | 16 +++++++++++-----
1 file changed, 11 insertions(+), 5 deletions(-)
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 08ccf0cebe8f..459c7d5cdeb3 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -2185,7 +2185,7 @@ static void __split_huge_page_tail(struct page *head, int tail,
* atomic_set() here would be safe on all archs (and not only on x86),
* it's safer to use atomic_inc()/atomic_add().
*/
- if (PageAnon(head)) {
+ if (PageAnon(head) && !PageSwapCache(head)) {
page_ref_inc(page_tail);
} else {
/* Additional pin to radix tree */
@@ -2196,6 +2196,7 @@ static void __split_huge_page_tail(struct page *head, int tail,
page_tail->flags |= (head->flags &
((1L << PG_referenced) |
(1L << PG_swapbacked) |
+ (1L << PG_swapcache) |
(1L << PG_mlocked) |
(1L << PG_uptodate) |
(1L << PG_active) |
@@ -2258,7 +2259,11 @@ static void __split_huge_page(struct page *page, struct list_head *list,
ClearPageCompound(head);
/* See comment in __split_huge_page_tail() */
if (PageAnon(head)) {
- page_ref_inc(head);
+ /* Additional pin to radix tree of swap cache */
+ if (PageSwapCache(head))
+ page_ref_add(head, 2);
+ else
+ page_ref_inc(head);
} else {
/* Additional pin to radix tree */
page_ref_add(head, 2);
@@ -2370,10 +2375,12 @@ int page_trans_huge_mapcount(struct page *page, int *total_mapcount)
/* Racy check whether the huge page can be split */
bool can_split_huge_page(struct page *page, int *pextra_pins)
{
- int extra_pins = 0;
+ int extra_pins;
/* Additional pins from radix tree */
- if (!PageAnon(page))
+ if (PageAnon(page))
+ extra_pins = PageSwapCache(page) ? HPAGE_PMD_NR : 0;
+ else
extra_pins = HPAGE_PMD_NR;
if (pextra_pins)
*pextra_pins = extra_pins;
@@ -2427,7 +2434,6 @@ int split_huge_page_to_list(struct page *page, struct list_head *list)
ret = -EBUSY;
goto out;
}
- extra_pins = 0;
mapping = NULL;
anon_vma_lock_write(anon_vma);
} else {
--
2.11.0
From: Huang Ying <[email protected]>
A variation of get_swap_page(), get_huge_swap_page(), is added to
allocate a swap cluster (HPAGE_PMD_NR swap slots) based on the swap
cluster allocation function. A fair simple algorithm is used, that is,
only the first swap device in priority list will be tried to allocate
the swap cluster. The function will fail if the trying is not
successful, and the caller will fallback to allocate a single swap slot
instead. This works good enough for normal cases.
This will be used for the THP (Transparent Huge Page) swap support.
Where get_huge_swap_page() will be used to allocate one swap cluster for
each THP swapped out.
Because of the algorithm adopted, if the difference of the number of the
free swap clusters among multiple swap devices is significant, it is
possible that some THPs are split earlier than necessary. For example,
this could be caused by big size difference among multiple swap devices.
Cc: Andrea Arcangeli <[email protected]>
Cc: Kirill A. Shutemov <[email protected]>
Cc: Hugh Dickins <[email protected]>
Cc: Shaohua Li <[email protected]>
Cc: Minchan Kim <[email protected]>
Cc: Rik van Riel <[email protected]>
Signed-off-by: "Huang, Ying" <[email protected]>
---
include/linux/swap.h | 19 ++++++++++++++++++-
mm/swap_slots.c | 5 +++--
mm/swapfile.c | 18 +++++++++++-------
3 files changed, 32 insertions(+), 10 deletions(-)
diff --git a/include/linux/swap.h b/include/linux/swap.h
index 278e1349a424..e3a7609a8989 100644
--- a/include/linux/swap.h
+++ b/include/linux/swap.h
@@ -388,7 +388,7 @@ static inline long get_nr_swap_pages(void)
extern void si_swapinfo(struct sysinfo *);
extern swp_entry_t get_swap_page(void);
extern swp_entry_t get_swap_page_of_type(int);
-extern int get_swap_pages(int n, swp_entry_t swp_entries[]);
+extern int get_swap_pages(int n, swp_entry_t swp_entries[], bool huge);
extern int add_swap_count_continuation(swp_entry_t, gfp_t);
extern void swap_shmem_alloc(swp_entry_t);
extern int swap_duplicate(swp_entry_t);
@@ -527,6 +527,23 @@ static inline swp_entry_t get_swap_page(void)
#endif /* CONFIG_SWAP */
+#ifdef CONFIG_THP_SWAP_CLUSTER
+static inline swp_entry_t get_huge_swap_page(void)
+{
+ swp_entry_t entry;
+
+ if (get_swap_pages(1, &entry, true))
+ return entry;
+ else
+ return (swp_entry_t) {0};
+}
+#else
+static inline swp_entry_t get_huge_swap_page(void)
+{
+ return (swp_entry_t) {0};
+}
+#endif
+
#ifdef CONFIG_MEMCG
static inline int mem_cgroup_swappiness(struct mem_cgroup *memcg)
{
diff --git a/mm/swap_slots.c b/mm/swap_slots.c
index 9b5bc86f96ad..075bb39e03c5 100644
--- a/mm/swap_slots.c
+++ b/mm/swap_slots.c
@@ -258,7 +258,8 @@ static int refill_swap_slots_cache(struct swap_slots_cache *cache)
cache->cur = 0;
if (swap_slot_cache_active)
- cache->nr = get_swap_pages(SWAP_SLOTS_CACHE_SIZE, cache->slots);
+ cache->nr = get_swap_pages(SWAP_SLOTS_CACHE_SIZE, cache->slots,
+ false);
return cache->nr;
}
@@ -334,7 +335,7 @@ swp_entry_t get_swap_page(void)
return entry;
}
- get_swap_pages(1, &entry);
+ get_swap_pages(1, &entry, false);
return entry;
}
diff --git a/mm/swapfile.c b/mm/swapfile.c
index 54480acbbeef..382e84541e16 100644
--- a/mm/swapfile.c
+++ b/mm/swapfile.c
@@ -904,13 +904,14 @@ static unsigned long scan_swap_map(struct swap_info_struct *si,
}
-int get_swap_pages(int n_goal, swp_entry_t swp_entries[])
+int get_swap_pages(int n_goal, swp_entry_t swp_entries[], bool huge)
{
struct swap_info_struct *si, *next;
long avail_pgs;
int n_ret = 0;
+ int nr_pages = huge_cluster_nr_entries(huge);
- avail_pgs = atomic_long_read(&nr_swap_pages);
+ avail_pgs = atomic_long_read(&nr_swap_pages) / nr_pages;
if (avail_pgs <= 0)
goto noswap;
@@ -920,7 +921,7 @@ int get_swap_pages(int n_goal, swp_entry_t swp_entries[])
if (n_goal > avail_pgs)
n_goal = avail_pgs;
- atomic_long_sub(n_goal, &nr_swap_pages);
+ atomic_long_sub(n_goal * nr_pages, &nr_swap_pages);
spin_lock(&swap_avail_lock);
@@ -946,10 +947,13 @@ int get_swap_pages(int n_goal, swp_entry_t swp_entries[])
spin_unlock(&si->lock);
goto nextsi;
}
- n_ret = scan_swap_map_slots(si, SWAP_HAS_CACHE,
- n_goal, swp_entries);
+ if (likely(!huge))
+ n_ret = scan_swap_map_slots(si, SWAP_HAS_CACHE,
+ n_goal, swp_entries);
+ else
+ n_ret = swap_alloc_huge_cluster(si, swp_entries);
spin_unlock(&si->lock);
- if (n_ret)
+ if (n_ret || unlikely(huge))
goto check_out;
pr_debug("scan_swap_map of si %d failed to find offset\n",
si->type);
@@ -975,7 +979,7 @@ int get_swap_pages(int n_goal, swp_entry_t swp_entries[])
check_out:
if (n_ret < n_goal)
- atomic_long_add((long) (n_goal-n_ret), &nr_swap_pages);
+ atomic_long_add((long)(n_goal-n_ret) * nr_pages, &nr_swap_pages);
noswap:
return n_ret;
}
--
2.11.0
On Tue, 28 Mar 2017 13:32:00 +0800 "Huang, Ying" <[email protected]> wrote:
> Hi, Andrew, could you help me to check whether the overall design is
> reasonable?
>
> Hi, Hugh, Shaohua, Minchan and Rik, could you help me to review the
> swap part of the patchset? Especially [1/9], [3/9], [4/9], [5/9],
> [6/9], [9/9].
>
> Hi, Andrea could you help me to review the THP part of the patchset?
> Especially [2/9], [7/9] and [8/9].
>
> Hi, Johannes, Michal and Vladimir, I am not very confident about the
> memory cgroup part, especially [2/9]. Could you help me to review it?
>
> And for all, Any comment is welcome!
>
>
> Recently, the performance of the storage devices improved so fast that
> we cannot saturate the disk bandwidth with single logical CPU when do
> page swap out even on a high-end server machine. Because the
> performance of the storage device improved faster than that of single
> logical CPU. And it seems that the trend will not change in the near
> future. On the other hand, the THP becomes more and more popular
> because of increased memory size. So it becomes necessary to optimize
> THP swap performance.
I'll merge this patchset for testing purposes, but I don't believe that
it has yet had sufficient review. And thanks for drawing our attention
to those parts where you believe close review is needed - that helps.
On Tue, Mar 28, 2017 at 01:32:01PM +0800, Huang, Ying wrote:
> From: Huang Ying <[email protected]>
>
> In this patch, the size of the swap cluster is changed to that of the
> THP (Transparent Huge Page) on x86_64 architecture (512). This is for
> the THP swap support on x86_64. Where one swap cluster will be used to
> hold the contents of each THP swapped out. And some information of the
> swapped out THP (such as compound map count) will be recorded in the
> swap_cluster_info data structure.
>
> For other architectures which want THP swap support,
> ARCH_USES_THP_SWAP_CLUSTER need to be selected in the Kconfig file for
> the architecture.
Intreseting case could be architecture with HPAGE_PMD_NR < 256.
Can current code pack more than one THP per claster.
If not we need to have BUILG_BUG_ON() to catch attempt of such enabling.
--
Kirill A. Shutemov
"Kirill A. Shutemov" <[email protected]> writes:
> On Tue, Mar 28, 2017 at 01:32:01PM +0800, Huang, Ying wrote:
>> From: Huang Ying <[email protected]>
>>
>> In this patch, the size of the swap cluster is changed to that of the
>> THP (Transparent Huge Page) on x86_64 architecture (512). This is for
>> the THP swap support on x86_64. Where one swap cluster will be used to
>> hold the contents of each THP swapped out. And some information of the
>> swapped out THP (such as compound map count) will be recorded in the
>> swap_cluster_info data structure.
>>
>> For other architectures which want THP swap support,
>> ARCH_USES_THP_SWAP_CLUSTER need to be selected in the Kconfig file for
>> the architecture.
>
> Intreseting case could be architecture with HPAGE_PMD_NR < 256.
> Can current code pack more than one THP per claster.
No. Only one THP for each swap cluster is supported. But in current
implementation, if HPAGE_PMD_NR < 256, the swap cluster will be < 256
too. The size of swap cluster will be exact same as HPAGE_PMD_NR.
Best Regards,
Huang, Ying
> If not we need to have BUILG_BUG_ON() to catch attempt of such enabling.
Hi, Andrew,
Andrew Morton <[email protected]> writes:
> On Tue, 28 Mar 2017 13:32:00 +0800 "Huang, Ying" <[email protected]> wrote:
>
>> Hi, Andrew, could you help me to check whether the overall design is
>> reasonable?
>>
>> Hi, Hugh, Shaohua, Minchan and Rik, could you help me to review the
>> swap part of the patchset? Especially [1/9], [3/9], [4/9], [5/9],
>> [6/9], [9/9].
>>
>> Hi, Andrea could you help me to review the THP part of the patchset?
>> Especially [2/9], [7/9] and [8/9].
>>
>> Hi, Johannes, Michal and Vladimir, I am not very confident about the
>> memory cgroup part, especially [2/9]. Could you help me to review it?
>>
>> And for all, Any comment is welcome!
>>
>>
>> Recently, the performance of the storage devices improved so fast that
>> we cannot saturate the disk bandwidth with single logical CPU when do
>> page swap out even on a high-end server machine. Because the
>> performance of the storage device improved faster than that of single
>> logical CPU. And it seems that the trend will not change in the near
>> future. On the other hand, the THP becomes more and more popular
>> because of increased memory size. So it becomes necessary to optimize
>> THP swap performance.
>
> I'll merge this patchset for testing purposes, but I don't believe that
> it has yet had sufficient review. And thanks for drawing our attention
> to those parts where you believe close review is needed - that helps.
Thanks a lot for your help! I believe the patchset will be better
tested in -mm tree. And hope people will have time to review it more
closely.
Best Regards,
Huang, Ying
On Tue, Mar 28, 2017 at 01:32:01PM +0800, Huang, Ying wrote:
> @@ -499,6 +499,19 @@ config FRONTSWAP
>
> If unsure, say Y to enable frontswap.
>
> +config ARCH_USES_THP_SWAP_CLUSTER
> + bool
> + default n
This is fine.
> +config THP_SWAP_CLUSTER
> + bool
> + depends on SWAP && TRANSPARENT_HUGEPAGE && ARCH_USES_THP_SWAP_CLUSTER
> + default y
> + help
> + Use one swap cluster to hold the contents of the THP
> + (Transparent Huge Page) swapped out. The size of the swap
> + cluster will be same as that of THP.
But this is a super weird thing to ask the user. How would they know
what to say, if we don't know? I don't think this should be a config
knob at all. Merge the two config items into a simple
config THP_SWAP_CLUSTER
bool
default n
and let the archs with reasonable THP sizes select it.
On Tue, Mar 28, 2017 at 01:32:02PM +0800, Huang, Ying wrote:
> @@ -5908,16 +5907,19 @@ void mem_cgroup_swapout(struct page *page, swp_entry_t entry)
> css_put(&memcg->css);
> }
>
> -/*
> - * mem_cgroup_try_charge_swap - try charging a swap entry
> +/**
> + * mem_cgroup_try_charge_swap - try charging a set of swap entries
> * @page: page being added to swap
> - * @entry: swap entry to charge
> + * @entry: the first swap entry to charge
> + * @nr_entries: the number of swap entries to charge
> *
> - * Try to charge @entry to the memcg that @page belongs to.
> + * Try to charge @nr_entries swap entries starting from @entry to the
> + * memcg that @page belongs to.
> *
> * Returns 0 on success, -ENOMEM on failure.
> */
> -int mem_cgroup_try_charge_swap(struct page *page, swp_entry_t entry)
> +int mem_cgroup_try_charge_swap(struct page *page, swp_entry_t entry,
> + unsigned int nr_entries)
I've pointed this out before, but there doesn't seem to be a reason to
pass @nr_entries when we have the struct page. Why can't this function
just check PageTransHuge() by itself?
On Tue, Mar 28, 2017 at 01:32:04PM +0800, Huang, Ying wrote:
> @@ -527,6 +527,23 @@ static inline swp_entry_t get_swap_page(void)
>
> #endif /* CONFIG_SWAP */
>
> +#ifdef CONFIG_THP_SWAP_CLUSTER
> +static inline swp_entry_t get_huge_swap_page(void)
> +{
> + swp_entry_t entry;
> +
> + if (get_swap_pages(1, &entry, true))
> + return entry;
> + else
> + return (swp_entry_t) {0};
> +}
> +#else
> +static inline swp_entry_t get_huge_swap_page(void)
> +{
> + return (swp_entry_t) {0};
> +}
> +#endif
Your introducing a function without a user, making it very hard to
judge whether the API is well-designed for the callers or not.
I pointed this out as a systemic problem with this patch series in v3,
along with other stuff, but with the way this series is structured I'm
having a hard time seeing whether you implemented my other feedback or
whether your counter arguments to them are justified.
I cannot review and ack these patches this way.
On Tue, Mar 28, 2017 at 01:32:09PM +0800, Huang, Ying wrote:
> @@ -183,12 +184,53 @@ void __delete_from_swap_cache(struct page *page)
> ADD_CACHE_INFO(del_total, nr);
> }
>
> +#ifdef CONFIG_THP_SWAP_CLUSTER
> +int add_to_swap_trans_huge(struct page *page, struct list_head *list)
> +{
> + swp_entry_t entry;
> + int ret = 0;
> +
> + /* cannot split, which may be needed during swap in, skip it */
> + if (!can_split_huge_page(page, NULL))
> + return -EBUSY;
> + /* fallback to split huge page firstly if no PMD map */
> + if (!compound_mapcount(page))
> + return 0;
> + entry = get_huge_swap_page();
> + if (!entry.val)
> + return 0;
> + if (mem_cgroup_try_charge_swap(page, entry, HPAGE_PMD_NR)) {
> + __swapcache_free(entry, true);
> + return -EOVERFLOW;
> + }
> + ret = add_to_swap_cache(page, entry,
> + __GFP_HIGH | __GFP_NOMEMALLOC|__GFP_NOWARN);
> + /* -ENOMEM radix-tree allocation failure */
> + if (ret) {
> + __swapcache_free(entry, true);
> + return 0;
> + }
> + ret = split_huge_page_to_list(page, list);
> + if (ret) {
> + delete_from_swap_cache(page);
> + return -EBUSY;
> + }
> + return 1;
> +}
> +#else
> +static inline int add_to_swap_trans_huge(struct page *page,
> + struct list_head *list)
> +{
> + return 0;
> +}
> +#endif
> +
> /**
> * add_to_swap - allocate swap space for a page
> * @page: page we want to move to swap
> *
> * Allocate swap space for the page and add the page to the
> - * swap cache. Caller needs to hold the page lock.
> + * swap cache. Caller needs to hold the page lock.
> */
> int add_to_swap(struct page *page, struct list_head *list)
> {
> @@ -198,6 +240,18 @@ int add_to_swap(struct page *page, struct list_head *list)
> VM_BUG_ON_PAGE(!PageLocked(page), page);
> VM_BUG_ON_PAGE(!PageUptodate(page), page);
>
> + if (unlikely(PageTransHuge(page))) {
> + err = add_to_swap_trans_huge(page, list);
> + switch (err) {
> + case 1:
> + return 1;
> + case 0:
> + /* fallback to split firstly if return 0 */
> + break;
> + default:
> + return 0;
> + }
> + }
> entry = get_swap_page();
> if (!entry.val)
> return 0;
add_to_swap_trans_huge() is too close a copy of add_to_swap(), which
makes the code error prone for future modifications to the swap slot
allocation protocol.
This should read:
retry:
entry = get_swap_page(page);
if (!entry.val) {
if (PageTransHuge(page)) {
split_huge_page_to_list(page, list);
goto retry;
}
return 0;
}
And get_swap_page(), mem_cgroup_try_charge_swap() etc. should all
check PageTransHuge() instead of having extra parameters or separate
code paths for the huge page case.
In general, don't try to tack this feature onto the side of the
VM. Because right now, this looks a bit like the hugetlb code, with
one big branch in the beginning that opens up an alternate
reality. Instead, these functions should handle THP all the way down
the stack, and without passing down redundant information.
Johannes Weiner <[email protected]> writes:
> On Tue, Mar 28, 2017 at 01:32:01PM +0800, Huang, Ying wrote:
>> @@ -499,6 +499,19 @@ config FRONTSWAP
>>
>> If unsure, say Y to enable frontswap.
>>
>> +config ARCH_USES_THP_SWAP_CLUSTER
>> + bool
>> + default n
>
> This is fine.
>
>> +config THP_SWAP_CLUSTER
>> + bool
>> + depends on SWAP && TRANSPARENT_HUGEPAGE && ARCH_USES_THP_SWAP_CLUSTER
>> + default y
>> + help
>> + Use one swap cluster to hold the contents of the THP
>> + (Transparent Huge Page) swapped out. The size of the swap
>> + cluster will be same as that of THP.
>
> But this is a super weird thing to ask the user. How would they know
> what to say, if we don't know? I don't think this should be a config
> knob at all. Merge the two config items into a simple
The user will not see this, because there is no string after "bool" to
let user to select it. The help here is for document only, so that
architecture developers could know what this is for.
> config THP_SWAP_CLUSTER
> bool
> default n
>
> and let the archs with reasonable THP sizes select it.
This will have same effect as the original solution except the document
is removed.
Best Regards,
Huang, Ying
Johannes Weiner <[email protected]> writes:
> On Tue, Mar 28, 2017 at 01:32:02PM +0800, Huang, Ying wrote:
>> @@ -5908,16 +5907,19 @@ void mem_cgroup_swapout(struct page *page, swp_entry_t entry)
>> css_put(&memcg->css);
>> }
>>
>> -/*
>> - * mem_cgroup_try_charge_swap - try charging a swap entry
>> +/**
>> + * mem_cgroup_try_charge_swap - try charging a set of swap entries
>> * @page: page being added to swap
>> - * @entry: swap entry to charge
>> + * @entry: the first swap entry to charge
>> + * @nr_entries: the number of swap entries to charge
>> *
>> - * Try to charge @entry to the memcg that @page belongs to.
>> + * Try to charge @nr_entries swap entries starting from @entry to the
>> + * memcg that @page belongs to.
>> *
>> * Returns 0 on success, -ENOMEM on failure.
>> */
>> -int mem_cgroup_try_charge_swap(struct page *page, swp_entry_t entry)
>> +int mem_cgroup_try_charge_swap(struct page *page, swp_entry_t entry,
>> + unsigned int nr_entries)
>
> I've pointed this out before,
Yes. And I have replied to your original comments too :-)
> but there doesn't seem to be a reason to
> pass @nr_entries when we have the struct page. Why can't this function
> just check PageTransHuge() by itself?
Because sometimes we need to charge one swap entry for a THP. Please
take a look at the original add_to_swap() implementation. For a THP,
one swap entry will be allocated and charged to the mem cgroup before
the THP is split. And I think it is not easy to change this, because we
don't want to split THP if the mem cgroup for swap exceeds its limit.
Best Regards,
Huang, Ying
Johannes Weiner <[email protected]> writes:
> On Tue, Mar 28, 2017 at 01:32:09PM +0800, Huang, Ying wrote:
>> @@ -183,12 +184,53 @@ void __delete_from_swap_cache(struct page *page)
>> ADD_CACHE_INFO(del_total, nr);
>> }
>>
>> +#ifdef CONFIG_THP_SWAP_CLUSTER
>> +int add_to_swap_trans_huge(struct page *page, struct list_head *list)
>> +{
>> + swp_entry_t entry;
>> + int ret = 0;
>> +
>> + /* cannot split, which may be needed during swap in, skip it */
>> + if (!can_split_huge_page(page, NULL))
>> + return -EBUSY;
>> + /* fallback to split huge page firstly if no PMD map */
>> + if (!compound_mapcount(page))
>> + return 0;
>> + entry = get_huge_swap_page();
>> + if (!entry.val)
>> + return 0;
>> + if (mem_cgroup_try_charge_swap(page, entry, HPAGE_PMD_NR)) {
>> + __swapcache_free(entry, true);
>> + return -EOVERFLOW;
>> + }
>> + ret = add_to_swap_cache(page, entry,
>> + __GFP_HIGH | __GFP_NOMEMALLOC|__GFP_NOWARN);
>> + /* -ENOMEM radix-tree allocation failure */
>> + if (ret) {
>> + __swapcache_free(entry, true);
>> + return 0;
>> + }
>> + ret = split_huge_page_to_list(page, list);
>> + if (ret) {
>> + delete_from_swap_cache(page);
>> + return -EBUSY;
>> + }
>> + return 1;
>> +}
>> +#else
>> +static inline int add_to_swap_trans_huge(struct page *page,
>> + struct list_head *list)
>> +{
>> + return 0;
>> +}
>> +#endif
>> +
>> /**
>> * add_to_swap - allocate swap space for a page
>> * @page: page we want to move to swap
>> *
>> * Allocate swap space for the page and add the page to the
>> - * swap cache. Caller needs to hold the page lock.
>> + * swap cache. Caller needs to hold the page lock.
>> */
>> int add_to_swap(struct page *page, struct list_head *list)
>> {
>> @@ -198,6 +240,18 @@ int add_to_swap(struct page *page, struct list_head *list)
>> VM_BUG_ON_PAGE(!PageLocked(page), page);
>> VM_BUG_ON_PAGE(!PageUptodate(page), page);
>>
>> + if (unlikely(PageTransHuge(page))) {
>> + err = add_to_swap_trans_huge(page, list);
>> + switch (err) {
>> + case 1:
>> + return 1;
>> + case 0:
>> + /* fallback to split firstly if return 0 */
>> + break;
>> + default:
>> + return 0;
>> + }
>> + }
>> entry = get_swap_page();
>> if (!entry.val)
>> return 0;
>
> add_to_swap_trans_huge() is too close a copy of add_to_swap(), which
> makes the code error prone for future modifications to the swap slot
> allocation protocol.
>
> This should read:
>
> retry:
> entry = get_swap_page(page);
> if (!entry.val) {
> if (PageTransHuge(page)) {
> split_huge_page_to_list(page, list);
> goto retry;
> }
> return 0;
> }
If the swap space is used up, that is, get_swap_page() cannot allocate
even 1 swap entry for a normal page. We will split THP unnecessarily
with the change, but in the original code, we just skip the THP. There
may be a performance regression here. Similar problem exists for
mem_cgroup_try_charge_swap() too. If the mem cgroup exceeds the swap
limit, the THP will be split unnecessary with the change too.
> And get_swap_page(), mem_cgroup_try_charge_swap() etc. should all
> check PageTransHuge() instead of having extra parameters or separate
> code paths for the huge page case.
>
> In general, don't try to tack this feature onto the side of the
> VM. Because right now, this looks a bit like the hugetlb code, with
> one big branch in the beginning that opens up an alternate
> reality. Instead, these functions should handle THP all the way down
> the stack, and without passing down redundant information.
Yes. We should share the code as much as possible. I just have some
questions as above. Could you help me on that?
Best Regards,
Huang, Ying
Johannes Weiner <[email protected]> writes:
> On Tue, Mar 28, 2017 at 01:32:04PM +0800, Huang, Ying wrote:
>> @@ -527,6 +527,23 @@ static inline swp_entry_t get_swap_page(void)
>>
>> #endif /* CONFIG_SWAP */
>>
>> +#ifdef CONFIG_THP_SWAP_CLUSTER
>> +static inline swp_entry_t get_huge_swap_page(void)
>> +{
>> + swp_entry_t entry;
>> +
>> + if (get_swap_pages(1, &entry, true))
>> + return entry;
>> + else
>> + return (swp_entry_t) {0};
>> +}
>> +#else
>> +static inline swp_entry_t get_huge_swap_page(void)
>> +{
>> + return (swp_entry_t) {0};
>> +}
>> +#endif
>
> Your introducing a function without a user, making it very hard to
> judge whether the API is well-designed for the callers or not.
>
> I pointed this out as a systemic problem with this patch series in v3,
> along with other stuff, but with the way this series is structured I'm
> having a hard time seeing whether you implemented my other feedback or
> whether your counter arguments to them are justified.
>
> I cannot review and ack these patches this way.
Sorry for inconvenience, I will send a new version to combine the
function definition and usage into one patch at least for you to
review. But I think we can continue our discussion in the comments your
raised so far firstly, what do you think about that?
Best Regards,
Huang, Ying
On Thu, Mar 30, 2017 at 12:15:13PM +0800, Huang, Ying wrote:
> Johannes Weiner <[email protected]> writes:
> > On Tue, Mar 28, 2017 at 01:32:09PM +0800, Huang, Ying wrote:
> >> @@ -198,6 +240,18 @@ int add_to_swap(struct page *page, struct list_head *list)
> >> VM_BUG_ON_PAGE(!PageLocked(page), page);
> >> VM_BUG_ON_PAGE(!PageUptodate(page), page);
> >>
> >> + if (unlikely(PageTransHuge(page))) {
> >> + err = add_to_swap_trans_huge(page, list);
> >> + switch (err) {
> >> + case 1:
> >> + return 1;
> >> + case 0:
> >> + /* fallback to split firstly if return 0 */
> >> + break;
> >> + default:
> >> + return 0;
> >> + }
> >> + }
> >> entry = get_swap_page();
> >> if (!entry.val)
> >> return 0;
> >
> > add_to_swap_trans_huge() is too close a copy of add_to_swap(), which
> > makes the code error prone for future modifications to the swap slot
> > allocation protocol.
> >
> > This should read:
> >
> > retry:
> > entry = get_swap_page(page);
> > if (!entry.val) {
> > if (PageTransHuge(page)) {
> > split_huge_page_to_list(page, list);
> > goto retry;
> > }
> > return 0;
> > }
>
> If the swap space is used up, that is, get_swap_page() cannot allocate
> even 1 swap entry for a normal page. We will split THP unnecessarily
> with the change, but in the original code, we just skip the THP. There
> may be a performance regression here. Similar problem exists for
> mem_cgroup_try_charge_swap() too. If the mem cgroup exceeds the swap
> limit, the THP will be split unnecessary with the change too.
If we skip the page, we're swapping out another page hotter than this
one. Giving THP preservation priority over LRU order is an issue best
kept for a separate patch set; this one is supposed to be a mechanical
implementation of THP swapping. Let's nail down the basics first.
Such a decision would need proof that splitting THPs on full swap
devices is a concern for real applications. I would assume that we're
pretty close to OOM anyway; it's much more likely that a single slot
frees up than a full cluster, at which point we'll be splitting THPs
anyway; etc. I have my doubts that this would be measurable.
But even if so, I don't think we'd have to duplicate the main code
flow to handle this corner case. You can extend get_swap_page() to
return an error code that tells add_to_swap() whether to split and
retry, or to fail and move on. So this way should be future proof.
On Thu, Mar 30, 2017 at 08:45:56AM +0800, Huang, Ying wrote:
> Johannes Weiner <[email protected]> writes:
>
> > On Tue, Mar 28, 2017 at 01:32:01PM +0800, Huang, Ying wrote:
> >> @@ -499,6 +499,19 @@ config FRONTSWAP
> >>
> >> If unsure, say Y to enable frontswap.
> >>
> >> +config ARCH_USES_THP_SWAP_CLUSTER
> >> + bool
> >> + default n
> >
> > This is fine.
> >
> >> +config THP_SWAP_CLUSTER
> >> + bool
> >> + depends on SWAP && TRANSPARENT_HUGEPAGE && ARCH_USES_THP_SWAP_CLUSTER
> >> + default y
> >> + help
> >> + Use one swap cluster to hold the contents of the THP
> >> + (Transparent Huge Page) swapped out. The size of the swap
> >> + cluster will be same as that of THP.
> >
> > But this is a super weird thing to ask the user. How would they know
> > what to say, if we don't know? I don't think this should be a config
> > knob at all. Merge the two config items into a simple
>
> The user will not see this, because there is no string after "bool" to
> let user to select it. The help here is for document only, so that
> architecture developers could know what this is for.
Oh, I missed that. My bad!
> > config THP_SWAP_CLUSTER
> > bool
> > default n
> >
> > and let the archs with reasonable THP sizes select it.
>
> This will have same effect as the original solution except the document
> is removed.
Then I still don't understand why we need two config symbols. Can't
archs select the documented THP_SWAP_CLUSTER directly?
The #ifdef in swapfile.c could check THP && THP_SWAP_CLUSTER.
Am I missing something?
On Thu, Mar 30, 2017 at 08:53:50AM +0800, Huang, Ying wrote:
> Johannes Weiner <[email protected]> writes:
> > but there doesn't seem to be a reason to
> > pass @nr_entries when we have the struct page. Why can't this function
> > just check PageTransHuge() by itself?
>
> Because sometimes we need to charge one swap entry for a THP. Please
> take a look at the original add_to_swap() implementation. For a THP,
> one swap entry will be allocated and charged to the mem cgroup before
> the THP is split. And I think it is not easy to change this, because we
> don't want to split THP if the mem cgroup for swap exceeds its limit.
I think we do. Let's continue this discussion in the 9/9 subthread.
On Thu, Mar 30, 2017 at 12:28:17PM +0800, Huang, Ying wrote:
> Johannes Weiner <[email protected]> writes:
> > On Tue, Mar 28, 2017 at 01:32:04PM +0800, Huang, Ying wrote:
> >> @@ -527,6 +527,23 @@ static inline swp_entry_t get_swap_page(void)
> >>
> >> #endif /* CONFIG_SWAP */
> >>
> >> +#ifdef CONFIG_THP_SWAP_CLUSTER
> >> +static inline swp_entry_t get_huge_swap_page(void)
> >> +{
> >> + swp_entry_t entry;
> >> +
> >> + if (get_swap_pages(1, &entry, true))
> >> + return entry;
> >> + else
> >> + return (swp_entry_t) {0};
> >> +}
> >> +#else
> >> +static inline swp_entry_t get_huge_swap_page(void)
> >> +{
> >> + return (swp_entry_t) {0};
> >> +}
> >> +#endif
> >
> > Your introducing a function without a user, making it very hard to
> > judge whether the API is well-designed for the callers or not.
> >
> > I pointed this out as a systemic problem with this patch series in v3,
> > along with other stuff, but with the way this series is structured I'm
> > having a hard time seeing whether you implemented my other feedback or
> > whether your counter arguments to them are justified.
> >
> > I cannot review and ack these patches this way.
>
> Sorry for inconvenience, I will send a new version to combine the
> function definition and usage into one patch at least for you to
> review.
We tried this before. I reviewed the self-contained patch and you
incorporated the feedback into the split-out structure that made it
impossible for me to verify the updates.
I'm not sure why you insist on preserving this series format. It's not
good for review, and it's not good for merging and git history.
> But I think we can continue our discussion in the comments your
> raised so far firstly, what do you think about that?
Yeah, let's finish the discussions before -v8.
Johannes Weiner <[email protected]> writes:
> On Thu, Mar 30, 2017 at 12:15:13PM +0800, Huang, Ying wrote:
>> Johannes Weiner <[email protected]> writes:
>> > On Tue, Mar 28, 2017 at 01:32:09PM +0800, Huang, Ying wrote:
>> >> @@ -198,6 +240,18 @@ int add_to_swap(struct page *page, struct list_head *list)
>> >> VM_BUG_ON_PAGE(!PageLocked(page), page);
>> >> VM_BUG_ON_PAGE(!PageUptodate(page), page);
>> >>
>> >> + if (unlikely(PageTransHuge(page))) {
>> >> + err = add_to_swap_trans_huge(page, list);
>> >> + switch (err) {
>> >> + case 1:
>> >> + return 1;
>> >> + case 0:
>> >> + /* fallback to split firstly if return 0 */
>> >> + break;
>> >> + default:
>> >> + return 0;
>> >> + }
>> >> + }
>> >> entry = get_swap_page();
>> >> if (!entry.val)
>> >> return 0;
>> >
>> > add_to_swap_trans_huge() is too close a copy of add_to_swap(), which
>> > makes the code error prone for future modifications to the swap slot
>> > allocation protocol.
>> >
>> > This should read:
>> >
>> > retry:
>> > entry = get_swap_page(page);
>> > if (!entry.val) {
>> > if (PageTransHuge(page)) {
>> > split_huge_page_to_list(page, list);
>> > goto retry;
>> > }
>> > return 0;
>> > }
>>
>> If the swap space is used up, that is, get_swap_page() cannot allocate
>> even 1 swap entry for a normal page. We will split THP unnecessarily
>> with the change, but in the original code, we just skip the THP. There
>> may be a performance regression here. Similar problem exists for
>> mem_cgroup_try_charge_swap() too. If the mem cgroup exceeds the swap
>> limit, the THP will be split unnecessary with the change too.
>
> If we skip the page, we're swapping out another page hotter than this
> one. Giving THP preservation priority over LRU order is an issue best
> kept for a separate patch set;
In my original patch, if we failed to allocate the swap space for a THP,
and we can allocate the swap space for a normal page, we will split the
THP. We skip the page only if we cannot allocate the swap space for a
normal page, that is, nr_swap_pages is 0. So we will not give THP
preservation priority over LRU order in the patch.
> this one is supposed to be a mechanical
> implementation of THP swapping. Let's nail down the basics first.
Yes. So I tried to keep the original behavior to deal with THP if we
cannot allocate the swap space (a swap cluster) for a whole THP.
Per my understanding, the difference between what you suggested and the
original behavior is that, when nr_swap_pages is 0, whether to split the
THP.
> Such a decision would need proof that splitting THPs on full swap
> devices is a concern for real applications. I would assume that we're
> pretty close to OOM anyway; it's much more likely that a single slot
> frees up than a full cluster, at which point we'll be splitting THPs
> anyway; etc. I have my doubts that this would be measurable.
>
> But even if so, I don't think we'd have to duplicate the main code
> flow to handle this corner case. You can extend get_swap_page() to
> return an error code that tells add_to_swap() whether to split and
> retry, or to fail and move on. So this way should be future proof.
Yes. I will try to merge add_to_swap_trans_huge() into add_to_swap() in
the next version. But if we want to keep the original behavior, we will
need an extra "nr_entries" parameter for mem_cgroup_try_charge_swap().
Best Regards,
Huang, Ying
Johannes Weiner <[email protected]> writes:
> On Thu, Mar 30, 2017 at 08:45:56AM +0800, Huang, Ying wrote:
>> Johannes Weiner <[email protected]> writes:
>>
>> > On Tue, Mar 28, 2017 at 01:32:01PM +0800, Huang, Ying wrote:
>> >> @@ -499,6 +499,19 @@ config FRONTSWAP
>> >>
>> >> If unsure, say Y to enable frontswap.
>> >>
>> >> +config ARCH_USES_THP_SWAP_CLUSTER
>> >> + bool
>> >> + default n
>> >
>> > This is fine.
>> >
>> >> +config THP_SWAP_CLUSTER
>> >> + bool
>> >> + depends on SWAP && TRANSPARENT_HUGEPAGE && ARCH_USES_THP_SWAP_CLUSTER
>> >> + default y
>> >> + help
>> >> + Use one swap cluster to hold the contents of the THP
>> >> + (Transparent Huge Page) swapped out. The size of the swap
>> >> + cluster will be same as that of THP.
>> >
>> > But this is a super weird thing to ask the user. How would they know
>> > what to say, if we don't know? I don't think this should be a config
>> > knob at all. Merge the two config items into a simple
>>
>> The user will not see this, because there is no string after "bool" to
>> let user to select it. The help here is for document only, so that
>> architecture developers could know what this is for.
>
> Oh, I missed that. My bad!
>
>> > config THP_SWAP_CLUSTER
>> > bool
>> > default n
>> >
>> > and let the archs with reasonable THP sizes select it.
>>
>> This will have same effect as the original solution except the document
>> is removed.
>
> Then I still don't understand why we need two config symbols. Can't
> archs select the documented THP_SWAP_CLUSTER directly?
>
> The #ifdef in swapfile.c could check THP && THP_SWAP_CLUSTER.
>
> Am I missing something?
I use two config symbols just to save some typing, instead of
#ifdef CONFIG_THP_SWAP_CLUSTER
it will be,
#if defined(CONFIG_TRANSPARENT_HUGEPAGE) && defined(CONFIG_THP_SWAP_CLUSTER)
or
#if defined(CONFIG_SWAP) && defined(CONFIG_TRANSPARENT_HUGEPAGE) && defined(CONFIG_THP_SWAP_CLUSTER)
Best Regards,
Huang, Ying
Johannes Weiner <[email protected]> writes:
> On Thu, Mar 30, 2017 at 12:28:17PM +0800, Huang, Ying wrote:
>> Johannes Weiner <[email protected]> writes:
>> > On Tue, Mar 28, 2017 at 01:32:04PM +0800, Huang, Ying wrote:
>> >> @@ -527,6 +527,23 @@ static inline swp_entry_t get_swap_page(void)
>> >>
>> >> #endif /* CONFIG_SWAP */
>> >>
>> >> +#ifdef CONFIG_THP_SWAP_CLUSTER
>> >> +static inline swp_entry_t get_huge_swap_page(void)
>> >> +{
>> >> + swp_entry_t entry;
>> >> +
>> >> + if (get_swap_pages(1, &entry, true))
>> >> + return entry;
>> >> + else
>> >> + return (swp_entry_t) {0};
>> >> +}
>> >> +#else
>> >> +static inline swp_entry_t get_huge_swap_page(void)
>> >> +{
>> >> + return (swp_entry_t) {0};
>> >> +}
>> >> +#endif
>> >
>> > Your introducing a function without a user, making it very hard to
>> > judge whether the API is well-designed for the callers or not.
>> >
>> > I pointed this out as a systemic problem with this patch series in v3,
>> > along with other stuff, but with the way this series is structured I'm
>> > having a hard time seeing whether you implemented my other feedback or
>> > whether your counter arguments to them are justified.
>> >
>> > I cannot review and ack these patches this way.
>>
>> Sorry for inconvenience, I will send a new version to combine the
>> function definition and usage into one patch at least for you to
>> review.
>
> We tried this before. I reviewed the self-contained patch and you
> incorporated the feedback into the split-out structure that made it
> impossible for me to verify the updates.
>
> I'm not sure why you insist on preserving this series format. It's not
> good for review, and it's not good for merging and git history.
I had thought some reviewers would prefer the original series format.
But I will use your suggested format in the future, unless more
reviewers prefer the original format.
Best Regards,
Huang, Ying
>> But I think we can continue our discussion in the comments your
>> raised so far firstly, what do you think about that?
>
> Yeah, let's finish the discussions before -v8.