From: Zi Yan <[email protected]>
Hi all,
THP migration is only enabled on x86_64 with a special
ARCH_ENABLE_THP_MIGRATION macro. This patchset enables THP migration for
all architectures that uses transparent hugepage, so that special macro can
be dropped. Instead, THP migration is enabled/disabled via
/sys/kernel/mm/transparent_hugepage/enable_thp_migration.
I grepped for TRANSPARENT_HUGEPAGE in arch folder and got 9 architectures that
are supporting transparent hugepage. I mechanically add __pmd_to_swp_entry() and
__swp_entry_to_pmd() based on existing __pte_to_swp_entry() and
__swp_entry_to_pte() for all these architectures, except tile which is going to
be dropped.
I have successfully compiled all these architectures, but have NOT tested them
due to lack of real hardware. I appreciate your help, if the maintainers of
these architectures can do a quick test with the code from
https://github.com/x-y-z/thp-migration-bench . Please apply patch 9 as well
to enable THP migration.
By enabling THP migration, migrating a 2MB THP on x86_64 machines takes only 1/3
time of migrating equivalent 512 4KB pages.
Hi Naoya, I also add soft dirty support for powerpc and s390. It would be great
if you can take a look at patch 6 & 7.
Feel free to give comments. Thanks.
Cc: [email protected]
Cc: Vineet Gupta <[email protected]>
Cc: [email protected]
Cc: Russell King <[email protected]>
Cc: Christoffer Dall <[email protected]>
Cc: Marc Zyngier <[email protected]>
Cc: [email protected]
Cc: Catalin Marinas <[email protected]>
Cc: Will Deacon <[email protected]>
Cc: Steve Capper <[email protected]>
Cc: Kristina Martsenko <[email protected]>
Cc: Dan Williams <[email protected]>
Cc: Thomas Gleixner <[email protected]>
Cc: Ingo Molnar <[email protected]>
Cc: "Kirill A. Shutemov" <[email protected]>
Cc: [email protected]
Cc: Ralf Baechle <[email protected]>
Cc: James Hogan <[email protected]>
Cc: Michal Hocko <[email protected]>
Cc: [email protected]
Cc: Benjamin Herrenschmidt <[email protected]>
Cc: Paul Mackerras <[email protected]>
Cc: Michael Ellerman <[email protected]>
Cc: "Aneesh Kumar K.V" <[email protected]>
Cc: Ram Pai <[email protected]>
Cc: Balbir Singh <[email protected]>
Cc: Naoya Horiguchi <[email protected]>
Cc: [email protected]
Cc: Martin Schwidefsky <[email protected]>
Cc: Heiko Carstens <[email protected]>
Cc: Janosch Frank <[email protected]>
Cc: [email protected]
Cc: "David S. Miller" <[email protected]>
Cc: [email protected]
Cc: "Huang, Ying" <[email protected]>
Zi Yan (9):
arc: mm: migrate: add pmd swap entry to support thp migration.
arm: mm: migrate: add pmd swap entry to support thp migration.
arm64: mm: migrate: add pmd swap entry to support thp migration.
i386: mm: migrate: add pmd swap entry to support thp migration.
mips: mm: migrate: add pmd swap entry to support thp migration.
powerpc: mm: migrate: add pmd swap entry to support thp migration.
s390: mm: migrate: add pmd swap entry to support thp migration.
sparc: mm: migrate: add pmd swap entry to support thp migration.
mm: migrate: enable thp migration for all possible architectures.
arch/arc/include/asm/pgtable.h | 2 ++
arch/arm/include/asm/pgtable.h | 2 ++
arch/arm64/include/asm/pgtable.h | 2 ++
arch/mips/include/asm/pgtable-64.h | 2 ++
arch/powerpc/include/asm/book3s/32/pgtable.h | 2 ++
arch/powerpc/include/asm/book3s/64/pgtable.h | 17 ++++++++++++
arch/powerpc/include/asm/nohash/32/pgtable.h | 2 ++
arch/powerpc/include/asm/nohash/64/pgtable.h | 2 ++
arch/s390/include/asm/pgtable.h | 5 ++++
arch/sparc/include/asm/pgtable_32.h | 2 ++
arch/sparc/include/asm/pgtable_64.h | 2 ++
arch/x86/Kconfig | 4 ---
arch/x86/include/asm/pgtable-2level.h | 2 ++
arch/x86/include/asm/pgtable-3level.h | 2 ++
arch/x86/include/asm/pgtable.h | 2 --
fs/proc/task_mmu.c | 2 --
include/asm-generic/pgtable.h | 21 ++-------------
include/linux/huge_mm.h | 9 +++----
include/linux/swapops.h | 4 +--
mm/Kconfig | 3 ---
mm/huge_memory.c | 27 +++++++++++++-------
mm/migrate.c | 6 ++---
mm/rmap.c | 5 ++--
23 files changed, 73 insertions(+), 54 deletions(-)
--
2.17.0
From: Zi Yan <[email protected]>
Signed-off-by: Zi Yan <[email protected]>
Cc: Catalin Marinas <[email protected]>
Cc: Will Deacon <[email protected]>
Cc: Steve Capper <[email protected]>
Cc: Marc Zyngier <[email protected]>
Cc: Kristina Martsenko <[email protected]>
Cc: Dan Williams <[email protected]>
Cc: [email protected]
Cc: [email protected]
---
arch/arm64/include/asm/pgtable.h | 2 ++
1 file changed, 2 insertions(+)
diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h
index 7e2c27e63cd8..1cdc9d3db2c7 100644
--- a/arch/arm64/include/asm/pgtable.h
+++ b/arch/arm64/include/asm/pgtable.h
@@ -742,7 +742,9 @@ extern pgd_t tramp_pg_dir[PTRS_PER_PGD];
#define __swp_entry(type,offset) ((swp_entry_t) { ((type) << __SWP_TYPE_SHIFT) | ((offset) << __SWP_OFFSET_SHIFT) })
#define __pte_to_swp_entry(pte) ((swp_entry_t) { pte_val(pte) })
+#define __pmd_to_swp_entry(pmd) ((swp_entry_t) { pmd_val(pmd) })
#define __swp_entry_to_pte(swp) ((pte_t) { (swp).val })
+#define __swp_entry_to_pmd(swp) ((pmd_t) { (swp).val })
/*
* Ensure that there are not more swap files than can be encoded in the kernel
--
2.17.0
From: Zi Yan <[email protected]>
Signed-off-by: Zi Yan <[email protected]>
Cc: Vineet Gupta <[email protected]>
Cc: [email protected]
Cc: [email protected]
---
arch/arc/include/asm/pgtable.h | 2 ++
1 file changed, 2 insertions(+)
diff --git a/arch/arc/include/asm/pgtable.h b/arch/arc/include/asm/pgtable.h
index 08fe33830d4b..246934105e61 100644
--- a/arch/arc/include/asm/pgtable.h
+++ b/arch/arc/include/asm/pgtable.h
@@ -383,7 +383,9 @@ void update_mmu_cache(struct vm_area_struct *vma, unsigned long address,
/* NOPs, to keep generic kernel happy */
#define __pte_to_swp_entry(pte) ((swp_entry_t) { pte_val(pte) })
+#define __pmd_to_swp_entry(pmd) ((swp_entry_t) { pmd_val(pmd) })
#define __swp_entry_to_pte(x) ((pte_t) { (x).val })
+#define __swp_entry_to_pmd(x) ((pmd_t) { (x).val })
#define kern_addr_valid(addr) (1)
--
2.17.0
From: Zi Yan <[email protected]>
Signed-off-by: Zi Yan <[email protected]>
Cc: Russell King <[email protected]>
Cc: Christoffer Dall <[email protected]>
Cc: Marc Zyngier <[email protected]>
Cc: [email protected]
Cc: [email protected]
---
arch/arm/include/asm/pgtable.h | 2 ++
1 file changed, 2 insertions(+)
diff --git a/arch/arm/include/asm/pgtable.h b/arch/arm/include/asm/pgtable.h
index a757401129f9..d4b35514e96a 100644
--- a/arch/arm/include/asm/pgtable.h
+++ b/arch/arm/include/asm/pgtable.h
@@ -347,7 +347,9 @@ static inline pte_t pte_modify(pte_t pte, pgprot_t newprot)
#define __swp_entry(type,offset) ((swp_entry_t) { ((type) << __SWP_TYPE_SHIFT) | ((offset) << __SWP_OFFSET_SHIFT) })
#define __pte_to_swp_entry(pte) ((swp_entry_t) { pte_val(pte) })
+#define __pmd_to_swp_entry(pmd) ((swp_entry_t) { pmd_val(pmd) })
#define __swp_entry_to_pte(swp) ((pte_t) { (swp).val })
+#define __swp_entry_to_pmd(swp) ((pmd_t) { (swp).val })
/*
* It is an error for the kernel to have more swap files than we can
--
2.17.0
From: Zi Yan <[email protected]>
Signed-off-by: Zi Yan <[email protected]>
Cc: Thomas Gleixner <[email protected]>
Cc: Ingo Molnar <[email protected]>
Cc: "Kirill A. Shutemov" <[email protected]>
Cc: [email protected]
Cc: [email protected]
---
arch/x86/include/asm/pgtable-2level.h | 2 ++
arch/x86/include/asm/pgtable-3level.h | 2 ++
2 files changed, 4 insertions(+)
diff --git a/arch/x86/include/asm/pgtable-2level.h b/arch/x86/include/asm/pgtable-2level.h
index 685ffe8a0eaf..fba4722ec2c2 100644
--- a/arch/x86/include/asm/pgtable-2level.h
+++ b/arch/x86/include/asm/pgtable-2level.h
@@ -93,6 +93,8 @@ static inline unsigned long pte_bitop(unsigned long value, unsigned int rightshi
((type) << (_PAGE_BIT_PRESENT + 1)) \
| ((offset) << SWP_OFFSET_SHIFT) })
#define __pte_to_swp_entry(pte) ((swp_entry_t) { (pte).pte_low })
+#define __pmd_to_swp_entry(pmd) ((swp_entry_t) { native_pmd_val(pmd) })
#define __swp_entry_to_pte(x) ((pte_t) { .pte = (x).val })
+#define __swp_entry_to_pmd(x) (native_make_pmd(x.val))
#endif /* _ASM_X86_PGTABLE_2LEVEL_H */
diff --git a/arch/x86/include/asm/pgtable-3level.h b/arch/x86/include/asm/pgtable-3level.h
index f24df59c40b2..9b7e3c74fbc0 100644
--- a/arch/x86/include/asm/pgtable-3level.h
+++ b/arch/x86/include/asm/pgtable-3level.h
@@ -246,7 +246,9 @@ static inline pud_t native_pudp_get_and_clear(pud_t *pudp)
#define __swp_offset(x) ((x).val >> 5)
#define __swp_entry(type, offset) ((swp_entry_t){(type) | (offset) << 5})
#define __pte_to_swp_entry(pte) ((swp_entry_t){ (pte).pte_high })
+#define __pmd_to_swp_entry(pmd) ((swp_entry_t){ (pmd).pmd_high })
#define __swp_entry_to_pte(x) ((pte_t){ { .pte_high = (x).val } })
+#define __swp_entry_to_pmd(x) ((pmd_t){ { .pmd_high = (x).val } })
#define gup_get_pte gup_get_pte
/*
--
2.17.0
From: Zi Yan <[email protected]>
pmd swap soft dirty support is added, too.
Signed-off-by: Zi Yan <[email protected]>
Cc: Benjamin Herrenschmidt <[email protected]>
Cc: Paul Mackerras <[email protected]>
Cc: Michael Ellerman <[email protected]>
Cc: "Aneesh Kumar K.V" <[email protected]>
Cc: Ram Pai <[email protected]>
Cc: Balbir Singh <[email protected]>
Cc: Naoya Horiguchi <[email protected]>
Cc: [email protected]
Cc: [email protected]
---
arch/powerpc/include/asm/book3s/32/pgtable.h | 2 ++
arch/powerpc/include/asm/book3s/64/pgtable.h | 17 +++++++++++++++++
arch/powerpc/include/asm/nohash/32/pgtable.h | 2 ++
arch/powerpc/include/asm/nohash/64/pgtable.h | 2 ++
4 files changed, 23 insertions(+)
diff --git a/arch/powerpc/include/asm/book3s/32/pgtable.h b/arch/powerpc/include/asm/book3s/32/pgtable.h
index c615abdce119..866b67a8abf0 100644
--- a/arch/powerpc/include/asm/book3s/32/pgtable.h
+++ b/arch/powerpc/include/asm/book3s/32/pgtable.h
@@ -294,7 +294,9 @@ static inline void __ptep_set_access_flags(struct mm_struct *mm,
#define __swp_offset(entry) ((entry).val >> 5)
#define __swp_entry(type, offset) ((swp_entry_t) { (type) | ((offset) << 5) })
#define __pte_to_swp_entry(pte) ((swp_entry_t) { pte_val(pte) >> 3 })
+#define __pmd_to_swp_entry(pmd) ((swp_entry_t) { pmd_val(pmd) >> 3 })
#define __swp_entry_to_pte(x) ((pte_t) { (x).val << 3 })
+#define __swp_entry_to_pmd(x) ((pmd_t) { (x).val << 3 })
int map_kernel_page(unsigned long va, phys_addr_t pa, int flags);
diff --git a/arch/powerpc/include/asm/book3s/64/pgtable.h b/arch/powerpc/include/asm/book3s/64/pgtable.h
index a6b9f1d74600..6b3c6492071d 100644
--- a/arch/powerpc/include/asm/book3s/64/pgtable.h
+++ b/arch/powerpc/include/asm/book3s/64/pgtable.h
@@ -726,7 +726,9 @@ static inline bool pte_user(pte_t pte)
* Clear bits not found in swap entries here.
*/
#define __pte_to_swp_entry(pte) ((swp_entry_t) { pte_val((pte)) & ~_PAGE_PTE })
+#define __pmd_to_swp_entry(pmd) ((swp_entry_t) { pmd_val((pmd)) & ~_PAGE_PTE })
#define __swp_entry_to_pte(x) __pte((x).val | _PAGE_PTE)
+#define __swp_entry_to_pmd(x) __pmd((x).val | _PAGE_PTE)
#ifdef CONFIG_MEM_SOFT_DIRTY
#define _PAGE_SWP_SOFT_DIRTY (1UL << (SWP_TYPE_BITS + _PAGE_BIT_SWAP_TYPE))
@@ -749,6 +751,21 @@ static inline pte_t pte_swp_clear_soft_dirty(pte_t pte)
{
return __pte(pte_val(pte) & ~_PAGE_SWP_SOFT_DIRTY);
}
+
+static inline pmd_t pmd_swp_mksoft_dirty(pmd_t pmd)
+{
+ return __pmd(pmd_val(pmd) | _PAGE_SWP_SOFT_DIRTY);
+}
+
+static inline bool pmd_swp_soft_dirty(pmd_t pmd)
+{
+ return !!(pmd_raw(pmd) & cpu_to_be64(_PAGE_SWP_SOFT_DIRTY));
+}
+
+static inline pmd_t pmd_swp_clear_soft_dirty(pmd_t pmd)
+{
+ return __pmd(pmd_val(pmd) & ~_PAGE_SWP_SOFT_DIRTY);
+}
#endif /* CONFIG_HAVE_ARCH_SOFT_DIRTY */
static inline bool check_pte_access(unsigned long access, unsigned long ptev)
diff --git a/arch/powerpc/include/asm/nohash/32/pgtable.h b/arch/powerpc/include/asm/nohash/32/pgtable.h
index 03bbd1149530..f6b0534a02d4 100644
--- a/arch/powerpc/include/asm/nohash/32/pgtable.h
+++ b/arch/powerpc/include/asm/nohash/32/pgtable.h
@@ -337,7 +337,9 @@ static inline void __ptep_set_access_flags(struct mm_struct *mm,
#define __swp_offset(entry) ((entry).val >> 5)
#define __swp_entry(type, offset) ((swp_entry_t) { (type) | ((offset) << 5) })
#define __pte_to_swp_entry(pte) ((swp_entry_t) { pte_val(pte) >> 3 })
+#define __pmd_to_swp_entry(pmd) ((swp_entry_t) { pmd_val(pte) >> 3 })
#define __swp_entry_to_pte(x) ((pte_t) { (x).val << 3 })
+#define __swp_entry_to_pmd(x) ((pmd_t) { (x).val << 3 })
int map_kernel_page(unsigned long va, phys_addr_t pa, int flags);
diff --git a/arch/powerpc/include/asm/nohash/64/pgtable.h b/arch/powerpc/include/asm/nohash/64/pgtable.h
index 5c5f75d005ad..5790763c07df 100644
--- a/arch/powerpc/include/asm/nohash/64/pgtable.h
+++ b/arch/powerpc/include/asm/nohash/64/pgtable.h
@@ -342,7 +342,9 @@ static inline void __ptep_set_access_flags(struct mm_struct *mm,
| ((offset) << PTE_RPN_SHIFT) })
#define __pte_to_swp_entry(pte) ((swp_entry_t) { pte_val((pte)) })
+#define __pmd_to_swp_entry(pmd) ((swp_entry_t) { pmd_val((pmd)) })
#define __swp_entry_to_pte(x) __pte((x).val)
+#define __swp_entry_to_pmd(x) __pmd((x).val)
extern int map_kernel_page(unsigned long ea, unsigned long pa,
unsigned long flags);
--
2.17.0
From: Zi Yan <[email protected]>
Signed-off-by: Zi Yan <[email protected]>
Cc: "David S. Miller" <[email protected]>
Cc: [email protected]
Cc: [email protected]
---
arch/sparc/include/asm/pgtable_32.h | 2 ++
arch/sparc/include/asm/pgtable_64.h | 2 ++
2 files changed, 4 insertions(+)
diff --git a/arch/sparc/include/asm/pgtable_32.h b/arch/sparc/include/asm/pgtable_32.h
index 4eebed6c6781..293bf9f8f949 100644
--- a/arch/sparc/include/asm/pgtable_32.h
+++ b/arch/sparc/include/asm/pgtable_32.h
@@ -367,7 +367,9 @@ static inline swp_entry_t __swp_entry(unsigned long type, unsigned long offset)
}
#define __pte_to_swp_entry(pte) ((swp_entry_t) { pte_val(pte) })
+#define __pmd_to_swp_entry(pmd) ((swp_entry_t) { pmd_val(pmd) })
#define __swp_entry_to_pte(x) ((pte_t) { (x).val })
+#define __swp_entry_to_pmd(x) ((pmd_t) { (x).val })
static inline unsigned long
__get_phys (unsigned long addr)
diff --git a/arch/sparc/include/asm/pgtable_64.h b/arch/sparc/include/asm/pgtable_64.h
index 339920fdf9ed..2811aef4a636 100644
--- a/arch/sparc/include/asm/pgtable_64.h
+++ b/arch/sparc/include/asm/pgtable_64.h
@@ -1031,7 +1031,9 @@ pgtable_t pgtable_trans_huge_withdraw(struct mm_struct *mm, pmd_t *pmdp);
((long)(offset) << (PAGE_SHIFT + 8UL))) \
} )
#define __pte_to_swp_entry(pte) ((swp_entry_t) { pte_val(pte) })
+#define __pmd_to_swp_entry(pmd) ((swp_entry_t) { pmd_val(pmd) })
#define __swp_entry_to_pte(x) ((pte_t) { (x).val })
+#define __swp_entry_to_pmd(x) ((pmd_t) { (x).val })
int page_in_phys_avail(unsigned long paddr);
--
2.17.0
From: Zi Yan <[email protected]>
pmd swap soft dirty support is added, too.
Signed-off-by: Zi Yan <[email protected]>
Cc: Martin Schwidefsky <[email protected]>
Cc: Heiko Carstens <[email protected]>
Cc: Janosch Frank <[email protected]>
Cc: Naoya Horiguchi <[email protected]>
Cc: [email protected]
Cc: [email protected]
---
arch/s390/include/asm/pgtable.h | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/arch/s390/include/asm/pgtable.h b/arch/s390/include/asm/pgtable.h
index 2d24d33bf188..215fbb34203e 100644
--- a/arch/s390/include/asm/pgtable.h
+++ b/arch/s390/include/asm/pgtable.h
@@ -798,18 +798,21 @@ static inline int pmd_soft_dirty(pmd_t pmd)
{
return pmd_val(pmd) & _SEGMENT_ENTRY_SOFT_DIRTY;
}
+#define pmd_swp_soft_dirty pmd_soft_dirty
static inline pmd_t pmd_mksoft_dirty(pmd_t pmd)
{
pmd_val(pmd) |= _SEGMENT_ENTRY_SOFT_DIRTY;
return pmd;
}
+#define pmd_swp_mksoft_dirty pmd_mksoft_dirty
static inline pmd_t pmd_clear_soft_dirty(pmd_t pmd)
{
pmd_val(pmd) &= ~_SEGMENT_ENTRY_SOFT_DIRTY;
return pmd;
}
+#define pmd_swp_clear_soft_dirty pmd_clear_soft_dirty
/*
* query functions pte_write/pte_dirty/pte_young only work if
@@ -1594,7 +1597,9 @@ static inline swp_entry_t __swp_entry(unsigned long type, unsigned long offset)
}
#define __pte_to_swp_entry(pte) ((swp_entry_t) { pte_val(pte) })
+#define __pmd_to_swp_entry(pte) ((swp_entry_t) { pmd_val(pmd) })
#define __swp_entry_to_pte(x) ((pte_t) { (x).val })
+#define __swp_entry_to_pmd(x) ((pmd_t) { (x).val })
#define kern_addr_valid(addr) (1)
--
2.17.0
From: Zi Yan <[email protected]>
Remove CONFIG_ARCH_ENABLE_THP_MIGRATION. thp migration is enabled along
with transparent hugepage and can be toggled via
/sys/kernel/mm/transparent_hugepage/enable_thp_migration.
Signed-off-by: Zi Yan <[email protected]>
Cc: [email protected]
Cc: Vineet Gupta <[email protected]>
Cc: [email protected]
Cc: Russell King <[email protected]>
Cc: Christoffer Dall <[email protected]>
Cc: Marc Zyngier <[email protected]>
Cc: [email protected]
Cc: Catalin Marinas <[email protected]>
Cc: Will Deacon <[email protected]>
Cc: Steve Capper <[email protected]>
Cc: Kristina Martsenko <[email protected]>
Cc: Dan Williams <[email protected]>
Cc: Thomas Gleixner <[email protected]>
Cc: Ingo Molnar <[email protected]>
Cc: "Kirill A. Shutemov" <[email protected]>
Cc: [email protected]
Cc: Ralf Baechle <[email protected]>
Cc: James Hogan <[email protected]>
Cc: Michal Hocko <[email protected]>
Cc: [email protected]
Cc: Benjamin Herrenschmidt <[email protected]>
Cc: Paul Mackerras <[email protected]>
Cc: Michael Ellerman <[email protected]>
Cc: "Aneesh Kumar K.V" <[email protected]>
Cc: Ram Pai <[email protected]>
Cc: Balbir Singh <[email protected]>
Cc: Naoya Horiguchi <[email protected]>
Cc: [email protected]
Cc: Martin Schwidefsky <[email protected]>
Cc: Heiko Carstens <[email protected]>
Cc: Janosch Frank <[email protected]>
Cc: [email protected]
Cc: "David S. Miller" <[email protected]>
Cc: [email protected]
Cc: "Huang, Ying" <[email protected]>
---
arch/x86/Kconfig | 4 ----
arch/x86/include/asm/pgtable.h | 2 --
fs/proc/task_mmu.c | 2 --
include/asm-generic/pgtable.h | 21 ++-------------------
include/linux/huge_mm.h | 9 ++++-----
include/linux/swapops.h | 4 +---
mm/Kconfig | 3 ---
mm/huge_memory.c | 27 ++++++++++++++++++---------
mm/migrate.c | 6 ++----
mm/rmap.c | 5 ++---
10 files changed, 29 insertions(+), 54 deletions(-)
diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index 0fa71a78ec99..e73954e3eef7 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -2423,10 +2423,6 @@ config ARCH_ENABLE_HUGEPAGE_MIGRATION
def_bool y
depends on X86_64 && HUGETLB_PAGE && MIGRATION
-config ARCH_ENABLE_THP_MIGRATION
- def_bool y
- depends on X86_64 && TRANSPARENT_HUGEPAGE
-
menu "Power management and ACPI options"
config ARCH_HIBERNATION_HEADER
diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h
index b444d83cfc95..f9f54d9b39e3 100644
--- a/arch/x86/include/asm/pgtable.h
+++ b/arch/x86/include/asm/pgtable.h
@@ -1193,7 +1193,6 @@ static inline pte_t pte_swp_clear_soft_dirty(pte_t pte)
return pte_clear_flags(pte, _PAGE_SWP_SOFT_DIRTY);
}
-#ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION
static inline pmd_t pmd_swp_mksoft_dirty(pmd_t pmd)
{
return pmd_set_flags(pmd, _PAGE_SWP_SOFT_DIRTY);
@@ -1209,7 +1208,6 @@ static inline pmd_t pmd_swp_clear_soft_dirty(pmd_t pmd)
return pmd_clear_flags(pmd, _PAGE_SWP_SOFT_DIRTY);
}
#endif
-#endif
#define PKRU_AD_BIT 0x1
#define PKRU_WD_BIT 0x2
diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
index dd1b2aeb01e8..07a2f028d29a 100644
--- a/fs/proc/task_mmu.c
+++ b/fs/proc/task_mmu.c
@@ -1326,7 +1326,6 @@ static int pagemap_pmd_range(pmd_t *pmdp, unsigned long addr, unsigned long end,
frame = pmd_pfn(pmd) +
((addr & ~PMD_MASK) >> PAGE_SHIFT);
}
-#ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION
else if (is_swap_pmd(pmd)) {
swp_entry_t entry = pmd_to_swp_entry(pmd);
unsigned long offset = swp_offset(entry);
@@ -1340,7 +1339,6 @@ static int pagemap_pmd_range(pmd_t *pmdp, unsigned long addr, unsigned long end,
VM_BUG_ON(!is_pmd_migration_entry(pmd));
page = migration_entry_to_page(entry);
}
-#endif
if (page && page_mapcount(page) == 1)
flags |= PM_MMAP_EXCLUSIVE;
diff --git a/include/asm-generic/pgtable.h b/include/asm-generic/pgtable.h
index f59639afaa39..9dacdd203131 100644
--- a/include/asm-generic/pgtable.h
+++ b/include/asm-generic/pgtable.h
@@ -674,24 +674,7 @@ static inline void ptep_modify_prot_commit(struct mm_struct *mm,
#define arch_start_context_switch(prev) do {} while (0)
#endif
-#ifdef CONFIG_HAVE_ARCH_SOFT_DIRTY
-#ifndef CONFIG_ARCH_ENABLE_THP_MIGRATION
-static inline pmd_t pmd_swp_mksoft_dirty(pmd_t pmd)
-{
- return pmd;
-}
-
-static inline int pmd_swp_soft_dirty(pmd_t pmd)
-{
- return 0;
-}
-
-static inline pmd_t pmd_swp_clear_soft_dirty(pmd_t pmd)
-{
- return pmd;
-}
-#endif
-#else /* !CONFIG_HAVE_ARCH_SOFT_DIRTY */
+#ifndef CONFIG_HAVE_ARCH_SOFT_DIRTY
static inline int pte_soft_dirty(pte_t pte)
{
return 0;
@@ -946,7 +929,7 @@ static inline int pmd_none_or_trans_huge_or_clear_bad(pmd_t *pmd)
* redundant with !pmd_present().
*/
if (pmd_none(pmdval) || pmd_trans_huge(pmdval) ||
- (IS_ENABLED(CONFIG_ARCH_ENABLE_THP_MIGRATION) && !pmd_present(pmdval)))
+ (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) && !pmd_present(pmdval)))
return 1;
if (unlikely(pmd_bad(pmdval))) {
pmd_clear_bad(pmd);
diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h
index a8a126259bc4..dc3144bdb7e5 100644
--- a/include/linux/huge_mm.h
+++ b/include/linux/huge_mm.h
@@ -59,6 +59,7 @@ enum transparent_hugepage_flag {
TRANSPARENT_HUGEPAGE_DEFRAG_REQ_MADV_FLAG,
TRANSPARENT_HUGEPAGE_DEFRAG_KHUGEPAGED_FLAG,
TRANSPARENT_HUGEPAGE_USE_ZERO_PAGE_FLAG,
+ TRANSPARENT_HUGEPAGE_MIGRATION_FLAG,
#ifdef CONFIG_DEBUG_VM
TRANSPARENT_HUGEPAGE_DEBUG_COW_FLAG,
#endif
@@ -126,6 +127,9 @@ static inline bool transparent_hugepage_enabled(struct vm_area_struct *vma)
#else /* CONFIG_DEBUG_VM */
#define transparent_hugepage_debug_cow() 0
#endif /* CONFIG_DEBUG_VM */
+#define thp_migration_supported() \
+ (transparent_hugepage_flags & \
+ (1<<TRANSPARENT_HUGEPAGE_MIGRATION_FLAG))
extern unsigned long thp_get_unmapped_area(struct file *filp,
unsigned long addr, unsigned long len, unsigned long pgoff,
@@ -240,11 +244,6 @@ void mm_put_huge_zero_page(struct mm_struct *mm);
#define mk_huge_pmd(page, prot) pmd_mkhuge(mk_pmd(page, prot))
-static inline bool thp_migration_supported(void)
-{
- return IS_ENABLED(CONFIG_ARCH_ENABLE_THP_MIGRATION);
-}
-
#else /* CONFIG_TRANSPARENT_HUGEPAGE */
#define HPAGE_PMD_SHIFT ({ BUILD_BUG(); 0; })
#define HPAGE_PMD_MASK ({ BUILD_BUG(); 0; })
diff --git a/include/linux/swapops.h b/include/linux/swapops.h
index 1d3877c39a00..1b723685f887 100644
--- a/include/linux/swapops.h
+++ b/include/linux/swapops.h
@@ -260,7 +260,7 @@ static inline int is_write_migration_entry(swp_entry_t entry)
struct page_vma_mapped_walk;
-#ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION
+#ifdef CONFIG_TRANSPARENT_HUGEPAGE
extern void set_pmd_migration_entry(struct page_vma_mapped_walk *pvmw,
struct page *page);
@@ -295,13 +295,11 @@ static inline int is_pmd_migration_entry(pmd_t pmd)
static inline void set_pmd_migration_entry(struct page_vma_mapped_walk *pvmw,
struct page *page)
{
- BUILD_BUG();
}
static inline void remove_migration_pmd(struct page_vma_mapped_walk *pvmw,
struct page *new)
{
- BUILD_BUG();
}
static inline void pmd_migration_entry_wait(struct mm_struct *m, pmd_t *p) { }
diff --git a/mm/Kconfig b/mm/Kconfig
index c782e8fb7235..7f29c5c2a8f6 100644
--- a/mm/Kconfig
+++ b/mm/Kconfig
@@ -262,9 +262,6 @@ config MIGRATION
config ARCH_ENABLE_HUGEPAGE_MIGRATION
bool
-config ARCH_ENABLE_THP_MIGRATION
- bool
-
config PHYS_ADDR_T_64BIT
def_bool 64BIT || ARCH_PHYS_ADDR_T_64BIT
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index a3a1815f8e11..80240bec2e11 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -55,7 +55,8 @@ unsigned long transparent_hugepage_flags __read_mostly =
#endif
(1<<TRANSPARENT_HUGEPAGE_DEFRAG_REQ_MADV_FLAG)|
(1<<TRANSPARENT_HUGEPAGE_DEFRAG_KHUGEPAGED_FLAG)|
- (1<<TRANSPARENT_HUGEPAGE_USE_ZERO_PAGE_FLAG);
+ (1<<TRANSPARENT_HUGEPAGE_USE_ZERO_PAGE_FLAG)|
+ (1<<TRANSPARENT_HUGEPAGE_MIGRATION_FLAG);
static struct shrinker deferred_split_shrinker;
@@ -288,6 +289,21 @@ static ssize_t use_zero_page_store(struct kobject *kobj,
static struct kobj_attribute use_zero_page_attr =
__ATTR(use_zero_page, 0644, use_zero_page_show, use_zero_page_store);
+static ssize_t thp_migration_show(struct kobject *kobj,
+ struct kobj_attribute *attr, char *buf)
+{
+ return single_hugepage_flag_show(kobj, attr, buf,
+ TRANSPARENT_HUGEPAGE_MIGRATION_FLAG);
+}
+static ssize_t thp_migration_store(struct kobject *kobj,
+ struct kobj_attribute *attr, const char *buf, size_t count)
+{
+ return single_hugepage_flag_store(kobj, attr, buf, count,
+ TRANSPARENT_HUGEPAGE_MIGRATION_FLAG);
+}
+static struct kobj_attribute thp_migration_attr =
+ __ATTR(enable_thp_migration, 0644, thp_migration_show, thp_migration_store);
+
static ssize_t hpage_pmd_size_show(struct kobject *kobj,
struct kobj_attribute *attr, char *buf)
{
@@ -319,6 +335,7 @@ static struct attribute *hugepage_attr[] = {
&defrag_attr.attr,
&use_zero_page_attr.attr,
&hpage_pmd_size_attr.attr,
+ &thp_migration_attr.attr,
#if defined(CONFIG_SHMEM) && defined(CONFIG_TRANSPARENT_HUGE_PAGECACHE)
&shmem_enabled_attr.attr,
#endif
@@ -924,7 +941,6 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm,
ret = -EAGAIN;
pmd = *src_pmd;
-#ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION
if (unlikely(is_swap_pmd(pmd))) {
swp_entry_t entry = pmd_to_swp_entry(pmd);
@@ -943,7 +959,6 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm,
ret = 0;
goto out_unlock;
}
-#endif
if (unlikely(!pmd_trans_huge(pmd))) {
pte_free(dst_mm, pgtable);
@@ -1857,7 +1872,6 @@ int change_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd,
preserve_write = prot_numa && pmd_write(*pmd);
ret = 1;
-#ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION
if (is_swap_pmd(*pmd)) {
swp_entry_t entry = pmd_to_swp_entry(*pmd);
@@ -1876,7 +1890,6 @@ int change_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd,
}
goto unlock;
}
-#endif
/*
* Avoid trapping faults against the zero page. The read-only
@@ -2128,7 +2141,6 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd,
*/
old_pmd = pmdp_invalidate(vma, haddr, pmd);
-#ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION
pmd_migration = is_pmd_migration_entry(old_pmd);
if (pmd_migration) {
swp_entry_t entry;
@@ -2136,7 +2148,6 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd,
entry = pmd_to_swp_entry(old_pmd);
page = pfn_to_page(swp_offset(entry));
} else
-#endif
page = pmd_page(old_pmd);
VM_BUG_ON_PAGE(!page_count(page), page);
page_ref_add(page, HPAGE_PMD_NR - 1);
@@ -2870,7 +2881,6 @@ static int __init split_huge_pages_debugfs(void)
late_initcall(split_huge_pages_debugfs);
#endif
-#ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION
void set_pmd_migration_entry(struct page_vma_mapped_walk *pvmw,
struct page *page)
{
@@ -2934,4 +2944,3 @@ void remove_migration_pmd(struct page_vma_mapped_walk *pvmw, struct page *new)
mlock_vma_page(new);
update_mmu_cache_pmd(vma, address, pvmw->pmd);
}
-#endif
diff --git a/mm/migrate.c b/mm/migrate.c
index 507cf9ba21bf..cb9c3af32614 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -220,14 +220,12 @@ static bool remove_migration_pte(struct page *page, struct vm_area_struct *vma,
new = page - pvmw.page->index +
linear_page_index(vma, pvmw.address);
-#ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION
/* PMD-mapped THP migration entry */
- if (!pvmw.pte) {
+ if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) && !pvmw.pte) {
VM_BUG_ON_PAGE(PageHuge(page) || !PageTransCompound(page), page);
remove_migration_pmd(&pvmw, new);
continue;
}
-#endif
get_page(new);
pte = pte_mkold(mk_pte(new, READ_ONCE(vma->vm_page_prot)));
@@ -353,7 +351,7 @@ void migration_entry_wait_huge(struct vm_area_struct *vma,
__migration_entry_wait(mm, pte, ptl);
}
-#ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION
+#ifdef CONFIG_TRANSPARENT_HUGEPAGE
void pmd_migration_entry_wait(struct mm_struct *mm, pmd_t *pmd)
{
spinlock_t *ptl;
diff --git a/mm/rmap.c b/mm/rmap.c
index 8d5337fed37b..f5434f4f3e06 100644
--- a/mm/rmap.c
+++ b/mm/rmap.c
@@ -1369,15 +1369,14 @@ static bool try_to_unmap_one(struct page *page, struct vm_area_struct *vma,
mmu_notifier_invalidate_range_start(vma->vm_mm, start, end);
while (page_vma_mapped_walk(&pvmw)) {
-#ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION
/* PMD-mapped THP migration entry */
- if (!pvmw.pte && (flags & TTU_MIGRATION)) {
+ if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) &&
+ !pvmw.pte && (flags & TTU_MIGRATION)) {
VM_BUG_ON_PAGE(PageHuge(page) || !PageTransCompound(page), page);
set_pmd_migration_entry(&pvmw, page);
continue;
}
-#endif
/*
* If the page is mlock()d, we cannot swap it out.
--
2.17.0
From: Zi Yan <[email protected]>
Signed-off-by: Zi Yan <[email protected]>
Cc: Ralf Baechle <[email protected]>
Cc: James Hogan <[email protected]>
Cc: Michal Hocko <[email protected]>
Cc: Ingo Molnar <[email protected]>
Cc: Andrew Morton <[email protected]>
Cc: [email protected]
Cc: [email protected]
---
arch/mips/include/asm/pgtable-64.h | 2 ++
1 file changed, 2 insertions(+)
diff --git a/arch/mips/include/asm/pgtable-64.h b/arch/mips/include/asm/pgtable-64.h
index 0036ea0c7173..ec72e5b12965 100644
--- a/arch/mips/include/asm/pgtable-64.h
+++ b/arch/mips/include/asm/pgtable-64.h
@@ -366,6 +366,8 @@ static inline pte_t mk_swap_pte(unsigned long type, unsigned long offset)
#define __swp_offset(x) ((x).val >> 24)
#define __swp_entry(type, offset) ((swp_entry_t) { pte_val(mk_swap_pte((type), (offset))) })
#define __pte_to_swp_entry(pte) ((swp_entry_t) { pte_val(pte) })
+#define __pmd_to_swp_entry(pmd) ((swp_entry_t) { pmd_val(pmd) })
#define __swp_entry_to_pte(x) ((pte_t) { (x).val })
+#define __swp_entry_to_pmd(x) ((pmd_t) { (x).val })
#endif /* _ASM_PGTABLE_64_H */
--
2.17.0
On Thu, 26 Apr 2018 10:27:55 -0400
Zi Yan <[email protected]> wrote:
> From: Zi Yan <[email protected]>
>
> Hi all,
>
> THP migration is only enabled on x86_64 with a special
> ARCH_ENABLE_THP_MIGRATION macro. This patchset enables THP migration for
> all architectures that uses transparent hugepage, so that special macro can
> be dropped. Instead, THP migration is enabled/disabled via
> /sys/kernel/mm/transparent_hugepage/enable_thp_migration.
>
> I grepped for TRANSPARENT_HUGEPAGE in arch folder and got 9 architectures that
> are supporting transparent hugepage. I mechanically add __pmd_to_swp_entry() and
> __swp_entry_to_pmd() based on existing __pte_to_swp_entry() and
> __swp_entry_to_pte() for all these architectures, except tile which is going to
> be dropped.
This will not work on s390, the pmd layout is very different from the pte
layout. Using __swp_entry/type/offset() on a pmd will go horribly wrong.
I currently don't see a chance to make this work for us, so please make/keep
this configurable, and do not configure it for s390.
Regards,
Gerald