This series enables THP migration on riscv via ARCH_ENABLE_THP_MIGRATION.
But first this adjusts PAGE_PROT_NONE to satisfy generic memory semantics
like the behavior of pmd_present() and pmd_trans_huge() when in
THP splitting or migration.
This feature can reduce the time of THP migration by not splits THP
before migration and can guarantee the pages after migration are still
contiguous.[1]
I have tested the below test case on qemu based on riscv after
enabling this feature, the throughput of THP migration gains 13x
performance improvement:
https://github.com/x-y-z/thp-migration-bench
I also have tested and passed the test cases under
tools/testing/selftests/vm.
[1]: https://lwn.net/Articles/723764/
Nanyong Sun (2):
riscv/mm: Adjust PAGE_PROT_NONE to comply with THP semantics
riscv/mm: Enable THP migration
arch/riscv/Kconfig | 1 +
arch/riscv/include/asm/pgtable-bits.h | 2 +-
arch/riscv/include/asm/pgtable.h | 16 +++++++++++-----
3 files changed, 13 insertions(+), 6 deletions(-)
--
2.25.1
This is a preparation for enabling THP migration.
As the commit b65399f6111b("arm64/mm: Change THP helpers
to comply with generic MM semantics") mentioned, pmd_present()
and pmd_trans_huge() are expected to behave in the following
manner:
-------------------------------------------------------------------------
| PMD states | pmd_present | pmd_trans_huge |
-------------------------------------------------------------------------
| Mapped | Yes | Yes |
-------------------------------------------------------------------------
| Splitting | Yes | Yes |
-------------------------------------------------------------------------
| Migration/Swap | No | No |
-------------------------------------------------------------------------
At present the PROT_NONE bit reuses the READ bit could not comply with
above semantics with two problems:
1. When splitting a PMD THP, PMD is first invalidated with
pmdp_invalidate()->pmd_mkinvalid(), which clears the PRESENT bit
and PROT_NONE bit/READ bit, if the PMD is read-only, then the PAGE_LEAF
property is also cleared, which results in pmd_present() return false.
2. When migrating, the swap entry only clear the PRESENT bit
and PROT_NONE bit/READ bit, the W/X bit may be set, so _PAGE_LEAF may be
true which results in pmd_present() return true.
Solution:
Adjust PROT_NONE bit from READ to GLOBAL bit can satisfy the above rules:
1. GLOBAL bit has no other meanings, not like the R/W/X bit, which is
also relative with _PAGE_LEAF property.
2. GLOBAL bit is at bit 5, making swap entry start from bit 6, bit 0-5
are zero, which means the PRESENT, PROT_NONE, and PAGE_LEAF are
all false, then the pmd_present() and pmd_trans_huge() return false when
in migration/swap.
Signed-off-by: Nanyong Sun <[email protected]>
---
arch/riscv/include/asm/pgtable-bits.h | 2 +-
arch/riscv/include/asm/pgtable.h | 11 ++++++-----
2 files changed, 7 insertions(+), 6 deletions(-)
diff --git a/arch/riscv/include/asm/pgtable-bits.h b/arch/riscv/include/asm/pgtable-bits.h
index 2ee413912926..a6b0c89824c2 100644
--- a/arch/riscv/include/asm/pgtable-bits.h
+++ b/arch/riscv/include/asm/pgtable-bits.h
@@ -31,7 +31,7 @@
* _PAGE_PROT_NONE is set on not-present pages (and ignored by the hardware) to
* distinguish them from swapped out pages
*/
-#define _PAGE_PROT_NONE _PAGE_READ
+#define _PAGE_PROT_NONE _PAGE_GLOBAL
#define _PAGE_PFN_SHIFT 10
diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h
index bf204e7c1f74..8bb2010f9a25 100644
--- a/arch/riscv/include/asm/pgtable.h
+++ b/arch/riscv/include/asm/pgtable.h
@@ -119,7 +119,7 @@
/* Page protection bits */
#define _PAGE_BASE (_PAGE_PRESENT | _PAGE_ACCESSED | _PAGE_USER)
-#define PAGE_NONE __pgprot(_PAGE_PROT_NONE)
+#define PAGE_NONE __pgprot(_PAGE_PROT_NONE | _PAGE_READ)
#define PAGE_READ __pgprot(_PAGE_BASE | _PAGE_READ)
#define PAGE_WRITE __pgprot(_PAGE_BASE | _PAGE_READ | _PAGE_WRITE)
#define PAGE_EXEC __pgprot(_PAGE_BASE | _PAGE_EXEC)
@@ -628,11 +628,12 @@ static inline pmd_t pmdp_establish(struct vm_area_struct *vma,
*
* Format of swap PTE:
* bit 0: _PAGE_PRESENT (zero)
- * bit 1: _PAGE_PROT_NONE (zero)
- * bits 2 to 6: swap type
- * bits 7 to XLEN-1: swap offset
+ * bit 1 to 3: _PAGE_LEAF (zero)
+ * bit 5: _PAGE_PROT_NONE (zero)
+ * bits 6 to 10: swap type
+ * bits 10 to XLEN-1: swap offset
*/
-#define __SWP_TYPE_SHIFT 2
+#define __SWP_TYPE_SHIFT 6
#define __SWP_TYPE_BITS 5
#define __SWP_TYPE_MASK ((1UL << __SWP_TYPE_BITS) - 1)
#define __SWP_OFFSET_SHIFT (__SWP_TYPE_BITS + __SWP_TYPE_SHIFT)
--
2.25.1
Hi Palmer,
This patchset has been sent for one month ago, I was wondering if you could
have a review and give a feedback please, thanks.
On 2021/11/23 22:06, Nanyong Sun wrote:
> This series enables THP migration on riscv via ARCH_ENABLE_THP_MIGRATION.
> But first this adjusts PAGE_PROT_NONE to satisfy generic memory semantics
> like the behavior of pmd_present() and pmd_trans_huge() when in
> THP splitting or migration.
>
> This feature can reduce the time of THP migration by not splits THP
> before migration and can guarantee the pages after migration are still
> contiguous.[1]
>
> I have tested the below test case on qemu based on riscv after
> enabling this feature, the throughput of THP migration gains 13x
> performance improvement:
> https://github.com/x-y-z/thp-migration-bench
>
> I also have tested and passed the test cases under
> tools/testing/selftests/vm.
>
> [1]: https://lwn.net/Articles/723764/
>
> Nanyong Sun (2):
> riscv/mm: Adjust PAGE_PROT_NONE to comply with THP semantics
> riscv/mm: Enable THP migration
>
> arch/riscv/Kconfig | 1 +
> arch/riscv/include/asm/pgtable-bits.h | 2 +-
> arch/riscv/include/asm/pgtable.h | 16 +++++++++++-----
> 3 files changed, 13 insertions(+), 6 deletions(-)
>
On Wed, 29 Dec 2021 22:45:57 PST (-0800), [email protected] wrote:
> Hi Palmer,
>
> This patchset has been sent for one month ago, I was wondering if you could
>
> have a review and give a feedback please, thanks.
Sorry for being slow, the queue got long over the holidays and I'm still
trying to dig out. I was kind of worried about losing some bits, but I
don't think it's actually that big of a deal -- it's not like there's
really any user ABI here, aside from being able to use less swap. If
someone wants to fix that I'm all ears, but I don't think it's worth
waiting.
This is on for-next.
Thanks!
> On 2021/11/23 22:06, Nanyong Sun wrote:
>> This series enables THP migration on riscv via ARCH_ENABLE_THP_MIGRATION.
>> But first this adjusts PAGE_PROT_NONE to satisfy generic memory semantics
>> like the behavior of pmd_present() and pmd_trans_huge() when in
>> THP splitting or migration.
>>
>> This feature can reduce the time of THP migration by not splits THP
>> before migration and can guarantee the pages after migration are still
>> contiguous.[1]
>>
>> I have tested the below test case on qemu based on riscv after
>> enabling this feature, the throughput of THP migration gains 13x
>> performance improvement:
>> https://github.com/x-y-z/thp-migration-bench
>>
>> I also have tested and passed the test cases under
>> tools/testing/selftests/vm.
>>
>> [1]: https://lwn.net/Articles/723764/
>>
>> Nanyong Sun (2):
>> riscv/mm: Adjust PAGE_PROT_NONE to comply with THP semantics
>> riscv/mm: Enable THP migration
>>
>> arch/riscv/Kconfig | 1 +
>> arch/riscv/include/asm/pgtable-bits.h | 2 +-
>> arch/riscv/include/asm/pgtable.h | 16 +++++++++++-----
>> 3 files changed, 13 insertions(+), 6 deletions(-)
>>