This patchset implements preserved-over-kexec memory storage or PKRAM as a
method for saving memory pages of the currently executing kernel so that
they may be restored after kexec into a new kernel. The patches are adapted
from an RFC patchset sent out in 2013 by Vladimir Davydov [1]. They
introduce the PKRAM kernel API and implement its use within tmpfs, allowing
tmpfs files to be preserved across kexec.
One use case for PKRAM is preserving guest memory and/or auxillary supporting
data (e.g. iommu data) across kexec in support of VMM Fast Restart[2].
VMM Fast Restart is currently using PKRAM to support preserving "Keep Alive
State" across reboot[3]. PKRAM provides a flexible way for doing this
without requiring that the amount of memory used by a fixed size created
a priori. Another use case is for databases to preserve their block caches
in shared memory across reboot.
Changes since RFC v1
- Rebased onto 5.12-rc4
- Refined the API to reduce the number of calls
and better support multithreading.
- Allow preserving byte data of arbitrary length
(was previously limited to one page).
- Build a new memblock reserved list with the
preserved ranges and then substitute it for
the existing one. (Mike Rapoport)
- Use mem_avoid_overlap() to avoid kaslr stepping
on preserved ranges. (Kees Cook)
-- Usage --
1) Mount tmpfs with 'pkram=NAME' option.
NAME is an arbitrary string specifying a preserved memory node.
Different tmpfs trees may be saved to PKRAM if different names are
passed.
# mkdir -p /mnt
# mount -t tmpfs -o pkram=mytmpfs none /mnt
2) Populate a file under /mnt
# head -c 2G /dev/urandom > /mnt/testfile
# md5sum /mnt/testfile
e281e2f019ac3bfa3bdb28aa08c4beb3 /mnt/testfile
3) Remount tmpfs to preserve files.
# mount -o remount,preserve,ro /mnt
4) Load the new kernel image.
Pass the PKRAM super block pfn via 'pkram' boot option. The pfn is
exported via the sysfs file /sys/kernel/pkram.
# kexec -s -l /boot/vmlinuz-$kernel --initrd=/boot/initramfs-$kernel.img \
--append="$(cat /proc/cmdline|sed -e 's/pkram=[^ ]*//g') pkram=$(cat /sys/kernel/pkram)"
5) Boot to the new kernel.
# systemctl kexec
6) Mount tmpfs with 'pkram=NAME' option.
It should find the PKRAM node with the tmpfs tree saved on previous
unmount and restore it.
# mount -t tmpfs -o pkram=mytmpfs none /mnt
7) Use the restored file under /mnt
# md5sum /mnt/testfile
e281e2f019ac3bfa3bdb28aa08c4beb3 /mnt/testfile
-- Implementation details --
* When a tmpfs filesystem is mounted the first time with the 'pkram=NAME'
option, a shmem_pkram_info is allocated to record NAME. The shmem_pkram_info
and whether the filesystem is in the preserved state are tracked by
shmem_sb_info.
* A PKRAM-enabled tmpfs filesystem is saved to PKRAM on remount when the
'preserve' mount option is specified and the filesystem is read-only.
* Saving a file to PKRAM is done by walking the pages of the file and
building a list of the pages and attributes needed to restore them later.
The pages containing this metadata as well as the target file pages have
their refcount incremented to prevent them from being freed even after
the last user puts the pages (i.e. the filesystem is unmounted).
* To aid in quickly finding contiguous ranges of memory containing
preserved pages a pseudo physical mapping pagetable is populated
with pages as they are preserved.
* If a page to be preserved is found to be in range of memory that was
previously reserved during early boot or in range of memory where the
kernel will be loaded to on kexec, the page will be copied to a page
outside of those ranges and the new page will be preserved. A compound
page will be copied to and preserved as individual base pages.
* A single page is allocated for the PKRAM super block. For the next kernel
kexec boot to find preserved memory metadata, the pfn of the PKRAM super
block, which is exported via /sys/kernel/pkram, is passed in the 'pkram'
boot option.
* In the newly booted kernel, PKRAM adds all preserved pages to the memblock
reserve list during early boot so that they will not be recycled.
* Since kexec may load the new kernel code to any memory region, it could
destroy preserved memory. When the kernel selects the memory region
(kexec_file_load syscall), kexec will avoid preserved pages. When the
user selects the kexec memory region to use (kexec_load syscall) , kexec
load will fail if there is conflict with preserved pages. Pages preserved
after a kexec kernel is loaded will be relocated if they conflict with
the selected memory region.
The current implementation has some restrictions:
* Only regular tmpfs files without multiple hard links can be preserved.
Save to PKRAM will abort and log an error if a directory or other file
type is encountered.
* Pages for PKRAM-enabled files are prevented from swapping out to avoid
the performance penalty of swapping in and the possibility of insufficient
memory.
-- Patches --
The patches are broken down into the following groups:
Patches 1-22 implement the API and supporting functionality.
Patches 23-27 implement the use of PKRAM within tmpfs
The remaining patches implement optimizations to the initialization of
preserved pages and to the preservation and restoration of shmem pages.
To give an idea of the improvement in performance here is an example
comparison with and without these patches when saving and loading a 100G
file:
Save a 100G file:
| No optimizations | Optimized (16 cpus) |
------------------------------------------------------
huge=never | 2265ms | 232ms |
------------------------------------------------------
huge=always | 58ms | 22ms |
Load a 100G file:
| No optimizations | Optimized (16 cpus) |
------------------------------------------------------
huge=never | 8833ms | 516ms |
------------------------------------------------------
huge=always | 752ms | 105ms |
Patches 28-31 Defer initialization of page structs for preserved pages
Patches 32-34 Implement multi-threading of shmem page preservation and
restoration.
Patches 35-37 Implement and use an API for inserting shmem pages in bulk
Patches 38-39: Reduce contention on the LRU lock by staging and adding pages
in bulk to the LRU
Patches 40-43: Reduce contention on the pagecache xarray lock by inserting
pages in bulk in certain cases
[1] https://lkml.org/lkml/2013/7/1/211
[2] https://www.youtube.com/watch?v=pBsHnf93tcQ
https://static.sched.com/hosted_files/kvmforum2019/66/VMM-fast-restart_kvmforum2019.pdf
[3] https://www.youtube.com/watch?v=pBsHnf93tcQ
https://static.sched.com/hosted_files/kvmforum2020/10/Device-Keepalive-State-KVMForum2020.pdf
Anthony Yznaga (43):
mm: add PKRAM API stubs and Kconfig
mm: PKRAM: implement node load and save functions
mm: PKRAM: implement object load and save functions
mm: PKRAM: implement page stream operations
mm: PKRAM: support preserving transparent hugepages
mm: PKRAM: implement byte stream operations
mm: PKRAM: link nodes by pfn before reboot
mm: PKRAM: introduce super block
PKRAM: track preserved pages in a physical mapping pagetable
PKRAM: pass a list of preserved ranges to the next kernel
PKRAM: prepare for adding preserved ranges to memblock reserved
mm: PKRAM: reserve preserved memory at boot
PKRAM: free the preserved ranges list
PKRAM: prevent inadvertent use of a stale superblock
PKRAM: provide a way to ban pages from use by PKRAM
kexec: PKRAM: prevent kexec clobbering preserved pages in some cases
PKRAM: provide a way to check if a memory range has preserved pages
kexec: PKRAM: avoid clobbering already preserved pages
mm: PKRAM: allow preserved memory to be freed from userspace
PKRAM: disable feature when running the kdump kernel
x86/KASLR: PKRAM: support physical kaslr
x86/boot/compressed/64: use 1GB pages for mappings
mm: shmem: introduce shmem_insert_page
mm: shmem: enable saving to PKRAM
mm: shmem: prevent swapping of PKRAM-enabled tmpfs pages
mm: shmem: specify the mm to use when inserting pages
mm: shmem: when inserting, handle pages already charged to a memcg
x86/mm/numa: add numa_isolate_memblocks()
PKRAM: ensure memblocks with preserved pages init'd for numa
memblock: PKRAM: mark memblocks that contain preserved pages
memblock, mm: defer initialization of preserved pages
shmem: preserve shmem files a chunk at a time
PKRAM: atomically add and remove link pages
shmem: PKRAM: multithread preserving and restoring shmem pages
shmem: introduce shmem_insert_pages()
PKRAM: add support for loading pages in bulk
shmem: PKRAM: enable bulk loading of preserved pages into shmem
mm: implement splicing a list of pages to the LRU
shmem: optimize adding pages to the LRU in shmem_insert_pages()
shmem: initial support for adding multiple pages to pagecache
XArray: add xas_export_node() and xas_import_node()
shmem: reduce time holding xa_lock when inserting pages
PKRAM: improve index alignment of pkram_link entries
documentation/core-api/xarray.rst | 8 +
arch/x86/boot/compressed/Makefile | 3 +
arch/x86/boot/compressed/ident_map_64.c | 9 +-
arch/x86/boot/compressed/kaslr.c | 10 +-
arch/x86/boot/compressed/misc.h | 10 +
arch/x86/boot/compressed/pkram.c | 109 ++
arch/x86/include/asm/numa.h | 4 +
arch/x86/kernel/setup.c | 3 +
arch/x86/mm/init_64.c | 2 +
arch/x86/mm/numa.c | 32 +-
include/linux/memblock.h | 6 +
include/linux/mm.h | 2 +-
include/linux/pkram.h | 120 ++
include/linux/shmem_fs.h | 28 +
include/linux/swap.h | 13 +
include/linux/xarray.h | 2 +
kernel/kexec.c | 9 +
kernel/kexec_core.c | 3 +
kernel/kexec_file.c | 15 +
lib/test_xarray.c | 45 +
lib/xarray.c | 100 ++
mm/Kconfig | 9 +
mm/Makefile | 1 +
mm/memblock.c | 11 +-
mm/page_alloc.c | 55 +-
mm/pkram.c | 1808 +++++++++++++++++++++++++++++++
mm/pkram_pagetable.c | 376 +++++++
mm/shmem.c | 494 ++++++++-
mm/shmem_pkram.c | 530 +++++++++
mm/swap.c | 86 ++
30 files changed, 3869 insertions(+), 34 deletions(-)
create mode 100644 arch/x86/boot/compressed/pkram.c
create mode 100644 include/linux/pkram.h
create mode 100644 mm/pkram.c
create mode 100644 mm/pkram_pagetable.c
create mode 100644 mm/shmem_pkram.c
--
1.8.3.1
In order to facilitate fast initialization of page structs for
preserved pages, memblocks with preserved pages must not cross
numa node boundaries and must have a node id assigned to them.
Signed-off-by: Anthony Yznaga <[email protected]>
---
mm/pkram.c | 9 +++++++++
1 file changed, 9 insertions(+)
diff --git a/mm/pkram.c b/mm/pkram.c
index aea069cc49be..b8d6b549fa6c 100644
--- a/mm/pkram.c
+++ b/mm/pkram.c
@@ -21,6 +21,7 @@
#include <linux/sysfs.h>
#include <linux/types.h>
+#include <asm/numa.h>
#include "internal.h"
#define PKRAM_MAGIC 0x706B726D
@@ -226,6 +227,14 @@ void __init pkram_reserve(void)
return;
}
+ /*
+ * Fix up the reserved memblock list to ensure the
+ * memblock regions are split along node boundaries
+ * and have a node ID set. This will allow the page
+ * structs for the preserved pages to be initialized
+ * more efficiently.
+ */
+ numa_isolate_memblocks();
done:
pr_info("PKRAM: %lu pages reserved\n", pkram_reserved_pages);
}
--
1.8.3.1
Calling memblock_reserve() repeatedly to add preserved ranges is
inefficient and risks clobbering preserved memory if the memblock
reserved regions array must be resized. Instead, calculate the size
needed to accomodate the preserved ranges, find a suitable range for
a new reserved regions array that does not overlap any preserved range,
and populate it with a new, merged regions array.
Signed-off-by: Anthony Yznaga <[email protected]>
---
mm/pkram.c | 241 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 241 insertions(+)
diff --git a/mm/pkram.c b/mm/pkram.c
index 4cfa236a4126..b4a14837946a 100644
--- a/mm/pkram.c
+++ b/mm/pkram.c
@@ -7,6 +7,7 @@
#include <linux/kernel.h>
#include <linux/kobject.h>
#include <linux/list.h>
+#include <linux/memblock.h>
#include <linux/mm.h>
#include <linux/module.h>
#include <linux/mutex.h>
@@ -1121,3 +1122,243 @@ static unsigned long pkram_populate_regions_list(void)
return priv.nr_regions;
}
+
+struct pkram_region *pkram_first_region(struct pkram_super_block *sb, struct pkram_region_list **rlp, int *idx)
+{
+ WARN_ON(!sb);
+ WARN_ON(!sb->region_list_pfn);
+
+ if (!sb || !sb->region_list_pfn)
+ return NULL;
+
+ *rlp = pfn_to_kaddr(sb->region_list_pfn);
+ *idx = 0;
+
+ return &(*rlp)->regions[0];
+}
+
+struct pkram_region *pkram_next_region(struct pkram_region_list **rlp, int *idx)
+{
+ struct pkram_region_list *rl = *rlp;
+ int i = *idx;
+
+ i++;
+ if (i >= PKRAM_REGIONS_LIST_MAX) {
+ if (!rl->next_pfn) {
+ pr_err("PKRAM: %s: no more pkram_region_list pages\n", __func__);
+ return NULL;
+ }
+ rl = pfn_to_kaddr(rl->next_pfn);
+ *rlp = rl;
+ i = 0;
+ }
+ *idx = i;
+
+ if (rl->regions[i].size == 0)
+ return NULL;
+
+ return &rl->regions[i];
+}
+
+struct pkram_region *pkram_first_region_topdown(struct pkram_super_block *sb, struct pkram_region_list **rlp, int *idx)
+{
+ struct pkram_region_list *rl;
+
+ WARN_ON(!sb);
+ WARN_ON(!sb->region_list_pfn);
+
+ if (!sb || !sb->region_list_pfn)
+ return NULL;
+
+ rl = pfn_to_kaddr(sb->region_list_pfn);
+ if (!rl->prev_pfn) {
+ WARN_ON(1);
+ return NULL;
+ }
+ rl = pfn_to_kaddr(rl->prev_pfn);
+
+ *rlp = rl;
+
+ *idx = (sb->nr_regions - 1) % PKRAM_REGIONS_LIST_MAX;
+
+ return &rl->regions[*idx];
+}
+
+struct pkram_region *pkram_next_region_topdown(struct pkram_region_list **rlp, int *idx)
+{
+ struct pkram_region_list *rl = *rlp;
+ int i = *idx;
+
+ if (i == 0) {
+ if (!rl->prev_pfn)
+ return NULL;
+ rl = pfn_to_kaddr(rl->prev_pfn);
+ *rlp = rl;
+ i = PKRAM_REGIONS_LIST_MAX - 1;
+ } else
+ i--;
+
+ *idx = i;
+
+ return &rl->regions[i];
+}
+
+/*
+ * Use the pkram regions list to find an available block of memory that does
+ * not overlap with preserved pages.
+ */
+phys_addr_t __init find_available_topdown(phys_addr_t size)
+{
+ phys_addr_t hole_start, hole_end, hole_size;
+ struct pkram_region_list *rl;
+ struct pkram_region *r;
+ phys_addr_t addr = 0;
+ int idx;
+
+ hole_end = memblock.current_limit;
+ r = pkram_first_region_topdown(pkram_sb, &rl, &idx);
+
+ while (r) {
+ hole_start = r->base + r->size;
+ hole_size = hole_end - hole_start;
+
+ if (hole_size >= size) {
+ addr = memblock_find_in_range(hole_start, hole_end,
+ size, PAGE_SIZE);
+ if (addr)
+ break;
+ }
+
+ hole_end = r->base;
+ r = pkram_next_region_topdown(&rl, &idx);
+ }
+
+ if (!addr)
+ addr = memblock_find_in_range(0, hole_end, size, PAGE_SIZE);
+
+ return addr;
+}
+
+int __init pkram_create_merged_reserved(struct memblock_type *new)
+{
+ unsigned long cnt_a;
+ unsigned long cnt_b;
+ long i, j, k;
+ struct memblock_region *r;
+ struct memblock_region *rgn;
+ struct pkram_region *pkr;
+ struct pkram_region_list *rl;
+ int idx;
+ unsigned long total_size = 0;
+ unsigned long nr_preserved = 0;
+
+ cnt_a = memblock.reserved.cnt;
+ cnt_b = pkram_sb->nr_regions;
+
+ i = 0;
+ j = 0;
+ k = 0;
+
+ pkr = pkram_first_region(pkram_sb, &rl, &idx);
+ if (!pkr)
+ return -EINVAL;
+ while (i < cnt_a && j < cnt_b && pkr) {
+ r = &memblock.reserved.regions[i];
+ rgn = &new->regions[k];
+
+ if (r->base + r->size <= pkr->base) {
+ *rgn = *r;
+ i++;
+ } else if (pkr->base + pkr->size <= r->base) {
+ rgn->base = pkr->base;
+ rgn->size = pkr->size;
+ memblock_set_region_node(rgn, MAX_NUMNODES);
+
+ nr_preserved += (rgn->size >> PAGE_SHIFT);
+ pkr = pkram_next_region(&rl, &idx);
+ j++;
+ } else {
+ pr_err("PKRAM: unexpected overlap:\n");
+ pr_err("PKRAM: reserved: base=%pa,size=%pa,flags=0x%x\n", &r->base, &r->size, (int)r->flags);
+ pr_err("PKRAM: pkram: base=%pa,size=%pa\n", &pkr->base, &pkr->size);
+ return -EBUSY;
+ }
+ total_size += rgn->size;
+ k++;
+ }
+
+ while (i < cnt_a) {
+ r = &memblock.reserved.regions[i];
+ rgn = &new->regions[k];
+
+ *rgn = *r;
+
+ total_size += rgn->size;
+ i++;
+ k++;
+ }
+ while (j < cnt_b && pkr) {
+ rgn = &new->regions[k];
+ rgn->base = pkr->base;
+ rgn->size = pkr->size;
+ memblock_set_region_node(rgn, MAX_NUMNODES);
+
+ nr_preserved += (rgn->size >> PAGE_SHIFT);
+ total_size += rgn->size;
+ pkr = pkram_next_region(&rl, &idx);
+ j++;
+ k++;
+ }
+
+ WARN_ON(cnt_a + cnt_b != k);
+ new->cnt = cnt_a + cnt_b;
+ new->total_size = total_size;
+
+ return 0;
+}
+
+/*
+ * Reserve pages that belong to preserved memory. This is accomplished by
+ * merging the existing reserved ranges with the preserved ranges into
+ * a new, sufficiently sized memblock reserved array.
+ *
+ * This function should be called at boot time as early as possible to prevent
+ * preserved memory from being recycled.
+ */
+int __init pkram_merge_with_reserved(void)
+{
+ struct memblock_type new;
+ unsigned long new_max;
+ phys_addr_t new_size;
+ phys_addr_t addr;
+ int err;
+
+ /*
+ * Need space to insert one more range into memblock.reserved
+ * without memblock_double_array() being called.
+ */
+ if (memblock.reserved.cnt == memblock.reserved.max) {
+ WARN_ONCE(1, "PKRAM: no space for new memblock list\n");
+ return -ENOMEM;
+ }
+
+ new_max = memblock.reserved.max + pkram_sb->nr_regions;
+ new_size = PAGE_ALIGN(sizeof (struct memblock_region) * new_max);
+
+ addr = find_available_topdown(new_size);
+ if (!addr || memblock_reserve(addr, new_size))
+ return -ENOMEM;
+
+ new.regions = __va(addr);
+ new.max = new_max;
+ err = pkram_create_merged_reserved(&new);
+ if (err)
+ return err;
+
+ memblock.reserved.cnt = new.cnt;
+ memblock.reserved.max = new.max;
+ memblock.reserved.total_size = new.total_size;
+ memblock.reserved.regions = new.regions;
+
+ return 0;
+}
--
1.8.3.1
To prepare for multithreading the work to preserve a shmem file,
divide the work into subranges of the total index range of the file.
The chunk size is a rather arbitrary 256k indices.
Signed-off-by: Anthony Yznaga <[email protected]>
---
mm/shmem_pkram.c | 64 +++++++++++++++++++++++++++++++++++++++++++++++++-------
1 file changed, 57 insertions(+), 7 deletions(-)
diff --git a/mm/shmem_pkram.c b/mm/shmem_pkram.c
index 8682b0c002c0..e52722b3a709 100644
--- a/mm/shmem_pkram.c
+++ b/mm/shmem_pkram.c
@@ -74,16 +74,14 @@ static int save_page(struct page *page, struct pkram_access *pa)
return err;
}
-static int save_file_content(struct pkram_stream *ps, struct address_space *mapping)
+static int save_file_content_range(struct pkram_access *pa,
+ struct address_space *mapping,
+ unsigned long start, unsigned long end)
{
- PKRAM_ACCESS(pa, ps, pages);
struct pagevec pvec;
- unsigned long start, end;
int err = 0;
int i;
- start = 0;
- end = DIV_ROUND_UP(i_size_read(mapping->host), PAGE_SIZE);
pagevec_init(&pvec);
for ( ; ; ) {
pvec.nr = find_get_pages_range(mapping, &start, end,
@@ -95,7 +93,7 @@ static int save_file_content(struct pkram_stream *ps, struct address_space *mapp
lock_page(page);
BUG_ON(page->mapping != mapping);
- err = save_page(page, &pa);
+ err = save_page(page, pa);
if (PageCompound(page)) {
start = page->index + compound_nr(page);
i += compound_nr(page);
@@ -113,10 +111,62 @@ static int save_file_content(struct pkram_stream *ps, struct address_space *mapp
cond_resched();
}
- pkram_finish_access(&pa, err == 0);
return err;
}
+struct shmem_pkram_arg {
+ struct pkram_stream *ps;
+ struct address_space *mapping;
+ struct mm_struct *mm;
+ atomic64_t next;
+};
+
+unsigned long shmem_pkram_max_index_range = 512 * 512;
+
+static int get_save_range(unsigned long max, atomic64_t *next, unsigned long *start, unsigned long *end)
+{
+ unsigned long index;
+
+ index = atomic64_fetch_add(shmem_pkram_max_index_range, next);
+ if (index >= max)
+ return -ENODATA;
+
+ *start = index;
+ *end = index + shmem_pkram_max_index_range - 1;
+
+ return 0;
+}
+
+static int do_save_file_content(struct pkram_stream *ps,
+ struct address_space *mapping,
+ atomic64_t *next)
+{
+ PKRAM_ACCESS(pa, ps, pages);
+ unsigned long start, end, max;
+ int ret;
+
+ max = DIV_ROUND_UP(i_size_read(mapping->host), PAGE_SIZE);
+
+ do {
+ ret = get_save_range(max, next, &start, &end);
+ if (!ret)
+ ret = save_file_content_range(&pa, mapping, start, end);
+ } while (!ret);
+
+ if (ret == -ENODATA)
+ ret = 0;
+
+ pkram_finish_access(&pa, ret == 0);
+ return ret;
+}
+
+static int save_file_content(struct pkram_stream *ps, struct address_space *mapping)
+{
+ struct shmem_pkram_arg arg = { ps, mapping, NULL, ATOMIC64_INIT(0) };
+
+ return do_save_file_content(arg.ps, arg.mapping, &arg.next);
+}
+
static int save_file(struct dentry *dentry, struct pkram_stream *ps)
{
PKRAM_ACCESS(pa_bytes, ps, bytes);
--
1.8.3.1
Ensure destination ranges of the kexec segments do not overlap
with any kernel pages marked to be preserved across kexec.
For kexec_load, return EADDRNOTAVAIL if overlap is detected.
For kexec_file_load, skip ranges containing preserved pages when
seaching for available ranges to use.
Signed-off-by: Anthony Yznaga <[email protected]>
---
kernel/kexec_core.c | 3 +++
kernel/kexec_file.c | 5 +++++
2 files changed, 8 insertions(+)
diff --git a/kernel/kexec_core.c b/kernel/kexec_core.c
index a0b6780740c8..fda4abb865ff 100644
--- a/kernel/kexec_core.c
+++ b/kernel/kexec_core.c
@@ -37,6 +37,7 @@
#include <linux/compiler.h>
#include <linux/hugetlb.h>
#include <linux/objtool.h>
+#include <linux/pkram.h>
#include <asm/page.h>
#include <asm/sections.h>
@@ -175,6 +176,8 @@ int sanity_check_segment_list(struct kimage *image)
return -EADDRNOTAVAIL;
if (mend >= KEXEC_DESTINATION_MEMORY_LIMIT)
return -EADDRNOTAVAIL;
+ if (pkram_has_preserved_pages(mstart, mend))
+ return -EADDRNOTAVAIL;
}
/* Verify our destination addresses do not overlap.
diff --git a/kernel/kexec_file.c b/kernel/kexec_file.c
index 1ec47a3c60dd..94109bcdbeff 100644
--- a/kernel/kexec_file.c
+++ b/kernel/kexec_file.c
@@ -516,6 +516,11 @@ static int locate_mem_hole_bottom_up(unsigned long start, unsigned long end,
continue;
}
+ if (pkram_has_preserved_pages(temp_start, temp_end + 1)) {
+ temp_start = temp_start - PAGE_SIZE;
+ continue;
+ }
+
/* We found a suitable memory range */
break;
} while (1);
--
1.8.3.1
Improve performance by multithreading the work to preserve and restore
shmem pages.
When preserving pages each thread saves non-overlapping ranges of a file
to a pkram_obj until all pages are preserved.
When restoring pages each thread loads pages using a local pkram_access.
Signed-off-by: Anthony Yznaga <[email protected]>
---
mm/shmem_pkram.c | 94 +++++++++++++++++++++++++++++++++++++++++++++++++++++---
1 file changed, 89 insertions(+), 5 deletions(-)
diff --git a/mm/shmem_pkram.c b/mm/shmem_pkram.c
index e52722b3a709..354c2b58962c 100644
--- a/mm/shmem_pkram.c
+++ b/mm/shmem_pkram.c
@@ -115,6 +115,7 @@ static int save_file_content_range(struct pkram_access *pa,
}
struct shmem_pkram_arg {
+ int *error;
struct pkram_stream *ps;
struct address_space *mapping;
struct mm_struct *mm;
@@ -137,6 +138,16 @@ static int get_save_range(unsigned long max, atomic64_t *next, unsigned long *st
return 0;
}
+/* Completion tracking for save_file_content_thr() threads */
+static atomic_t pkram_save_n_undone;
+static DECLARE_COMPLETION(pkram_save_all_done_comp);
+
+static inline void pkram_save_report_one_done(void)
+{
+ if (atomic_dec_and_test(&pkram_save_n_undone))
+ complete(&pkram_save_all_done_comp);
+}
+
static int do_save_file_content(struct pkram_stream *ps,
struct address_space *mapping,
atomic64_t *next)
@@ -160,11 +171,40 @@ static int do_save_file_content(struct pkram_stream *ps,
return ret;
}
-static int save_file_content(struct pkram_stream *ps, struct address_space *mapping)
+static int save_file_content_thr(void *data)
{
- struct shmem_pkram_arg arg = { ps, mapping, NULL, ATOMIC64_INIT(0) };
-
- return do_save_file_content(arg.ps, arg.mapping, &arg.next);
+ struct shmem_pkram_arg *arg = data;
+ int ret;
+
+ ret = do_save_file_content(arg->ps, arg->mapping, &arg->next);
+ if (ret && !*arg->error)
+ *arg->error = ret;
+
+ pkram_save_report_one_done();
+ return 0;
+}
+
+static int shmem_pkram_max_threads = 16;
+
+static int save_file_content(struct pkram_stream *ps, struct address_space *mapping)
+ {
+ int err = 0;
+ struct shmem_pkram_arg arg = { &err, ps, mapping, NULL, ATOMIC64_INIT(0) };
+ unsigned int thr, nr_threads;
+
+ nr_threads = num_online_cpus() - 1;
+ nr_threads = clamp_val(shmem_pkram_max_threads, 1, nr_threads);
+
+ if (nr_threads == 1)
+ return do_save_file_content(arg.ps, arg.mapping, &arg.next);
+
+ atomic_set(&pkram_save_n_undone, nr_threads);
+ for (thr = 0; thr < nr_threads; thr++)
+ kthread_run(save_file_content_thr, &arg, "pkram_save%d", thr);
+
+ wait_for_completion(&pkram_save_all_done_comp);
+
+ return err;
}
static int save_file(struct dentry *dentry, struct pkram_stream *ps)
@@ -275,7 +315,17 @@ int shmem_save_pkram(struct super_block *sb)
return err;
}
-static int load_file_content(struct pkram_stream *ps, struct address_space *mapping, struct mm_struct *mm)
+/* Completion tracking for load_file_content_thr() threads */
+static atomic_t pkram_load_n_undone;
+static DECLARE_COMPLETION(pkram_load_all_done_comp);
+
+static inline void pkram_load_report_one_done(void)
+{
+ if (atomic_dec_and_test(&pkram_load_n_undone))
+ complete(&pkram_load_all_done_comp);
+}
+
+static int do_load_file_content(struct pkram_stream *ps, struct address_space *mapping, struct mm_struct *mm)
{
PKRAM_ACCESS(pa, ps, pages);
unsigned long index;
@@ -296,6 +346,40 @@ static int load_file_content(struct pkram_stream *ps, struct address_space *mapp
return err;
}
+static int load_file_content_thr(void *data)
+{
+ struct shmem_pkram_arg *arg = data;
+ int ret;
+
+ ret = do_load_file_content(arg->ps, arg->mapping, arg->mm);
+ if (ret && !*arg->error)
+ *arg->error = ret;
+
+ pkram_load_report_one_done();
+ return 0;
+}
+
+static int load_file_content(struct pkram_stream *ps, struct address_space *mapping, struct mm_struct *mm)
+{
+ int err = 0;
+ struct shmem_pkram_arg arg = { &err, ps, mapping, mm };
+ unsigned int thr, nr_threads;
+
+ nr_threads = num_online_cpus() - 1;
+ nr_threads = clamp_val(shmem_pkram_max_threads, 1, nr_threads);
+
+ if (nr_threads == 1)
+ return do_load_file_content(ps, mapping, mm);
+
+ atomic_set(&pkram_load_n_undone, nr_threads);
+ for (thr = 0; thr < nr_threads; thr++)
+ kthread_run(load_file_content_thr, &arg, "pkram_load%d", thr);
+
+ wait_for_completion(&pkram_load_all_done_comp);
+
+ return err;
+}
+
static int load_file(struct dentry *parent, struct pkram_stream *ps,
char *buf, size_t bufsize)
{
--
1.8.3.1
Calling shmem_insert_page() to insert one page at a time does
not scale well when multiple threads are inserting pages into
the same shmem segment. This is primarily due to the locking needed
when adding to the pagecache and LRU but also due to contention
on the shmem_inode_info lock. To address the shmem_inode_info lock
and prepare for future optimizations, introduce shmem_insert_pages()
which allows a caller to pass an array of pages to be inserted into a
shmem segment.
Signed-off-by: Anthony Yznaga <[email protected]>
---
include/linux/shmem_fs.h | 3 +-
mm/shmem.c | 93 ++++++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 95 insertions(+), 1 deletion(-)
diff --git a/include/linux/shmem_fs.h b/include/linux/shmem_fs.h
index 78149d702a62..bc116c4fe145 100644
--- a/include/linux/shmem_fs.h
+++ b/include/linux/shmem_fs.h
@@ -112,7 +112,8 @@ extern int shmem_getpage(struct inode *inode, pgoff_t index,
extern int shmem_insert_page(struct mm_struct *mm, struct inode *inode,
pgoff_t index, struct page *page);
-
+extern int shmem_insert_pages(struct mm_struct *mm, struct inode *inode,
+ pgoff_t index, struct page *pages[], int npages);
#ifdef CONFIG_PKRAM
extern int shmem_parse_pkram(const char *str, struct shmem_pkram_info **pkram);
extern void shmem_show_pkram(struct seq_file *seq, struct shmem_pkram_info *pkram,
diff --git a/mm/shmem.c b/mm/shmem.c
index 44cc158ab34d..c3fa72061d8a 100644
--- a/mm/shmem.c
+++ b/mm/shmem.c
@@ -838,6 +838,99 @@ int shmem_insert_page(struct mm_struct *mm, struct inode *inode, pgoff_t index,
return err;
}
+int shmem_insert_pages(struct mm_struct *charge_mm, struct inode *inode,
+ pgoff_t index, struct page *pages[], int npages)
+{
+ struct address_space *mapping = inode->i_mapping;
+ struct shmem_inode_info *info = SHMEM_I(inode);
+ struct shmem_sb_info *sbinfo = SHMEM_SB(inode->i_sb);
+ gfp_t gfp = mapping_gfp_mask(mapping);
+ int i, err;
+ int nr = 0;
+
+ for (i = 0; i < npages; i++)
+ nr += thp_nr_pages(pages[i]);
+
+ if (index + nr - 1 > (MAX_LFS_FILESIZE >> PAGE_SHIFT))
+ return -EFBIG;
+
+retry:
+ err = 0;
+ if (!shmem_inode_acct_block(inode, nr))
+ err = -ENOSPC;
+ if (err) {
+ int retry = 5;
+
+ /*
+ * Try to reclaim some space by splitting a huge page
+ * beyond i_size on the filesystem.
+ */
+ while (retry--) {
+ int ret;
+
+ ret = shmem_unused_huge_shrink(sbinfo, NULL, 1);
+ if (ret == SHRINK_STOP)
+ break;
+ if (ret)
+ goto retry;
+ }
+ goto failed;
+ }
+
+ for (i = 0; i < npages; i++) {
+ if (!PageLRU(pages[i])) {
+ __SetPageLocked(pages[i]);
+ __SetPageSwapBacked(pages[i]);
+ } else {
+ lock_page(pages[i]);
+ }
+
+ __SetPageReferenced(pages[i]);
+ }
+
+ for (i = 0; i < npages; i++) {
+ bool ischarged = page_memcg(pages[i]) ? true : false;
+
+ err = shmem_add_to_page_cache(pages[i], mapping, index,
+ NULL, gfp & GFP_RECLAIM_MASK,
+ charge_mm, ischarged);
+ if (err)
+ goto out_release;
+
+ index += thp_nr_pages(pages[i]);
+ }
+
+ spin_lock(&info->lock);
+ info->alloced += nr;
+ inode->i_blocks += BLOCKS_PER_PAGE * nr;
+ shmem_recalc_inode(inode);
+ spin_unlock(&info->lock);
+
+ for (i = 0; i < npages; i++) {
+ if (!PageLRU(pages[i]))
+ lru_cache_add(pages[i]);
+
+ flush_dcache_page(pages[i]);
+ SetPageUptodate(pages[i]);
+ set_page_dirty(pages[i]);
+
+ unlock_page(pages[i]);
+ }
+
+ return 0;
+
+out_release:
+ while (--i >= 0)
+ delete_from_page_cache(pages[i]);
+
+ for (i = 0; i < npages; i++)
+ unlock_page(pages[i]);
+
+ shmem_inode_unacct_blocks(inode, nr);
+failed:
+ return err;
+}
+
/*
* Remove swap entry from page cache, free the swap and its page cache.
*/
--
1.8.3.1
Make use of new interfaces for loading and inserting preserved pages
into a shmem file in bulk.
Signed-off-by: Anthony Yznaga <[email protected]>
---
mm/shmem_pkram.c | 23 +++++++++++++++++------
1 file changed, 17 insertions(+), 6 deletions(-)
diff --git a/mm/shmem_pkram.c b/mm/shmem_pkram.c
index 354c2b58962c..24a1ebb4af59 100644
--- a/mm/shmem_pkram.c
+++ b/mm/shmem_pkram.c
@@ -328,20 +328,31 @@ static inline void pkram_load_report_one_done(void)
static int do_load_file_content(struct pkram_stream *ps, struct address_space *mapping, struct mm_struct *mm)
{
PKRAM_ACCESS(pa, ps, pages);
+ struct page **pages;
+ unsigned int nr_pages;
unsigned long index;
- struct page *page;
- int err = 0;
+ int i, err;
+
+ pages = kzalloc(PKRAM_PAGES_BUFSIZE, GFP_KERNEL);
+ if (!pages)
+ return -ENOMEM;
do {
- page = pkram_load_file_page(&pa, &index);
- if (!page)
+ err = pkram_load_file_pages(&pa, pages, &nr_pages, &index);
+ if (err) {
+ if (err == -ENODATA)
+ err = 0;
break;
+ }
+
+ err = shmem_insert_pages(mm, mapping->host, index, pages, nr_pages);
- err = shmem_insert_page(mm, mapping->host, index, page);
- put_page(page);
+ for (i = 0; i < nr_pages; i++)
+ put_page(pages[i]);
cond_resched();
} while (!err);
+ kfree(pages);
pkram_finish_access(&pa, err == 0);
return err;
}
--
1.8.3.1
Preserved memory is divided into nodes which can be saved and loaded
independently of each other. PKRAM nodes are kept on a list and
identified by unique names. Whenever a save operation is initiated by
calling pkram_prepare_save(), a new node is created and linked to the
list. When the save operation has been committed by calling
pkram_finish_save(), the node becomes loadable. A load operation can be
then initiated by calling pkram_prepare_load() which deletes the node
from the list and prepares the corresponding stream for loading data
from it. After the load has been finished, the pkram_finish_load()
function must be called to free the node. Nodes are also deleted when a
save operation is discarded, i.e. pkram_discard_save() is called instead
of pkram_finish_save().
Originally-by: Vladimir Davydov <[email protected]>
Signed-off-by: Anthony Yznaga <[email protected]>
---
include/linux/pkram.h | 8 ++-
mm/pkram.c | 148 ++++++++++++++++++++++++++++++++++++++++++++++++--
2 files changed, 150 insertions(+), 6 deletions(-)
diff --git a/include/linux/pkram.h b/include/linux/pkram.h
index a575da2d6c79..01055a876450 100644
--- a/include/linux/pkram.h
+++ b/include/linux/pkram.h
@@ -6,6 +6,8 @@
#include <linux/types.h>
#include <linux/mm_types.h>
+struct pkram_node;
+
/**
* enum pkram_data_flags - definition of data types contained in a pkram obj
* @PKRAM_DATA_none: No data types configured
@@ -14,7 +16,11 @@ enum pkram_data_flags {
PKRAM_DATA_none = 0x0, /* No data types configured */
};
-struct pkram_stream;
+struct pkram_stream {
+ gfp_t gfp_mask;
+ struct pkram_node *node;
+};
+
struct pkram_access;
#define PKRAM_NAME_MAX 256 /* including nul */
diff --git a/mm/pkram.c b/mm/pkram.c
index 59e4661b2fb7..21976df6e0ea 100644
--- a/mm/pkram.c
+++ b/mm/pkram.c
@@ -2,16 +2,85 @@
#include <linux/err.h>
#include <linux/gfp.h>
#include <linux/kernel.h>
+#include <linux/list.h>
#include <linux/mm.h>
+#include <linux/mutex.h>
#include <linux/pkram.h>
+#include <linux/string.h>
#include <linux/types.h>
+/*
+ * Preserved memory is divided into nodes that can be saved or loaded
+ * independently of each other. The nodes are identified by unique name
+ * strings.
+ *
+ * The structure occupies a memory page.
+ */
+struct pkram_node {
+ __u32 flags;
+
+ __u8 name[PKRAM_NAME_MAX];
+};
+
+#define PKRAM_SAVE 1
+#define PKRAM_LOAD 2
+#define PKRAM_ACCMODE_MASK 3
+
+static LIST_HEAD(pkram_nodes); /* linked through page::lru */
+static DEFINE_MUTEX(pkram_mutex); /* serializes open/close */
+
+static inline struct page *pkram_alloc_page(gfp_t gfp_mask)
+{
+ return alloc_page(gfp_mask);
+}
+
+static inline void pkram_free_page(void *addr)
+{
+ free_page((unsigned long)addr);
+}
+
+static inline void pkram_insert_node(struct pkram_node *node)
+{
+ list_add(&virt_to_page(node)->lru, &pkram_nodes);
+}
+
+static inline void pkram_delete_node(struct pkram_node *node)
+{
+ list_del(&virt_to_page(node)->lru);
+}
+
+static struct pkram_node *pkram_find_node(const char *name)
+{
+ struct page *page;
+ struct pkram_node *node;
+
+ list_for_each_entry(page, &pkram_nodes, lru) {
+ node = page_address(page);
+ if (strcmp(node->name, name) == 0)
+ return node;
+ }
+ return NULL;
+}
+
+static void pkram_stream_init(struct pkram_stream *ps,
+ struct pkram_node *node, gfp_t gfp_mask)
+{
+ memset(ps, 0, sizeof(*ps));
+ ps->gfp_mask = gfp_mask;
+ ps->node = node;
+}
+
/**
* Create a preserved memory node with name @name and initialize stream @ps
* for saving data to it.
*
* @gfp_mask specifies the memory allocation mask to be used when saving data.
*
+ * Error values:
+ * %ENAMETOOLONG: name len >= PKRAM_NAME_MAX
+ * %ENOMEM: insufficient memory available
+ * %EEXIST: node with specified name already exists
+ *
* Returns 0 on success, -errno on failure.
*
* After the save has finished, pkram_finish_save() (or pkram_discard_save() in
@@ -19,7 +88,34 @@
*/
int pkram_prepare_save(struct pkram_stream *ps, const char *name, gfp_t gfp_mask)
{
- return -ENOSYS;
+ struct page *page;
+ struct pkram_node *node;
+ int err = 0;
+
+ if (strlen(name) >= PKRAM_NAME_MAX)
+ return -ENAMETOOLONG;
+
+ page = pkram_alloc_page(gfp_mask | __GFP_ZERO);
+ if (!page)
+ return -ENOMEM;
+ node = page_address(page);
+
+ node->flags = PKRAM_SAVE;
+ strcpy(node->name, name);
+
+ mutex_lock(&pkram_mutex);
+ if (!pkram_find_node(name))
+ pkram_insert_node(node);
+ else
+ err = -EEXIST;
+ mutex_unlock(&pkram_mutex);
+ if (err) {
+ pkram_free_page(node);
+ return err;
+ }
+
+ pkram_stream_init(ps, node, gfp_mask);
+ return 0;
}
/**
@@ -50,7 +146,12 @@ void pkram_finish_save_obj(struct pkram_stream *ps)
*/
void pkram_finish_save(struct pkram_stream *ps)
{
- BUG();
+ struct pkram_node *node = ps->node;
+
+ BUG_ON((node->flags & PKRAM_ACCMODE_MASK) != PKRAM_SAVE);
+
+ smp_wmb();
+ node->flags &= ~PKRAM_ACCMODE_MASK;
}
/**
@@ -60,7 +161,15 @@ void pkram_finish_save(struct pkram_stream *ps)
*/
void pkram_discard_save(struct pkram_stream *ps)
{
- BUG();
+ struct pkram_node *node = ps->node;
+
+ BUG_ON((node->flags & PKRAM_ACCMODE_MASK) != PKRAM_SAVE);
+
+ mutex_lock(&pkram_mutex);
+ pkram_delete_node(node);
+ mutex_unlock(&pkram_mutex);
+
+ pkram_free_page(node);
}
/**
@@ -69,11 +178,36 @@ void pkram_discard_save(struct pkram_stream *ps)
*
* Returns 0 on success, -errno on failure.
*
+ * Error values:
+ * %ENOENT: node with specified name does not exist
+ * %EBUSY: save to required node has not finished yet
+ *
* After the load has finished, pkram_finish_load() is to be called.
*/
int pkram_prepare_load(struct pkram_stream *ps, const char *name)
{
- return -ENOSYS;
+ struct pkram_node *node;
+ int err = 0;
+
+ mutex_lock(&pkram_mutex);
+ node = pkram_find_node(name);
+ if (!node) {
+ err = -ENOENT;
+ goto out_unlock;
+ }
+ if (node->flags & PKRAM_ACCMODE_MASK) {
+ err = -EBUSY;
+ goto out_unlock;
+ }
+ pkram_delete_node(node);
+out_unlock:
+ mutex_unlock(&pkram_mutex);
+ if (err)
+ return err;
+
+ node->flags |= PKRAM_LOAD;
+ pkram_stream_init(ps, node, 0);
+ return 0;
}
/**
@@ -106,7 +240,11 @@ void pkram_finish_load_obj(struct pkram_stream *ps)
*/
void pkram_finish_load(struct pkram_stream *ps)
{
- BUG();
+ struct pkram_node *node = ps->node;
+
+ BUG_ON((node->flags & PKRAM_ACCMODE_MASK) != PKRAM_LOAD);
+
+ pkram_free_page(node);
}
/**
--
1.8.3.1
Considerable contention on the LRU lock happens when multiple threads
are used to insert pages into a shmem file in parallel. To alleviate this
provide a way for pages to be added to the same LRU to be staged so that
they can be added by splicing lists and updating stats once with the lock
held. For now only unevictable pages are supported.
Signed-off-by: Anthony Yznaga <[email protected]>
---
include/linux/swap.h | 13 ++++++++
mm/swap.c | 86 ++++++++++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 99 insertions(+)
diff --git a/include/linux/swap.h b/include/linux/swap.h
index 4cc6ec3bf0ab..254c9c8d71d0 100644
--- a/include/linux/swap.h
+++ b/include/linux/swap.h
@@ -351,6 +351,19 @@ extern void lru_note_cost(struct lruvec *lruvec, bool file,
extern void lru_cache_add_inactive_or_unevictable(struct page *page,
struct vm_area_struct *vma);
+struct lru_splice {
+ struct list_head splice;
+ struct list_head *lru_head;
+ struct lruvec *lruvec;
+ enum lru_list lru;
+ unsigned long nr_pages[MAX_NR_ZONES];
+ unsigned long pgculled;
+};
+#define LRU_SPLICE_INIT(name) { .splice = LIST_HEAD_INIT(name.splice) }
+#define LRU_SPLICE(name) \
+ struct lru_splice name = LRU_SPLICE_INIT(name)
+extern void lru_splice_add(struct page *page, struct lru_splice *splice);
+extern void add_splice_to_lru_list(struct lru_splice *splice);
/* linux/mm/vmscan.c */
extern unsigned long zone_reclaimable_pages(struct zone *zone);
diff --git a/mm/swap.c b/mm/swap.c
index 31b844d4ed94..a1db6a748608 100644
--- a/mm/swap.c
+++ b/mm/swap.c
@@ -200,6 +200,92 @@ int get_kernel_page(unsigned long start, int write, struct page **pages)
}
EXPORT_SYMBOL_GPL(get_kernel_page);
+/*
+ * Update stats and move accumulated pages from an lru_splice to the lru.
+ */
+void add_splice_to_lru_list(struct lru_splice *splice)
+{
+ struct lruvec *lruvec = splice->lruvec;
+ enum lru_list lru = splice->lru;
+ unsigned long flags = 0;
+ int zid;
+
+ if (list_empty(&splice->splice))
+ return;
+
+ spin_lock_irqsave(&lruvec->lru_lock, flags);
+ for (zid = 0; zid < MAX_NR_ZONES; zid++) {
+ if (splice->nr_pages[zid])
+ update_lru_size(lruvec, lru, zid, splice->nr_pages[zid]);
+ }
+ count_vm_events(UNEVICTABLE_PGCULLED, splice->pgculled);
+ list_splice_init(&splice->splice, splice->lru_head);
+ spin_unlock_irqrestore(&lruvec->lru_lock, flags);
+}
+
+static void add_page_to_lru_splice(struct page *page, struct lru_splice *splice,
+ struct lruvec *lruvec, enum lru_list lru)
+{
+ if (list_empty(&splice->splice)) {
+ int zid;
+
+ splice->lruvec = lruvec;
+ splice->lru_head = &lruvec->lists[lru];
+ splice->lru = lru;
+ for (zid = 0; zid < MAX_NR_ZONES; zid++)
+ splice->nr_pages[zid] = 0;
+ splice->pgculled = 0;
+ }
+
+ BUG_ON(splice->lruvec != lruvec);
+ BUG_ON(splice->lru_head != &lruvec->lists[lru]);
+
+ list_add(&page->lru, &splice->splice);
+ splice->nr_pages[page_zonenum(page)] += thp_nr_pages(page);
+}
+
+/*
+ * Similar in functionality to __pagevec_lru_add_fn() but here the page is
+ * being added to an lru_splice and the LRU lock is not held.
+ */
+static void page_lru_splice_add(struct page *page, struct lru_splice *splice, struct lruvec *lruvec)
+{
+ enum lru_list lru;
+ int was_unevictable = TestClearPageUnevictable(page);
+ int nr_pages = thp_nr_pages(page);
+
+ VM_BUG_ON_PAGE(PageLRU(page), page);
+ /* XXX only supports unevictable pages at the moment */
+ VM_BUG_ON_PAGE(was_unevictable, page);
+
+ SetPageLRU(page);
+ smp_mb__after_atomic();
+
+ lru = LRU_UNEVICTABLE;
+ ClearPageActive(page);
+ SetPageUnevictable(page);
+ if (!was_unevictable)
+ splice->pgculled += nr_pages;
+
+ add_page_to_lru_splice(page, splice, lruvec, lru);
+ trace_mm_lru_insertion(page);
+}
+
+void lru_splice_add(struct page *page, struct lru_splice *splice)
+{
+ struct lruvec *lruvec;
+
+ VM_BUG_ON_PAGE(PageActive(page) && PageUnevictable(page), page);
+ VM_BUG_ON_PAGE(PageLRU(page), page);
+
+ get_page(page);
+ lruvec = mem_cgroup_page_lruvec(page, page_pgdat(page));
+ if (lruvec != splice->lruvec)
+ add_splice_to_lru_list(splice);
+ page_lru_splice_add(page, splice, lruvec);
+ put_page(page);
+}
+
static void pagevec_lru_move_fn(struct pagevec *pvec,
void (*move_fn)(struct page *page, struct lruvec *lruvec))
{
--
1.8.3.1
Free the pages used to pass the preserved ranges to the new boot.
Signed-off-by: Anthony Yznaga <[email protected]>
---
arch/x86/mm/init_64.c | 1 +
include/linux/pkram.h | 2 ++
mm/pkram.c | 20 ++++++++++++++++++++
3 files changed, 23 insertions(+)
diff --git a/arch/x86/mm/init_64.c b/arch/x86/mm/init_64.c
index 8efb2fb2a88b..69bd71996b8b 100644
--- a/arch/x86/mm/init_64.c
+++ b/arch/x86/mm/init_64.c
@@ -1294,6 +1294,7 @@ void __init mem_init(void)
after_bootmem = 1;
x86_init.hyper.init_after_bootmem();
+ pkram_cleanup();
totalram_pages_add(pkram_reserved_pages);
/*
* Must be done after boot memory is put on freelist, because here we
diff --git a/include/linux/pkram.h b/include/linux/pkram.h
index 8d3d780d9fe1..c2099a4f2004 100644
--- a/include/linux/pkram.h
+++ b/include/linux/pkram.h
@@ -102,9 +102,11 @@ int pkram_prepare_save(struct pkram_stream *ps, const char *name,
#ifdef CONFIG_PKRAM
extern unsigned long pkram_reserved_pages;
void pkram_reserve(void);
+void pkram_cleanup(void);
#else
#define pkram_reserved_pages 0UL
static inline void pkram_reserve(void) { }
+static inline void pkram_cleanup(void) { }
#endif
#endif /* _LINUX_PKRAM_H */
diff --git a/mm/pkram.c b/mm/pkram.c
index 03731bb6af26..dab6657080bf 100644
--- a/mm/pkram.c
+++ b/mm/pkram.c
@@ -1434,3 +1434,23 @@ int __init pkram_merge_with_reserved(void)
return 0;
}
+
+void __init pkram_cleanup(void)
+{
+ struct pkram_region_list *rl;
+ unsigned long next_pfn;
+
+ if (!pkram_sb || !pkram_reserved_pages)
+ return;
+
+ next_pfn = pkram_sb->region_list_pfn;
+
+ while (next_pfn) {
+ struct page *page = pfn_to_page(next_pfn);
+
+ rl = pfn_to_kaddr(next_pfn);
+ next_pfn = rl->next_pfn;
+ __free_pages_core(page, 0);
+ pkram_reserved_pages--;
+ }
+}
--
1.8.3.1
To take advantage of optimizations when adding pages to the page cache
via shmem_insert_pages(), improve the likelihood that the pages array
passed to shmem_insert_pages() starts on an aligned index. Do this
when preserving pages by starting a new pkram_link page when the current
page is aligned and the next aligned page will not fit on the pkram_link
page.
Signed-off-by: Anthony Yznaga <[email protected]>
---
mm/pkram.c | 13 ++++++++++++-
1 file changed, 12 insertions(+), 1 deletion(-)
diff --git a/mm/pkram.c b/mm/pkram.c
index b63b2a3958e7..3f43809c8a85 100644
--- a/mm/pkram.c
+++ b/mm/pkram.c
@@ -911,9 +911,20 @@ static int __pkram_save_page(struct pkram_access *pa, struct page *page,
{
struct pkram_data_stream *pds = &pa->pds;
struct pkram_link *link = pds->link;
+ int align, align_cnt;
+
+ if (PageTransHuge(page)) {
+ align = 1 << (HPAGE_PMD_ORDER + XA_CHUNK_SHIFT - (HPAGE_PMD_ORDER % XA_CHUNK_SHIFT));
+ align_cnt = align >> HPAGE_PMD_ORDER;
+ } else {
+ align = XA_CHUNK_SIZE;
+ align_cnt = XA_CHUNK_SIZE;
+ }
if (!link || pds->entry_idx >= PKRAM_LINK_ENTRIES_MAX ||
- index != pa->pages.next_index) {
+ index != pa->pages.next_index ||
+ (IS_ALIGNED(index, align) &&
+ (pds->entry_idx + align_cnt > PKRAM_LINK_ENTRIES_MAX))) {
link = pkram_new_link(pds, pa->ps->gfp_mask);
if (!link)
return -ENOMEM;
--
1.8.3.1
Rather than adding one page at a time to the page cache and taking the
page cache xarray lock each time, where possible add pages in bulk by
first populating an xarray node outside of the page cache before taking
the lock to insert it.
When a group of pages to be inserted will fill an xarray node, add them
to a local xarray, export the xarray node, and then take the lock on the
page cache xarray and insert the node.
Signed-off-by: Anthony Yznaga <[email protected]>
---
mm/shmem.c | 162 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++---
1 file changed, 156 insertions(+), 6 deletions(-)
diff --git a/mm/shmem.c b/mm/shmem.c
index f495af51042e..a7c23b43b57f 100644
--- a/mm/shmem.c
+++ b/mm/shmem.c
@@ -827,17 +827,149 @@ static void shmem_delete_from_page_cache(struct page *page, void *radswap)
BUG_ON(error);
}
+static int shmem_add_aligned_to_page_cache(struct page *pages[], int npages,
+ struct address_space *mapping,
+ pgoff_t index, gfp_t gfp, int order,
+ struct mm_struct *charge_mm)
+{
+ int xa_shift = order + XA_CHUNK_SHIFT - (order % XA_CHUNK_SHIFT);
+ XA_STATE_ORDER(xas, &mapping->i_pages, index, xa_shift);
+ struct xarray xa_tmp;
+ /*
+ * Specify order so xas_create_range() only needs to be called once
+ * to allocate the entire range. This guarantees that xas_store()
+ * will not fail due to lack of memory.
+ * Specify index == 0 so the minimum necessary nodes are allocated.
+ */
+ XA_STATE_ORDER(xas_tmp, &xa_tmp, 0, xa_shift);
+ unsigned long nr = 1UL << order;
+ struct xa_node *node;
+ int i, error;
+
+ if (npages * nr != 1 << xa_shift) {
+ WARN_ONCE(1, "npages (%d) not aligned to xa_shift\n", npages);
+ return -EINVAL;
+ }
+ if (!IS_ALIGNED(index, 1 << xa_shift)) {
+ WARN_ONCE(1, "index (%lu) not aligned to xa_shift\n", index);
+ return -EINVAL;
+ }
+
+ for (i = 0; i < npages; i++) {
+ bool skipcharge = page_memcg(pages[i]) ? true : false;
+
+ VM_BUG_ON_PAGE(PageTail(pages[i]), pages[i]);
+ VM_BUG_ON_PAGE(!PageLocked(pages[i]), pages[i]);
+ VM_BUG_ON_PAGE(!PageSwapBacked(pages[i]), pages[i]);
+
+ page_ref_add(pages[i], nr);
+ pages[i]->mapping = mapping;
+ pages[i]->index = index + (i * nr);
+
+ if (!skipcharge && !PageSwapCache(pages[i])) {
+ error = mem_cgroup_charge(pages[i], charge_mm, gfp);
+ if (error) {
+ if (PageTransHuge(pages[i])) {
+ count_vm_event(THP_FILE_FALLBACK);
+ count_vm_event(THP_FILE_FALLBACK_CHARGE);
+ }
+ goto error;
+ }
+ }
+ cgroup_throttle_swaprate(pages[i], gfp);
+ }
+
+ xa_init(&xa_tmp);
+ do {
+ xas_lock(&xas_tmp);
+ xas_create_range(&xas_tmp);
+ if (xas_error(&xas_tmp))
+ goto unlock;
+ for (i = 0; i < npages; i++) {
+ int j = 0;
+next:
+ xas_store(&xas_tmp, pages[i]);
+ if (++j < nr) {
+ xas_next(&xas_tmp);
+ goto next;
+ }
+ if (i < npages - 1)
+ xas_next(&xas_tmp);
+ }
+ xas_set_order(&xas_tmp, 0, xa_shift);
+ node = xas_export_node(&xas_tmp);
+unlock:
+ xas_unlock(&xas_tmp);
+ } while (xas_nomem(&xas_tmp, gfp));
+
+ if (xas_error(&xas_tmp)) {
+ error = xas_error(&xas_tmp);
+ i = npages - 1;
+ goto error;
+ }
+
+ do {
+ xas_lock_irq(&xas);
+ xas_import_node(&xas, node);
+ if (xas_error(&xas))
+ goto unlock1;
+ mapping->nrpages += nr * npages;
+ xas_unlock(&xas);
+ for (i = 0; i < npages; i++) {
+ __mod_lruvec_page_state(pages[i], NR_FILE_PAGES, nr);
+ __mod_lruvec_page_state(pages[i], NR_SHMEM, nr);
+ if (PageTransHuge(pages[i])) {
+ count_vm_event(THP_FILE_ALLOC);
+ __inc_node_page_state(pages[i], NR_SHMEM_THPS);
+ }
+ }
+ local_irq_enable();
+ break;
+unlock1:
+ xas_unlock_irq(&xas);
+ } while (xas_nomem(&xas, gfp));
+
+ if (xas_error(&xas)) {
+ error = xas_error(&xas);
+ goto error;
+ }
+
+ return 0;
+error:
+ while (i != 0) {
+ pages[i]->mapping = NULL;
+ page_ref_sub(pages[i], nr);
+ i--;
+ }
+ return error;
+}
+
static int shmem_add_pages_to_cache(struct page *pages[], int npages,
struct address_space *mapping,
pgoff_t start, gfp_t gfp,
struct mm_struct *charge_mm)
{
pgoff_t index = start;
- int i, err;
+ int i, j, err;
i = 0;
while (i < npages) {
if (PageTransHuge(pages[i])) {
+ if (IS_ALIGNED(index, 4096) && i+8 <= npages) {
+ for (j = 1; j < 8; j++) {
+ if (!PageTransHuge(pages[i+j]))
+ break;
+ }
+ if (j == 8) {
+ err = shmem_add_aligned_to_page_cache(&pages[i], 8, mapping, index, gfp, HPAGE_PMD_ORDER, charge_mm);
+ if (err)
+ goto out_release;
+ index += HPAGE_PMD_NR * 8;
+ i += 8;
+ continue;
+ }
+ }
+
err = shmem_add_to_page_cache_fast(pages[i], mapping, index, gfp, charge_mm, page_memcg(pages[i]) ? true : false);
if (err)
goto out_release;
@@ -846,11 +978,29 @@ static int shmem_add_pages_to_cache(struct page *pages[], int npages,
continue;
}
- err = shmem_add_to_page_cache_fast(pages[i], mapping, index, gfp, charge_mm, page_memcg(pages[i]) ? true : false);
- if (err)
- goto out_release;
- index++;
- i++;
+ for (j = 1; i + j < npages; j++) {
+ if (PageTransHuge(pages[i + j]))
+ break;
+ }
+
+ while (j > 0) {
+ if (IS_ALIGNED(index, 64) && j >= 64) {
+ err = shmem_add_aligned_to_page_cache(&pages[i], 64, mapping, index, gfp, 0, charge_mm);
+ if (err)
+ goto out_release;
+ index += 64;
+ i += 64;
+ j -= 64;
+ continue;
+ }
+
+ err = shmem_add_to_page_cache_fast(pages[i], mapping, index, gfp, charge_mm, page_memcg(pages[i]) ? true : false);
+ if (err)
+ goto out_release;
+ index++;
+ i++;
+ j--;
+ }
}
return 0;
--
1.8.3.1
Preserved pages are represented in the memblock reserved list, but page
structs for pages in the reserved list are initialized early while boot
is single threaded which means that a large number of preserved pages
can impact boot time. To mitigate, defer initialization of preserved
pages by skipping them when other reserved pages are initialized and
initializing them later with a separate kernel thread.
Signed-off-by: Anthony Yznaga <[email protected]>
---
arch/x86/mm/init_64.c | 1 -
include/linux/mm.h | 2 +-
mm/memblock.c | 11 +++++++++--
mm/page_alloc.c | 55 +++++++++++++++++++++++++++++++++++++++++++--------
4 files changed, 57 insertions(+), 12 deletions(-)
diff --git a/arch/x86/mm/init_64.c b/arch/x86/mm/init_64.c
index 69bd71996b8b..8efb2fb2a88b 100644
--- a/arch/x86/mm/init_64.c
+++ b/arch/x86/mm/init_64.c
@@ -1294,7 +1294,6 @@ void __init mem_init(void)
after_bootmem = 1;
x86_init.hyper.init_after_bootmem();
- pkram_cleanup();
totalram_pages_add(pkram_reserved_pages);
/*
* Must be done after boot memory is put on freelist, because here we
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 64a71bf20536..2a93b2a6ec8d 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -2337,7 +2337,7 @@ extern unsigned long free_reserved_area(void *start, void *end,
extern void adjust_managed_page_count(struct page *page, long count);
extern void mem_init_print_info(const char *str);
-extern void reserve_bootmem_region(phys_addr_t start, phys_addr_t end);
+extern void reserve_bootmem_region(phys_addr_t start, phys_addr_t end, int nid);
/* Free the reserved page into the buddy system, so it gets managed. */
static inline void free_reserved_page(struct page *page)
diff --git a/mm/memblock.c b/mm/memblock.c
index afaefa8fc6ab..461ea0f85495 100644
--- a/mm/memblock.c
+++ b/mm/memblock.c
@@ -2007,11 +2007,18 @@ static unsigned long __init free_low_memory_core_early(void)
unsigned long count = 0;
phys_addr_t start, end;
u64 i;
+ struct memblock_region *r;
memblock_clear_hotplug(0, -1);
- for_each_reserved_mem_range(i, &start, &end)
- reserve_bootmem_region(start, end);
+ for_each_reserved_mem_region(r) {
+ if (IS_ENABLED(CONFIG_DEFERRED_STRUCT_PAGE_INIT) && memblock_is_preserved(r))
+ continue;
+
+ start = r->base;
+ end = r->base + r->size;
+ reserve_bootmem_region(start, end, NUMA_NO_NODE);
+ }
/*
* We need to use NUMA_NO_NODE instead of NODE_DATA(0)->node_id
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index cfc72873961d..999fcc8fe907 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -72,6 +72,7 @@
#include <linux/padata.h>
#include <linux/khugepaged.h>
#include <linux/buffer_head.h>
+#include <linux/pkram.h>
#include <asm/sections.h>
#include <asm/tlbflush.h>
@@ -1475,15 +1476,18 @@ static void __meminit __init_single_page(struct page *page, unsigned long pfn,
}
#ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT
-static void __meminit init_reserved_page(unsigned long pfn)
+static void __meminit init_reserved_page(unsigned long pfn, int nid)
{
pg_data_t *pgdat;
- int nid, zid;
+ int zid;
- if (!early_page_uninitialised(pfn))
- return;
+ if (nid == NUMA_NO_NODE) {
+ if (!early_page_uninitialised(pfn))
+ return;
+
+ nid = early_pfn_to_nid(pfn);
+ }
- nid = early_pfn_to_nid(pfn);
pgdat = NODE_DATA(nid);
for (zid = 0; zid < MAX_NR_ZONES; zid++) {
@@ -1495,7 +1499,7 @@ static void __meminit init_reserved_page(unsigned long pfn)
__init_single_page(pfn_to_page(pfn), pfn, zid, nid);
}
#else
-static inline void init_reserved_page(unsigned long pfn)
+static inline void init_reserved_page(unsigned long pfn, int nid)
{
}
#endif /* CONFIG_DEFERRED_STRUCT_PAGE_INIT */
@@ -1506,7 +1510,7 @@ static inline void init_reserved_page(unsigned long pfn)
* marks the pages PageReserved. The remaining valid pages are later
* sent to the buddy page allocator.
*/
-void __meminit reserve_bootmem_region(phys_addr_t start, phys_addr_t end)
+void __meminit reserve_bootmem_region(phys_addr_t start, phys_addr_t end, int nid)
{
unsigned long start_pfn = PFN_DOWN(start);
unsigned long end_pfn = PFN_UP(end);
@@ -1515,7 +1519,7 @@ void __meminit reserve_bootmem_region(phys_addr_t start, phys_addr_t end)
if (pfn_valid(start_pfn)) {
struct page *page = pfn_to_page(start_pfn);
- init_reserved_page(start_pfn);
+ init_reserved_page(start_pfn, nid);
/* Avoid false-positive PageTail() */
INIT_LIST_HEAD(&page->lru);
@@ -2008,6 +2012,35 @@ static int __init deferred_init_memmap(void *data)
return 0;
}
+#ifdef CONFIG_PKRAM
+static int __init deferred_init_preserved(void *dummy)
+{
+ unsigned long start = jiffies;
+ unsigned long nr_pages = 0;
+ struct memblock_region *r;
+ phys_addr_t spa, epa;
+ int nid;
+
+ for_each_reserved_mem_region(r) {
+ if (!memblock_is_preserved(r))
+ continue;
+
+ spa = r->base;
+ epa = r->base + r->size;
+ nid = memblock_get_region_node(r);
+
+ reserve_bootmem_region(spa, epa, nid);
+ nr_pages += ((epa - spa) >> PAGE_SHIFT);
+ }
+
+ pr_info("initialised %lu preserved pages in %ums\n", nr_pages,
+ jiffies_to_msecs(jiffies - start));
+
+ pgdat_init_report_one_done();
+ return 0;
+}
+#endif /* CONFIG_PKRAM */
+
/*
* If this zone has deferred pages, try to grow it by initializing enough
* deferred pages to satisfy the allocation specified by order, rounded up to
@@ -2107,6 +2140,10 @@ void __init page_alloc_init_late(void)
/* There will be num_node_state(N_MEMORY) threads */
atomic_set(&pgdat_init_n_undone, num_node_state(N_MEMORY));
+#ifdef CONFIG_PKRAM
+ atomic_inc(&pgdat_init_n_undone);
+ kthread_run(deferred_init_preserved, NULL, "pgdatainit_preserved");
+#endif
for_each_node_state(nid, N_MEMORY) {
kthread_run(deferred_init_memmap, NODE_DATA(nid), "pgdatinit%d", nid);
}
@@ -2114,6 +2151,8 @@ void __init page_alloc_init_late(void)
/* Block until all are initialised */
wait_for_completion(&pgdat_init_all_done_comp);
+ pkram_cleanup();
+
/*
* The number of managed pages has changed due to the initialisation
* so the pcpu batch and high limits needs to be updated or the limits
--
1.8.3.1
pkram kaslr code can incur multiple page faults when it walks its
preserved ranges list called via mem_avoid_overlap(). The multiple
faults can easily end up using up the small number of pages available
to be allocated for page table pages.
This patch hacks things so that mappings are 1GB which results in the need
for far fewer page table pages. As is this breaks AMD SEV-ES which expects
the mappings to be 2M. This could possibly be fixed by updating split
code to split 1GB page if the aren't any other issues with using 1GB
mappings.
Signed-off-by: Anthony Yznaga <[email protected]>
---
arch/x86/boot/compressed/ident_map_64.c | 9 +++++----
1 file changed, 5 insertions(+), 4 deletions(-)
diff --git a/arch/x86/boot/compressed/ident_map_64.c b/arch/x86/boot/compressed/ident_map_64.c
index f7213d0943b8..6ff02da4cc1a 100644
--- a/arch/x86/boot/compressed/ident_map_64.c
+++ b/arch/x86/boot/compressed/ident_map_64.c
@@ -95,8 +95,8 @@ static void add_identity_map(unsigned long start, unsigned long end)
int ret;
/* Align boundary to 2M. */
- start = round_down(start, PMD_SIZE);
- end = round_up(end, PMD_SIZE);
+ start = round_down(start, PUD_SIZE);
+ end = round_up(end, PUD_SIZE);
if (start >= end)
return;
@@ -119,6 +119,7 @@ void initialize_identity_maps(void *rmode)
mapping_info.context = &pgt_data;
mapping_info.page_flag = __PAGE_KERNEL_LARGE_EXEC | sme_me_mask;
mapping_info.kernpg_flag = _KERNPG_TABLE;
+ mapping_info.direct_gbpages = true;
/*
* It should be impossible for this not to already be true,
@@ -329,8 +330,8 @@ void do_boot_page_fault(struct pt_regs *regs, unsigned long error_code)
ghcb_fault = sev_es_check_ghcb_fault(address);
- address &= PMD_MASK;
- end = address + PMD_SIZE;
+ address &= PUD_MASK;
+ end = address + PUD_SIZE;
/*
* Check for unexpected error codes. Unexpected are:
--
1.8.3.1
Reduce LRU lock contention when inserting shmem pages by staging pages
to be added to the same LRU and adding them en masse.
Signed-off-by: Anthony Yznaga <[email protected]>
---
mm/shmem.c | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/mm/shmem.c b/mm/shmem.c
index c3fa72061d8a..63299da75166 100644
--- a/mm/shmem.c
+++ b/mm/shmem.c
@@ -845,6 +845,7 @@ int shmem_insert_pages(struct mm_struct *charge_mm, struct inode *inode,
struct shmem_inode_info *info = SHMEM_I(inode);
struct shmem_sb_info *sbinfo = SHMEM_SB(inode->i_sb);
gfp_t gfp = mapping_gfp_mask(mapping);
+ LRU_SPLICE(splice);
int i, err;
int nr = 0;
@@ -908,7 +909,7 @@ int shmem_insert_pages(struct mm_struct *charge_mm, struct inode *inode,
for (i = 0; i < npages; i++) {
if (!PageLRU(pages[i]))
- lru_cache_add(pages[i]);
+ lru_splice_add(pages[i], &splice);
flush_dcache_page(pages[i]);
SetPageUptodate(pages[i]);
@@ -917,6 +918,8 @@ int shmem_insert_pages(struct mm_struct *charge_mm, struct inode *inode,
unlock_page(pages[i]);
}
+ add_splice_to_lru_list(&splice);
+
return 0;
out_release:
--
1.8.3.1
If shmem_insert_page() is called to insert a page that was preserved
using PKRAM on the current boot (i.e. preserved page is restored without
an intervening kexec boot), the page will still be charged to a memory
cgroup because it is never freed. Don't try to charge it again.
Signed-off-by: Anthony Yznaga <[email protected]>
---
mm/shmem.c | 14 ++++++++------
1 file changed, 8 insertions(+), 6 deletions(-)
diff --git a/mm/shmem.c b/mm/shmem.c
index 8dfe80aeee97..44cc158ab34d 100644
--- a/mm/shmem.c
+++ b/mm/shmem.c
@@ -671,7 +671,7 @@ static inline bool is_huge_enabled(struct shmem_sb_info *sbinfo)
static int shmem_add_to_page_cache(struct page *page,
struct address_space *mapping,
pgoff_t index, void *expected, gfp_t gfp,
- struct mm_struct *charge_mm)
+ struct mm_struct *charge_mm, bool skipcharge)
{
XA_STATE_ORDER(xas, &mapping->i_pages, index, compound_order(page));
unsigned long i = 0;
@@ -688,7 +688,7 @@ static int shmem_add_to_page_cache(struct page *page,
page->mapping = mapping;
page->index = index;
- if (!PageSwapCache(page)) {
+ if (!skipcharge && !PageSwapCache(page)) {
error = mem_cgroup_charge(page, charge_mm, gfp);
if (error) {
if (PageTransHuge(page)) {
@@ -770,6 +770,7 @@ int shmem_insert_page(struct mm_struct *mm, struct inode *inode, pgoff_t index,
int nr;
pgoff_t hindex = index;
bool on_lru = PageLRU(page);
+ bool ischarged = page_memcg(page) ? true : false;
if (index > (MAX_LFS_FILESIZE >> PAGE_SHIFT))
return -EFBIG;
@@ -809,7 +810,8 @@ int shmem_insert_page(struct mm_struct *mm, struct inode *inode, pgoff_t index,
__SetPageReferenced(page);
err = shmem_add_to_page_cache(page, mapping, hindex,
- NULL, gfp & GFP_RECLAIM_MASK, mm);
+ NULL, gfp & GFP_RECLAIM_MASK,
+ mm, ischarged);
if (err)
goto out_unlock;
@@ -1829,7 +1831,7 @@ static int shmem_swapin_page(struct inode *inode, pgoff_t index,
error = shmem_add_to_page_cache(page, mapping, index,
swp_to_radix_entry(swap), gfp,
- charge_mm);
+ charge_mm, false);
if (error)
goto failed;
@@ -2009,7 +2011,7 @@ static int shmem_getpage_gfp(struct inode *inode, pgoff_t index,
error = shmem_add_to_page_cache(page, mapping, hindex,
NULL, gfp & GFP_RECLAIM_MASK,
- charge_mm);
+ charge_mm, false);
if (error)
goto unacct;
lru_cache_add(page);
@@ -2500,7 +2502,7 @@ static int shmem_mfill_atomic_pte(struct mm_struct *dst_mm,
goto out_release;
ret = shmem_add_to_page_cache(page, mapping, pgoff, NULL,
- gfp & GFP_RECLAIM_MASK, dst_mm);
+ gfp & GFP_RECLAIM_MASK, dst_mm, false);
if (ret)
goto out_release;
--
1.8.3.1
Provide a way for a caller external to numa to ensure memblocks in the
memblock reserved list do not cross node boundaries and have a node id
assigned to them. This will be used by PKRAM to ensure initialization of
page structs for preserved pages can be deferred and multithreaded
efficiently.
Signed-off-by: Anthony Yznaga <[email protected]>
---
arch/x86/include/asm/numa.h | 4 ++++
arch/x86/mm/numa.c | 32 ++++++++++++++++++++------------
2 files changed, 24 insertions(+), 12 deletions(-)
diff --git a/arch/x86/include/asm/numa.h b/arch/x86/include/asm/numa.h
index e3bae2b60a0d..632b5b6d8cb3 100644
--- a/arch/x86/include/asm/numa.h
+++ b/arch/x86/include/asm/numa.h
@@ -41,6 +41,7 @@ static inline void set_apicid_to_node(int apicid, s16 node)
}
extern int numa_cpu_node(int cpu);
+extern void __init numa_isolate_memblocks(void);
#else /* CONFIG_NUMA */
static inline void set_apicid_to_node(int apicid, s16 node)
@@ -51,6 +52,9 @@ static inline int numa_cpu_node(int cpu)
{
return NUMA_NO_NODE;
}
+static inline void numa_isolate_memblocks(void)
+{
+}
#endif /* CONFIG_NUMA */
#ifdef CONFIG_X86_32
diff --git a/arch/x86/mm/numa.c b/arch/x86/mm/numa.c
index 5eb4dc2b97da..dd85098f9d72 100644
--- a/arch/x86/mm/numa.c
+++ b/arch/x86/mm/numa.c
@@ -473,6 +473,25 @@ static bool __init numa_meminfo_cover_memory(const struct numa_meminfo *mi)
return true;
}
+void __init numa_isolate_memblocks(void)
+{
+ int i;
+
+ /*
+ * Iterate over all memory known to the x86 architecture,
+ * and use those ranges to set the nid in memblock.reserved.
+ * This will split up the memblock regions along node
+ * boundaries and will set the node IDs as well.
+ */
+ for (i = 0; i < numa_meminfo.nr_blks; i++) {
+ struct numa_memblk *mb = numa_meminfo.blk + i;
+ int ret;
+
+ ret = memblock_set_node(mb->start, mb->end - mb->start, &memblock.reserved, mb->nid);
+ WARN_ON_ONCE(ret);
+ }
+}
+
/*
* Mark all currently memblock-reserved physical memory (which covers the
* kernel's own memory ranges) as hot-unswappable.
@@ -491,19 +510,8 @@ static void __init numa_clear_kernel_node_hotplug(void)
* used by the kernel, but those regions are not split up
* along node boundaries yet, and don't necessarily have their
* node ID set yet either.
- *
- * So iterate over all memory known to the x86 architecture,
- * and use those ranges to set the nid in memblock.reserved.
- * This will split up the memblock regions along node
- * boundaries and will set the node IDs as well.
*/
- for (i = 0; i < numa_meminfo.nr_blks; i++) {
- struct numa_memblk *mb = numa_meminfo.blk + i;
- int ret;
-
- ret = memblock_set_node(mb->start, mb->end - mb->start, &memblock.reserved, mb->nid);
- WARN_ON_ONCE(ret);
- }
+ numa_isolate_memblocks();
/*
* Now go over all reserved memblock regions, to construct a
--
1.8.3.1
The kdump kernel should not preserve or restore pages.
Signed-off-by: Anthony Yznaga <[email protected]>
---
mm/pkram.c | 8 ++++++--
1 file changed, 6 insertions(+), 2 deletions(-)
diff --git a/mm/pkram.c b/mm/pkram.c
index 8700fd77dc67..aea069cc49be 100644
--- a/mm/pkram.c
+++ b/mm/pkram.c
@@ -1,4 +1,5 @@
// SPDX-License-Identifier: GPL-2.0
+#include <linux/crash_dump.h>
#include <linux/err.h>
#include <linux/gfp.h>
#include <linux/highmem.h>
@@ -189,7 +190,7 @@ void __init pkram_reserve(void)
{
int err = 0;
- if (!pkram_sb_pfn)
+ if (!pkram_sb_pfn || is_kdump_kernel())
return;
pr_info("PKRAM: Examining preserved memory...\n");
@@ -286,6 +287,9 @@ static void pkram_show_banned(void)
int i;
unsigned long n, total = 0;
+ if (is_kdump_kernel())
+ return;
+
pr_info("PKRAM: banned regions:\n");
for (i = 0; i < nr_banned; i++) {
n = banned[i].end - banned[i].start + 1;
@@ -1315,7 +1319,7 @@ static int __init pkram_init_sb(void)
static int __init pkram_init(void)
{
- if (pkram_init_sb()) {
+ if (!is_kdump_kernel() && pkram_init_sb()) {
register_reboot_notifier(&pkram_reboot_notifier);
register_shrinker(&banned_pages_shrinker);
sysfs_update_group(kernel_kobj, &pkram_attr_group);
--
1.8.3.1
To support deferred initialization of page structs for preserved pages,
separate memblocks containing preserved pages by setting a new flag
when adding them to the memblock reserved list.
Signed-off-by: Anthony Yznaga <[email protected]>
---
include/linux/memblock.h | 6 ++++++
mm/pkram.c | 2 ++
2 files changed, 8 insertions(+)
diff --git a/include/linux/memblock.h b/include/linux/memblock.h
index d13e3cd938b4..39c53d08d9f7 100644
--- a/include/linux/memblock.h
+++ b/include/linux/memblock.h
@@ -37,6 +37,7 @@ enum memblock_flags {
MEMBLOCK_HOTPLUG = 0x1, /* hotpluggable region */
MEMBLOCK_MIRROR = 0x2, /* mirrored region */
MEMBLOCK_NOMAP = 0x4, /* don't add to kernel direct mapping */
+ MEMBLOCK_PRESERVED = 0x8, /* preserved pages region */
};
/**
@@ -248,6 +249,11 @@ static inline bool memblock_is_nomap(struct memblock_region *m)
return m->flags & MEMBLOCK_NOMAP;
}
+static inline bool memblock_is_preserved(struct memblock_region *m)
+{
+ return m->flags & MEMBLOCK_PRESERVED;
+}
+
int memblock_search_pfn_nid(unsigned long pfn, unsigned long *start_pfn,
unsigned long *end_pfn);
void __next_mem_pfn_range(int *idx, int nid, unsigned long *out_start_pfn,
diff --git a/mm/pkram.c b/mm/pkram.c
index b8d6b549fa6c..08144c18d425 100644
--- a/mm/pkram.c
+++ b/mm/pkram.c
@@ -1607,6 +1607,7 @@ int __init pkram_create_merged_reserved(struct memblock_type *new)
} else if (pkr->base + pkr->size <= r->base) {
rgn->base = pkr->base;
rgn->size = pkr->size;
+ rgn->flags = MEMBLOCK_PRESERVED;
memblock_set_region_node(rgn, MAX_NUMNODES);
nr_preserved += (rgn->size >> PAGE_SHIFT);
@@ -1636,6 +1637,7 @@ int __init pkram_create_merged_reserved(struct memblock_type *new)
rgn = &new->regions[k];
rgn->base = pkr->base;
rgn->size = pkr->size;
+ rgn->flags = MEMBLOCK_PRESERVED;
memblock_set_region_node(rgn, MAX_NUMNODES);
nr_preserved += (rgn->size >> PAGE_SHIFT);
--
1.8.3.1
Work around the limitation that shmem pages must be in memory in order
to be preserved by preventing them from being swapped out in the first
place. Do this by marking shmem pages associated with a PKRAM node
as unevictable.
Signed-off-by: Anthony Yznaga <[email protected]>
---
mm/shmem.c | 2 ++
1 file changed, 2 insertions(+)
diff --git a/mm/shmem.c b/mm/shmem.c
index c1c5760465f2..8dfe80aeee97 100644
--- a/mm/shmem.c
+++ b/mm/shmem.c
@@ -2400,6 +2400,8 @@ static struct inode *shmem_get_inode(struct super_block *sb, const struct inode
INIT_LIST_HEAD(&info->swaplist);
simple_xattrs_init(&info->xattrs);
cache_no_acl(inode);
+ if (sbinfo->pkram)
+ mapping_set_unevictable(inode->i_mapping);
switch (mode & S_IFMT) {
default:
--
1.8.3.1
Implement a new API function, pkram_load_file_pages(), to support
loading pages in bulk. A caller provided buffer not smaller than
PKRAM_PAGES_BUFSIZE is populated with pages pointers that are contiguous
by their original mapping index values. The number of pages in the buffer
and the mapping index of the first page are provided to the caller.
Signed-off-by: Anthony Yznaga <[email protected]>
---
include/linux/pkram.h | 4 ++++
mm/pkram.c | 46 ++++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 50 insertions(+)
diff --git a/include/linux/pkram.h b/include/linux/pkram.h
index 977cf45a1bcf..ca46e5eafe71 100644
--- a/include/linux/pkram.h
+++ b/include/linux/pkram.h
@@ -96,6 +96,10 @@ int pkram_prepare_save(struct pkram_stream *ps, const char *name,
int pkram_save_file_page(struct pkram_access *pa, struct page *page);
struct page *pkram_load_file_page(struct pkram_access *pa, unsigned long *index);
+#define PKRAM_PAGES_BUFSIZE PAGE_SIZE
+
+int pkram_load_file_pages(struct pkram_access *pa, struct page *pages[], unsigned int *nr_pages, unsigned long *index);
+
ssize_t pkram_write(struct pkram_access *pa, const void *buf, size_t count);
size_t pkram_read(struct pkram_access *pa, void *buf, size_t count);
diff --git a/mm/pkram.c b/mm/pkram.c
index 382ccf6f789f..b63b2a3958e7 100644
--- a/mm/pkram.c
+++ b/mm/pkram.c
@@ -1099,6 +1099,52 @@ struct page *pkram_load_file_page(struct pkram_access *pa, unsigned long *index)
}
/**
+ * Load pages from the preserved memory node and object associated with
+ * pkram stream access @pa. The stream must have been initialized with
+ * pkram_prepare_load() and pkram_prepare_load_obj() and access initialized
+ * with PKRAM_ACCESS().
+ * The page entries of a single pkram_link are processed, and @pages is
+ * populated with the page pointers. @nr_pages is set to the number of
+ * pages, and @index is set to the mapping index of the first page.
+ *
+ * Returns 0 if one or more pages are loaded or -ENODATA if there are no
+ * pages to load.
+ *
+ * The pages loaded have an incremented refcount either because the page
+ * was initialized with a refcount of 1 at boot or because the page was
+ * subsequently preserved which increased the refcount.
+ */
+int pkram_load_file_pages(struct pkram_access *pa, struct page *pages[], unsigned int *nr_pages, unsigned long *index)
+{
+ struct pkram_data_stream *pds = &pa->pds;
+ struct pkram_link *link;
+ int nr_entries = 0;
+ int i, ret;
+
+ ret = pkram_next_link(pds, &link);
+ if (ret)
+ return ret;
+
+ for (i = 0; i < PKRAM_LINK_ENTRIES_MAX; i++) {
+ unsigned long p = link->entry[i];
+
+ if (!p)
+ break;
+
+ pages[i] = __pkram_prep_load_page(p);
+ nr_entries++;
+ }
+
+ *nr_pages = nr_entries;
+ *index = link->index;
+
+ pkram_free_page(link);
+ pds->link = NULL;
+
+ return 0;
+}
+
+/**
* Copy @count bytes from @buf to the preserved memory node and object
* associated with pkram stream access @pa. The stream must have been
* initialized with pkram_prepare_save() and pkram_prepare_save_obj()
--
1.8.3.1
shmem_insert_pages() currently loops over the array of pages passed
to it and calls shmem_add_to_page_cache() for each one. Prepare
for adding pages to the pagecache in bulk by adding and using a
shmem_add_pages_to_cache() call. For now it just iterates over
an array and adds pages individually, but improvements in performance
when multiple threads are adding to the same pagecache are achieved
by calling a new shmem_add_to_page_cache_fast() function that does
not check for conflicts and drops the xarray lock before updating stats.
Signed-off-by: Anthony Yznaga <[email protected]>
---
mm/shmem.c | 123 +++++++++++++++++++++++++++++++++++++++++++++++++++++--------
1 file changed, 108 insertions(+), 15 deletions(-)
diff --git a/mm/shmem.c b/mm/shmem.c
index 63299da75166..f495af51042e 100644
--- a/mm/shmem.c
+++ b/mm/shmem.c
@@ -738,6 +738,74 @@ static int shmem_add_to_page_cache(struct page *page,
return error;
}
+static int shmem_add_to_page_cache_fast(struct page *page,
+ struct address_space *mapping,
+ pgoff_t index, gfp_t gfp,
+ struct mm_struct *charge_mm, bool skipcharge)
+{
+ XA_STATE_ORDER(xas, &mapping->i_pages, index, thp_order(page));
+ unsigned long nr = thp_nr_pages(page);
+ unsigned long i = 0;
+ int error;
+
+ VM_BUG_ON_PAGE(PageTail(page), page);
+ VM_BUG_ON_PAGE(index != round_down(index, nr), page);
+ VM_BUG_ON_PAGE(!PageLocked(page), page);
+ VM_BUG_ON_PAGE(!PageSwapBacked(page), page);
+
+ page_ref_add(page, nr);
+ page->mapping = mapping;
+ page->index = index;
+
+ if (!skipcharge && !PageSwapCache(page)) {
+ error = mem_cgroup_charge(page, charge_mm, gfp);
+ if (error) {
+ if (PageTransHuge(page)) {
+ count_vm_event(THP_FILE_FALLBACK);
+ count_vm_event(THP_FILE_FALLBACK_CHARGE);
+ }
+ goto error;
+ }
+ }
+ cgroup_throttle_swaprate(page, gfp);
+
+ do {
+ xas_lock_irq(&xas);
+ xas_create_range(&xas);
+ if (xas_error(&xas))
+ goto unlock;
+next:
+ xas_store(&xas, page);
+ if (++i < nr) {
+ xas_next(&xas);
+ goto next;
+ }
+ mapping->nrpages += nr;
+ xas_unlock(&xas);
+ if (PageTransHuge(page)) {
+ count_vm_event(THP_FILE_ALLOC);
+ __inc_node_page_state(page, NR_SHMEM_THPS);
+ }
+ __mod_lruvec_page_state(page, NR_FILE_PAGES, nr);
+ __mod_lruvec_page_state(page, NR_SHMEM, nr);
+ local_irq_enable();
+ break;
+unlock:
+ xas_unlock_irq(&xas);
+ } while (xas_nomem(&xas, gfp));
+
+ if (xas_error(&xas)) {
+ error = xas_error(&xas);
+ goto error;
+ }
+
+ return 0;
+error:
+ page->mapping = NULL;
+ page_ref_sub(page, nr);
+ return error;
+}
+
/*
* Like delete_from_page_cache, but substitutes swap for page.
*/
@@ -759,6 +827,41 @@ static void shmem_delete_from_page_cache(struct page *page, void *radswap)
BUG_ON(error);
}
+static int shmem_add_pages_to_cache(struct page *pages[], int npages,
+ struct address_space *mapping,
+ pgoff_t start, gfp_t gfp,
+ struct mm_struct *charge_mm)
+{
+ pgoff_t index = start;
+ int i, err;
+
+ i = 0;
+ while (i < npages) {
+ if (PageTransHuge(pages[i])) {
+ err = shmem_add_to_page_cache_fast(pages[i], mapping, index, gfp, charge_mm, page_memcg(pages[i]) ? true : false);
+ if (err)
+ goto out_release;
+ index += thp_nr_pages(pages[i]);
+ i++;
+ continue;
+ }
+
+ err = shmem_add_to_page_cache_fast(pages[i], mapping, index, gfp, charge_mm, page_memcg(pages[i]) ? true : false);
+ if (err)
+ goto out_release;
+ index++;
+ i++;
+ }
+ return 0;
+
+out_release:
+ while (i < npages) {
+ delete_from_page_cache(pages[i]);
+ i--;
+ }
+ return err;
+}
+
int shmem_insert_page(struct mm_struct *mm, struct inode *inode, pgoff_t index,
struct page *page)
{
@@ -889,17 +992,10 @@ int shmem_insert_pages(struct mm_struct *charge_mm, struct inode *inode,
__SetPageReferenced(pages[i]);
}
- for (i = 0; i < npages; i++) {
- bool ischarged = page_memcg(pages[i]) ? true : false;
-
- err = shmem_add_to_page_cache(pages[i], mapping, index,
- NULL, gfp & GFP_RECLAIM_MASK,
- charge_mm, ischarged);
- if (err)
- goto out_release;
-
- index += thp_nr_pages(pages[i]);
- }
+ err = shmem_add_pages_to_cache(pages, npages, mapping, index,
+ gfp & GFP_RECLAIM_MASK, charge_mm);
+ if (err)
+ goto out_unlock;
spin_lock(&info->lock);
info->alloced += nr;
@@ -922,10 +1018,7 @@ int shmem_insert_pages(struct mm_struct *charge_mm, struct inode *inode,
return 0;
-out_release:
- while (--i >= 0)
- delete_from_page_cache(pages[i]);
-
+out_unlock:
for (i = 0; i < npages; i++)
unlock_page(pages[i]);
--
1.8.3.1
Add and remove pkram_link pages from a pkram_obj atomically to prepare
for multithreading.
Signed-off-by: Anthony Yznaga <[email protected]>
---
mm/pkram.c | 39 ++++++++++++++++++++++++---------------
1 file changed, 24 insertions(+), 15 deletions(-)
diff --git a/mm/pkram.c b/mm/pkram.c
index 08144c18d425..382ccf6f789f 100644
--- a/mm/pkram.c
+++ b/mm/pkram.c
@@ -535,33 +535,42 @@ static void pkram_truncate(void)
static void pkram_add_link(struct pkram_link *link, struct pkram_data_stream *pds)
{
__u64 link_pfn = page_to_pfn(virt_to_page(link));
+ __u64 *tail = pds->tail_link_pfnp;
+ __u64 tail_pfn;
- if (!*pds->head_link_pfnp) {
+ do {
+ tail_pfn = *tail;
+ } while (cmpxchg64(tail, tail_pfn, link_pfn) != tail_pfn);
+
+ if (!tail_pfn) {
*pds->head_link_pfnp = link_pfn;
- *pds->tail_link_pfnp = link_pfn;
} else {
- struct pkram_link *tail = pfn_to_kaddr(*pds->tail_link_pfnp);
+ struct pkram_link *prev_tail = pfn_to_kaddr(tail_pfn);
- tail->link_pfn = link_pfn;
- *pds->tail_link_pfnp = link_pfn;
+ prev_tail->link_pfn = link_pfn;
}
}
static struct pkram_link *pkram_remove_link(struct pkram_data_stream *pds)
{
- struct pkram_link *link;
+ __u64 *head = pds->head_link_pfnp;
+ __u64 head_pfn = *head;
- if (!*pds->head_link_pfnp)
- return NULL;
+ while (head_pfn) {
+ struct pkram_link *link = pfn_to_kaddr(head_pfn);
- link = pfn_to_kaddr(*pds->head_link_pfnp);
- *pds->head_link_pfnp = link->link_pfn;
- if (!*pds->head_link_pfnp)
- *pds->tail_link_pfnp = 0;
- else
- link->link_pfn = 0;
+ if (cmpxchg64(head, head_pfn, link->link_pfn) == head_pfn) {
+ if (!*head)
+ *pds->tail_link_pfnp = 0;
+ else
+ link->link_pfn = 0;
+ return link;
+ }
- return link;
+ head_pfn = *head;
+ }
+
+ return NULL;
}
static struct pkram_link *pkram_new_link(struct pkram_data_stream *pds, gfp_t gfp_mask)
--
1.8.3.1
Contention on the xarray lock when multiple threads are adding to the
same xarray can be mitigated by providing a way to add entries in
bulk.
Allow a caller to allocate and populate an xarray node outside of
the target xarray and then only take the xarray lock long enough to
import the node into it.
Signed-off-by: Anthony Yznaga <[email protected]>
---
Documentation/core-api/xarray.rst | 8 +++
include/linux/xarray.h | 2 +
lib/test_xarray.c | 45 +++++++++++++++++
lib/xarray.c | 100 ++++++++++++++++++++++++++++++++++++++
4 files changed, 155 insertions(+)
diff --git a/Documentation/core-api/xarray.rst b/Documentation/core-api/xarray.rst
index a137a0e6d068..12ec59038fc8 100644
--- a/Documentation/core-api/xarray.rst
+++ b/Documentation/core-api/xarray.rst
@@ -444,6 +444,14 @@ called each time the XArray updates a node. This is used by the page
cache workingset code to maintain its list of nodes which contain only
shadow entries.
+xas_export_node() is used to remove and return a node from an XArray
+while xas_import_node() is used to add a node to an XArray. Together
+these can be used, for example, to reduce lock contention when multiple
+threads are updating an XArray by allowing a caller to allocate and
+populate a node outside of the target XArray in a local XArray, export
+the node, and then take the target XArray lock just long enough to import
+the node.
+
Multi-Index Entries
-------------------
diff --git a/include/linux/xarray.h b/include/linux/xarray.h
index 92c0160b3352..1eda38cbe020 100644
--- a/include/linux/xarray.h
+++ b/include/linux/xarray.h
@@ -1506,6 +1506,8 @@ static inline bool xas_retry(struct xa_state *xas, const void *entry)
void xas_pause(struct xa_state *);
void xas_create_range(struct xa_state *);
+struct xa_node *xas_export_node(struct xa_state *xas);
+void xas_import_node(struct xa_state *xas, struct xa_node *node);
#ifdef CONFIG_XARRAY_MULTI
int xa_get_order(struct xarray *, unsigned long index);
diff --git a/lib/test_xarray.c b/lib/test_xarray.c
index 8294f43f4981..9cca0921cf9b 100644
--- a/lib/test_xarray.c
+++ b/lib/test_xarray.c
@@ -1765,6 +1765,50 @@ static noinline void check_destroy(struct xarray *xa)
#endif
}
+static noinline void check_export_import_1(struct xarray *xa,
+ unsigned long index, unsigned int order)
+{
+ int xa_shift = order + XA_CHUNK_SHIFT - (order % XA_CHUNK_SHIFT);
+ XA_STATE(xas, xa, index);
+ struct xa_node *node;
+ unsigned long i;
+
+ xa_store_many_order(xa, index, xa_shift);
+
+ xas_lock(&xas);
+ xas_set_order(&xas, index, xa_shift);
+ node = xas_export_node(&xas);
+ xas_unlock(&xas);
+
+ XA_BUG_ON(xa, !xa_empty(xa));
+
+ do {
+ xas_lock(&xas);
+ xas_set_order(&xas, index, xa_shift);
+ xas_import_node(&xas, node);
+ xas_unlock(&xas);
+ } while (xas_nomem(&xas, GFP_KERNEL));
+
+ for (i = index; i < index + (1UL << xa_shift); i++)
+ xa_erase_index(xa, i);
+
+ XA_BUG_ON(xa, !xa_empty(xa));
+}
+
+static noinline void check_export_import(struct xarray *xa)
+{
+ unsigned int order;
+ unsigned int max_order = IS_ENABLED(CONFIG_XARRAY_MULTI) ? 12 : 1;
+
+ for (order = 0; order < max_order; order += XA_CHUNK_SHIFT) {
+ int xa_shift = order + XA_CHUNK_SHIFT;
+ unsigned long j;
+
+ for (j = 0; j < XA_CHUNK_SIZE; j++)
+ check_export_import_1(xa, j << xa_shift, order);
+ }
+}
+
static DEFINE_XARRAY(array);
static int xarray_checks(void)
@@ -1797,6 +1841,7 @@ static int xarray_checks(void)
check_workingset(&array, 0);
check_workingset(&array, 64);
check_workingset(&array, 4096);
+ check_export_import(&array);
printk("XArray: %u of %u tests passed\n", tests_passed, tests_run);
return (tests_run == tests_passed) ? 0 : -EINVAL;
diff --git a/lib/xarray.c b/lib/xarray.c
index 5fa51614802a..58d58333f0d0 100644
--- a/lib/xarray.c
+++ b/lib/xarray.c
@@ -510,6 +510,30 @@ static void xas_delete_node(struct xa_state *xas)
xas_shrink(xas);
}
+static void xas_unlink_node(struct xa_state *xas)
+{
+ struct xa_node *node = xas->xa_node;
+ struct xa_node *parent;
+
+ parent = xa_parent_locked(xas->xa, node);
+ xas->xa_node = parent;
+ xas->xa_offset = node->offset;
+
+ if (!parent) {
+ xas->xa->xa_head = NULL;
+ xas->xa_node = XAS_BOUNDS;
+ return;
+ }
+
+ parent->slots[xas->xa_offset] = NULL;
+ parent->count--;
+ XA_NODE_BUG_ON(parent, parent->count > XA_CHUNK_SIZE);
+
+ xas_update(xas, parent);
+
+ xas_delete_node(xas);
+}
+
/**
* xas_free_nodes() - Free this node and all nodes that it references
* @xas: Array operation state.
@@ -1690,6 +1714,82 @@ static void xas_set_range(struct xa_state *xas, unsigned long first,
}
/**
+ * xas_export_node() - remove and return a node from an XArray
+ * @xas: XArray operation state
+ *
+ * The range covered by @xas must be aligned to and cover a single node
+ * at any level of the tree.
+ *
+ * Return: On success, returns the removed node. If the range is invalid,
+ * returns %NULL and sets -EINVAL in @xas. Otherwise returns %NULL if the
+ * node does not exist.
+ */
+struct xa_node *xas_export_node(struct xa_state *xas)
+{
+ struct xa_node *node;
+
+ if (!xas->xa_shift || xas->xa_sibs) {
+ xas_set_err(xas, -EINVAL);
+ return NULL;
+ }
+
+ xas->xa_shift -= XA_CHUNK_SHIFT;
+
+ if (!xas_find(xas, xas->xa_index))
+ return NULL;
+ node = xas->xa_node;
+ xas_unlink_node(xas);
+ node->parent = NULL;
+
+ return node;
+}
+
+/**
+ * xas_import_node() - add a node to an XArray
+ * @xas: XArray operation state
+ * @node: The node to add
+ *
+ * The range covered by @xas must be aligned to and cover a single node
+ * at any level of the tree. No nodes should already exist within the
+ * range.
+ * Sets an error in @xas if the range is invalid or xas_create() fails
+ */
+void xas_import_node(struct xa_state *xas, struct xa_node *node)
+{
+ struct xa_node *parent = NULL;
+ void __rcu **slot = &xas->xa->xa_head;
+ int count = 0;
+
+ if (!xas->xa_shift || xas->xa_sibs) {
+ xas_set_err(xas, -EINVAL);
+ return;
+ }
+
+ if (xas->xa_index || xa_head_locked(xas->xa)) {
+ xas_set_order(xas, xas->xa_index, node->shift + XA_CHUNK_SHIFT);
+ xas_create(xas, true);
+
+ if (xas_invalid(xas))
+ return;
+
+ parent = xas->xa_node;
+ }
+
+ if (parent) {
+ slot = &parent->slots[xas->xa_offset];
+ node->offset = xas->xa_offset;
+ count++;
+ }
+
+ RCU_INIT_POINTER(node->parent, parent);
+ node->array = xas->xa;
+
+ rcu_assign_pointer(*slot, xa_mk_node(node));
+
+ update_node(xas, parent, count, 0);
+}
+
+/**
* xa_store_range() - Store this entry at a range of indices in the XArray.
* @xa: XArray.
* @first: First index to affect.
--
1.8.3.1
On 3/30/21 5:35 PM, Anthony Yznaga wrote:
> This patchset implements preserved-over-kexec memory storage or PKRAM as a
> method for saving memory pages of the currently executing kernel so that
> they may be restored after kexec into a new kernel. The patches are adapted
> from an RFC patchset sent out in 2013 by Vladimir Davydov [1]. They
> introduce the PKRAM kernel API and implement its use within tmpfs, allowing
> tmpfs files to be preserved across kexec.
>
> One use case for PKRAM is preserving guest memory and/or auxillary supporting
> data (e.g. iommu data) across kexec in support of VMM Fast Restart[2].
> VMM Fast Restart is currently using PKRAM to support preserving "Keep Alive
> State" across reboot[3]. PKRAM provides a flexible way for doing this
> without requiring that the amount of memory used by a fixed size created
> a priori. Another use case is for databases to preserve their block caches
> in shared memory across reboot.
Hi Anthony,
I have several concerns about preserving arbitrary not prereserved segments across reboot.
1. PKRAM does not work across firmware reboots
With emulated persistent memory it is possible to do reboot through firmware and not loose the preserved-memory. The firmware can be modified to mark the required ranges pages as PRAM, and Linux will treat them as such. The benefit of this is that it works for both cases kexec and reboot through firmware. The disadvantage is that you have to know in advance how much memory needs to be preserved. However, with the ability to hot-plug/hot-remove the PMEM, the second point becomes moot as it is possible to mark a large chunk of memory as PMEM if needed. I have designed something like this for one of our projects, and it is already been used in the fleet. Reboot through firmware, allows us to service firmware in addition to kernel.
2. Boot failures due to memory fragmentation
We also considered using PRAM instead of PMEM. PRAM was one of the previous attempts to do the persistent memory thing via tmpfs flag: mount -t tmpfs -o pram=mytmpfs none /mnt/crdump"; that project was never upstreamed. However, we gave up with that idea because in addition to loosing possibility to reboot through the firmware, it also adds memory fragmentation. For example, if the new kernel require larger contiguous memory chunks to be allocated during boot than the previous kernel (i.e. the next kernel has new drivers, or some debug feature enabled), the boot might simply fail because of the extra memory ranges being reserved.
3. New intra-kernel dependencies
Kexec reboot is when one Linux kernel works as a bootloader for the next one. Currently, there is very little information that is passed from the old kernel to the next kernel. Adding more information that two independent kernels must know about each other is not a very good thing from architectural point of view. It limits the flexibility of kexec.
However, we do need PKRAM and ability to preserve kernel memory across reboot for fast hypervisor updates or such. User pages can already be preserved across reboot on emulated or real persistent memory. The easiest way is via DAXFS placed on that memory.
Kernel cannot preserve its memory on PMEM across the reboot. However, functionality can be extended so kernel memory can be preserved on both emulated persistent memory or on real persistent memory. PKRAM could provide an interface to save kernel data to a file, and that file could be placed on any filesystem including DAXFS. When placed on DAXFS, that file can be used as iommu data, as it is actually located in physical memory and not moving anywhere. It is preserved across firmware/kexec reboot with having the devices survive the reboot state intact. During boot, have the device drivers that use PKRAM preserve functionality map saved files from DAXFS in order to have IOMMU functionality working again.
Thank you,
Pasha