Here's respin of my huge ext4 patchset on top of Matthew's patchset with
few changes and fixes (see below).
Please review and consider applying.
I don't see any xfstests regressions with huge pages enabled. Patch with
new configurations for xfstests-bld is below.
The basics are the same as with tmpfs[1] which is in Linus' tree now and
ext4 built on top of it. The main difference is that we need to handle
read out from and write-back to backing storage.
As with other THPs, the implementation is build around compound pages:
a naturally aligned collection of pages that memory management subsystem
[in most cases] treat as a single entity:
- head page (the first subpage) on LRU represents whole huge page;
- head page's flags represent state of whole huge page (with few
exceptions);
- mm can't migrate subpages of the compound page individually;
For THP, we use PMD-sized huge pages.
Head page links buffer heads for whole huge page. Dirty/writeback/etc.
tracking happens on per-hugepage level as all subpages share the same page
flags.
lock_page() on any subpage would lock whole hugepage for the same reason.
On radix-tree, a huge page represented as a multi-order entry of the same
order (HPAGE_PMD_ORDER). This allows us to track dirty/writeback on
radix-tree tags with the same granularity as on struct page.
On IO via syscalls, we are still limited by copying upto PAGE_SIZE per
iteration. The limitation here comes from how copy_page_to_iter() and
copy_page_from_iter() work wrt. highmem: it can only handle one small
page a time.
On write side, we also have problem with assuming small pages: write
length and offset within page calculated before we know if small or huge
page is allocated. It's not easy to fix. Looks like it would require
change in ->write_begin() interface to accept len > PAGE_SIZE.
On split_huge_page() we need to free buffers before splitting the page.
Page buffers takes additional pin on the page and can be a vector to mess
with the page during split. We want to avoid this.
If try_to_free_buffers() fails, split_huge_page() would return -EBUSY.
Readahead doesn't play with huge pages well: 128k max readahead window,
assumption on page size, PageReadahead() to track hit/miss. I've got it
to allocate huge pages, but it doesn't provide any readahead as such.
I don't know how to do this right. It's not clear at this point if we
really need readahead with huge pages. I guess it's good enough for now.
Shadow entries ignored on allocation -- recently evicted page is not
promoted to active list. Not sure if current workingset logic is adequate
for huge pages. On eviction, we split the huge page and setup 4k shadow
entries as usual.
Unlike tmpfs, ext4 makes use of tags in radix-tree. The approach I used
for tmpfs -- 512 entries in radix-tree per-hugepages -- doesn't work well
if we want to have coherent view on tags. So the first patch converts
tmpfs to use multi-order entries in radix-tree. The same infrastructure
used for ext4.
Encryption doesn't handle huge pages yet. To avoid regressions we just
disable huge pages for the inode if it has EXT4_INODE_ENCRYPT.
Tested with 4k, 1k, encryption and bigalloc. All with and without
huge=always. I think it's reasonable coverage.
The patchset is also in git:
git://git.kernel.org/pub/scm/linux/kernel/git/kas/linux.git hugeext4/v5
[1] http://lkml.kernel.org/r/1465222029-45942-1-git-send-email-kirill.shutemov@linux.intel.com
Changes since v4:
- Rebase onto updated radix-tree interface;
- Change interface to page cache lookups wrt. multi-order entries;
- Do not mess with BIO_MAX_PAGES: ext4_mpage_readpages() now uses
block_read_full_page() for THP read out;
- Fix work with memcg enabled;
- Drop bogus VM_BUG_ON() from wp_huge_pmd();
Changes since v3:
- account huge page to dirty/writeback/reclaimable/etc. according to its
size. It fixes background writback.
- move code that adds huge page to radix-tree to
page_cache_tree_insert() (Jan);
- make ramdisk work with huge pages;
- fix unaccont of shadow entries (Jan);
- use try_to_release_page() instead of try_to_free_buffers() in
split_huge_page() (Jan);
- make thp_get_unmapped_area() respect S_HUGE_MODE;
- use huge-page aligned address to zap page range in wp_huge_pmd();
- use ext4_kvmalloc in ext4_mpage_readpages() instead of
kmalloc() (Andreas);
Changes since v2:
- fix intermittent crash in generic/299;
- typo (condition inversion) in do_generic_file_read(),
reported by Jitendra;
TODO:
- on IO via syscalls, copy more than PAGE_SIZE per iteration to/from
userspace;
- readahead ?;
- wire up madvise()/fadvise();
- encryption with huge pages;
- reclaim of file huge pages can be optimized -- split_huge_page() is not
required for pages with backing storage;
>From f523dd3aad026f5a3f8cbabc0ec69958a0618f6b Mon Sep 17 00:00:00 2001
From: "Kirill A. Shutemov" <[email protected]>
Date: Fri, 12 Aug 2016 19:44:30 +0300
Subject: [PATCH] Add few more configurations to test ext4 with huge pages
Four new configurations: huge_4k, huge_1k, huge_bigalloc, huge_encrypt.
Signed-off-by: Kirill A. Shutemov <[email protected]>
---
.../test-appliance/files/root/fs/ext4/cfg/all.list | 4 ++++
.../test-appliance/files/root/fs/ext4/cfg/huge_1k | 6 ++++++
.../test-appliance/files/root/fs/ext4/cfg/huge_4k | 6 ++++++
.../test-appliance/files/root/fs/ext4/cfg/huge_bigalloc | 14 ++++++++++++++
.../files/root/fs/ext4/cfg/huge_bigalloc.exclude | 7 +++++++
.../test-appliance/files/root/fs/ext4/cfg/huge_encrypt | 5 +++++
.../files/root/fs/ext4/cfg/huge_encrypt.exclude | 16 ++++++++++++++++
kvm-xfstests/util/parse_cli | 1 +
8 files changed, 59 insertions(+)
create mode 100644 kvm-xfstests/test-appliance/files/root/fs/ext4/cfg/huge_1k
create mode 100644 kvm-xfstests/test-appliance/files/root/fs/ext4/cfg/huge_4k
create mode 100644 kvm-xfstests/test-appliance/files/root/fs/ext4/cfg/huge_bigalloc
create mode 100644 kvm-xfstests/test-appliance/files/root/fs/ext4/cfg/huge_bigalloc.exclude
create mode 100644 kvm-xfstests/test-appliance/files/root/fs/ext4/cfg/huge_encrypt
create mode 100644 kvm-xfstests/test-appliance/files/root/fs/ext4/cfg/huge_encrypt.exclude
diff --git a/kvm-xfstests/test-appliance/files/root/fs/ext4/cfg/all.list b/kvm-xfstests/test-appliance/files/root/fs/ext4/cfg/all.list
index 7ec37f4bafaa..14a8e72d2e6e 100644
--- a/kvm-xfstests/test-appliance/files/root/fs/ext4/cfg/all.list
+++ b/kvm-xfstests/test-appliance/files/root/fs/ext4/cfg/all.list
@@ -9,3 +9,7 @@ dioread_nolock
data_journal
bigalloc
bigalloc_1k
+huge_4k
+huge_1k
+huge_bigalloc
+huge_encrypt
diff --git a/kvm-xfstests/test-appliance/files/root/fs/ext4/cfg/huge_1k b/kvm-xfstests/test-appliance/files/root/fs/ext4/cfg/huge_1k
new file mode 100644
index 000000000000..209c76a8a6c1
--- /dev/null
+++ b/kvm-xfstests/test-appliance/files/root/fs/ext4/cfg/huge_1k
@@ -0,0 +1,6 @@
+export FS=ext4
+export TEST_DEV=$SM_TST_DEV
+export TEST_DIR=$SM_TST_MNT
+export MKFS_OPTIONS="-q -b 1024"
+export EXT_MOUNT_OPTIONS="huge=always"
+TESTNAME="Ext4 1k block with huge pages"
diff --git a/kvm-xfstests/test-appliance/files/root/fs/ext4/cfg/huge_4k b/kvm-xfstests/test-appliance/files/root/fs/ext4/cfg/huge_4k
new file mode 100644
index 000000000000..bae901cb2bab
--- /dev/null
+++ b/kvm-xfstests/test-appliance/files/root/fs/ext4/cfg/huge_4k
@@ -0,0 +1,6 @@
+export FS=ext4
+export TEST_DEV=$PRI_TST_DEV
+export TEST_DIR=$PRI_TST_MNT
+export MKFS_OPTIONS="-q"
+export EXT_MOUNT_OPTIONS="huge=always"
+TESTNAME="Ext4 4k block with huge pages"
diff --git a/kvm-xfstests/test-appliance/files/root/fs/ext4/cfg/huge_bigalloc b/kvm-xfstests/test-appliance/files/root/fs/ext4/cfg/huge_bigalloc
new file mode 100644
index 000000000000..b3d87562bce6
--- /dev/null
+++ b/kvm-xfstests/test-appliance/files/root/fs/ext4/cfg/huge_bigalloc
@@ -0,0 +1,14 @@
+SIZE=large
+export MKFS_OPTIONS="-O bigalloc"
+export EXT_MOUNT_OPTIONS="huge=always"
+
+# Until we can teach xfstests the difference between cluster size and
+# block size, avoid collapse_range, insert_range, and zero_range since
+# these will fail due the fact that these operations require
+# cluster-aligned ranges.
+export FSX_AVOID="-C -I -z"
+export FSSTRESS_AVOID="-f collapse=0 -f insert=0 -f zero=0"
+export XFS_IO_AVOID="fcollapse finsert zero"
+
+TESTNAME="Ext4 4k block w/bigalloc"
+
diff --git a/kvm-xfstests/test-appliance/files/root/fs/ext4/cfg/huge_bigalloc.exclude b/kvm-xfstests/test-appliance/files/root/fs/ext4/cfg/huge_bigalloc.exclude
new file mode 100644
index 000000000000..bd779be99518
--- /dev/null
+++ b/kvm-xfstests/test-appliance/files/root/fs/ext4/cfg/huge_bigalloc.exclude
@@ -0,0 +1,7 @@
+# bigalloc does not support on-line defrag
+ext4/301
+ext4/302
+ext4/303
+ext4/304
+ext4/307
+ext4/308
diff --git a/kvm-xfstests/test-appliance/files/root/fs/ext4/cfg/huge_encrypt b/kvm-xfstests/test-appliance/files/root/fs/ext4/cfg/huge_encrypt
new file mode 100644
index 000000000000..29f058ba937d
--- /dev/null
+++ b/kvm-xfstests/test-appliance/files/root/fs/ext4/cfg/huge_encrypt
@@ -0,0 +1,5 @@
+SIZE=small
+export MKFS_OPTIONS=""
+export EXT_MOUNT_OPTIONS="test_dummy_encryption,huge=always"
+REQUIRE_FEATURE=encryption
+TESTNAME="Ext4 encryption"
diff --git a/kvm-xfstests/test-appliance/files/root/fs/ext4/cfg/huge_encrypt.exclude b/kvm-xfstests/test-appliance/files/root/fs/ext4/cfg/huge_encrypt.exclude
new file mode 100644
index 000000000000..b91cc58b5aa3
--- /dev/null
+++ b/kvm-xfstests/test-appliance/files/root/fs/ext4/cfg/huge_encrypt.exclude
@@ -0,0 +1,16 @@
+ext4/004 # dump/restore doesn't handle quotas
+
+# encryption doesn't play well with quota
+generic/082
+generic/219
+generic/230
+generic/231
+generic/232
+generic/233
+generic/235
+generic/270
+
+# generic/204 tests ENOSPC handling; it doesn't correctly
+# anticipate the external extended attribute required when
+# using a 1k block size
+generic/204
diff --git a/kvm-xfstests/util/parse_cli b/kvm-xfstests/util/parse_cli
index 83400ea71985..ba64ce5df016 100644
--- a/kvm-xfstests/util/parse_cli
+++ b/kvm-xfstests/util/parse_cli
@@ -36,6 +36,7 @@ print_help ()
echo "Common file system configurations are:"
echo " 4k 1k ext3 nojournal ext3conv metacsum dioread_nolock "
echo " data_journal bigalloc bigalloc_1k inline"
+ echo " huge_4k huge_1k huge_bigalloc huge_encrypt"
echo ""
echo "xfstest names have the form: ext4/NNN generic/NNN shared/NNN"
echo ""
--
2.9.3
Kirill A. Shutemov (35):
mm, shmem: swich huge tmpfs to multi-order radix-tree entries
Revert "radix-tree: implement radix_tree_maybe_preload_order()"
page-flags: relax page flag policy for few flags
mm, rmap: account file thp pages
thp: try to free page's buffers before attempt split
thp: handle write-protection faults for file THP
filemap: allocate huge page in page_cache_read(), if allowed
filemap: handle huge pages in do_generic_file_read()
filemap: allocate huge page in pagecache_get_page(), if allowed
filemap: handle huge pages in filemap_fdatawait_range()
HACK: readahead: alloc huge pages, if allowed
brd: make it handle huge pages
mm: make write_cache_pages() work on huge pages
thp: introduce hpage_size() and hpage_mask()
thp: do not threat slab pages as huge in hpage_{nr_pages,size,mask}
thp: make thp_get_unmapped_area() respect S_HUGE_MODE
fs: make block_read_full_page() be able to read huge page
fs: make block_write_{begin,end}() be able to handle huge pages
fs: make block_page_mkwrite() aware about huge pages
truncate: make truncate_inode_pages_range() aware about huge pages
truncate: make invalidate_inode_pages2_range() aware about huge pages
mm: account huge pages to dirty, writaback, reclaimable, etc.
ext4: make ext4_mpage_readpages() hugepage-aware
ext4: make ext4_writepage() work on huge pages
ext4: handle huge pages in ext4_page_mkwrite()
ext4: handle huge pages in __ext4_block_zero_page_range()
ext4: make ext4_block_write_begin() aware about huge pages
ext4: handle huge pages in ext4_da_write_end()
ext4: make ext4_da_page_release_reservation() aware about huge pages
ext4: handle writeback with huge pages
ext4: make EXT4_IOC_MOVE_EXT work with huge pages
ext4: fix SEEK_DATA/SEEK_HOLE for huge pages
ext4: make fallocate() operations work with huge pages
mm, fs, ext4: expand use of page_mapping() and page_to_pgoff()
ext4, vfs: add huge= mount option
Naoya Horiguchi (1):
mm, hugetlb: switch hugetlbfs to multi-order radix-tree entries
drivers/base/node.c | 6 +
drivers/block/brd.c | 17 +-
fs/buffer.c | 106 ++++++-----
fs/ext4/ext4.h | 5 +
fs/ext4/extents.c | 10 +-
fs/ext4/file.c | 18 +-
fs/ext4/inode.c | 175 +++++++++++------
fs/ext4/move_extent.c | 12 +-
fs/ext4/page-io.c | 11 +-
fs/ext4/readpage.c | 2 +-
fs/ext4/super.c | 24 +++
fs/fs-writeback.c | 10 +-
fs/hugetlbfs/inode.c | 22 +--
fs/proc/meminfo.c | 4 +
fs/proc/task_mmu.c | 5 +-
include/linux/backing-dev.h | 10 +
include/linux/buffer_head.h | 10 +-
include/linux/fs.h | 5 +
include/linux/huge_mm.h | 18 +-
include/linux/memcontrol.h | 22 +--
include/linux/mm.h | 10 +-
include/linux/mmzone.h | 2 +
include/linux/page-flags.h | 12 +-
include/linux/pagemap.h | 54 +++---
include/linux/radix-tree.h | 1 -
lib/radix-tree.c | 74 --------
mm/filemap.c | 447 ++++++++++++++++++++++++++++----------------
mm/huge_memory.c | 74 ++++++--
mm/hugetlb.c | 19 +-
mm/khugepaged.c | 26 +--
mm/memory.c | 12 +-
mm/migrate.c | 1 +
mm/page-writeback.c | 99 ++++++----
mm/page_alloc.c | 5 +
mm/readahead.c | 17 +-
mm/rmap.c | 16 +-
mm/shmem.c | 117 +++++-------
mm/truncate.c | 137 ++++++++++----
mm/vmstat.c | 2 +
39 files changed, 998 insertions(+), 619 deletions(-)
--
2.10.2
Let's add FileHugePages and FilePmdMapped fields into meminfo and smaps.
It indicates how many times we allocate and map file THP.
Signed-off-by: Kirill A. Shutemov <[email protected]>
---
drivers/base/node.c | 6 ++++++
fs/proc/meminfo.c | 4 ++++
fs/proc/task_mmu.c | 5 ++++-
include/linux/mmzone.h | 2 ++
mm/filemap.c | 3 ++-
mm/huge_memory.c | 5 ++++-
mm/page_alloc.c | 5 +++++
mm/rmap.c | 12 ++++++++----
mm/vmstat.c | 2 ++
9 files changed, 37 insertions(+), 7 deletions(-)
diff --git a/drivers/base/node.c b/drivers/base/node.c
index 5548f9686016..45be0ddb84ed 100644
--- a/drivers/base/node.c
+++ b/drivers/base/node.c
@@ -116,6 +116,8 @@ static ssize_t node_read_meminfo(struct device *dev,
"Node %d AnonHugePages: %8lu kB\n"
"Node %d ShmemHugePages: %8lu kB\n"
"Node %d ShmemPmdMapped: %8lu kB\n"
+ "Node %d FileHugePages: %8lu kB\n"
+ "Node %d FilePmdMapped: %8lu kB\n"
#endif
,
nid, K(node_page_state(pgdat, NR_FILE_DIRTY)),
@@ -139,6 +141,10 @@ static ssize_t node_read_meminfo(struct device *dev,
nid, K(node_page_state(pgdat, NR_SHMEM_THPS) *
HPAGE_PMD_NR),
nid, K(node_page_state(pgdat, NR_SHMEM_PMDMAPPED) *
+ HPAGE_PMD_NR),
+ nid, K(node_page_state(pgdat, NR_FILE_THPS) *
+ HPAGE_PMD_NR),
+ nid, K(node_page_state(pgdat, NR_FILE_PMDMAPPED) *
HPAGE_PMD_NR));
#else
nid, K(sum_zone_node_page_state(nid, NR_SLAB_UNRECLAIMABLE)));
diff --git a/fs/proc/meminfo.c b/fs/proc/meminfo.c
index 8a428498d6b2..8396843be7a7 100644
--- a/fs/proc/meminfo.c
+++ b/fs/proc/meminfo.c
@@ -146,6 +146,10 @@ static int meminfo_proc_show(struct seq_file *m, void *v)
global_node_page_state(NR_SHMEM_THPS) * HPAGE_PMD_NR);
show_val_kb(m, "ShmemPmdMapped: ",
global_node_page_state(NR_SHMEM_PMDMAPPED) * HPAGE_PMD_NR);
+ show_val_kb(m, "FileHugePages: ",
+ global_node_page_state(NR_FILE_THPS) * HPAGE_PMD_NR);
+ show_val_kb(m, "FilePmdMapped: ",
+ global_node_page_state(NR_FILE_PMDMAPPED) * HPAGE_PMD_NR);
#endif
#ifdef CONFIG_CMA
diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
index d47c723e7bc2..06840421fae3 100644
--- a/fs/proc/task_mmu.c
+++ b/fs/proc/task_mmu.c
@@ -442,6 +442,7 @@ struct mem_size_stats {
unsigned long anonymous;
unsigned long anonymous_thp;
unsigned long shmem_thp;
+ unsigned long file_thp;
unsigned long swap;
unsigned long shared_hugetlb;
unsigned long private_hugetlb;
@@ -581,7 +582,7 @@ static void smaps_pmd_entry(pmd_t *pmd, unsigned long addr,
else if (is_zone_device_page(page))
/* pass */;
else
- VM_BUG_ON_PAGE(1, page);
+ mss->file_thp += HPAGE_PMD_SIZE;
mss->rss_pmd += PMD_SIZE;
smaps_account(mss, page, true, pmd_young(*pmd), pmd_dirty(*pmd));
}
@@ -848,6 +849,7 @@ static int show_smap(struct seq_file *m, void *v, int is_pid)
"Anonymous: %8lu kB\n"
"AnonHugePages: %8lu kB\n"
"ShmemPmdMapped: %8lu kB\n"
+ "FilePmdMapped: %8lu kB\n"
"Shared_Hugetlb: %8lu kB\n"
"Private_Hugetlb: %7lu kB\n"
"Swap: %8lu kB\n"
@@ -866,6 +868,7 @@ static int show_smap(struct seq_file *m, void *v, int is_pid)
mss.anonymous >> 10,
mss.anonymous_thp >> 10,
mss.shmem_thp >> 10,
+ mss.file_thp >> 10,
mss.shared_hugetlb >> 10,
mss.private_hugetlb >> 10,
mss.swap >> 10,
diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index 0f088f3a2fed..44a43f576d52 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -163,6 +163,8 @@ enum node_stat_item {
NR_SHMEM, /* shmem pages (included tmpfs/GEM pages) */
NR_SHMEM_THPS,
NR_SHMEM_PMDMAPPED,
+ NR_FILE_THPS,
+ NR_FILE_PMDMAPPED,
NR_ANON_THPS,
NR_UNSTABLE_NFS, /* NFS unstable pages */
NR_VMSCAN_WRITE,
diff --git a/mm/filemap.c b/mm/filemap.c
index f8607ab7b7e4..16d39340c106 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -240,7 +240,8 @@ void __delete_from_page_cache(struct page *page, void *shadow)
if (PageTransHuge(page))
__dec_node_page_state(page, NR_SHMEM_THPS);
} else {
- VM_BUG_ON_PAGE(PageTransHuge(page) && !PageHuge(page), page);
+ if (PageTransHuge(page) && !PageHuge(page))
+ __dec_node_page_state(page, NR_FILE_THPS);
}
/*
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 7680797b287e..91dbab9644be 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -1936,7 +1936,10 @@ static void __split_huge_page(struct page *page, struct list_head *list,
struct radix_tree_iter iter;
void **slot;
- __dec_node_page_state(head, NR_SHMEM_THPS);
+ if (PageSwapBacked(page))
+ __dec_node_page_state(page, NR_SHMEM_THPS);
+ else
+ __dec_node_page_state(page, NR_FILE_THPS);
radix_tree_split(&mapping->page_tree, head->index, 0);
radix_tree_for_each_slot(slot, &mapping->page_tree, &iter,
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index bb668eab5ee4..916be2f49f9a 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -4336,6 +4336,8 @@ void show_free_areas(unsigned int filter)
#ifdef CONFIG_TRANSPARENT_HUGEPAGE
" shmem_thp: %lukB"
" shmem_pmdmapped: %lukB"
+ " file_thp: %lukB"
+ " file_pmdmapped: %lukB"
" anon_thp: %lukB"
#endif
" writeback_tmp:%lukB"
@@ -4358,6 +4360,9 @@ void show_free_areas(unsigned int filter)
K(node_page_state(pgdat, NR_SHMEM_THPS) * HPAGE_PMD_NR),
K(node_page_state(pgdat, NR_SHMEM_PMDMAPPED)
* HPAGE_PMD_NR),
+ K(node_page_state(pgdat, NR_FILE_THPS) * HPAGE_PMD_NR),
+ K(node_page_state(pgdat, NR_FILE_PMDMAPPED)
+ * HPAGE_PMD_NR),
K(node_page_state(pgdat, NR_ANON_THPS) * HPAGE_PMD_NR),
#endif
K(node_page_state(pgdat, NR_SHMEM)),
diff --git a/mm/rmap.c b/mm/rmap.c
index 1ef36404e7b2..48c7310639bd 100644
--- a/mm/rmap.c
+++ b/mm/rmap.c
@@ -1281,8 +1281,10 @@ void page_add_file_rmap(struct page *page, bool compound)
}
if (!atomic_inc_and_test(compound_mapcount_ptr(page)))
goto out;
- VM_BUG_ON_PAGE(!PageSwapBacked(page), page);
- __inc_node_page_state(page, NR_SHMEM_PMDMAPPED);
+ if (PageSwapBacked(page))
+ __inc_node_page_state(page, NR_SHMEM_PMDMAPPED);
+ else
+ __inc_node_page_state(page, NR_FILE_PMDMAPPED);
} else {
if (PageTransCompound(page) && page_mapping(page)) {
VM_WARN_ON_ONCE(!PageLocked(page));
@@ -1322,8 +1324,10 @@ static void page_remove_file_rmap(struct page *page, bool compound)
}
if (!atomic_add_negative(-1, compound_mapcount_ptr(page)))
goto out;
- VM_BUG_ON_PAGE(!PageSwapBacked(page), page);
- __dec_node_page_state(page, NR_SHMEM_PMDMAPPED);
+ if (PageSwapBacked(page))
+ __dec_node_page_state(page, NR_SHMEM_PMDMAPPED);
+ else
+ __dec_node_page_state(page, NR_FILE_PMDMAPPED);
} else {
if (!atomic_add_negative(-1, &page->_mapcount))
goto out;
diff --git a/mm/vmstat.c b/mm/vmstat.c
index 604f26a4f696..04dc6bd8ee43 100644
--- a/mm/vmstat.c
+++ b/mm/vmstat.c
@@ -967,6 +967,8 @@ const char * const vmstat_text[] = {
"nr_shmem",
"nr_shmem_hugepages",
"nr_shmem_pmdmapped",
+ "nr_file_hugepaged",
+ "nr_file_pmdmapped",
"nr_anon_transparent_hugepages",
"nr_unstable",
"nr_vmscan_write",
--
2.10.2
For filesystems that wants to be write-notified (has mkwrite), we will
encount write-protection faults for huge PMDs in shared mappings.
The easiest way to handle them is to clear the PMD and let it refault as
wriable.
Signed-off-by: Kirill A. Shutemov <[email protected]>
Reviewed-by: Jan Kara <[email protected]>
---
mm/memory.c | 10 +++++++++-
1 file changed, 9 insertions(+), 1 deletion(-)
diff --git a/mm/memory.c b/mm/memory.c
index 455c3e628d52..e3d7cea8cc6a 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -3495,8 +3495,16 @@ static int wp_huge_pmd(struct vm_fault *vmf, pmd_t orig_pmd)
return vmf->vma->vm_ops->pmd_fault(vmf->vma, vmf->address,
vmf->pmd, vmf->flags);
+ if (vmf->vma->vm_flags & VM_SHARED) {
+ /* Clear PMD */
+ zap_page_range_single(vmf->vma, vmf->address & HPAGE_PMD_MASK,
+ HPAGE_PMD_SIZE, NULL);
+
+ /* Refault to establish writable PMD */
+ return 0;
+ }
+
/* COW handled on pte level: split pmd */
- VM_BUG_ON_VMA(vmf->vma->vm_flags & VM_SHARED, vmf->vma);
__split_huge_pmd(vmf->vma, vmf->pmd, vmf->address, false, NULL);
return VM_FAULT_FALLBACK;
--
2.10.2
Do not assume length of bio segment is never larger than PAGE_SIZE.
With huge pages it's HPAGE_PMD_SIZE (2M on x86-64).
Signed-off-by: Kirill A. Shutemov <[email protected]>
---
drivers/block/brd.c | 17 ++++++++++++-----
1 file changed, 12 insertions(+), 5 deletions(-)
diff --git a/drivers/block/brd.c b/drivers/block/brd.c
index ad793f35632c..fd050445c5d7 100644
--- a/drivers/block/brd.c
+++ b/drivers/block/brd.c
@@ -202,12 +202,15 @@ static int copy_to_brd_setup(struct brd_device *brd, sector_t sector, size_t n)
size_t copy;
copy = min_t(size_t, n, PAGE_SIZE - offset);
+ n -= copy;
if (!brd_insert_page(brd, sector))
return -ENOSPC;
- if (copy < n) {
+ while (n) {
sector += copy >> SECTOR_SHIFT;
if (!brd_insert_page(brd, sector))
return -ENOSPC;
+ copy = min_t(size_t, n, PAGE_SIZE);
+ n -= copy;
}
return 0;
}
@@ -242,6 +245,7 @@ static void copy_to_brd(struct brd_device *brd, const void *src,
size_t copy;
copy = min_t(size_t, n, PAGE_SIZE - offset);
+ n -= copy;
page = brd_lookup_page(brd, sector);
BUG_ON(!page);
@@ -249,10 +253,11 @@ static void copy_to_brd(struct brd_device *brd, const void *src,
memcpy(dst + offset, src, copy);
kunmap_atomic(dst);
- if (copy < n) {
+ while (n) {
src += copy;
sector += copy >> SECTOR_SHIFT;
- copy = n - copy;
+ copy = min_t(size_t, n, PAGE_SIZE);
+ n -= copy;
page = brd_lookup_page(brd, sector);
BUG_ON(!page);
@@ -274,6 +279,7 @@ static void copy_from_brd(void *dst, struct brd_device *brd,
size_t copy;
copy = min_t(size_t, n, PAGE_SIZE - offset);
+ n -= copy;
page = brd_lookup_page(brd, sector);
if (page) {
src = kmap_atomic(page);
@@ -282,10 +288,11 @@ static void copy_from_brd(void *dst, struct brd_device *brd,
} else
memset(dst, 0, copy);
- if (copy < n) {
+ while (n) {
dst += copy;
sector += copy >> SECTOR_SHIFT;
- copy = n - copy;
+ copy = min_t(size_t, n, PAGE_SIZE);
+ n -= copy;
page = brd_lookup_page(brd, sector);
if (page) {
src = kmap_atomic(page);
--
2.10.2
Write path allocate pages using pagecache_get_page(). We should be able
to allocate huge pages there, if it's allowed. As usually, fallback to
small pages, if failed.
Signed-off-by: Kirill A. Shutemov <[email protected]>
---
mm/filemap.c | 17 +++++++++++++++--
1 file changed, 15 insertions(+), 2 deletions(-)
diff --git a/mm/filemap.c b/mm/filemap.c
index 6a2f9ea521fb..ec976ddcb88a 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -1237,13 +1237,16 @@ struct page *pagecache_get_page(struct address_space *mapping, pgoff_t offset,
no_page:
if (!page && (fgp_flags & FGP_CREAT)) {
+ pgoff_t hoffset;
int err;
if ((fgp_flags & FGP_WRITE) && mapping_cap_account_dirty(mapping))
gfp_mask |= __GFP_WRITE;
if (fgp_flags & FGP_NOFS)
gfp_mask &= ~__GFP_FS;
- page = __page_cache_alloc(gfp_mask);
+ page = page_cache_alloc_huge(mapping, offset, gfp_mask);
+no_huge: if (!page)
+ page = __page_cache_alloc(gfp_mask);
if (!page)
return NULL;
@@ -1254,9 +1257,19 @@ struct page *pagecache_get_page(struct address_space *mapping, pgoff_t offset,
if (fgp_flags & FGP_ACCESSED)
__SetPageReferenced(page);
- err = add_to_page_cache_lru(page, mapping, offset,
+ if (PageTransHuge(page))
+ hoffset = round_down(offset, HPAGE_PMD_NR);
+ else
+ hoffset = offset;
+
+ err = add_to_page_cache_lru(page, mapping, hoffset,
gfp_mask & GFP_RECLAIM_MASK);
if (unlikely(err)) {
+ if (PageTransHuge(page)) {
+ put_page(page);
+ page = NULL;
+ goto no_huge;
+ }
put_page(page);
page = NULL;
if (err == -EEXIST)
--
2.10.2
This reverts commit 356e1c23292a4f63cfdf1daf0e0ddada51f32de8.
After conversion of huge tmpfs to multi-order entries, we don't need
this anymore.
Signed-off-by: Kirill A. Shutemov <[email protected]>
---
include/linux/radix-tree.h | 1 -
lib/radix-tree.c | 74 ----------------------------------------------
2 files changed, 75 deletions(-)
diff --git a/include/linux/radix-tree.h b/include/linux/radix-tree.h
index d0690691d9bf..6563fe64cf69 100644
--- a/include/linux/radix-tree.h
+++ b/include/linux/radix-tree.h
@@ -323,7 +323,6 @@ unsigned int radix_tree_gang_lookup_slot(struct radix_tree_root *root,
unsigned long first_index, unsigned int max_items);
int radix_tree_preload(gfp_t gfp_mask);
int radix_tree_maybe_preload(gfp_t gfp_mask);
-int radix_tree_maybe_preload_order(gfp_t gfp_mask, int order);
void radix_tree_init(void);
void *radix_tree_tag_set(struct radix_tree_root *root,
unsigned long index, unsigned int tag);
diff --git a/lib/radix-tree.c b/lib/radix-tree.c
index 5e8fc32697b1..d298ddbbbfec 100644
--- a/lib/radix-tree.c
+++ b/lib/radix-tree.c
@@ -39,9 +39,6 @@
#include <linux/string.h>
-/* Number of nodes in fully populated tree of given height */
-static unsigned long height_to_maxnodes[RADIX_TREE_MAX_PATH + 1] __read_mostly;
-
/*
* Radix tree node cache.
*/
@@ -523,51 +520,6 @@ int radix_tree_split_preload(unsigned int old_order, unsigned int new_order,
}
#endif
-/*
- * The same as function above, but preload number of nodes required to insert
- * (1 << order) continuous naturally-aligned elements.
- */
-int radix_tree_maybe_preload_order(gfp_t gfp_mask, int order)
-{
- unsigned long nr_subtrees;
- int nr_nodes, subtree_height;
-
- /* Preloading doesn't help anything with this gfp mask, skip it */
- if (!gfpflags_allow_blocking(gfp_mask)) {
- preempt_disable();
- return 0;
- }
-
- /*
- * Calculate number and height of fully populated subtrees it takes to
- * store (1 << order) elements.
- */
- nr_subtrees = 1 << order;
- for (subtree_height = 0; nr_subtrees > RADIX_TREE_MAP_SIZE;
- subtree_height++)
- nr_subtrees >>= RADIX_TREE_MAP_SHIFT;
-
- /*
- * The worst case is zero height tree with a single item at index 0 and
- * then inserting items starting at ULONG_MAX - (1 << order).
- *
- * This requires RADIX_TREE_MAX_PATH nodes to build branch from root to
- * 0-index item.
- */
- nr_nodes = RADIX_TREE_MAX_PATH;
-
- /* Plus branch to fully populated subtrees. */
- nr_nodes += RADIX_TREE_MAX_PATH - subtree_height;
-
- /* Root node is shared. */
- nr_nodes--;
-
- /* Plus nodes required to build subtrees. */
- nr_nodes += nr_subtrees * height_to_maxnodes[subtree_height];
-
- return __radix_tree_preload(gfp_mask, nr_nodes);
-}
-
static unsigned radix_tree_load_root(struct radix_tree_root *root,
struct radix_tree_node **nodep, unsigned long *maxindex)
{
@@ -2454,31 +2406,6 @@ radix_tree_node_ctor(void *arg)
INIT_LIST_HEAD(&node->private_list);
}
-static __init unsigned long __maxindex(unsigned int height)
-{
- unsigned int width = height * RADIX_TREE_MAP_SHIFT;
- int shift = RADIX_TREE_INDEX_BITS - width;
-
- if (shift < 0)
- return ~0UL;
- if (shift >= BITS_PER_LONG)
- return 0UL;
- return ~0UL >> shift;
-}
-
-static __init void radix_tree_init_maxnodes(void)
-{
- unsigned long height_to_maxindex[RADIX_TREE_MAX_PATH + 1];
- unsigned int i, j;
-
- for (i = 0; i < ARRAY_SIZE(height_to_maxindex); i++)
- height_to_maxindex[i] = __maxindex(i);
- for (i = 0; i < ARRAY_SIZE(height_to_maxnodes); i++) {
- for (j = i; j > 0; j--)
- height_to_maxnodes[i] += height_to_maxindex[j - 1] + 1;
- }
-}
-
static int radix_tree_cpu_dead(unsigned int cpu)
{
struct radix_tree_preload *rtp;
@@ -2502,7 +2429,6 @@ void __init radix_tree_init(void)
sizeof(struct radix_tree_node), 0,
SLAB_PANIC | SLAB_RECLAIM_ACCOUNT,
radix_tree_node_ctor);
- radix_tree_init_maxnodes();
ret = cpuhp_setup_state_nocalls(CPUHP_RADIX_DEAD, "lib/radix:dead",
NULL, radix_tree_cpu_dead);
WARN_ON(ret < 0);
--
2.10.2
We writeback whole huge page a time. Let's adjust iteration this way.
Signed-off-by: Kirill A. Shutemov <[email protected]>
---
include/linux/mm.h | 1 +
include/linux/pagemap.h | 1 +
mm/page-writeback.c | 17 ++++++++++++-----
3 files changed, 14 insertions(+), 5 deletions(-)
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 4424784ac374..582844ca0b23 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -1045,6 +1045,7 @@ extern pgoff_t __page_file_index(struct page *page);
*/
static inline pgoff_t page_index(struct page *page)
{
+ page = compound_head(page);
if (unlikely(PageSwapCache(page)))
return __page_file_index(page);
return page->index;
diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h
index e530e7b3b6b2..faa3fa173939 100644
--- a/include/linux/pagemap.h
+++ b/include/linux/pagemap.h
@@ -546,6 +546,7 @@ static inline void wait_on_page_locked(struct page *page)
*/
static inline void wait_on_page_writeback(struct page *page)
{
+ page = compound_head(page);
if (PageWriteback(page))
wait_on_page_bit(page, PG_writeback);
}
diff --git a/mm/page-writeback.c b/mm/page-writeback.c
index 290e8b7d3181..47d5b12c460e 100644
--- a/mm/page-writeback.c
+++ b/mm/page-writeback.c
@@ -2209,7 +2209,7 @@ int write_cache_pages(struct address_space *mapping,
* mapping. However, page->index will not change
* because we have a reference on the page.
*/
- if (page->index > end) {
+ if (page_to_pgoff(page) > end) {
/*
* can't be range_cyclic (1st pass) because
* end == -1 in that case.
@@ -2218,7 +2218,12 @@ int write_cache_pages(struct address_space *mapping,
break;
}
- done_index = page->index;
+ done_index = page_to_pgoff(page);
+ if (PageTransCompound(page)) {
+ index = round_up(index + 1, HPAGE_PMD_NR);
+ i += HPAGE_PMD_NR -
+ done_index % HPAGE_PMD_NR - 1;
+ }
lock_page(page);
@@ -2230,7 +2235,7 @@ int write_cache_pages(struct address_space *mapping,
* even if there is now a new, dirty page at the same
* pagecache address.
*/
- if (unlikely(page->mapping != mapping)) {
+ if (unlikely(page_mapping(page) != mapping)) {
continue_unlock:
unlock_page(page);
continue;
@@ -2268,7 +2273,8 @@ int write_cache_pages(struct address_space *mapping,
* not be suitable for data integrity
* writeout).
*/
- done_index = page->index + 1;
+ done_index = compound_head(page)->index
+ + hpage_nr_pages(page);
done = 1;
break;
}
@@ -2280,7 +2286,8 @@ int write_cache_pages(struct address_space *mapping,
* keep going until we have written all the pages
* we tagged for writeback prior to entering this loop.
*/
- if (--wbc->nr_to_write <= 0 &&
+ wbc->nr_to_write -= hpage_nr_pages(page);
+ if (wbc->nr_to_write <= 0 &&
wbc->sync_mode == WB_SYNC_NONE) {
done = 1;
break;
--
2.10.2
Change ext4_writepage() and underlying ext4_bio_write_page().
It basically removes assumption on page size, infer it from struct page
instead.
Signed-off-by: Kirill A. Shutemov <[email protected]>
---
fs/ext4/inode.c | 10 +++++-----
fs/ext4/page-io.c | 11 +++++++++--
2 files changed, 14 insertions(+), 7 deletions(-)
diff --git a/fs/ext4/inode.c b/fs/ext4/inode.c
index ebccc535b15e..fa4467e4b129 100644
--- a/fs/ext4/inode.c
+++ b/fs/ext4/inode.c
@@ -2037,10 +2037,10 @@ static int ext4_writepage(struct page *page,
trace_ext4_writepage(page);
size = i_size_read(inode);
- if (page->index == size >> PAGE_SHIFT)
- len = size & ~PAGE_MASK;
- else
- len = PAGE_SIZE;
+
+ len = hpage_size(page);
+ if (page->index + hpage_nr_pages(page) - 1 == size >> PAGE_SHIFT)
+ len = size & ~hpage_mask(page);
page_bufs = page_buffers(page);
/*
@@ -2064,7 +2064,7 @@ static int ext4_writepage(struct page *page,
ext4_bh_delay_or_unwritten)) {
redirty_page_for_writepage(wbc, page);
if ((current->flags & PF_MEMALLOC) ||
- (inode->i_sb->s_blocksize == PAGE_SIZE)) {
+ (inode->i_sb->s_blocksize == hpage_size(page))) {
/*
* For memory cleaning there's no point in writing only
* some buffers. So just bail out. Warn if we came here
diff --git a/fs/ext4/page-io.c b/fs/ext4/page-io.c
index d83b0f3c5fe9..360c74daec5c 100644
--- a/fs/ext4/page-io.c
+++ b/fs/ext4/page-io.c
@@ -413,6 +413,7 @@ int ext4_bio_write_page(struct ext4_io_submit *io,
BUG_ON(!PageLocked(page));
BUG_ON(PageWriteback(page));
+ BUG_ON(PageTail(page));
if (keep_towrite)
set_page_writeback_keepwrite(page);
@@ -429,8 +430,14 @@ int ext4_bio_write_page(struct ext4_io_submit *io,
* the page size, the remaining memory is zeroed when mapped, and
* writes to that region are not written out to the file."
*/
- if (len < PAGE_SIZE)
- zero_user_segment(page, len, PAGE_SIZE);
+ if (len < hpage_size(page)) {
+ page += len / PAGE_SIZE;
+ if (len % PAGE_SIZE)
+ zero_user_segment(page, len % PAGE_SIZE, PAGE_SIZE);
+ while (page + 1 == compound_head(page))
+ clear_highpage(++page);
+ page = compound_head(page);
+ }
/*
* In the first loop we prepare and mark buffers to submit. We have to
* mark all buffers in the page before submitting so that
--
2.10.2
Call ext4_da_should_update_i_disksize() for head page with offset
relative to head page.
Signed-off-by: Kirill A. Shutemov <[email protected]>
---
fs/ext4/inode.c | 7 +++----
1 file changed, 3 insertions(+), 4 deletions(-)
diff --git a/fs/ext4/inode.c b/fs/ext4/inode.c
index 21662bcbbbcb..e89249c03d2f 100644
--- a/fs/ext4/inode.c
+++ b/fs/ext4/inode.c
@@ -3019,7 +3019,6 @@ static int ext4_da_write_end(struct file *file,
int ret = 0, ret2;
handle_t *handle = ext4_journal_current_handle();
loff_t new_i_size;
- unsigned long start, end;
int write_mode = (int)(unsigned long)fsdata;
if (write_mode == FALL_BACK_TO_NONDELALLOC)
@@ -3027,8 +3026,6 @@ static int ext4_da_write_end(struct file *file,
len, copied, page, fsdata);
trace_ext4_da_write_end(inode, pos, len, copied);
- start = pos & (PAGE_SIZE - 1);
- end = start + copied - 1;
/*
* generic_write_end() will run mark_inode_dirty() if i_size
@@ -3037,8 +3034,10 @@ static int ext4_da_write_end(struct file *file,
*/
new_i_size = pos + copied;
if (copied && new_i_size > EXT4_I(inode)->i_disksize) {
+ struct page *head = compound_head(page);
+ unsigned long end = (pos & ~hpage_mask(head)) + copied - 1;
if (ext4_has_inline_data(inode) ||
- ext4_da_should_update_i_disksize(page, end)) {
+ ext4_da_should_update_i_disksize(head, end)) {
ext4_update_i_disksize(inode, new_i_size);
/* We need to mark inode dirty even if
* new_i_size is less that inode->i_size
--
2.10.2
It simply matches changes to __block_write_begin_int().
Signed-off-by: Kirill A. Shutemov <[email protected]>
---
fs/ext4/inode.c | 35 +++++++++++++++++++++--------------
1 file changed, 21 insertions(+), 14 deletions(-)
diff --git a/fs/ext4/inode.c b/fs/ext4/inode.c
index d3143dfe9962..21662bcbbbcb 100644
--- a/fs/ext4/inode.c
+++ b/fs/ext4/inode.c
@@ -1094,9 +1094,8 @@ int do_journal_get_write_access(handle_t *handle,
static int ext4_block_write_begin(struct page *page, loff_t pos, unsigned len,
get_block_t *get_block)
{
- unsigned from = pos & (PAGE_SIZE - 1);
- unsigned to = from + len;
- struct inode *inode = page->mapping->host;
+ unsigned from, to;
+ struct inode *inode = page_mapping(page)->host;
unsigned block_start, block_end;
sector_t block;
int err = 0;
@@ -1104,10 +1103,14 @@ static int ext4_block_write_begin(struct page *page, loff_t pos, unsigned len,
unsigned bbits;
struct buffer_head *bh, *head, *wait[2], **wait_bh = wait;
bool decrypt = false;
+ bool uptodate = PageUptodate(page);
+ page = compound_head(page);
+ from = pos & ~hpage_mask(page);
+ to = from + len;
BUG_ON(!PageLocked(page));
- BUG_ON(from > PAGE_SIZE);
- BUG_ON(to > PAGE_SIZE);
+ BUG_ON(from > hpage_size(page));
+ BUG_ON(to > hpage_size(page));
BUG_ON(from > to);
if (!page_has_buffers(page))
@@ -1120,10 +1123,8 @@ static int ext4_block_write_begin(struct page *page, loff_t pos, unsigned len,
block++, block_start = block_end, bh = bh->b_this_page) {
block_end = block_start + blocksize;
if (block_end <= from || block_start >= to) {
- if (PageUptodate(page)) {
- if (!buffer_uptodate(bh))
- set_buffer_uptodate(bh);
- }
+ if (uptodate && !buffer_uptodate(bh))
+ set_buffer_uptodate(bh);
continue;
}
if (buffer_new(bh))
@@ -1135,19 +1136,25 @@ static int ext4_block_write_begin(struct page *page, loff_t pos, unsigned len,
break;
if (buffer_new(bh)) {
clean_bdev_bh_alias(bh);
- if (PageUptodate(page)) {
+ if (uptodate) {
clear_buffer_new(bh);
set_buffer_uptodate(bh);
mark_buffer_dirty(bh);
continue;
}
- if (block_end > to || block_start < from)
- zero_user_segments(page, to, block_end,
- block_start, from);
+ if (block_end > to || block_start < from) {
+ BUG_ON(to - from > PAGE_SIZE);
+ zero_user_segments(page +
+ block_start / PAGE_SIZE,
+ to % PAGE_SIZE,
+ (block_start % PAGE_SIZE) + blocksize,
+ block_start % PAGE_SIZE,
+ from % PAGE_SIZE);
+ }
continue;
}
}
- if (PageUptodate(page)) {
+ if (uptodate) {
if (!buffer_uptodate(bh))
set_buffer_uptodate(bh);
continue;
--
2.10.2
Most page cache allocation happens via readahead (sync or async), so if
we want to have significant number of huge pages in page cache we need
to find a ways to allocate them from readahead.
Unfortunately, huge pages doesn't fit into current readahead design:
128 max readahead window, assumption on page size, PageReadahead() to
track hit/miss.
I haven't found a ways to get it right yet.
This patch just allocates huge page if allowed, but doesn't really
provide any readahead if huge page is allocated. We read out 2M a time
and I would expect spikes in latancy without readahead.
Therefore HACK.
Having that said, I don't think it should prevent huge page support to
be applied. Future will show if lacking readahead is a big deal with
huge pages in page cache.
Any suggestions are welcome.
Signed-off-by: Kirill A. Shutemov <[email protected]>
---
mm/readahead.c | 17 ++++++++++++++++-
1 file changed, 16 insertions(+), 1 deletion(-)
diff --git a/mm/readahead.c b/mm/readahead.c
index fb4c99f85618..87e38b522645 100644
--- a/mm/readahead.c
+++ b/mm/readahead.c
@@ -174,6 +174,21 @@ int __do_page_cache_readahead(struct address_space *mapping, struct file *filp,
if (page_offset > end_index)
break;
+ if (IS_ENABLED(CONFIG_TRANSPARENT_HUGE_PAGECACHE) &&
+ (!page_idx || !(page_offset % HPAGE_PMD_NR)) &&
+ page_cache_allow_huge(mapping, page_offset)) {
+ page = __page_cache_alloc_order(gfp_mask | __GFP_COMP,
+ HPAGE_PMD_ORDER);
+ if (page) {
+ prep_transhuge_page(page);
+ page->index = round_down(page_offset,
+ HPAGE_PMD_NR);
+ list_add(&page->lru, &page_pool);
+ ret++;
+ goto start_io;
+ }
+ }
+
rcu_read_lock();
page = radix_tree_lookup(&mapping->page_tree, page_offset);
rcu_read_unlock();
@@ -189,7 +204,7 @@ int __do_page_cache_readahead(struct address_space *mapping, struct file *filp,
SetPageReadahead(page);
ret++;
}
-
+start_io:
/*
* Now start the IO. We ignore I/O errors - if the page is not
* uptodate then the caller will launch readpage again, and
--
2.10.2
We want mmap(NULL) to return PMD-aligned address if the inode can have
huge pages in page cache.
Signed-off-by: Kirill A. Shutemov <[email protected]>
---
mm/huge_memory.c | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index a15d566b14f6..9c6ba124ba50 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -518,10 +518,12 @@ unsigned long thp_get_unmapped_area(struct file *filp, unsigned long addr,
unsigned long len, unsigned long pgoff, unsigned long flags)
{
loff_t off = (loff_t)pgoff << PAGE_SHIFT;
+ struct inode *inode = filp->f_mapping->host;
if (addr)
goto out;
- if (!IS_DAX(filp->f_mapping->host) || !IS_ENABLED(CONFIG_FS_DAX_PMD))
+ if ((inode->i_flags & S_HUGE_MODE) == S_HUGE_NEVER &&
+ (!IS_DAX(inode) || !IS_ENABLED(CONFIG_FS_DAX_PMD)))
goto out;
addr = __thp_get_unmapped_area(filp, len, off, flags, PMD_SIZE);
--
2.10.2
We need to account huge pages according to its size to get background
writaback work properly.
Signed-off-by: Kirill A. Shutemov <[email protected]>
---
fs/fs-writeback.c | 10 +++---
include/linux/backing-dev.h | 10 ++++++
include/linux/memcontrol.h | 22 ++-----------
mm/migrate.c | 1 +
mm/page-writeback.c | 80 +++++++++++++++++++++++++++++----------------
mm/rmap.c | 4 +--
6 files changed, 74 insertions(+), 53 deletions(-)
diff --git a/fs/fs-writeback.c b/fs/fs-writeback.c
index ef600591d96f..e1c9faddc9e1 100644
--- a/fs/fs-writeback.c
+++ b/fs/fs-writeback.c
@@ -366,8 +366,9 @@ static void inode_switch_wbs_work_fn(struct work_struct *work)
struct page *page = radix_tree_deref_slot_protected(slot,
&mapping->tree_lock);
if (likely(page) && PageDirty(page)) {
- __dec_wb_stat(old_wb, WB_RECLAIMABLE);
- __inc_wb_stat(new_wb, WB_RECLAIMABLE);
+ int nr = hpage_nr_pages(page);
+ __add_wb_stat(old_wb, WB_RECLAIMABLE, -nr);
+ __add_wb_stat(new_wb, WB_RECLAIMABLE, nr);
}
}
@@ -376,9 +377,10 @@ static void inode_switch_wbs_work_fn(struct work_struct *work)
struct page *page = radix_tree_deref_slot_protected(slot,
&mapping->tree_lock);
if (likely(page)) {
+ int nr = hpage_nr_pages(page);
WARN_ON_ONCE(!PageWriteback(page));
- __dec_wb_stat(old_wb, WB_WRITEBACK);
- __inc_wb_stat(new_wb, WB_WRITEBACK);
+ __add_wb_stat(old_wb, WB_WRITEBACK, -nr);
+ __add_wb_stat(new_wb, WB_WRITEBACK, nr);
}
}
diff --git a/include/linux/backing-dev.h b/include/linux/backing-dev.h
index 43b93a947e61..e63487f78824 100644
--- a/include/linux/backing-dev.h
+++ b/include/linux/backing-dev.h
@@ -61,6 +61,16 @@ static inline void __add_wb_stat(struct bdi_writeback *wb,
__percpu_counter_add(&wb->stat[item], amount, WB_STAT_BATCH);
}
+static inline void add_wb_stat(struct bdi_writeback *wb,
+ enum wb_stat_item item, s64 amount)
+{
+ unsigned long flags;
+
+ local_irq_save(flags);
+ __add_wb_stat(wb, item, amount);
+ local_irq_restore(flags);
+}
+
static inline void __inc_wb_stat(struct bdi_writeback *wb,
enum wb_stat_item item)
{
diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h
index 61d20c17f3b7..df014eff82da 100644
--- a/include/linux/memcontrol.h
+++ b/include/linux/memcontrol.h
@@ -29,6 +29,7 @@
#include <linux/mmzone.h>
#include <linux/writeback.h>
#include <linux/page-flags.h>
+#include <linux/mm.h>
struct mem_cgroup;
struct page;
@@ -503,18 +504,6 @@ static inline void mem_cgroup_update_page_stat(struct page *page,
this_cpu_add(page->mem_cgroup->stat->count[idx], val);
}
-static inline void mem_cgroup_inc_page_stat(struct page *page,
- enum mem_cgroup_stat_index idx)
-{
- mem_cgroup_update_page_stat(page, idx, 1);
-}
-
-static inline void mem_cgroup_dec_page_stat(struct page *page,
- enum mem_cgroup_stat_index idx)
-{
- mem_cgroup_update_page_stat(page, idx, -1);
-}
-
unsigned long mem_cgroup_soft_limit_reclaim(pg_data_t *pgdat, int order,
gfp_t gfp_mask,
unsigned long *total_scanned);
@@ -719,13 +708,8 @@ static inline bool mem_cgroup_oom_synchronize(bool wait)
return false;
}
-static inline void mem_cgroup_inc_page_stat(struct page *page,
- enum mem_cgroup_stat_index idx)
-{
-}
-
-static inline void mem_cgroup_dec_page_stat(struct page *page,
- enum mem_cgroup_stat_index idx)
+static inline void mem_cgroup_update_page_stat(struct page *page,
+ enum mem_cgroup_stat_index idx, int val)
{
}
diff --git a/mm/migrate.c b/mm/migrate.c
index 0ed24b1fa77b..c274f9d8ac2b 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -505,6 +505,7 @@ int migrate_page_move_mapping(struct address_space *mapping,
* are mapped to swap space.
*/
if (newzone != oldzone) {
+ BUG_ON(PageTransHuge(page));
__dec_node_state(oldzone->zone_pgdat, NR_FILE_PAGES);
__inc_node_state(newzone->zone_pgdat, NR_FILE_PAGES);
if (PageSwapBacked(page) && !PageSwapCache(page)) {
diff --git a/mm/page-writeback.c b/mm/page-writeback.c
index 47d5b12c460e..d7b905d66add 100644
--- a/mm/page-writeback.c
+++ b/mm/page-writeback.c
@@ -2430,19 +2430,22 @@ void account_page_dirtied(struct page *page, struct address_space *mapping)
if (mapping_cap_account_dirty(mapping)) {
struct bdi_writeback *wb;
+ struct zone *zone = page_zone(page);
+ pg_data_t *pgdat = page_pgdat(page);
+ int nr = hpage_nr_pages(page);
inode_attach_wb(inode, page);
wb = inode_to_wb(inode);
- mem_cgroup_inc_page_stat(page, MEM_CGROUP_STAT_DIRTY);
- __inc_node_page_state(page, NR_FILE_DIRTY);
- __inc_zone_page_state(page, NR_ZONE_WRITE_PENDING);
- __inc_node_page_state(page, NR_DIRTIED);
- __inc_wb_stat(wb, WB_RECLAIMABLE);
- __inc_wb_stat(wb, WB_DIRTIED);
- task_io_account_write(PAGE_SIZE);
- current->nr_dirtied++;
- this_cpu_inc(bdp_ratelimits);
+ mem_cgroup_update_page_stat(page, MEM_CGROUP_STAT_DIRTY, nr);
+ __mod_node_page_state(pgdat, NR_FILE_DIRTY, nr);
+ __mod_zone_page_state(zone, NR_ZONE_WRITE_PENDING, nr);
+ __mod_node_page_state(pgdat, NR_DIRTIED, nr);
+ __add_wb_stat(wb, WB_RECLAIMABLE, nr);
+ __add_wb_stat(wb, WB_DIRTIED, nr);
+ task_io_account_write(nr * PAGE_SIZE);
+ current->nr_dirtied += nr;
+ this_cpu_add(bdp_ratelimits, nr);
}
}
EXPORT_SYMBOL(account_page_dirtied);
@@ -2456,11 +2459,15 @@ void account_page_cleaned(struct page *page, struct address_space *mapping,
struct bdi_writeback *wb)
{
if (mapping_cap_account_dirty(mapping)) {
- mem_cgroup_dec_page_stat(page, MEM_CGROUP_STAT_DIRTY);
- dec_node_page_state(page, NR_FILE_DIRTY);
- dec_zone_page_state(page, NR_ZONE_WRITE_PENDING);
- dec_wb_stat(wb, WB_RECLAIMABLE);
- task_io_account_cancelled_write(PAGE_SIZE);
+ struct zone *zone = page_zone(page);
+ pg_data_t *pgdat = page_pgdat(page);
+ int nr = hpage_nr_pages(page);
+
+ mem_cgroup_update_page_stat(page, MEM_CGROUP_STAT_DIRTY, -nr);
+ mod_node_page_state(pgdat, NR_FILE_DIRTY, -nr);
+ mod_zone_page_state(zone, NR_ZONE_WRITE_PENDING, -nr);
+ add_wb_stat(wb, WB_RECLAIMABLE, -nr);
+ task_io_account_cancelled_write(PAGE_SIZE * nr);
}
}
@@ -2520,14 +2527,16 @@ void account_page_redirty(struct page *page)
struct address_space *mapping = page->mapping;
if (mapping && mapping_cap_account_dirty(mapping)) {
+ pg_data_t *pgdat = page_pgdat(page);
+ int nr = hpage_nr_pages(page);
struct inode *inode = mapping->host;
struct bdi_writeback *wb;
bool locked;
wb = unlocked_inode_to_wb_begin(inode, &locked);
- current->nr_dirtied--;
- dec_node_page_state(page, NR_DIRTIED);
- dec_wb_stat(wb, WB_DIRTIED);
+ current->nr_dirtied -= nr;
+ mod_node_page_state(pgdat, NR_DIRTIED, -nr);
+ add_wb_stat(wb, WB_DIRTIED, -nr);
unlocked_inode_to_wb_end(inode, locked);
}
}
@@ -2713,10 +2722,15 @@ int clear_page_dirty_for_io(struct page *page)
*/
wb = unlocked_inode_to_wb_begin(inode, &locked);
if (TestClearPageDirty(page)) {
- mem_cgroup_dec_page_stat(page, MEM_CGROUP_STAT_DIRTY);
- dec_node_page_state(page, NR_FILE_DIRTY);
- dec_zone_page_state(page, NR_ZONE_WRITE_PENDING);
- dec_wb_stat(wb, WB_RECLAIMABLE);
+ struct zone *zone = page_zone(page);
+ pg_data_t *pgdat = page_pgdat(page);
+ int nr = hpage_nr_pages(page);
+
+ mem_cgroup_update_page_stat(page,
+ MEM_CGROUP_STAT_DIRTY, -nr);
+ mod_node_page_state(pgdat, NR_FILE_DIRTY, -nr);
+ mod_zone_page_state(zone, NR_ZONE_WRITE_PENDING, -nr);
+ add_wb_stat(wb, WB_RECLAIMABLE, -nr);
ret = 1;
}
unlocked_inode_to_wb_end(inode, locked);
@@ -2760,10 +2774,15 @@ int test_clear_page_writeback(struct page *page)
ret = TestClearPageWriteback(page);
}
if (ret) {
- mem_cgroup_dec_page_stat(page, MEM_CGROUP_STAT_WRITEBACK);
- dec_node_page_state(page, NR_WRITEBACK);
- dec_zone_page_state(page, NR_ZONE_WRITE_PENDING);
- inc_node_page_state(page, NR_WRITTEN);
+ struct zone *zone = page_zone(page);
+ pg_data_t *pgdat = page_pgdat(page);
+ int nr = hpage_nr_pages(page);
+
+ mem_cgroup_update_page_stat(page,
+ MEM_CGROUP_STAT_WRITEBACK, -nr);
+ mod_node_page_state(pgdat, NR_WRITEBACK, -nr);
+ mod_zone_page_state(zone, NR_ZONE_WRITE_PENDING, -nr);
+ mod_node_page_state(pgdat, NR_WRITTEN, nr);
}
unlock_page_memcg(page);
return ret;
@@ -2815,9 +2834,14 @@ int __test_set_page_writeback(struct page *page, bool keep_write)
ret = TestSetPageWriteback(page);
}
if (!ret) {
- mem_cgroup_inc_page_stat(page, MEM_CGROUP_STAT_WRITEBACK);
- inc_node_page_state(page, NR_WRITEBACK);
- inc_zone_page_state(page, NR_ZONE_WRITE_PENDING);
+ struct zone *zone = page_zone(page);
+ pg_data_t *pgdat = page_pgdat(page);
+ int nr = hpage_nr_pages(page);
+
+ mem_cgroup_update_page_stat(page,
+ MEM_CGROUP_STAT_WRITEBACK, nr);
+ mod_node_page_state(pgdat, NR_WRITEBACK, nr);
+ mod_zone_page_state(zone, NR_ZONE_WRITE_PENDING, nr);
}
unlock_page_memcg(page);
return ret;
diff --git a/mm/rmap.c b/mm/rmap.c
index 48c7310639bd..b9570e784405 100644
--- a/mm/rmap.c
+++ b/mm/rmap.c
@@ -1297,7 +1297,7 @@ void page_add_file_rmap(struct page *page, bool compound)
goto out;
}
__mod_node_page_state(page_pgdat(page), NR_FILE_MAPPED, nr);
- mem_cgroup_inc_page_stat(page, MEM_CGROUP_STAT_FILE_MAPPED);
+ mem_cgroup_update_page_stat(page, MEM_CGROUP_STAT_FILE_MAPPED, nr);
out:
unlock_page_memcg(page);
}
@@ -1339,7 +1339,7 @@ static void page_remove_file_rmap(struct page *page, bool compound)
* pte lock(a spinlock) is held, which implies preemption disabled.
*/
__mod_node_page_state(page_pgdat(page), NR_FILE_MAPPED, -nr);
- mem_cgroup_dec_page_stat(page, MEM_CGROUP_STAT_FILE_MAPPED);
+ mem_cgroup_update_page_stat(page, MEM_CGROUP_STAT_FILE_MAPPED, -nr);
if (unlikely(PageMlocked(page)))
clear_page_mlock(page);
--
2.10.2
As with shmem_undo_range(), truncate_inode_pages_range() removes huge
pages, if it fully within range.
Partial truncate of huge pages zero out this part of THP.
Unlike with shmem, it doesn't prevent us having holes in the middle of
huge page we still can skip writeback not touched buffers.
With memory-mapped IO we would loose holes in some cases when we have
THP in page cache, since we cannot track access on 4k level in this
case.
Signed-off-by: Kirill A. Shutemov <[email protected]>
---
fs/buffer.c | 2 +-
include/linux/mm.h | 9 +++++-
mm/truncate.c | 86 ++++++++++++++++++++++++++++++++++++++++++++----------
3 files changed, 80 insertions(+), 17 deletions(-)
diff --git a/fs/buffer.c b/fs/buffer.c
index 8e000021513c..24daf7b9bdb0 100644
--- a/fs/buffer.c
+++ b/fs/buffer.c
@@ -1534,7 +1534,7 @@ void block_invalidatepage(struct page *page, unsigned int offset,
/*
* Check for overflow
*/
- BUG_ON(stop > PAGE_SIZE || stop < length);
+ BUG_ON(stop > hpage_size(page) || stop < length);
head = page_buffers(page);
bh = head;
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 582844ca0b23..59e74dc57359 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -1328,8 +1328,15 @@ int get_kernel_page(unsigned long start, int write, struct page **pages);
struct page *get_dump_page(unsigned long addr);
extern int try_to_release_page(struct page * page, gfp_t gfp_mask);
-extern void do_invalidatepage(struct page *page, unsigned int offset,
+extern void __do_invalidatepage(struct page *page, unsigned int offset,
unsigned int length);
+static inline void do_invalidatepage(struct page *page, unsigned int offset,
+ unsigned int length)
+{
+ if (page_has_private(page))
+ __do_invalidatepage(page, offset, length);
+}
+
int __set_page_dirty_nobuffers(struct page *page);
int __set_page_dirty_no_writeback(struct page *page);
diff --git a/mm/truncate.c b/mm/truncate.c
index eb3a3a45feb6..d2d95f283ec3 100644
--- a/mm/truncate.c
+++ b/mm/truncate.c
@@ -70,12 +70,12 @@ static void clear_exceptional_entry(struct address_space *mapping,
* point. Because the caller is about to free (and possibly reuse) those
* blocks on-disk.
*/
-void do_invalidatepage(struct page *page, unsigned int offset,
+void __do_invalidatepage(struct page *page, unsigned int offset,
unsigned int length)
{
void (*invalidatepage)(struct page *, unsigned int, unsigned int);
- invalidatepage = page->mapping->a_ops->invalidatepage;
+ invalidatepage = page_mapping(page)->a_ops->invalidatepage;
#ifdef CONFIG_BLOCK
if (!invalidatepage)
invalidatepage = block_invalidatepage;
@@ -100,8 +100,7 @@ truncate_complete_page(struct address_space *mapping, struct page *page)
if (page->mapping != mapping)
return -EIO;
- if (page_has_private(page))
- do_invalidatepage(page, 0, PAGE_SIZE);
+ do_invalidatepage(page, 0, hpage_size(page));
/*
* Some filesystems seem to re-dirty the page even after
@@ -273,13 +272,35 @@ void truncate_inode_pages_range(struct address_space *mapping,
unlock_page(page);
continue;
}
+
+ if (PageTransHuge(page)) {
+ int j, first = 0, last = HPAGE_PMD_NR - 1;
+
+ if (start > page->index)
+ first = start & (HPAGE_PMD_NR - 1);
+ if (index == round_down(end, HPAGE_PMD_NR))
+ last = (end - 1) & (HPAGE_PMD_NR - 1);
+
+ /* Range starts or ends in the middle of THP */
+ if (first != 0 || last != HPAGE_PMD_NR - 1) {
+ int off, len;
+ for (j = first; j <= last; j++)
+ clear_highpage(page + j);
+ off = first * PAGE_SIZE;
+ len = (last + 1) * PAGE_SIZE - off;
+ do_invalidatepage(page, off, len);
+ unlock_page(page);
+ continue;
+ }
+ }
+
truncate_inode_page(mapping, page);
unlock_page(page);
}
pagevec_remove_exceptionals(&pvec);
+ index += pvec.nr ? hpage_nr_pages(pvec.pages[pvec.nr - 1]) : 1;
pagevec_release(&pvec);
cond_resched();
- index++;
}
if (partial_start) {
@@ -294,9 +315,12 @@ void truncate_inode_pages_range(struct address_space *mapping,
wait_on_page_writeback(page);
zero_user_segment(page, partial_start, top);
cleancache_invalidate_page(mapping, page);
- if (page_has_private(page))
- do_invalidatepage(page, partial_start,
- top - partial_start);
+ if (page_has_private(page)) {
+ int off = page - compound_head(page);
+ do_invalidatepage(compound_head(page),
+ off * PAGE_SIZE + partial_start,
+ top - partial_start);
+ }
unlock_page(page);
put_page(page);
}
@@ -307,9 +331,12 @@ void truncate_inode_pages_range(struct address_space *mapping,
wait_on_page_writeback(page);
zero_user_segment(page, 0, partial_end);
cleancache_invalidate_page(mapping, page);
- if (page_has_private(page))
- do_invalidatepage(page, 0,
- partial_end);
+ if (page_has_private(page)) {
+ int off = page - compound_head(page);
+ do_invalidatepage(compound_head(page),
+ off * PAGE_SIZE,
+ partial_end);
+ }
unlock_page(page);
put_page(page);
}
@@ -323,7 +350,7 @@ void truncate_inode_pages_range(struct address_space *mapping,
index = start;
for ( ; ; ) {
- cond_resched();
+restart: cond_resched();
if (!pagevec_lookup_entries(&pvec, mapping, index,
min(end - index, (pgoff_t)PAGEVEC_SIZE), indices)) {
/* If all gone from start onwards, we're done */
@@ -346,8 +373,8 @@ void truncate_inode_pages_range(struct address_space *mapping,
index = indices[i];
if (index >= end) {
/* Restart punch to make sure all gone */
- index = start - 1;
- break;
+ index = start;
+ goto restart;
}
if (radix_tree_exceptional_entry(page)) {
@@ -358,12 +385,41 @@ void truncate_inode_pages_range(struct address_space *mapping,
lock_page(page);
WARN_ON(page_to_index(page) != index);
wait_on_page_writeback(page);
+
+ if (PageTransHuge(page)) {
+ int j, first = 0, last = HPAGE_PMD_NR - 1;
+
+ if (start > page->index)
+ first = start & (HPAGE_PMD_NR - 1);
+ if (index == round_down(end, HPAGE_PMD_NR))
+ last = (end - 1) & (HPAGE_PMD_NR - 1);
+
+ /*
+ * On Partial thp truncate due 'start' in
+ * middle of THP: don't need to look on these
+ * pages again on !pvec.nr restart.
+ */
+ start = page->index + HPAGE_PMD_NR;
+
+ /* Range starts or ends in the middle of THP */
+ if (first != 0 || last != HPAGE_PMD_NR - 1) {
+ int off, len;
+ for (j = first; j <= last; j++)
+ clear_highpage(page + j);
+ off = first * PAGE_SIZE;
+ len = (last + 1) * PAGE_SIZE - off;
+ do_invalidatepage(page, off, len);
+ unlock_page(page);
+ continue;
+ }
+ }
+
truncate_inode_page(mapping, page);
unlock_page(page);
}
pagevec_remove_exceptionals(&pvec);
+ index += pvec.nr ? hpage_nr_pages(pvec.pages[pvec.nr - 1]) : 1;
pagevec_release(&pvec);
- index++;
}
cleancache_invalidate_inode(mapping);
}
--
2.10.2
From: Naoya Horiguchi <[email protected]>
Currently, hugetlb pages are linked to page cache on the basis of hugepage
offset (derived from vma_hugecache_offset()) for historical reason, which
doesn't match to the generic usage of page cache and requires some routines
to covert page offset <=> hugepage offset in common path. This patch
adjusts code for multi-order radix-tree to avoid the situation.
Main change is on the behavior of page->index for hugetlbfs. Before this
patch, it represented hugepage offset, but with this patch it represents
page offset. So index-related code have to be updated.
Note that hugetlb_fault_mutex_hash() and reservation region handling are
still working with hugepage offset.
Signed-off-by: Naoya Horiguchi <[email protected]>
[[email protected]: reject fixed]
Signed-off-by: Kirill A. Shutemov <[email protected]>
---
fs/hugetlbfs/inode.c | 22 ++++++++++------------
include/linux/pagemap.h | 23 +++--------------------
mm/filemap.c | 12 +++++-------
mm/hugetlb.c | 19 ++++++-------------
mm/truncate.c | 8 ++++----
5 files changed, 28 insertions(+), 56 deletions(-)
diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c
index 4fb7b10f3a05..45992c839794 100644
--- a/fs/hugetlbfs/inode.c
+++ b/fs/hugetlbfs/inode.c
@@ -388,8 +388,8 @@ static void remove_inode_hugepages(struct inode *inode, loff_t lstart,
{
struct hstate *h = hstate_inode(inode);
struct address_space *mapping = &inode->i_data;
- const pgoff_t start = lstart >> huge_page_shift(h);
- const pgoff_t end = lend >> huge_page_shift(h);
+ const pgoff_t start = lstart >> PAGE_SHIFT;
+ const pgoff_t end = lend >> PAGE_SHIFT;
struct vm_area_struct pseudo_vma;
struct pagevec pvec;
pgoff_t next;
@@ -446,8 +446,7 @@ static void remove_inode_hugepages(struct inode *inode, loff_t lstart,
i_mmap_lock_write(mapping);
hugetlb_vmdelete_list(&mapping->i_mmap,
- next * pages_per_huge_page(h),
- (next + 1) * pages_per_huge_page(h));
+ next, next + 1);
i_mmap_unlock_write(mapping);
}
@@ -466,7 +465,8 @@ static void remove_inode_hugepages(struct inode *inode, loff_t lstart,
freed++;
if (!truncate_op) {
if (unlikely(hugetlb_unreserve_pages(inode,
- next, next + 1, 1)))
+ (next) << huge_page_order(h),
+ (next + 1) << huge_page_order(h), 1)))
hugetlb_fix_reserve_counts(inode);
}
@@ -550,8 +550,6 @@ static long hugetlbfs_fallocate(struct file *file, int mode, loff_t offset,
struct hstate *h = hstate_inode(inode);
struct vm_area_struct pseudo_vma;
struct mm_struct *mm = current->mm;
- loff_t hpage_size = huge_page_size(h);
- unsigned long hpage_shift = huge_page_shift(h);
pgoff_t start, index, end;
int error;
u32 hash;
@@ -567,8 +565,8 @@ static long hugetlbfs_fallocate(struct file *file, int mode, loff_t offset,
* For this range, start is rounded down and end is rounded up
* as well as being converted to page offsets.
*/
- start = offset >> hpage_shift;
- end = (offset + len + hpage_size - 1) >> hpage_shift;
+ start = offset >> PAGE_SHIFT;
+ end = (offset + len + huge_page_size(h) - 1) >> PAGE_SHIFT;
inode_lock(inode);
@@ -586,7 +584,7 @@ static long hugetlbfs_fallocate(struct file *file, int mode, loff_t offset,
pseudo_vma.vm_flags = (VM_HUGETLB | VM_MAYSHARE | VM_SHARED);
pseudo_vma.vm_file = file;
- for (index = start; index < end; index++) {
+ for (index = start; index < end; index += pages_per_huge_page(h)) {
/*
* This is supposed to be the vaddr where the page is being
* faulted in, but we have no vaddr here.
@@ -607,10 +605,10 @@ static long hugetlbfs_fallocate(struct file *file, int mode, loff_t offset,
}
/* Set numa allocation policy based on index */
- hugetlb_set_vma_policy(&pseudo_vma, inode, index);
+ hugetlb_set_vma_policy(&pseudo_vma, inode, index >> huge_page_order(h));
/* addr is the offset within the file (zero based) */
- addr = index * hpage_size;
+ addr = index << PAGE_SHIFT & ~huge_page_mask(h);
/* mutex taken here, fault path and hole punch */
hash = hugetlb_fault_mutex_hash(h, mm, &pseudo_vma, mapping,
diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h
index faa3fa173939..bb0b7022421e 100644
--- a/include/linux/pagemap.h
+++ b/include/linux/pagemap.h
@@ -398,10 +398,9 @@ static inline struct page *read_mapping_page(struct address_space *mapping,
}
/*
- * Get index of the page with in radix-tree
- * (TODO: remove once hugetlb pages will have ->index in PAGE_SIZE)
+ * Get the offset in PAGE_SIZE.
*/
-static inline pgoff_t page_to_index(struct page *page)
+static inline pgoff_t page_to_pgoff(struct page *page)
{
pgoff_t pgoff;
@@ -418,18 +417,6 @@ static inline pgoff_t page_to_index(struct page *page)
}
/*
- * Get the offset in PAGE_SIZE.
- * (TODO: hugepage should have ->index in PAGE_SIZE)
- */
-static inline pgoff_t page_to_pgoff(struct page *page)
-{
- if (unlikely(PageHeadHuge(page)))
- return page->index << compound_order(page);
-
- return page_to_index(page);
-}
-
-/*
* Return byte-offset into filesystem object for page.
*/
static inline loff_t page_offset(struct page *page)
@@ -442,15 +429,11 @@ static inline loff_t page_file_offset(struct page *page)
return ((loff_t)page_index(page)) << PAGE_SHIFT;
}
-extern pgoff_t linear_hugepage_index(struct vm_area_struct *vma,
- unsigned long address);
-
static inline pgoff_t linear_page_index(struct vm_area_struct *vma,
unsigned long address)
{
pgoff_t pgoff;
- if (unlikely(is_vm_hugetlb_page(vma)))
- return linear_hugepage_index(vma, address);
+
pgoff = (address - vma->vm_start) >> PAGE_SHIFT;
pgoff += vma->vm_pgoff;
return pgoff;
diff --git a/mm/filemap.c b/mm/filemap.c
index 52be2b457208..33974ad1a8ec 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -165,10 +165,7 @@ static void page_cache_tree_delete(struct address_space *mapping,
{
struct radix_tree_node *node;
void **slot;
- int nr;
-
- /* hugetlb pages are represented by one entry in the radix tree */
- nr = PageHuge(page) ? 1 : hpage_nr_pages(page);
+ int nr = hpage_nr_pages(page);
VM_BUG_ON_PAGE(!PageLocked(page), page);
VM_BUG_ON_PAGE(PageTail(page), page);
@@ -1420,16 +1417,17 @@ unsigned find_get_pages(struct address_space *mapping, pgoff_t start,
}
/* For multi-order entries, find relevant subpage */
- if (PageTransHuge(page)) {
+ if (PageCompound(page)) {
VM_BUG_ON(index - page->index < 0);
- VM_BUG_ON(index - page->index >= HPAGE_PMD_NR);
+ VM_BUG_ON(index - page->index >=
+ 1 << compound_order(page));
page += index - page->index;
}
pages[ret] = page;
if (++ret == nr_pages)
break;
- if (!PageTransCompound(page))
+ if (PageHuge(page) || !PageTransCompound(page))
continue;
for (refs = 0; ret < nr_pages &&
(index + 1) % HPAGE_PMD_NR;
diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 3faec05b1875..f359653f31ff 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -622,13 +622,6 @@ static pgoff_t vma_hugecache_offset(struct hstate *h,
(vma->vm_pgoff >> huge_page_order(h));
}
-pgoff_t linear_hugepage_index(struct vm_area_struct *vma,
- unsigned long address)
-{
- return vma_hugecache_offset(hstate_vma(vma), vma, address);
-}
-EXPORT_SYMBOL_GPL(linear_hugepage_index);
-
/*
* Return the size of the pages allocated when backing a VMA. In the majority
* cases this will be same size as used by the page table entries.
@@ -3658,7 +3651,7 @@ static struct page *hugetlbfs_pagecache_page(struct hstate *h,
pgoff_t idx;
mapping = vma->vm_file->f_mapping;
- idx = vma_hugecache_offset(h, vma, address);
+ idx = linear_page_index(vma, address);
return find_lock_page(mapping, idx);
}
@@ -3675,7 +3668,7 @@ static bool hugetlbfs_pagecache_present(struct hstate *h,
struct page *page;
mapping = vma->vm_file->f_mapping;
- idx = vma_hugecache_offset(h, vma, address);
+ idx = linear_page_index(vma, address);
page = find_get_page(mapping, idx);
if (page)
@@ -3730,7 +3723,7 @@ static int hugetlb_no_page(struct mm_struct *mm, struct vm_area_struct *vma,
retry:
page = find_lock_page(mapping, idx);
if (!page) {
- size = i_size_read(mapping->host) >> huge_page_shift(h);
+ size = i_size_read(mapping->host) >> PAGE_SHIFT;
if (idx >= size)
goto out;
page = alloc_huge_page(vma, address, 0);
@@ -3791,7 +3784,7 @@ static int hugetlb_no_page(struct mm_struct *mm, struct vm_area_struct *vma,
}
ptl = huge_pte_lock(h, mm, ptep);
- size = i_size_read(mapping->host) >> huge_page_shift(h);
+ size = i_size_read(mapping->host) >> PAGE_SHIFT;
if (idx >= size)
goto backout;
@@ -3839,7 +3832,7 @@ u32 hugetlb_fault_mutex_hash(struct hstate *h, struct mm_struct *mm,
if (vma->vm_flags & VM_SHARED) {
key[0] = (unsigned long) mapping;
- key[1] = idx;
+ key[1] = idx >> huge_page_order(h);
} else {
key[0] = (unsigned long) mm;
key[1] = address >> huge_page_shift(h);
@@ -3895,7 +3888,7 @@ int hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma,
}
mapping = vma->vm_file->f_mapping;
- idx = vma_hugecache_offset(h, vma, address);
+ idx = linear_page_index(vma, address);
/*
* Serialize hugepage allocation and instantiation, so that we don't
diff --git a/mm/truncate.c b/mm/truncate.c
index 6df4b06a190f..7508c2c7e4ed 100644
--- a/mm/truncate.c
+++ b/mm/truncate.c
@@ -267,7 +267,7 @@ void truncate_inode_pages_range(struct address_space *mapping,
if (!trylock_page(page))
continue;
- WARN_ON(page_to_index(page) != index);
+ WARN_ON(page_to_pgoff(page) != index);
if (PageWriteback(page)) {
unlock_page(page);
continue;
@@ -383,7 +383,7 @@ restart: cond_resched();
}
lock_page(page);
- WARN_ON(page_to_index(page) != index);
+ WARN_ON(page_to_pgoff(page) != index);
wait_on_page_writeback(page);
if (PageTransHuge(page)) {
@@ -533,7 +533,7 @@ unsigned long invalidate_mapping_pages(struct address_space *mapping,
if (!trylock_page(page))
continue;
- WARN_ON(page_to_index(page) != index);
+ WARN_ON(page_to_pgoff(page) != index);
/* Is 'start' or 'end' in the middle of THP ? */
if (PageTransHuge(page) &&
@@ -650,7 +650,7 @@ int invalidate_inode_pages2_range(struct address_space *mapping,
}
lock_page(page);
- WARN_ON(page_to_index(page) != index);
+ WARN_ON(page_to_pgoff(page) != index);
if (page->mapping != mapping) {
unlock_page(page);
continue;
--
2.10.2
We writeback whole huge page a time.
Signed-off-by: Kirill A. Shutemov <[email protected]>
---
mm/filemap.c | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/mm/filemap.c b/mm/filemap.c
index ec976ddcb88a..52be2b457208 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -405,9 +405,14 @@ static int __filemap_fdatawait_range(struct address_space *mapping,
if (page->index > end)
continue;
+ page = compound_head(page);
wait_on_page_writeback(page);
if (TestClearPageError(page))
ret = -EIO;
+ if (PageTransHuge(page)) {
+ index = page->index + HPAGE_PMD_NR;
+ i += index - pvec.pages[i]->index - 1;
+ }
}
pagevec_release(&pvec);
cond_resched();
--
2.10.2
This patch adds basic functionality to put huge page into page cache.
At the moment we only put huge pages into radix-tree if the range covered
by the huge page is empty.
We ignore shadow entires for now, just remove them from the tree before
inserting huge page.
Later we can add logic to accumulate information from shadow entires to
return to caller (average eviction time?).
Signed-off-by: Kirill A. Shutemov <[email protected]>
---
include/linux/fs.h | 5 ++
include/linux/pagemap.h | 21 ++++++-
mm/filemap.c | 155 ++++++++++++++++++++++++++++++++++++++----------
3 files changed, 147 insertions(+), 34 deletions(-)
diff --git a/include/linux/fs.h b/include/linux/fs.h
index 03a5a398ae83..be94b922a22f 100644
--- a/include/linux/fs.h
+++ b/include/linux/fs.h
@@ -1799,6 +1799,11 @@ struct super_operations {
#else
#define S_DAX 0 /* Make all the DAX code disappear */
#endif
+#define S_HUGE_MODE 0xc000
+#define S_HUGE_NEVER 0x0000
+#define S_HUGE_ALWAYS 0x4000
+#define S_HUGE_WITHIN_SIZE 0x8000
+#define S_HUGE_ADVISE 0xc000
/*
* Note that nosuid etc flags are inode-specific: setting some file-system
diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h
index f88d69e2419d..e530e7b3b6b2 100644
--- a/include/linux/pagemap.h
+++ b/include/linux/pagemap.h
@@ -201,14 +201,20 @@ static inline int page_cache_add_speculative(struct page *page, int count)
}
#ifdef CONFIG_NUMA
-extern struct page *__page_cache_alloc(gfp_t gfp);
+extern struct page *__page_cache_alloc_order(gfp_t gfp, unsigned int order);
#else
-static inline struct page *__page_cache_alloc(gfp_t gfp)
+static inline struct page *__page_cache_alloc_order(gfp_t gfp,
+ unsigned int order)
{
- return alloc_pages(gfp, 0);
+ return alloc_pages(gfp, order);
}
#endif
+static inline struct page *__page_cache_alloc(gfp_t gfp)
+{
+ return __page_cache_alloc_order(gfp, 0);
+}
+
static inline struct page *page_cache_alloc(struct address_space *x)
{
return __page_cache_alloc(mapping_gfp_mask(x));
@@ -225,6 +231,15 @@ static inline gfp_t readahead_gfp_mask(struct address_space *x)
__GFP_COLD | __GFP_NORETRY | __GFP_NOWARN;
}
+extern bool __page_cache_allow_huge(struct address_space *x, pgoff_t offset);
+static inline bool page_cache_allow_huge(struct address_space *x,
+ pgoff_t offset)
+{
+ if (!IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE))
+ return false;
+ return __page_cache_allow_huge(x, offset);
+}
+
typedef int filler_t(void *, struct page *);
pgoff_t page_cache_next_hole(struct address_space *mapping,
diff --git a/mm/filemap.c b/mm/filemap.c
index 16d39340c106..74341f8b831e 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -113,37 +113,50 @@
static int page_cache_tree_insert(struct address_space *mapping,
struct page *page, void **shadowp)
{
- struct radix_tree_node *node;
- void **slot;
+ struct radix_tree_iter iter;
+ void **slot, *p;
int error;
- error = __radix_tree_create(&mapping->page_tree, page->index, 0,
- &node, &slot);
- if (error)
- return error;
- if (*slot) {
- void *p;
+ /* Wipe shadow entires */
+ radix_tree_for_each_slot(slot, &mapping->page_tree, &iter,
+ page->index) {
+ if (iter.index >= page->index + hpage_nr_pages(page))
+ break;
p = radix_tree_deref_slot_protected(slot, &mapping->tree_lock);
- if (!radix_tree_exceptional_entry(p))
+ if (!p)
+ continue;
+
+ if (!radix_tree_exception(p))
return -EEXIST;
+ __radix_tree_replace(&mapping->page_tree, iter.node, slot, NULL,
+ workingset_update_node, mapping);
+
mapping->nrexceptional--;
- if (!dax_mapping(mapping)) {
- if (shadowp)
- *shadowp = p;
- } else {
+ if (dax_mapping(mapping)) {
/* DAX can replace empty locked entry with a hole */
WARN_ON_ONCE(p !=
dax_radix_locked_entry(0, RADIX_DAX_EMPTY));
/* Wakeup waiters for exceptional entry lock */
dax_wake_mapping_entry_waiter(mapping, page->index, p,
false);
+ } else if (!PageTransHuge(page) && shadowp) {
+ *shadowp = p;
}
}
- __radix_tree_replace(&mapping->page_tree, node, slot, page,
- workingset_update_node, mapping);
- mapping->nrpages++;
+
+ error = __radix_tree_insert(&mapping->page_tree,
+ page->index, compound_order(page), page);
+ /* This shouldn't happen */
+ if (WARN_ON_ONCE(error))
+ return error;
+
+ mapping->nrpages += hpage_nr_pages(page);
+ if (PageTransHuge(page) && !PageHuge(page)) {
+ count_vm_event(THP_FILE_ALLOC);
+ __inc_node_page_state(page, NR_FILE_THPS);
+ }
return 0;
}
@@ -600,14 +613,14 @@ static int __add_to_page_cache_locked(struct page *page,
pgoff_t offset, gfp_t gfp_mask,
void **shadowp)
{
- int huge = PageHuge(page);
+ int hugetlb = PageHuge(page);
struct mem_cgroup *memcg;
int error;
VM_BUG_ON_PAGE(!PageLocked(page), page);
VM_BUG_ON_PAGE(PageSwapBacked(page), page);
- if (!huge) {
+ if (!hugetlb) {
error = mem_cgroup_try_charge(page, current->mm,
gfp_mask, &memcg, false);
if (error)
@@ -616,7 +629,7 @@ static int __add_to_page_cache_locked(struct page *page,
error = radix_tree_maybe_preload(gfp_mask & ~__GFP_HIGHMEM);
if (error) {
- if (!huge)
+ if (!hugetlb)
mem_cgroup_cancel_charge(page, memcg, false);
return error;
}
@@ -632,10 +645,11 @@ static int __add_to_page_cache_locked(struct page *page,
goto err_insert;
/* hugetlb pages do not participate in page cache accounting. */
- if (!huge)
- __inc_node_page_state(page, NR_FILE_PAGES);
+ if (!hugetlb)
+ __mod_node_page_state(page_pgdat(page), NR_FILE_PAGES,
+ hpage_nr_pages(page));
spin_unlock_irq(&mapping->tree_lock);
- if (!huge)
+ if (!hugetlb)
mem_cgroup_commit_charge(page, memcg, false, false);
trace_mm_filemap_add_to_page_cache(page);
return 0;
@@ -643,7 +657,7 @@ static int __add_to_page_cache_locked(struct page *page,
page->mapping = NULL;
/* Leave page->index set: truncation relies upon it */
spin_unlock_irq(&mapping->tree_lock);
- if (!huge)
+ if (!hugetlb)
mem_cgroup_cancel_charge(page, memcg, false);
put_page(page);
return error;
@@ -700,7 +714,7 @@ int add_to_page_cache_lru(struct page *page, struct address_space *mapping,
EXPORT_SYMBOL_GPL(add_to_page_cache_lru);
#ifdef CONFIG_NUMA
-struct page *__page_cache_alloc(gfp_t gfp)
+struct page *__page_cache_alloc_order(gfp_t gfp, unsigned int order)
{
int n;
struct page *page;
@@ -710,14 +724,14 @@ struct page *__page_cache_alloc(gfp_t gfp)
do {
cpuset_mems_cookie = read_mems_allowed_begin();
n = cpuset_mem_spread_node();
- page = __alloc_pages_node(n, gfp, 0);
+ page = __alloc_pages_node(n, gfp, order);
} while (!page && read_mems_allowed_retry(cpuset_mems_cookie));
return page;
}
- return alloc_pages(gfp, 0);
+ return alloc_pages(gfp, order);
}
-EXPORT_SYMBOL(__page_cache_alloc);
+EXPORT_SYMBOL(__page_cache_alloc_order);
#endif
/*
@@ -1102,6 +1116,69 @@ struct page *find_lock_entry(struct address_space *mapping, pgoff_t offset)
}
EXPORT_SYMBOL(find_lock_entry);
+bool __page_cache_allow_huge(struct address_space *mapping, pgoff_t offset)
+{
+ struct inode *inode = mapping->host;
+ struct radix_tree_iter iter;
+ void **slot;
+ struct page *page;
+
+ if (!IS_ENABLED(CONFIG_TRANSPARENT_HUGE_PAGECACHE))
+ return false;
+
+ offset = round_down(offset, HPAGE_PMD_NR);
+
+ switch (inode->i_flags & S_HUGE_MODE) {
+ case S_HUGE_NEVER:
+ return false;
+ case S_HUGE_ALWAYS:
+ break;
+ case S_HUGE_WITHIN_SIZE:
+ if (DIV_ROUND_UP(i_size_read(inode), PAGE_SIZE) <
+ offset + HPAGE_PMD_NR)
+ return false;
+ break;
+ case S_HUGE_ADVISE:
+ /* TODO */
+ return false;
+ default:
+ WARN_ON_ONCE(1);
+ return false;
+ }
+
+ rcu_read_lock();
+ radix_tree_for_each_slot(slot, &mapping->page_tree, &iter, offset) {
+ if (iter.index >= offset + HPAGE_PMD_NR)
+ break;
+
+ /* Shadow entires are fine */
+ page = radix_tree_deref_slot(slot);
+ if (page && !radix_tree_exception(page)) {
+ rcu_read_unlock();
+ return false;
+ }
+ }
+ rcu_read_unlock();
+
+ return true;
+
+}
+
+static struct page *page_cache_alloc_huge(struct address_space *mapping,
+ pgoff_t offset, gfp_t gfp_mask)
+{
+ struct page *page;
+
+ if (!page_cache_allow_huge(mapping, offset))
+ return NULL;
+
+ gfp_mask |= __GFP_COMP | __GFP_NORETRY | __GFP_NOWARN;
+ page = __page_cache_alloc_order(gfp_mask, HPAGE_PMD_ORDER);
+ if (page)
+ prep_transhuge_page(page);
+ return page;
+}
+
/**
* pagecache_get_page - find and get a page reference
* @mapping: the address_space to search
@@ -1941,18 +2018,34 @@ static int page_cache_read(struct file *file, pgoff_t offset, gfp_t gfp_mask)
{
struct address_space *mapping = file->f_mapping;
struct page *page;
+ pgoff_t hoffset;
int ret;
do {
- page = __page_cache_alloc(gfp_mask|__GFP_COLD);
+ page = page_cache_alloc_huge(mapping, offset, gfp_mask);
+no_huge:
+ if (!page)
+ page = __page_cache_alloc(gfp_mask|__GFP_COLD);
if (!page)
return -ENOMEM;
- ret = add_to_page_cache_lru(page, mapping, offset, gfp_mask & GFP_KERNEL);
- if (ret == 0)
+ if (PageTransHuge(page))
+ hoffset = round_down(offset, HPAGE_PMD_NR);
+ else
+ hoffset = offset;
+
+ ret = add_to_page_cache_lru(page, mapping, hoffset,
+ gfp_mask & GFP_KERNEL);
+
+ if (ret == -EEXIST && PageTransHuge(page)) {
+ put_page(page);
+ page = NULL;
+ goto no_huge;
+ } else if (ret == 0) {
ret = mapping->a_ops->readpage(file, page);
- else if (ret == -EEXIST)
+ } else if (ret == -EEXIST) {
ret = 0; /* losing race to add is OK */
+ }
put_page(page);
--
2.10.2
We want page to be isolated from the rest of the system before spliting
it. We rely on page count to be 2 for file pages to make sure nobody
uses the page: one pin to caller, one to radix-tree.
Filesystems with backing storage can have page count increased if it has
buffers.
Let's try to free them, before attempt split. And remove one guarding
VM_BUG_ON_PAGE().
Signed-off-by: Kirill A. Shutemov <[email protected]>
---
include/linux/buffer_head.h | 1 +
mm/huge_memory.c | 19 ++++++++++++++++++-
2 files changed, 19 insertions(+), 1 deletion(-)
diff --git a/include/linux/buffer_head.h b/include/linux/buffer_head.h
index d67ab83823ad..fd4134ce9c54 100644
--- a/include/linux/buffer_head.h
+++ b/include/linux/buffer_head.h
@@ -400,6 +400,7 @@ extern int __set_page_dirty_buffers(struct page *page);
#else /* CONFIG_BLOCK */
static inline void buffer_init(void) {}
+static inline int page_has_buffers(struct page *page) { return 0; }
static inline int try_to_free_buffers(struct page *page) { return 1; }
static inline int inode_has_buffers(struct inode *inode) { return 0; }
static inline void invalidate_inode_buffers(struct inode *inode) {}
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 91dbab9644be..a15d566b14f6 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -30,6 +30,7 @@
#include <linux/userfaultfd_k.h>
#include <linux/page_idle.h>
#include <linux/shmem_fs.h>
+#include <linux/buffer_head.h>
#include <asm/tlb.h>
#include <asm/pgalloc.h>
@@ -2111,7 +2112,6 @@ int split_huge_page_to_list(struct page *page, struct list_head *list)
VM_BUG_ON_PAGE(is_huge_zero_page(page), page);
VM_BUG_ON_PAGE(!PageLocked(page), page);
- VM_BUG_ON_PAGE(!PageSwapBacked(page), page);
VM_BUG_ON_PAGE(!PageCompound(page), page);
if (PageAnon(head)) {
@@ -2140,6 +2140,23 @@ int split_huge_page_to_list(struct page *page, struct list_head *list)
goto out;
}
+ /* Try to free buffers before attempt split */
+ if (!PageSwapBacked(head) && PagePrivate(page)) {
+ /*
+ * We cannot trigger writeback from here due possible
+ * recursion if triggered from vmscan, only wait.
+ *
+ * Caller can trigger writeback it on its own, if safe.
+ */
+ wait_on_page_writeback(head);
+
+ if (page_has_buffers(head) && !try_to_release_page(head,
+ GFP_KERNEL)) {
+ ret = -EBUSY;
+ goto out;
+ }
+ }
+
/* Addidional pin from radix tree */
extra_pins = 1;
anon_vma = NULL;
--
2.10.2
Most of work happans on head page. Only when we need to do copy data to
userspace we find relevant subpage.
We are still limited by PAGE_SIZE per iteration. Lifting this limitation
would require some more work.
Signed-off-by: Kirill A. Shutemov <[email protected]>
---
mm/filemap.c | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/mm/filemap.c b/mm/filemap.c
index 74341f8b831e..6a2f9ea521fb 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -1749,6 +1749,7 @@ static ssize_t do_generic_file_read(struct file *filp, loff_t *ppos,
if (unlikely(page == NULL))
goto no_cached_page;
}
+ page = compound_head(page);
if (PageReadahead(page)) {
page_cache_async_readahead(mapping,
ra, filp, page,
@@ -1830,7 +1831,8 @@ static ssize_t do_generic_file_read(struct file *filp, loff_t *ppos,
* now we can copy it to user space...
*/
- ret = copy_page_to_iter(page, offset, nr, iter);
+ ret = copy_page_to_iter(page + index - page->index, offset,
+ nr, iter);
offset += ret;
index += offset >> PAGE_SHIFT;
offset &= ~PAGE_MASK;
@@ -2248,6 +2250,7 @@ int filemap_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
* because there really aren't any performance issues here
* and we need to check for errors.
*/
+ page = compound_head(page);
ClearPageError(page);
error = mapping->a_ops->readpage(file, page);
if (!error) {
--
2.10.2
__ext4_block_zero_page_range() adjusted to calculate starting iblock
correctry for huge pages.
ext4_{collapse,insert}_range() requires page cache invalidation. We need
the invalidation to be aligning to huge page border if huge pages are
possible in page cache.
Signed-off-by: Kirill A. Shutemov <[email protected]>
---
fs/ext4/extents.c | 10 ++++++++--
fs/ext4/inode.c | 3 +--
2 files changed, 9 insertions(+), 4 deletions(-)
diff --git a/fs/ext4/extents.c b/fs/ext4/extents.c
index 7d29ef0f5f5c..99632945b3f5 100644
--- a/fs/ext4/extents.c
+++ b/fs/ext4/extents.c
@@ -5501,7 +5501,10 @@ int ext4_collapse_range(struct inode *inode, loff_t offset, loff_t len)
* Need to round down offset to be aligned with page size boundary
* for page size > block size.
*/
- ioffset = round_down(offset, PAGE_SIZE);
+ if (IS_ENABLED(CONFIG_TRANSPARENT_HUGE_PAGECACHE))
+ ioffset = round_down(offset, HPAGE_PMD_SIZE);
+ else
+ ioffset = round_down(offset, PAGE_SIZE);
/*
* Write tail of the last page before removed range since it will get
* removed from the page cache below.
@@ -5650,7 +5653,10 @@ int ext4_insert_range(struct inode *inode, loff_t offset, loff_t len)
* Need to round down to align start offset to page size boundary
* for page size > block size.
*/
- ioffset = round_down(offset, PAGE_SIZE);
+ if (IS_ENABLED(CONFIG_TRANSPARENT_HUGE_PAGECACHE))
+ ioffset = round_down(offset, HPAGE_PMD_SIZE);
+ else
+ ioffset = round_down(offset, PAGE_SIZE);
/* Write out all dirty pages */
ret = filemap_write_and_wait_range(inode->i_mapping, ioffset,
LLONG_MAX);
diff --git a/fs/ext4/inode.c b/fs/ext4/inode.c
index ff4f460d3625..263b53ace613 100644
--- a/fs/ext4/inode.c
+++ b/fs/ext4/inode.c
@@ -3807,7 +3807,6 @@ void ext4_set_aops(struct inode *inode)
static int __ext4_block_zero_page_range(handle_t *handle,
struct address_space *mapping, loff_t from, loff_t length)
{
- ext4_fsblk_t index = from >> PAGE_SHIFT;
unsigned offset;
unsigned blocksize, pos;
ext4_lblk_t iblock;
@@ -3826,7 +3825,7 @@ static int __ext4_block_zero_page_range(handle_t *handle,
blocksize = inode->i_sb->s_blocksize;
- iblock = index << (PAGE_SHIFT - inode->i_sb->s_blocksize_bits);
+ iblock = page->index << (PAGE_SHIFT - inode->i_sb->s_blocksize_bits);
if (!page_has_buffers(page))
create_empty_buffers(page, blocksize, 0);
--
2.10.2
The same four values as in tmpfs case.
Encyption code is not yet ready to handle huge page, so we disable huge
pages support if the inode has EXT4_INODE_ENCRYPT.
Signed-off-by: Kirill A. Shutemov <[email protected]>
---
fs/ext4/ext4.h | 5 +++++
fs/ext4/inode.c | 30 +++++++++++++++++++++++-------
fs/ext4/super.c | 24 ++++++++++++++++++++++++
3 files changed, 52 insertions(+), 7 deletions(-)
diff --git a/fs/ext4/ext4.h b/fs/ext4/ext4.h
index aff204f040fc..fb3f81863b53 100644
--- a/fs/ext4/ext4.h
+++ b/fs/ext4/ext4.h
@@ -1133,6 +1133,11 @@ struct ext4_inode_info {
#define EXT4_MOUNT_DIOREAD_NOLOCK 0x400000 /* Enable support for dio read nolocking */
#define EXT4_MOUNT_JOURNAL_CHECKSUM 0x800000 /* Journal checksums */
#define EXT4_MOUNT_JOURNAL_ASYNC_COMMIT 0x1000000 /* Journal Async Commit */
+#define EXT4_MOUNT_HUGE_MODE 0x6000000 /* Huge support mode: */
+#define EXT4_MOUNT_HUGE_NEVER 0x0000000
+#define EXT4_MOUNT_HUGE_ALWAYS 0x2000000
+#define EXT4_MOUNT_HUGE_WITHIN_SIZE 0x4000000
+#define EXT4_MOUNT_HUGE_ADVISE 0x6000000
#define EXT4_MOUNT_DELALLOC 0x8000000 /* Delalloc support */
#define EXT4_MOUNT_DATA_ERR_ABORT 0x10000000 /* Abort on file data write */
#define EXT4_MOUNT_BLOCK_VALIDITY 0x20000000 /* Block validity checking */
diff --git a/fs/ext4/inode.c b/fs/ext4/inode.c
index 17a767c21dc3..4c37fd9fb219 100644
--- a/fs/ext4/inode.c
+++ b/fs/ext4/inode.c
@@ -4472,7 +4472,7 @@ int ext4_get_inode_loc(struct inode *inode, struct ext4_iloc *iloc)
void ext4_set_inode_flags(struct inode *inode)
{
unsigned int flags = EXT4_I(inode)->i_flags;
- unsigned int new_fl = 0;
+ unsigned int mask, new_fl = 0;
if (flags & EXT4_SYNC_FL)
new_fl |= S_SYNC;
@@ -4484,12 +4484,28 @@ void ext4_set_inode_flags(struct inode *inode)
new_fl |= S_NOATIME;
if (flags & EXT4_DIRSYNC_FL)
new_fl |= S_DIRSYNC;
- if (test_opt(inode->i_sb, DAX) && S_ISREG(inode->i_mode) &&
- !ext4_should_journal_data(inode) && !ext4_has_inline_data(inode) &&
- !ext4_encrypted_inode(inode))
- new_fl |= S_DAX;
- inode_set_flags(inode, new_fl,
- S_SYNC|S_APPEND|S_IMMUTABLE|S_NOATIME|S_DIRSYNC|S_DAX);
+ if (S_ISREG(inode->i_mode) && !ext4_encrypted_inode(inode)) {
+ if (test_opt(inode->i_sb, DAX) &&
+ !ext4_should_journal_data(inode) &&
+ !ext4_has_inline_data(inode))
+ new_fl |= S_DAX;
+ switch (test_opt(inode->i_sb, HUGE_MODE)) {
+ case EXT4_MOUNT_HUGE_NEVER:
+ break;
+ case EXT4_MOUNT_HUGE_ALWAYS:
+ new_fl |= S_HUGE_ALWAYS;
+ break;
+ case EXT4_MOUNT_HUGE_WITHIN_SIZE:
+ new_fl |= S_HUGE_WITHIN_SIZE;
+ break;
+ case EXT4_MOUNT_HUGE_ADVISE:
+ new_fl |= S_HUGE_ADVISE;
+ break;
+ }
+ }
+ mask = S_SYNC | S_APPEND | S_IMMUTABLE | S_NOATIME |
+ S_DIRSYNC | S_DAX | S_HUGE_MODE;
+ inode_set_flags(inode, new_fl, mask);
}
/* Propagate flags from i_flags to EXT4_I(inode)->i_flags */
diff --git a/fs/ext4/super.c b/fs/ext4/super.c
index 72b459d2b244..127ddfeae1e0 100644
--- a/fs/ext4/super.c
+++ b/fs/ext4/super.c
@@ -1296,6 +1296,7 @@ enum {
Opt_dioread_nolock, Opt_dioread_lock,
Opt_discard, Opt_nodiscard, Opt_init_itable, Opt_noinit_itable,
Opt_max_dir_size_kb, Opt_nojournal_checksum,
+ Opt_huge_never, Opt_huge_always, Opt_huge_within_size, Opt_huge_advise,
};
static const match_table_t tokens = {
@@ -1376,6 +1377,10 @@ static const match_table_t tokens = {
{Opt_init_itable, "init_itable"},
{Opt_noinit_itable, "noinit_itable"},
{Opt_max_dir_size_kb, "max_dir_size_kb=%u"},
+ {Opt_huge_never, "huge=never"},
+ {Opt_huge_always, "huge=always"},
+ {Opt_huge_within_size, "huge=within_size"},
+ {Opt_huge_advise, "huge=advise"},
{Opt_test_dummy_encryption, "test_dummy_encryption"},
{Opt_removed, "check=none"}, /* mount option from ext2/3 */
{Opt_removed, "nocheck"}, /* mount option from ext2/3 */
@@ -1494,6 +1499,11 @@ static int clear_qf_name(struct super_block *sb, int qtype)
#define MOPT_NO_EXT3 0x0200
#define MOPT_EXT4_ONLY (MOPT_NO_EXT2 | MOPT_NO_EXT3)
#define MOPT_STRING 0x0400
+#ifdef CONFIG_TRANSPARENT_HUGE_PAGECACHE
+#define MOPT_HUGE 0x1000
+#else
+#define MOPT_HUGE MOPT_NOSUPPORT
+#endif
static const struct mount_opts {
int token;
@@ -1581,6 +1591,10 @@ static const struct mount_opts {
{Opt_jqfmt_vfsv0, QFMT_VFS_V0, MOPT_QFMT},
{Opt_jqfmt_vfsv1, QFMT_VFS_V1, MOPT_QFMT},
{Opt_max_dir_size_kb, 0, MOPT_GTE0},
+ {Opt_huge_never, EXT4_MOUNT_HUGE_NEVER, MOPT_HUGE},
+ {Opt_huge_always, EXT4_MOUNT_HUGE_ALWAYS, MOPT_HUGE},
+ {Opt_huge_within_size, EXT4_MOUNT_HUGE_WITHIN_SIZE, MOPT_HUGE},
+ {Opt_huge_advise, EXT4_MOUNT_HUGE_ADVISE, MOPT_HUGE},
{Opt_test_dummy_encryption, 0, MOPT_GTE0},
{Opt_err, 0, 0}
};
@@ -1662,6 +1676,16 @@ static int handle_mount_opt(struct super_block *sb, char *opt, int token,
} else
return -1;
}
+ if (MOPT_HUGE != MOPT_NOSUPPORT && m->flags & MOPT_HUGE) {
+ sbi->s_mount_opt &= ~EXT4_MOUNT_HUGE_MODE;
+ sbi->s_mount_opt |= m->mount_opt;
+ if (m->mount_opt) {
+ ext4_msg(sb, KERN_WARNING, "Warning: "
+ "Support of huge pages is EXPERIMENTAL,"
+ " use at your own risk");
+ }
+ return 1;
+ }
if (m->flags & MOPT_CLEAR_ERR)
clear_opt(sb, ERRORS_MASK);
if (token == Opt_noquota && sb_any_quota_loaded(sb)) {
--
2.10.2
These flags are in use for filesystems with backing storage: PG_error,
PG_writeback and PG_readahead.
Signed-off-by: Kirill A. Shutemov <[email protected]>
---
include/linux/page-flags.h | 10 +++++-----
1 file changed, 5 insertions(+), 5 deletions(-)
diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h
index 74e4dda91238..a2bef9a41bcf 100644
--- a/include/linux/page-flags.h
+++ b/include/linux/page-flags.h
@@ -253,7 +253,7 @@ static inline int TestClearPage##uname(struct page *page) { return 0; }
TESTSETFLAG_FALSE(uname) TESTCLEARFLAG_FALSE(uname)
__PAGEFLAG(Locked, locked, PF_NO_TAIL)
-PAGEFLAG(Error, error, PF_NO_COMPOUND) TESTCLEARFLAG(Error, error, PF_NO_COMPOUND)
+PAGEFLAG(Error, error, PF_NO_TAIL) TESTCLEARFLAG(Error, error, PF_NO_TAIL)
PAGEFLAG(Referenced, referenced, PF_HEAD)
TESTCLEARFLAG(Referenced, referenced, PF_HEAD)
__SETPAGEFLAG(Referenced, referenced, PF_HEAD)
@@ -293,15 +293,15 @@ PAGEFLAG(OwnerPriv1, owner_priv_1, PF_ANY)
* Only test-and-set exist for PG_writeback. The unconditional operators are
* risky: they bypass page accounting.
*/
-TESTPAGEFLAG(Writeback, writeback, PF_NO_COMPOUND)
- TESTSCFLAG(Writeback, writeback, PF_NO_COMPOUND)
+TESTPAGEFLAG(Writeback, writeback, PF_NO_TAIL)
+ TESTSCFLAG(Writeback, writeback, PF_NO_TAIL)
PAGEFLAG(MappedToDisk, mappedtodisk, PF_NO_TAIL)
/* PG_readahead is only used for reads; PG_reclaim is only for writes */
PAGEFLAG(Reclaim, reclaim, PF_NO_TAIL)
TESTCLEARFLAG(Reclaim, reclaim, PF_NO_TAIL)
-PAGEFLAG(Readahead, reclaim, PF_NO_COMPOUND)
- TESTCLEARFLAG(Readahead, reclaim, PF_NO_COMPOUND)
+PAGEFLAG(Readahead, reclaim, PF_NO_TAIL)
+ TESTCLEARFLAG(Readahead, reclaim, PF_NO_TAIL)
#ifdef CONFIG_HIGHMEM
/*
--
2.10.2
With huge pages in page cache we see tail pages in more code paths.
This patch replaces direct access to struct page fields with macros
which can handle tail pages properly.
Signed-off-by: Kirill A. Shutemov <[email protected]>
---
fs/buffer.c | 2 +-
fs/ext4/inode.c | 4 ++--
mm/filemap.c | 24 +++++++++++++-----------
mm/memory.c | 2 +-
mm/page-writeback.c | 2 +-
mm/truncate.c | 5 +++--
6 files changed, 21 insertions(+), 18 deletions(-)
diff --git a/fs/buffer.c b/fs/buffer.c
index 24daf7b9bdb0..c7fe6c9bae25 100644
--- a/fs/buffer.c
+++ b/fs/buffer.c
@@ -631,7 +631,7 @@ static void __set_page_dirty(struct page *page, struct address_space *mapping,
unsigned long flags;
spin_lock_irqsave(&mapping->tree_lock, flags);
- if (page->mapping) { /* Race with truncate? */
+ if (page_mapping(page)) { /* Race with truncate? */
WARN_ON_ONCE(warn && !PageUptodate(page));
account_page_dirtied(page, mapping);
radix_tree_tag_set(&mapping->page_tree,
diff --git a/fs/ext4/inode.c b/fs/ext4/inode.c
index 263b53ace613..17a767c21dc3 100644
--- a/fs/ext4/inode.c
+++ b/fs/ext4/inode.c
@@ -1237,7 +1237,7 @@ static int ext4_write_begin(struct file *file, struct address_space *mapping,
}
lock_page(page);
- if (page->mapping != mapping) {
+ if (page_mapping(page) != mapping) {
/* The page got truncated from under us */
unlock_page(page);
put_page(page);
@@ -2974,7 +2974,7 @@ static int ext4_da_write_begin(struct file *file, struct address_space *mapping,
}
lock_page(page);
- if (page->mapping != mapping) {
+ if (page_mapping(page) != mapping) {
/* The page got truncated from under us */
unlock_page(page);
put_page(page);
diff --git a/mm/filemap.c b/mm/filemap.c
index 33974ad1a8ec..be8ccadb915f 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -399,7 +399,7 @@ static int __filemap_fdatawait_range(struct address_space *mapping,
struct page *page = pvec.pages[i];
/* until radix tree lookup accepts end_index */
- if (page->index > end)
+ if (page_to_pgoff(page) > end)
continue;
page = compound_head(page);
@@ -1227,7 +1227,7 @@ struct page *pagecache_get_page(struct address_space *mapping, pgoff_t offset,
}
/* Has the page been truncated? */
- if (unlikely(page->mapping != mapping)) {
+ if (unlikely(page_mapping(page) != mapping)) {
unlock_page(page);
put_page(page);
goto repeat;
@@ -1504,7 +1504,8 @@ unsigned find_get_pages_contig(struct address_space *mapping, pgoff_t start,
* otherwise we can get both false positives and false
* negatives, which is just confusing to the caller.
*/
- if (page->mapping == NULL || page_to_pgoff(page) != index) {
+ if (page_mapping(page) == NULL ||
+ page_to_pgoff(page) != index) {
put_page(page);
break;
}
@@ -1792,7 +1793,7 @@ static ssize_t do_generic_file_read(struct file *filp, loff_t *ppos,
if (!trylock_page(page))
goto page_not_up_to_date;
/* Did it get truncated before we got the lock? */
- if (!page->mapping)
+ if (!page_mapping(page))
goto page_not_up_to_date_locked;
if (!mapping->a_ops->is_partially_uptodate(page,
offset, iter->count))
@@ -1872,7 +1873,7 @@ static ssize_t do_generic_file_read(struct file *filp, loff_t *ppos,
page_not_up_to_date_locked:
/* Did it get truncated before we got the lock? */
- if (!page->mapping) {
+ if (!page_mapping(page)) {
unlock_page(page);
put_page(page);
continue;
@@ -1908,7 +1909,7 @@ static ssize_t do_generic_file_read(struct file *filp, loff_t *ppos,
if (unlikely(error))
goto readpage_error;
if (!PageUptodate(page)) {
- if (page->mapping == NULL) {
+ if (page_mapping(page) == NULL) {
/*
* invalidate_mapping_pages got it
*/
@@ -2207,12 +2208,12 @@ int filemap_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
}
/* Did it get truncated? */
- if (unlikely(page->mapping != mapping)) {
+ if (unlikely(page_mapping(page) != mapping)) {
unlock_page(page);
put_page(page);
goto retry_find;
}
- VM_BUG_ON_PAGE(page->index != offset, page);
+ VM_BUG_ON_PAGE(page_to_pgoff(page) != offset, page);
/*
* We have a locked page in the page cache, now we need to check
@@ -2388,7 +2389,7 @@ int filemap_page_mkwrite(struct vm_area_struct *vma, struct vm_fault *vmf)
sb_start_pagefault(inode->i_sb);
file_update_time(vma->vm_file);
lock_page(page);
- if (page->mapping != inode->i_mapping) {
+ if (page_mapping(page) != inode->i_mapping) {
unlock_page(page);
ret = VM_FAULT_NOPAGE;
goto out;
@@ -2537,7 +2538,7 @@ static struct page *do_read_cache_page(struct address_space *mapping,
lock_page(page);
/* Case c or d, restart the operation */
- if (!page->mapping) {
+ if (!page_mapping(page)) {
unlock_page(page);
put_page(page);
goto repeat;
@@ -2993,12 +2994,13 @@ EXPORT_SYMBOL(generic_file_write_iter);
*/
int try_to_release_page(struct page *page, gfp_t gfp_mask)
{
- struct address_space * const mapping = page->mapping;
+ struct address_space * const mapping = page_mapping(page);
BUG_ON(!PageLocked(page));
if (PageWriteback(page))
return 0;
+ page = compound_head(page);
if (mapping && mapping->a_ops->releasepage)
return mapping->a_ops->releasepage(page, gfp_mask);
return try_to_free_buffers(page);
diff --git a/mm/memory.c b/mm/memory.c
index e3d7cea8cc6a..804b0e972bd3 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -2049,7 +2049,7 @@ static int do_page_mkwrite(struct vm_fault *vmf)
return ret;
if (unlikely(!(ret & VM_FAULT_LOCKED))) {
lock_page(page);
- if (!page->mapping) {
+ if (!page_mapping(page)) {
unlock_page(page);
return 0; /* retry */
}
diff --git a/mm/page-writeback.c b/mm/page-writeback.c
index d7b905d66add..3ebbac70681f 100644
--- a/mm/page-writeback.c
+++ b/mm/page-writeback.c
@@ -2869,7 +2869,7 @@ EXPORT_SYMBOL(mapping_tagged);
*/
void wait_for_stable_page(struct page *page)
{
- if (bdi_cap_stable_pages_required(inode_to_bdi(page->mapping->host)))
+ if (bdi_cap_stable_pages_required(inode_to_bdi(page_mapping(page)->host)))
wait_on_page_writeback(page);
}
EXPORT_SYMBOL_GPL(wait_for_stable_page);
diff --git a/mm/truncate.c b/mm/truncate.c
index 7508c2c7e4ed..8cc0c17d95d5 100644
--- a/mm/truncate.c
+++ b/mm/truncate.c
@@ -575,6 +575,7 @@ invalidate_complete_page2(struct address_space *mapping, struct page *page)
{
unsigned long flags;
+ page = compound_head(page);
if (page->mapping != mapping)
return 0;
@@ -603,7 +604,7 @@ static int do_launder_page(struct address_space *mapping, struct page *page)
{
if (!PageDirty(page))
return 0;
- if (page->mapping != mapping || mapping->a_ops->launder_page == NULL)
+ if (page_mapping(page) != mapping || mapping->a_ops->launder_page == NULL)
return 0;
return mapping->a_ops->launder_page(page);
}
@@ -651,7 +652,7 @@ int invalidate_inode_pages2_range(struct address_space *mapping,
lock_page(page);
WARN_ON(page_to_pgoff(page) != index);
- if (page->mapping != mapping) {
+ if (page_mapping(page) != mapping) {
unlock_page(page);
continue;
}
--
2.10.2
ext4_find_unwritten_pgoff() needs few tweaks to work with huge pages.
Mostly trivial page_mapping()/page_to_pgoff() and adjustment to how we
find relevant block.
Signe-off-by: Kirill A. Shutemov <[email protected]>
---
fs/ext4/file.c | 18 ++++++++++++++----
1 file changed, 14 insertions(+), 4 deletions(-)
diff --git a/fs/ext4/file.c b/fs/ext4/file.c
index b5f184493c57..7998ac1483c4 100644
--- a/fs/ext4/file.c
+++ b/fs/ext4/file.c
@@ -547,7 +547,7 @@ static int ext4_find_unwritten_pgoff(struct inode *inode,
* range, it will be a hole.
*/
if (lastoff < endoff && whence == SEEK_HOLE &&
- page->index > end) {
+ page_to_pgoff(page) > end) {
found = 1;
*offset = lastoff;
goto out;
@@ -555,7 +555,7 @@ static int ext4_find_unwritten_pgoff(struct inode *inode,
lock_page(page);
- if (unlikely(page->mapping != inode->i_mapping)) {
+ if (unlikely(page_mapping(page) != inode->i_mapping)) {
unlock_page(page);
continue;
}
@@ -566,8 +566,12 @@ static int ext4_find_unwritten_pgoff(struct inode *inode,
}
if (page_has_buffers(page)) {
+ int diff;
lastoff = page_offset(page);
bh = head = page_buffers(page);
+ diff = (page - compound_head(page)) << inode->i_blkbits;
+ while (diff--)
+ bh = bh->b_this_page;
do {
if (buffer_uptodate(bh) ||
buffer_unwritten(bh)) {
@@ -588,8 +592,12 @@ static int ext4_find_unwritten_pgoff(struct inode *inode,
} while (bh != head);
}
- lastoff = page_offset(page) + PAGE_SIZE;
+ lastoff = page_offset(page) + hpage_size(page);
unlock_page(page);
+ if (PageTransCompound(page)) {
+ i++;
+ break;
+ }
}
/*
@@ -602,7 +610,9 @@ static int ext4_find_unwritten_pgoff(struct inode *inode,
break;
}
- index = pvec.pages[i - 1]->index + 1;
+ index = page_to_pgoff(pvec.pages[i - 1]) + 1;
+ if (PageTransCompound(pvec.pages[i - 1]))
+ index = round_up(index, HPAGE_PMD_NR);
pagevec_release(&pvec);
} while (index <= end);
--
2.10.2
Adjust how we find relevant block within page and how we clear the
required part of the page.
Signed-off-by: Kirill A. Shutemov <[email protected]>
---
fs/ext4/move_extent.c | 12 +++++++++---
1 file changed, 9 insertions(+), 3 deletions(-)
diff --git a/fs/ext4/move_extent.c b/fs/ext4/move_extent.c
index 6fc14def0c70..2efa9deb47a9 100644
--- a/fs/ext4/move_extent.c
+++ b/fs/ext4/move_extent.c
@@ -210,7 +210,9 @@ mext_page_mkuptodate(struct page *page, unsigned from, unsigned to)
return err;
}
if (!buffer_mapped(bh)) {
- zero_user(page, block_start, blocksize);
+ zero_user(page + block_start / PAGE_SIZE,
+ block_start % PAGE_SIZE,
+ blocksize);
set_buffer_uptodate(bh);
continue;
}
@@ -267,10 +269,11 @@ move_extent_per_page(struct file *o_filp, struct inode *donor_inode,
unsigned int tmp_data_size, data_size, replaced_size;
int i, err2, jblocks, retries = 0;
int replaced_count = 0;
- int from = data_offset_in_page << orig_inode->i_blkbits;
+ int from;
int blocks_per_page = PAGE_SIZE >> orig_inode->i_blkbits;
struct super_block *sb = orig_inode->i_sb;
struct buffer_head *bh = NULL;
+ int diff;
/*
* It needs twice the amount of ordinary journal buffers because
@@ -355,6 +358,9 @@ move_extent_per_page(struct file *o_filp, struct inode *donor_inode,
goto unlock_pages;
}
data_copy:
+ diff = (pagep[0] - compound_head(pagep[0])) * blocks_per_page;
+ from = (data_offset_in_page + diff) << orig_inode->i_blkbits;
+ pagep[0] = compound_head(pagep[0]);
*err = mext_page_mkuptodate(pagep[0], from, from + replaced_size);
if (*err)
goto unlock_pages;
@@ -384,7 +390,7 @@ move_extent_per_page(struct file *o_filp, struct inode *donor_inode,
if (!page_has_buffers(pagep[0]))
create_empty_buffers(pagep[0], 1 << orig_inode->i_blkbits, 0);
bh = page_buffers(pagep[0]);
- for (i = 0; i < data_offset_in_page; i++)
+ for (i = 0; i < data_offset_in_page + diff; i++)
bh = bh->b_this_page;
for (i = 0; i < block_len_in_page; i++) {
*err = ext4_get_block(orig_inode, orig_blk_offset + i, bh, 0);
--
2.10.2
Modify mpage_map_and_submit_buffers() and mpage_release_unused_pages()
to deal with huge pages.
Mostly result of try-and-error. Critical view would be appriciated.
Signed-off-by: Kirill A. Shutemov <[email protected]>
---
fs/ext4/inode.c | 61 ++++++++++++++++++++++++++++++++++++++++-----------------
1 file changed, 43 insertions(+), 18 deletions(-)
diff --git a/fs/ext4/inode.c b/fs/ext4/inode.c
index 035256019e16..ff4f460d3625 100644
--- a/fs/ext4/inode.c
+++ b/fs/ext4/inode.c
@@ -1666,20 +1666,32 @@ static void mpage_release_unused_pages(struct mpage_da_data *mpd,
if (nr_pages == 0)
break;
for (i = 0; i < nr_pages; i++) {
- struct page *page = pvec.pages[i];
+ struct page *page = compound_head(pvec.pages[i]);
+
if (page->index > end)
break;
BUG_ON(!PageLocked(page));
BUG_ON(PageWriteback(page));
if (invalidate) {
+ unsigned long offset, len;
+
+ offset = (index % hpage_nr_pages(page));
+ len = min_t(unsigned long, end - page->index,
+ hpage_nr_pages(page));
+
if (page_mapped(page))
clear_page_dirty_for_io(page);
- block_invalidatepage(page, 0, PAGE_SIZE);
+ block_invalidatepage(page, offset << PAGE_SHIFT,
+ len << PAGE_SHIFT);
ClearPageUptodate(page);
}
unlock_page(page);
+ if (PageTransHuge(page))
+ break;
}
- index = pvec.pages[nr_pages - 1]->index + 1;
+ index = page_to_pgoff(pvec.pages[nr_pages - 1]) + 1;
+ if (PageTransCompound(pvec.pages[nr_pages - 1]))
+ index = round_up(index, HPAGE_PMD_NR);
pagevec_release(&pvec);
}
}
@@ -2113,16 +2125,16 @@ static int mpage_submit_page(struct mpage_da_data *mpd, struct page *page)
loff_t size = i_size_read(mpd->inode);
int err;
- BUG_ON(page->index != mpd->first_page);
- if (page->index == size >> PAGE_SHIFT)
- len = size & ~PAGE_MASK;
- else
- len = PAGE_SIZE;
+ page = compound_head(page);
+ len = hpage_size(page);
+ if (page->index + hpage_nr_pages(page) - 1 == size >> PAGE_SHIFT)
+ len = size & ~hpage_mask(page);
+
clear_page_dirty_for_io(page);
err = ext4_bio_write_page(&mpd->io_submit, page, len, mpd->wbc, false);
if (!err)
- mpd->wbc->nr_to_write--;
- mpd->first_page++;
+ mpd->wbc->nr_to_write -= hpage_nr_pages(page);
+ mpd->first_page = round_up(mpd->first_page + 1, hpage_nr_pages(page));
return err;
}
@@ -2270,12 +2282,16 @@ static int mpage_map_and_submit_buffers(struct mpage_da_data *mpd)
break;
for (i = 0; i < nr_pages; i++) {
struct page *page = pvec.pages[i];
+ unsigned long diff;
- if (page->index > end)
+ if (page_to_pgoff(page) > end)
break;
/* Up to 'end' pages must be contiguous */
- BUG_ON(page->index != start);
+ BUG_ON(page_to_pgoff(page) != start);
+ diff = (page - compound_head(page)) << bpp_bits;
bh = head = page_buffers(page);
+ while (diff--)
+ bh = bh->b_this_page;
do {
if (lblk < mpd->map.m_lblk)
continue;
@@ -2312,7 +2328,10 @@ static int mpage_map_and_submit_buffers(struct mpage_da_data *mpd)
* supports blocksize < pagesize as we will try to
* convert potentially unmapped parts of inode.
*/
- mpd->io_submit.io_end->size += PAGE_SIZE;
+ if (PageTransCompound(page))
+ mpd->io_submit.io_end->size += HPAGE_PMD_SIZE;
+ else
+ mpd->io_submit.io_end->size += PAGE_SIZE;
/* Page fully mapped - let IO run! */
err = mpage_submit_page(mpd, page);
if (err < 0) {
@@ -2320,6 +2339,10 @@ static int mpage_map_and_submit_buffers(struct mpage_da_data *mpd)
return err;
}
start++;
+ if (PageTransCompound(page)) {
+ start = round_up(start, HPAGE_PMD_NR);
+ break;
+ }
}
pagevec_release(&pvec);
}
@@ -2556,7 +2579,7 @@ static int mpage_prepare_extent_to_map(struct mpage_da_data *mpd)
* mapping. However, page->index will not change
* because we have a reference on the page.
*/
- if (page->index > end)
+ if (page_to_pgoff(page) > end)
goto out;
/*
@@ -2571,7 +2594,7 @@ static int mpage_prepare_extent_to_map(struct mpage_da_data *mpd)
goto out;
/* If we can't merge this page, we are done. */
- if (mpd->map.m_len > 0 && mpd->next_page != page->index)
+ if (mpd->map.m_len > 0 && mpd->next_page != page_to_pgoff(page))
goto out;
lock_page(page);
@@ -2585,7 +2608,7 @@ static int mpage_prepare_extent_to_map(struct mpage_da_data *mpd)
if (!PageDirty(page) ||
(PageWriteback(page) &&
(mpd->wbc->sync_mode == WB_SYNC_NONE)) ||
- unlikely(page->mapping != mapping)) {
+ unlikely(page_mapping(page) != mapping)) {
unlock_page(page);
continue;
}
@@ -2594,8 +2617,10 @@ static int mpage_prepare_extent_to_map(struct mpage_da_data *mpd)
BUG_ON(PageWriteback(page));
if (mpd->map.m_len == 0)
- mpd->first_page = page->index;
- mpd->next_page = page->index + 1;
+ mpd->first_page = page_to_pgoff(page);
+ page = compound_head(page);
+ mpd->next_page = round_up(page->index + 1,
+ hpage_nr_pages(page));
/* Add all dirty buffers to mpd */
lblk = ((ext4_lblk_t)page->index) <<
(PAGE_SHIFT - blkbits);
--
2.10.2
It's more or less straight-forward.
Most changes are around getting offset/len withing page right and zero
out desired part of the page.
Signed-off-by: Kirill A. Shutemov <[email protected]>
---
fs/buffer.c | 70 +++++++++++++++++++++++++++++++++++--------------------------
1 file changed, 40 insertions(+), 30 deletions(-)
diff --git a/fs/buffer.c b/fs/buffer.c
index 090f7edfa6b7..7d333621ccfb 100644
--- a/fs/buffer.c
+++ b/fs/buffer.c
@@ -1902,6 +1902,7 @@ void page_zero_new_buffers(struct page *page, unsigned from, unsigned to)
{
unsigned int block_start, block_end;
struct buffer_head *head, *bh;
+ bool uptodate = PageUptodate(page);
BUG_ON(!PageLocked(page));
if (!page_has_buffers(page))
@@ -1912,21 +1913,21 @@ void page_zero_new_buffers(struct page *page, unsigned from, unsigned to)
do {
block_end = block_start + bh->b_size;
- if (buffer_new(bh)) {
- if (block_end > from && block_start < to) {
- if (!PageUptodate(page)) {
- unsigned start, size;
+ if (buffer_new(bh) && block_end > from && block_start < to) {
+ if (!uptodate) {
+ unsigned start, size;
- start = max(from, block_start);
- size = min(to, block_end) - start;
+ start = max(from, block_start);
+ size = min(to, block_end) - start;
- zero_user(page, start, size);
- set_buffer_uptodate(bh);
- }
-
- clear_buffer_new(bh);
- mark_buffer_dirty(bh);
+ zero_user(page + block_start / PAGE_SIZE,
+ start % PAGE_SIZE,
+ size);
+ set_buffer_uptodate(bh);
}
+
+ clear_buffer_new(bh);
+ mark_buffer_dirty(bh);
}
block_start = block_end;
@@ -1992,18 +1993,21 @@ iomap_to_bh(struct inode *inode, sector_t block, struct buffer_head *bh,
int __block_write_begin_int(struct page *page, loff_t pos, unsigned len,
get_block_t *get_block, struct iomap *iomap)
{
- unsigned from = pos & (PAGE_SIZE - 1);
- unsigned to = from + len;
- struct inode *inode = page->mapping->host;
+ unsigned from, to;
+ struct inode *inode = page_mapping(page)->host;
unsigned block_start, block_end;
sector_t block;
int err = 0;
unsigned blocksize, bbits;
struct buffer_head *bh, *head, *wait[2], **wait_bh=wait;
+ bool uptodate = PageUptodate(page);
+ page = compound_head(page);
+ from = pos & ~hpage_mask(page);
+ to = from + len;
BUG_ON(!PageLocked(page));
- BUG_ON(from > PAGE_SIZE);
- BUG_ON(to > PAGE_SIZE);
+ BUG_ON(from > hpage_size(page));
+ BUG_ON(to > hpage_size(page));
BUG_ON(from > to);
head = create_page_buffers(page, inode, 0);
@@ -2016,10 +2020,8 @@ int __block_write_begin_int(struct page *page, loff_t pos, unsigned len,
block++, block_start=block_end, bh = bh->b_this_page) {
block_end = block_start + blocksize;
if (block_end <= from || block_start >= to) {
- if (PageUptodate(page)) {
- if (!buffer_uptodate(bh))
- set_buffer_uptodate(bh);
- }
+ if (uptodate && !buffer_uptodate(bh))
+ set_buffer_uptodate(bh);
continue;
}
if (buffer_new(bh))
@@ -2036,23 +2038,28 @@ int __block_write_begin_int(struct page *page, loff_t pos, unsigned len,
if (buffer_new(bh)) {
clean_bdev_bh_alias(bh);
- if (PageUptodate(page)) {
+ if (uptodate) {
clear_buffer_new(bh);
set_buffer_uptodate(bh);
mark_buffer_dirty(bh);
continue;
}
- if (block_end > to || block_start < from)
- zero_user_segments(page,
- to, block_end,
- block_start, from);
+ if (block_end > to || block_start < from) {
+ BUG_ON(to - from > PAGE_SIZE);
+ zero_user_segments(page +
+ block_start / PAGE_SIZE,
+ to % PAGE_SIZE,
+ (block_start % PAGE_SIZE) + blocksize,
+ block_start % PAGE_SIZE,
+ from % PAGE_SIZE);
+ }
continue;
}
}
- if (PageUptodate(page)) {
+ if (uptodate) {
if (!buffer_uptodate(bh))
set_buffer_uptodate(bh);
- continue;
+ continue;
}
if (!buffer_uptodate(bh) && !buffer_delay(bh) &&
!buffer_unwritten(bh) &&
@@ -2089,6 +2096,7 @@ static int __block_commit_write(struct inode *inode, struct page *page,
unsigned blocksize;
struct buffer_head *bh, *head;
+ VM_BUG_ON_PAGE(PageTail(page), page);
bh = head = page_buffers(page);
blocksize = bh->b_size;
@@ -2102,7 +2110,8 @@ static int __block_commit_write(struct inode *inode, struct page *page,
set_buffer_uptodate(bh);
mark_buffer_dirty(bh);
}
- clear_buffer_new(bh);
+ if (buffer_new(bh))
+ clear_buffer_new(bh);
block_start = block_end;
bh = bh->b_this_page;
@@ -2155,7 +2164,8 @@ int block_write_end(struct file *file, struct address_space *mapping,
struct inode *inode = mapping->host;
unsigned start;
- start = pos & (PAGE_SIZE - 1);
+ page = compound_head(page);
+ start = pos & ~hpage_mask(page);
if (unlikely(copied < len)) {
/*
--
2.10.2
For huge pages 'stop' must be within HPAGE_PMD_SIZE.
Let's use hpage_size() in the BUG_ON().
We also need to change how we calculate lblk for cluster deallocation.
Signed-off-by: Kirill A. Shutemov <[email protected]>
---
fs/ext4/inode.c | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/fs/ext4/inode.c b/fs/ext4/inode.c
index e89249c03d2f..035256019e16 100644
--- a/fs/ext4/inode.c
+++ b/fs/ext4/inode.c
@@ -1572,7 +1572,7 @@ static void ext4_da_page_release_reservation(struct page *page,
int num_clusters;
ext4_fsblk_t lblk;
- BUG_ON(stop > PAGE_SIZE || stop < length);
+ BUG_ON(stop > hpage_size(page) || stop < length);
head = page_buffers(page);
bh = head;
@@ -1607,7 +1607,8 @@ static void ext4_da_page_release_reservation(struct page *page,
* need to release the reserved space for that cluster. */
num_clusters = EXT4_NUM_B2C(sbi, to_release);
while (num_clusters > 0) {
- lblk = (page->index << (PAGE_SHIFT - inode->i_blkbits)) +
+ lblk = ((page->index + offset / PAGE_SIZE) <<
+ (PAGE_SHIFT - inode->i_blkbits)) +
((num_clusters - 1) << sbi->s_cluster_bits);
if (sbi->s_cluster_ratio == 1 ||
!ext4_find_delalloc_cluster(inode, lblk))
--
2.10.2
As the function handles zeroing range only within one block, the
required changes are trivial, just remove assuption on page size.
Signed-off-by: Kirill A. Shutemov <[email protected]>
---
fs/ext4/inode.c | 7 +++++--
1 file changed, 5 insertions(+), 2 deletions(-)
diff --git a/fs/ext4/inode.c b/fs/ext4/inode.c
index 387aa857770b..d3143dfe9962 100644
--- a/fs/ext4/inode.c
+++ b/fs/ext4/inode.c
@@ -3776,7 +3776,7 @@ static int __ext4_block_zero_page_range(handle_t *handle,
struct address_space *mapping, loff_t from, loff_t length)
{
ext4_fsblk_t index = from >> PAGE_SHIFT;
- unsigned offset = from & (PAGE_SIZE-1);
+ unsigned offset;
unsigned blocksize, pos;
ext4_lblk_t iblock;
struct inode *inode = mapping->host;
@@ -3789,6 +3789,9 @@ static int __ext4_block_zero_page_range(handle_t *handle,
if (!page)
return -ENOMEM;
+ page = compound_head(page);
+ offset = from & ~hpage_mask(page);
+
blocksize = inode->i_sb->s_blocksize;
iblock = index << (PAGE_SHIFT - inode->i_sb->s_blocksize_bits);
@@ -3845,7 +3848,7 @@ static int __ext4_block_zero_page_range(handle_t *handle,
if (err)
goto unlock;
}
- zero_user(page, offset, length);
+ zero_user(page + offset / PAGE_SIZE, offset % PAGE_SIZE, length);
BUFFER_TRACE(bh, "zeroed end of block");
if (ext4_should_journal_data(inode)) {
--
2.10.2
Trivial: remove assumption on page size.
Signed-off-by: Kirill A. Shutemov <[email protected]>
---
fs/ext4/inode.c | 13 +++++++------
1 file changed, 7 insertions(+), 6 deletions(-)
diff --git a/fs/ext4/inode.c b/fs/ext4/inode.c
index fa4467e4b129..387aa857770b 100644
--- a/fs/ext4/inode.c
+++ b/fs/ext4/inode.c
@@ -5759,7 +5759,7 @@ static int ext4_bh_unmapped(handle_t *handle, struct buffer_head *bh)
int ext4_page_mkwrite(struct vm_area_struct *vma, struct vm_fault *vmf)
{
- struct page *page = vmf->page;
+ struct page *page = compound_head(vmf->page);
loff_t size;
unsigned long len;
int ret;
@@ -5795,10 +5795,10 @@ int ext4_page_mkwrite(struct vm_area_struct *vma, struct vm_fault *vmf)
goto out;
}
- if (page->index == size >> PAGE_SHIFT)
- len = size & ~PAGE_MASK;
- else
- len = PAGE_SIZE;
+ len = hpage_size(page);
+ if (page->index + hpage_nr_pages(page) - 1 == size >> PAGE_SHIFT)
+ len = size & ~hpage_mask(page);
+
/*
* Return if we have all the buffers mapped. This avoids the need to do
* journal_start/journal_stop which can block and take a long time
@@ -5829,7 +5829,8 @@ int ext4_page_mkwrite(struct vm_area_struct *vma, struct vm_fault *vmf)
ret = block_page_mkwrite(vma, vmf, get_block);
if (!ret && ext4_should_journal_data(inode)) {
if (ext4_walk_page_buffers(handle, page_buffers(page), 0,
- PAGE_SIZE, NULL, do_journal_get_write_access)) {
+ hpage_size(page), NULL,
+ do_journal_get_write_access)) {
unlock_page(page);
ret = VM_FAULT_SIGBUS;
ext4_journal_stop(handle);
--
2.10.2
As BIO_MAX_PAGES is smaller (on x86) than HPAGE_PMD_NR, we cannot use
the optimization ext4_mpage_readpages() provides.
So, for huge pages, we fallback directly to block_read_full_page().
This should be re-visited once we get multipage bvec upstream.
Signed-off-by: Kirill A. Shutemov <[email protected]>
---
fs/ext4/readpage.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/fs/ext4/readpage.c b/fs/ext4/readpage.c
index a81b829d56de..b865df0c0973 100644
--- a/fs/ext4/readpage.c
+++ b/fs/ext4/readpage.c
@@ -134,7 +134,7 @@ int ext4_mpage_readpages(struct address_space *mapping,
goto next_page;
}
- if (page_has_buffers(page))
+ if (page_has_buffers(page) || PageTransHuge(page))
goto confused;
block_in_file = (sector_t)page->index << (PAGE_SHIFT - blkbits);
--
2.10.2
The approach is straight-forward: for compound pages we read out whole
huge page.
For huge page we cannot have array of buffer head pointers on stack --
it's 4096 pointers on x86-64 -- 'arr' is allocated with kmalloc() for
huge pages.
Signed-off-by: Kirill A. Shutemov <[email protected]>
---
fs/buffer.c | 22 +++++++++++++++++-----
include/linux/buffer_head.h | 9 +++++----
include/linux/page-flags.h | 2 +-
3 files changed, 23 insertions(+), 10 deletions(-)
diff --git a/fs/buffer.c b/fs/buffer.c
index d21771fcf7d3..090f7edfa6b7 100644
--- a/fs/buffer.c
+++ b/fs/buffer.c
@@ -871,7 +871,7 @@ struct buffer_head *alloc_page_buffers(struct page *page, unsigned long size,
try_again:
head = NULL;
- offset = PAGE_SIZE;
+ offset = hpage_size(page);
while ((offset -= size) >= 0) {
bh = alloc_buffer_head(GFP_NOFS);
if (!bh)
@@ -1466,7 +1466,7 @@ void set_bh_page(struct buffer_head *bh,
struct page *page, unsigned long offset)
{
bh->b_page = page;
- BUG_ON(offset >= PAGE_SIZE);
+ BUG_ON(offset >= hpage_size(page));
if (PageHighMem(page))
/*
* This catches illegal uses and preserves the offset:
@@ -2280,11 +2280,13 @@ int block_read_full_page(struct page *page, get_block_t *get_block)
{
struct inode *inode = page->mapping->host;
sector_t iblock, lblock;
- struct buffer_head *bh, *head, *arr[MAX_BUF_PER_PAGE];
+ struct buffer_head *arr_on_stack[MAX_BUF_PER_PAGE];
+ struct buffer_head *bh, *head, **arr = arr_on_stack;
unsigned int blocksize, bbits;
int nr, i;
int fully_mapped = 1;
+ VM_BUG_ON_PAGE(PageTail(page), page);
head = create_page_buffers(page, inode, 0);
blocksize = head->b_size;
bbits = block_size_bits(blocksize);
@@ -2295,6 +2297,11 @@ int block_read_full_page(struct page *page, get_block_t *get_block)
nr = 0;
i = 0;
+ if (PageTransHuge(page)) {
+ arr = kmalloc(sizeof(struct buffer_head *) * HPAGE_PMD_NR *
+ MAX_BUF_PER_PAGE, GFP_NOFS);
+ }
+
do {
if (buffer_uptodate(bh))
continue;
@@ -2310,7 +2317,9 @@ int block_read_full_page(struct page *page, get_block_t *get_block)
SetPageError(page);
}
if (!buffer_mapped(bh)) {
- zero_user(page, i * blocksize, blocksize);
+ zero_user(page + (i * blocksize / PAGE_SIZE),
+ i * blocksize % PAGE_SIZE,
+ blocksize);
if (!err)
set_buffer_uptodate(bh);
continue;
@@ -2336,7 +2345,7 @@ int block_read_full_page(struct page *page, get_block_t *get_block)
if (!PageError(page))
SetPageUptodate(page);
unlock_page(page);
- return 0;
+ goto out;
}
/* Stage two: lock the buffers */
@@ -2358,6 +2367,9 @@ int block_read_full_page(struct page *page, get_block_t *get_block)
else
submit_bh(REQ_OP_READ, 0, bh);
}
+out:
+ if (arr != arr_on_stack)
+ kfree(arr);
return 0;
}
EXPORT_SYMBOL(block_read_full_page);
diff --git a/include/linux/buffer_head.h b/include/linux/buffer_head.h
index fd4134ce9c54..f12f6293ed44 100644
--- a/include/linux/buffer_head.h
+++ b/include/linux/buffer_head.h
@@ -131,13 +131,14 @@ BUFFER_FNS(Meta, meta)
BUFFER_FNS(Prio, prio)
BUFFER_FNS(Defer_Completion, defer_completion)
-#define bh_offset(bh) ((unsigned long)(bh)->b_data & ~PAGE_MASK)
+#define bh_offset(bh) ((unsigned long)(bh)->b_data & ~hpage_mask(bh->b_page))
/* If we *know* page->private refers to buffer_heads */
-#define page_buffers(page) \
+#define page_buffers(__page) \
({ \
- BUG_ON(!PagePrivate(page)); \
- ((struct buffer_head *)page_private(page)); \
+ struct page *p = compound_head(__page); \
+ BUG_ON(!PagePrivate(p)); \
+ ((struct buffer_head *)page_private(p)); \
})
#define page_has_buffers(page) PagePrivate(page)
diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h
index a2bef9a41bcf..20b7684e9298 100644
--- a/include/linux/page-flags.h
+++ b/include/linux/page-flags.h
@@ -730,7 +730,7 @@ static inline void ClearPageSlabPfmemalloc(struct page *page)
*/
static inline int page_has_private(struct page *page)
{
- return !!(page->flags & PAGE_FLAGS_PRIVATE);
+ return !!(compound_head(page)->flags & PAGE_FLAGS_PRIVATE);
}
#undef PF_ANY
--
2.10.2
For huge pages we need to unmap whole range covered by the huge page.
Signed-off-by: Kirill A. Shutemov <[email protected]>
---
mm/truncate.c | 23 ++++++++++++++---------
1 file changed, 14 insertions(+), 9 deletions(-)
diff --git a/mm/truncate.c b/mm/truncate.c
index d2d95f283ec3..6df4b06a190f 100644
--- a/mm/truncate.c
+++ b/mm/truncate.c
@@ -656,27 +656,32 @@ int invalidate_inode_pages2_range(struct address_space *mapping,
continue;
}
wait_on_page_writeback(page);
+
if (page_mapped(page)) {
+ loff_t begin, len;
+
+ begin = index << PAGE_SHIFT;
if (!did_range_unmap) {
/*
* Zap the rest of the file in one hit.
*/
+ len = (loff_t)(1 + end - index) <<
+ PAGE_SHIFT;
+ if (len < hpage_size(page))
+ len = hpage_size(page);
unmap_mapping_range(mapping,
- (loff_t)index << PAGE_SHIFT,
- (loff_t)(1 + end - index)
- << PAGE_SHIFT,
- 0);
+ begin, len, 0);
did_range_unmap = 1;
} else {
/*
* Just zap this page
*/
- unmap_mapping_range(mapping,
- (loff_t)index << PAGE_SHIFT,
- PAGE_SIZE, 0);
+ len = hpage_size(page);
+ unmap_mapping_range(mapping, begin,
+ len, 0);
}
}
- BUG_ON(page_mapped(page));
+ VM_BUG_ON_PAGE(page_mapped(page), page);
ret2 = do_launder_page(mapping, page);
if (ret2 == 0) {
if (!invalidate_complete_page2(mapping, page))
@@ -687,9 +692,9 @@ int invalidate_inode_pages2_range(struct address_space *mapping,
unlock_page(page);
}
pagevec_remove_exceptionals(&pvec);
+ index += pvec.nr ? hpage_nr_pages(pvec.pages[pvec.nr - 1]) : 1;
pagevec_release(&pvec);
cond_resched();
- index++;
}
cleancache_invalidate_inode(mapping);
return ret;
--
2.10.2
Adjust check on whether part of the page beyond file size and apply
compound_head() and page_mapping() where appropriate.
Signed-off-by: Kirill A. Shutemov <[email protected]>
---
fs/buffer.c | 10 +++++-----
1 file changed, 5 insertions(+), 5 deletions(-)
diff --git a/fs/buffer.c b/fs/buffer.c
index 7d333621ccfb..8e000021513c 100644
--- a/fs/buffer.c
+++ b/fs/buffer.c
@@ -2544,7 +2544,7 @@ EXPORT_SYMBOL(block_commit_write);
int block_page_mkwrite(struct vm_area_struct *vma, struct vm_fault *vmf,
get_block_t get_block)
{
- struct page *page = vmf->page;
+ struct page *page = compound_head(vmf->page);
struct inode *inode = file_inode(vma->vm_file);
unsigned long end;
loff_t size;
@@ -2552,7 +2552,7 @@ int block_page_mkwrite(struct vm_area_struct *vma, struct vm_fault *vmf,
lock_page(page);
size = i_size_read(inode);
- if ((page->mapping != inode->i_mapping) ||
+ if ((page_mapping(page) != inode->i_mapping) ||
(page_offset(page) > size)) {
/* We overload EFAULT to mean page got truncated */
ret = -EFAULT;
@@ -2560,10 +2560,10 @@ int block_page_mkwrite(struct vm_area_struct *vma, struct vm_fault *vmf,
}
/* page is wholly or partially inside EOF */
- if (((page->index + 1) << PAGE_SHIFT) > size)
- end = size & ~PAGE_MASK;
+ if (((page->index + hpage_nr_pages(page)) << PAGE_SHIFT) > size)
+ end = size & ~hpage_mask(page);
else
- end = PAGE_SIZE;
+ end = hpage_size(page);
ret = __block_write_begin(page, 0, end, get_block);
if (!ret)
--
2.10.2
We would need to use multi-order radix-tree entires for ext4 and other
filesystems to have coherent view on tags (dirty/towrite) in the tree.
This patch converts huge tmpfs implementation to multi-order entries, so
we will be able to use the same code patch for all filesystems.
We also change interface for page-cache lookup function:
- functions that lookup for pages[1] would return subpages of THP
relevant for requested indexes;
- functions that lookup for entries[2] would return one entry per-THP
and index will point to index of head page (basically, round down to
HPAGE_PMD_NR);
This would provide balanced exposure of multi-order entires to the rest
of the kernel.
[1] find_get_pages(), pagecache_get_page(), pagevec_lookup(), etc.
[2] find_get_entry(), find_get_entries(), pagevec_lookup_entries(), etc.
Signed-off-by: Kirill A. Shutemov <[email protected]>
---
include/linux/pagemap.h | 9 ++
mm/filemap.c | 236 ++++++++++++++++++++++++++----------------------
mm/huge_memory.c | 48 +++++++---
mm/khugepaged.c | 26 ++----
mm/shmem.c | 117 ++++++++++--------------
mm/truncate.c | 15 ++-
6 files changed, 235 insertions(+), 216 deletions(-)
diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h
index 7dbe9148b2f8..f88d69e2419d 100644
--- a/include/linux/pagemap.h
+++ b/include/linux/pagemap.h
@@ -332,6 +332,15 @@ static inline struct page *grab_cache_page_nowait(struct address_space *mapping,
mapping_gfp_mask(mapping));
}
+static inline struct page *find_subpage(struct page *page, pgoff_t offset)
+{
+ VM_BUG_ON_PAGE(PageTail(page), page);
+ VM_BUG_ON_PAGE(page->index > offset, page);
+ VM_BUG_ON_PAGE(page->index + (1 << compound_order(page)) < offset,
+ page);
+ return page - page->index + offset;
+}
+
struct page *find_get_entry(struct address_space *mapping, pgoff_t offset);
struct page *find_lock_entry(struct address_space *mapping, pgoff_t offset);
unsigned find_get_entries(struct address_space *mapping, pgoff_t start,
diff --git a/mm/filemap.c b/mm/filemap.c
index 235021e361eb..f8607ab7b7e4 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -150,7 +150,9 @@ static int page_cache_tree_insert(struct address_space *mapping,
static void page_cache_tree_delete(struct address_space *mapping,
struct page *page, void *shadow)
{
- int i, nr;
+ struct radix_tree_node *node;
+ void **slot;
+ int nr;
/* hugetlb pages are represented by one entry in the radix tree */
nr = PageHuge(page) ? 1 : hpage_nr_pages(page);
@@ -159,19 +161,12 @@ static void page_cache_tree_delete(struct address_space *mapping,
VM_BUG_ON_PAGE(PageTail(page), page);
VM_BUG_ON_PAGE(nr != 1 && shadow, page);
- for (i = 0; i < nr; i++) {
- struct radix_tree_node *node;
- void **slot;
+ __radix_tree_lookup(&mapping->page_tree, page->index, &node, &slot);
+ VM_BUG_ON_PAGE(!node && nr != 1, page);
- __radix_tree_lookup(&mapping->page_tree, page->index + i,
- &node, &slot);
-
- VM_BUG_ON_PAGE(!node && nr != 1, page);
-
- radix_tree_clear_tags(&mapping->page_tree, node, slot);
- __radix_tree_replace(&mapping->page_tree, node, slot, shadow,
- workingset_update_node, mapping);
- }
+ radix_tree_clear_tags(&mapping->page_tree, node, slot);
+ __radix_tree_replace(&mapping->page_tree, node, slot, shadow,
+ workingset_update_node, mapping);
if (shadow) {
mapping->nrexceptional += nr;
@@ -285,12 +280,7 @@ void delete_from_page_cache(struct page *page)
if (freepage)
freepage(page);
- if (PageTransHuge(page) && !PageHuge(page)) {
- page_ref_sub(page, HPAGE_PMD_NR);
- VM_BUG_ON_PAGE(page_count(page) <= 0, page);
- } else {
- put_page(page);
- }
+ put_page(page);
}
EXPORT_SYMBOL(delete_from_page_cache);
@@ -1035,7 +1025,7 @@ EXPORT_SYMBOL(page_cache_prev_hole);
struct page *find_get_entry(struct address_space *mapping, pgoff_t offset)
{
void **pagep;
- struct page *head, *page;
+ struct page *page;
rcu_read_lock();
repeat:
@@ -1056,15 +1046,8 @@ struct page *find_get_entry(struct address_space *mapping, pgoff_t offset)
goto out;
}
- head = compound_head(page);
- if (!page_cache_get_speculative(head))
- goto repeat;
-
- /* The page was split under us? */
- if (compound_head(page) != head) {
- put_page(head);
+ if (!page_cache_get_speculative(page))
goto repeat;
- }
/*
* Has the page moved?
@@ -1072,7 +1055,7 @@ struct page *find_get_entry(struct address_space *mapping, pgoff_t offset)
* include/linux/pagemap.h for details.
*/
if (unlikely(page != *pagep)) {
- put_page(head);
+ put_page(page);
goto repeat;
}
}
@@ -1113,7 +1096,6 @@ struct page *find_lock_entry(struct address_space *mapping, pgoff_t offset)
put_page(page);
goto repeat;
}
- VM_BUG_ON_PAGE(page_to_pgoff(page) != offset, page);
}
return page;
}
@@ -1170,7 +1152,6 @@ struct page *pagecache_get_page(struct address_space *mapping, pgoff_t offset,
put_page(page);
goto repeat;
}
- VM_BUG_ON_PAGE(page->index != offset, page);
}
if (page && (fgp_flags & FGP_ACCESSED))
@@ -1205,6 +1186,8 @@ struct page *pagecache_get_page(struct address_space *mapping, pgoff_t offset,
}
}
+ if (page)
+ page = find_subpage(page, offset);
return page;
}
EXPORT_SYMBOL(pagecache_get_page);
@@ -1245,7 +1228,7 @@ unsigned find_get_entries(struct address_space *mapping,
rcu_read_lock();
radix_tree_for_each_slot(slot, &mapping->page_tree, &iter, start) {
- struct page *head, *page;
+ struct page *page;
repeat:
page = radix_tree_deref_slot(slot);
if (unlikely(!page))
@@ -1263,19 +1246,12 @@ unsigned find_get_entries(struct address_space *mapping,
goto export;
}
- head = compound_head(page);
- if (!page_cache_get_speculative(head))
+ if (!page_cache_get_speculative(page))
goto repeat;
- /* The page was split under us? */
- if (compound_head(page) != head) {
- put_page(head);
- goto repeat;
- }
-
/* Has the page moved? */
if (unlikely(page != *slot)) {
- put_page(head);
+ put_page(page);
goto repeat;
}
export:
@@ -1309,14 +1285,17 @@ unsigned find_get_pages(struct address_space *mapping, pgoff_t start,
{
struct radix_tree_iter iter;
void **slot;
- unsigned ret = 0;
+ unsigned refs, ret = 0;
if (unlikely(!nr_pages))
return 0;
rcu_read_lock();
radix_tree_for_each_slot(slot, &mapping->page_tree, &iter, start) {
- struct page *head, *page;
+ struct page *page;
+ unsigned long index = iter.index;
+ if (index < start)
+ index = start;
repeat:
page = radix_tree_deref_slot(slot);
if (unlikely(!page))
@@ -1335,25 +1314,35 @@ unsigned find_get_pages(struct address_space *mapping, pgoff_t start,
continue;
}
- head = compound_head(page);
- if (!page_cache_get_speculative(head))
- goto repeat;
-
- /* The page was split under us? */
- if (compound_head(page) != head) {
- put_page(head);
+ if (!page_cache_get_speculative(page))
goto repeat;
- }
/* Has the page moved? */
if (unlikely(page != *slot)) {
- put_page(head);
+ put_page(page);
goto repeat;
}
+ /* For multi-order entries, find relevant subpage */
+ if (PageTransHuge(page)) {
+ VM_BUG_ON(index - page->index < 0);
+ VM_BUG_ON(index - page->index >= HPAGE_PMD_NR);
+ page += index - page->index;
+ }
+
pages[ret] = page;
if (++ret == nr_pages)
break;
+ if (!PageTransCompound(page))
+ continue;
+ for (refs = 0; ret < nr_pages &&
+ (index + 1) % HPAGE_PMD_NR;
+ ret++, refs++, index++)
+ pages[ret] = ++page;
+ if (refs)
+ page_ref_add(compound_head(page), refs);
+ if (ret == nr_pages)
+ break;
}
rcu_read_unlock();
@@ -1363,7 +1352,7 @@ unsigned find_get_pages(struct address_space *mapping, pgoff_t start,
/**
* find_get_pages_contig - gang contiguous pagecache lookup
* @mapping: The address_space to search
- * @index: The starting page index
+ * @start: The starting page index
* @nr_pages: The maximum number of pages
* @pages: Where the resulting pages are placed
*
@@ -1372,19 +1361,22 @@ unsigned find_get_pages(struct address_space *mapping, pgoff_t start,
*
* find_get_pages_contig() returns the number of pages which were found.
*/
-unsigned find_get_pages_contig(struct address_space *mapping, pgoff_t index,
+unsigned find_get_pages_contig(struct address_space *mapping, pgoff_t start,
unsigned int nr_pages, struct page **pages)
{
struct radix_tree_iter iter;
void **slot;
- unsigned int ret = 0;
+ unsigned int refs, ret = 0;
if (unlikely(!nr_pages))
return 0;
rcu_read_lock();
- radix_tree_for_each_contig(slot, &mapping->page_tree, &iter, index) {
- struct page *head, *page;
+ radix_tree_for_each_contig(slot, &mapping->page_tree, &iter, start) {
+ struct page *page;
+ unsigned long index = iter.index;
+ if (index < start)
+ index = start;
repeat:
page = radix_tree_deref_slot(slot);
/* The hole, there no reason to continue */
@@ -1404,19 +1396,12 @@ unsigned find_get_pages_contig(struct address_space *mapping, pgoff_t index,
break;
}
- head = compound_head(page);
- if (!page_cache_get_speculative(head))
+ if (!page_cache_get_speculative(page))
goto repeat;
- /* The page was split under us? */
- if (compound_head(page) != head) {
- put_page(head);
- goto repeat;
- }
-
/* Has the page moved? */
if (unlikely(page != *slot)) {
- put_page(head);
+ put_page(page);
goto repeat;
}
@@ -1425,14 +1410,31 @@ unsigned find_get_pages_contig(struct address_space *mapping, pgoff_t index,
* otherwise we can get both false positives and false
* negatives, which is just confusing to the caller.
*/
- if (page->mapping == NULL || page_to_pgoff(page) != iter.index) {
+ if (page->mapping == NULL || page_to_pgoff(page) != index) {
put_page(page);
break;
}
+ /* For multi-order entries, find relevant subpage */
+ if (PageTransHuge(page)) {
+ VM_BUG_ON(index - page->index < 0);
+ VM_BUG_ON(index - page->index >= HPAGE_PMD_NR);
+ page += index - page->index;
+ }
+
pages[ret] = page;
if (++ret == nr_pages)
break;
+ if (!PageTransCompound(page))
+ continue;
+ for (refs = 0; ret < nr_pages &&
+ (index + 1) % HPAGE_PMD_NR;
+ ret++, refs++, index++)
+ pages[ret] = ++page;
+ if (refs)
+ page_ref_add(compound_head(page), refs);
+ if (ret == nr_pages)
+ break;
}
rcu_read_unlock();
return ret;
@@ -1442,7 +1444,7 @@ EXPORT_SYMBOL(find_get_pages_contig);
/**
* find_get_pages_tag - find and return pages that match @tag
* @mapping: the address_space to search
- * @index: the starting page index
+ * @indexp: the starting page index
* @tag: the tag index
* @nr_pages: the maximum number of pages
* @pages: where the resulting pages are placed
@@ -1450,20 +1452,23 @@ EXPORT_SYMBOL(find_get_pages_contig);
* Like find_get_pages, except we only return pages which are tagged with
* @tag. We update @index to index the next page for the traversal.
*/
-unsigned find_get_pages_tag(struct address_space *mapping, pgoff_t *index,
+unsigned find_get_pages_tag(struct address_space *mapping, pgoff_t *indexp,
int tag, unsigned int nr_pages, struct page **pages)
{
struct radix_tree_iter iter;
void **slot;
- unsigned ret = 0;
+ unsigned refs, ret = 0;
if (unlikely(!nr_pages))
return 0;
rcu_read_lock();
radix_tree_for_each_tagged(slot, &mapping->page_tree,
- &iter, *index, tag) {
- struct page *head, *page;
+ &iter, *indexp, tag) {
+ struct page *page;
+ unsigned long index = iter.index;
+ if (index < *indexp)
+ index = *indexp;
repeat:
page = radix_tree_deref_slot(slot);
if (unlikely(!page))
@@ -1488,31 +1493,41 @@ unsigned find_get_pages_tag(struct address_space *mapping, pgoff_t *index,
continue;
}
- head = compound_head(page);
- if (!page_cache_get_speculative(head))
- goto repeat;
-
- /* The page was split under us? */
- if (compound_head(page) != head) {
- put_page(head);
+ if (!page_cache_get_speculative(page))
goto repeat;
- }
/* Has the page moved? */
if (unlikely(page != *slot)) {
- put_page(head);
+ put_page(page);
goto repeat;
}
+ /* For multi-order entries, find relevant subpage */
+ if (PageTransHuge(page)) {
+ VM_BUG_ON(index - page->index < 0);
+ VM_BUG_ON(index - page->index >= HPAGE_PMD_NR);
+ page += index - page->index;
+ }
+
pages[ret] = page;
if (++ret == nr_pages)
break;
+ if (!PageTransCompound(page))
+ continue;
+ for (refs = 0; ret < nr_pages &&
+ (index + 1) % HPAGE_PMD_NR;
+ ret++, refs++, index++)
+ pages[ret] = ++page;
+ if (refs)
+ page_ref_add(compound_head(page), refs);
+ if (ret == nr_pages)
+ break;
}
rcu_read_unlock();
if (ret)
- *index = pages[ret - 1]->index + 1;
+ *indexp = page_to_pgoff(pages[ret - 1]) + 1;
return ret;
}
@@ -1544,7 +1559,7 @@ unsigned find_get_entries_tag(struct address_space *mapping, pgoff_t start,
rcu_read_lock();
radix_tree_for_each_tagged(slot, &mapping->page_tree,
&iter, start, tag) {
- struct page *head, *page;
+ struct page *page;
repeat:
page = radix_tree_deref_slot(slot);
if (unlikely(!page))
@@ -1563,19 +1578,12 @@ unsigned find_get_entries_tag(struct address_space *mapping, pgoff_t start,
goto export;
}
- head = compound_head(page);
- if (!page_cache_get_speculative(head))
- goto repeat;
-
- /* The page was split under us? */
- if (compound_head(page) != head) {
- put_page(head);
+ if (!page_cache_get_speculative(page))
goto repeat;
- }
/* Has the page moved? */
if (unlikely(page != *slot)) {
- put_page(head);
+ put_page(page);
goto repeat;
}
export:
@@ -2173,12 +2181,15 @@ void filemap_map_pages(struct vm_fault *vmf,
struct address_space *mapping = file->f_mapping;
pgoff_t last_pgoff = start_pgoff;
loff_t size;
- struct page *head, *page;
+ struct page *page;
rcu_read_lock();
radix_tree_for_each_slot(slot, &mapping->page_tree, &iter,
start_pgoff) {
- if (iter.index > end_pgoff)
+ unsigned long index = iter.index;
+ if (index < start_pgoff)
+ index = start_pgoff;
+ if (index > end_pgoff)
break;
repeat:
page = radix_tree_deref_slot(slot);
@@ -2189,25 +2200,26 @@ void filemap_map_pages(struct vm_fault *vmf,
slot = radix_tree_iter_retry(&iter);
continue;
}
+ page = NULL;
goto next;
}
- head = compound_head(page);
- if (!page_cache_get_speculative(head))
+ if (!page_cache_get_speculative(page))
goto repeat;
- /* The page was split under us? */
- if (compound_head(page) != head) {
- put_page(head);
- goto repeat;
- }
-
/* Has the page moved? */
if (unlikely(page != *slot)) {
- put_page(head);
+ put_page(page);
goto repeat;
}
+ /* For multi-order entries, find relevant subpage */
+ if (PageTransHuge(page)) {
+ VM_BUG_ON(index - page->index < 0);
+ VM_BUG_ON(index - page->index >= HPAGE_PMD_NR);
+ page += index - page->index;
+ }
+
if (!PageUptodate(page) ||
PageReadahead(page) ||
PageHWPoison(page))
@@ -2215,20 +2227,20 @@ void filemap_map_pages(struct vm_fault *vmf,
if (!trylock_page(page))
goto skip;
- if (page->mapping != mapping || !PageUptodate(page))
+ if (page_mapping(page) != mapping || !PageUptodate(page))
goto unlock;
size = round_up(i_size_read(mapping->host), PAGE_SIZE);
- if (page->index >= size >> PAGE_SHIFT)
+ if (compound_head(page)->index >= size >> PAGE_SHIFT)
goto unlock;
if (file->f_ra.mmap_miss > 0)
file->f_ra.mmap_miss--;
- vmf->address += (iter.index - last_pgoff) << PAGE_SHIFT;
+ vmf->address += (index - last_pgoff) << PAGE_SHIFT;
if (vmf->pte)
- vmf->pte += iter.index - last_pgoff;
- last_pgoff = iter.index;
+ vmf->pte += index - last_pgoff;
+ last_pgoff = index;
if (alloc_set_pte(vmf, NULL, page))
goto unlock;
unlock_page(page);
@@ -2241,8 +2253,14 @@ void filemap_map_pages(struct vm_fault *vmf,
/* Huge page is mapped? No need to proceed. */
if (pmd_trans_huge(*vmf->pmd))
break;
- if (iter.index == end_pgoff)
+ if (index == end_pgoff)
break;
+ if (page && PageTransCompound(page) &&
+ (index & (HPAGE_PMD_NR - 1)) !=
+ HPAGE_PMD_NR - 1) {
+ index++;
+ goto repeat;
+ }
}
rcu_read_unlock();
}
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 0957e654b3c9..7680797b287e 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -1922,6 +1922,7 @@ static void __split_huge_page(struct page *page, struct list_head *list,
struct page *head = compound_head(page);
struct zone *zone = page_zone(head);
struct lruvec *lruvec;
+ struct page *subpage;
pgoff_t end = -1;
int i;
@@ -1930,8 +1931,27 @@ static void __split_huge_page(struct page *page, struct list_head *list,
/* complete memcg works before add pages to LRU */
mem_cgroup_split_huge_fixup(head);
- if (!PageAnon(page))
- end = DIV_ROUND_UP(i_size_read(head->mapping->host), PAGE_SIZE);
+ if (!PageAnon(head)) {
+ struct address_space *mapping = head->mapping;
+ struct radix_tree_iter iter;
+ void **slot;
+
+ __dec_node_page_state(head, NR_SHMEM_THPS);
+
+ radix_tree_split(&mapping->page_tree, head->index, 0);
+ radix_tree_for_each_slot(slot, &mapping->page_tree, &iter,
+ head->index) {
+ if (iter.index >= head->index + HPAGE_PMD_NR)
+ break;
+ subpage = head + iter.index - head->index;
+ radix_tree_replace_slot(&mapping->page_tree,
+ slot, subpage);
+ VM_BUG_ON_PAGE(compound_head(subpage) != head, subpage);
+ }
+ radix_tree_preload_end();
+
+ end = DIV_ROUND_UP(i_size_read(mapping->host), PAGE_SIZE);
+ }
for (i = HPAGE_PMD_NR - 1; i >= 1; i--) {
__split_huge_page_tail(head, i, lruvec, list);
@@ -1960,7 +1980,7 @@ static void __split_huge_page(struct page *page, struct list_head *list,
unfreeze_page(head);
for (i = 0; i < HPAGE_PMD_NR; i++) {
- struct page *subpage = head + i;
+ subpage = head + i;
if (subpage == page)
continue;
unlock_page(subpage);
@@ -2117,8 +2137,8 @@ int split_huge_page_to_list(struct page *page, struct list_head *list)
goto out;
}
- /* Addidional pins from radix tree */
- extra_pins = HPAGE_PMD_NR;
+ /* Addidional pin from radix tree */
+ extra_pins = 1;
anon_vma = NULL;
i_mmap_lock_read(mapping);
}
@@ -2140,6 +2160,12 @@ int split_huge_page_to_list(struct page *page, struct list_head *list)
if (mlocked)
lru_add_drain();
+ if (mapping && radix_tree_split_preload(HPAGE_PMD_ORDER, 0,
+ GFP_KERNEL)) {
+ ret = -ENOMEM;
+ goto unfreeze;
+ }
+
/* prevent PageLRU to go away from under us, and freeze lru stats */
spin_lock_irqsave(zone_lru_lock(page_zone(head)), flags);
@@ -2149,10 +2175,7 @@ int split_huge_page_to_list(struct page *page, struct list_head *list)
spin_lock(&mapping->tree_lock);
pslot = radix_tree_lookup_slot(&mapping->page_tree,
page_index(head));
- /*
- * Check if the head page is present in radix tree.
- * We assume all tail are present too, if head is there.
- */
+ /* Check if the page is present in radix tree */
if (radix_tree_deref_slot_protected(pslot,
&mapping->tree_lock) != head)
goto fail;
@@ -2167,8 +2190,6 @@ int split_huge_page_to_list(struct page *page, struct list_head *list)
pgdata->split_queue_len--;
list_del(page_deferred_list(head));
}
- if (mapping)
- __dec_node_page_state(page, NR_SHMEM_THPS);
spin_unlock(&pgdata->split_queue_lock);
__split_huge_page(page, list, flags);
ret = 0;
@@ -2182,9 +2203,12 @@ int split_huge_page_to_list(struct page *page, struct list_head *list)
BUG();
}
spin_unlock(&pgdata->split_queue_lock);
-fail: if (mapping)
+fail: if (mapping) {
spin_unlock(&mapping->tree_lock);
+ radix_tree_preload_end();
+ }
spin_unlock_irqrestore(zone_lru_lock(page_zone(head)), flags);
+unfreeze:
unfreeze_page(head);
ret = -EBUSY;
}
diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index e32389a97030..7e9ec33d3575 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -1368,10 +1368,8 @@ static void collapse_shmem(struct mm_struct *mm,
break;
}
nr_none += n;
- for (; index < min(iter.index, end); index++) {
- radix_tree_insert(&mapping->page_tree, index,
- new_page + (index % HPAGE_PMD_NR));
- }
+ for (; index < min(iter.index, end); index++)
+ radix_tree_insert(&mapping->page_tree, index, new_page);
/* We are done. */
if (index >= end)
@@ -1443,8 +1441,7 @@ static void collapse_shmem(struct mm_struct *mm,
list_add_tail(&page->lru, &pagelist);
/* Finally, replace with the new page. */
- radix_tree_replace_slot(&mapping->page_tree, slot,
- new_page + (index % HPAGE_PMD_NR));
+ radix_tree_replace_slot(&mapping->page_tree, slot, new_page);
slot = radix_tree_iter_resume(slot, &iter);
index++;
@@ -1462,24 +1459,17 @@ static void collapse_shmem(struct mm_struct *mm,
break;
}
- /*
- * Handle hole in radix tree at the end of the range.
- * This code only triggers if there's nothing in radix tree
- * beyond 'end'.
- */
- if (result == SCAN_SUCCEED && index < end) {
+ if (result == SCAN_SUCCEED) {
int n = end - index;
- if (!shmem_charge(mapping->host, n)) {
+ if (n && !shmem_charge(mapping->host, n)) {
result = SCAN_FAIL;
goto tree_locked;
}
-
- for (; index < end; index++) {
- radix_tree_insert(&mapping->page_tree, index,
- new_page + (index % HPAGE_PMD_NR));
- }
nr_none += n;
+
+ radix_tree_join(&mapping->page_tree, start,
+ HPAGE_PMD_ORDER, new_page);
}
tree_locked:
diff --git a/mm/shmem.c b/mm/shmem.c
index 0dd83bbe44a8..183d2937157e 100644
--- a/mm/shmem.c
+++ b/mm/shmem.c
@@ -544,33 +544,14 @@ static int shmem_add_to_page_cache(struct page *page,
VM_BUG_ON_PAGE(!PageSwapBacked(page), page);
VM_BUG_ON(expected && PageTransHuge(page));
- page_ref_add(page, nr);
+ get_page(page);
page->mapping = mapping;
page->index = index;
spin_lock_irq(&mapping->tree_lock);
- if (PageTransHuge(page)) {
- void __rcu **results;
- pgoff_t idx;
- int i;
-
- error = 0;
- if (radix_tree_gang_lookup_slot(&mapping->page_tree,
- &results, &idx, index, 1) &&
- idx < index + HPAGE_PMD_NR) {
- error = -EEXIST;
- }
-
- if (!error) {
- for (i = 0; i < HPAGE_PMD_NR; i++) {
- error = radix_tree_insert(&mapping->page_tree,
- index + i, page + i);
- VM_BUG_ON(error);
- }
- count_vm_event(THP_FILE_ALLOC);
- }
- } else if (!expected) {
- error = radix_tree_insert(&mapping->page_tree, index, page);
+ if (!expected) {
+ error = __radix_tree_insert(&mapping->page_tree, index,
+ compound_order(page), page);
} else {
error = shmem_radix_tree_replace(mapping, index, expected,
page);
@@ -578,15 +559,17 @@ static int shmem_add_to_page_cache(struct page *page,
if (!error) {
mapping->nrpages += nr;
- if (PageTransHuge(page))
+ if (PageTransHuge(page)) {
+ count_vm_event(THP_FILE_ALLOC);
__inc_node_page_state(page, NR_SHMEM_THPS);
+ }
__mod_node_page_state(page_pgdat(page), NR_FILE_PAGES, nr);
__mod_node_page_state(page_pgdat(page), NR_SHMEM, nr);
spin_unlock_irq(&mapping->tree_lock);
} else {
page->mapping = NULL;
spin_unlock_irq(&mapping->tree_lock);
- page_ref_sub(page, nr);
+ put_page(page);
}
return error;
}
@@ -727,8 +710,9 @@ void shmem_unlock_mapping(struct address_space *mapping)
PAGEVEC_SIZE, pvec.pages, indices);
if (!pvec.nr)
break;
- index = indices[pvec.nr - 1] + 1;
pagevec_remove_exceptionals(&pvec);
+ index = indices[pvec.nr - 1] +
+ hpage_nr_pages(pvec.pages[pvec.nr - 1]);
check_move_unevictable_pages(pvec.pages, pvec.nr);
pagevec_release(&pvec);
cond_resched();
@@ -785,23 +769,25 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, loff_t lend,
if (!trylock_page(page))
continue;
- if (PageTransTail(page)) {
- /* Middle of THP: zero out the page */
- clear_highpage(page);
- unlock_page(page);
- continue;
- } else if (PageTransHuge(page)) {
+ if (PageTransHuge(page)) {
+ /* Range starts in the middle of THP */
+ if (start > page->index) {
+ pgoff_t i;
+ index += HPAGE_PMD_NR;
+ page += start - page->index;
+ for (i = start; i < index; i++, page++)
+ clear_highpage(page);
+ unlock_page(page - 1);
+ continue;
+ }
+
+ /* Range ends in the middle of THP */
if (index == round_down(end, HPAGE_PMD_NR)) {
- /*
- * Range ends in the middle of THP:
- * zero out the page
- */
- clear_highpage(page);
+ while (index++ < end)
+ clear_highpage(page++);
unlock_page(page);
continue;
}
- index += HPAGE_PMD_NR - 1;
- i += HPAGE_PMD_NR - 1;
}
if (!unfalloc || !PageUptodate(page)) {
@@ -814,9 +800,9 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, loff_t lend,
unlock_page(page);
}
pagevec_remove_exceptionals(&pvec);
+ index += pvec.nr ? hpage_nr_pages(pvec.pages[pvec.nr - 1]) : 1;
pagevec_release(&pvec);
cond_resched();
- index++;
}
if (partial_start) {
@@ -874,8 +860,7 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, loff_t lend,
continue;
if (shmem_free_swap(mapping, index, page)) {
/* Swap was replaced by page: retry */
- index--;
- break;
+ goto retry;
}
nr_swaps_freed++;
continue;
@@ -883,30 +868,24 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, loff_t lend,
lock_page(page);
- if (PageTransTail(page)) {
- /* Middle of THP: zero out the page */
- clear_highpage(page);
- unlock_page(page);
- /*
- * Partial thp truncate due 'start' in middle
- * of THP: don't need to look on these pages
- * again on !pvec.nr restart.
- */
- if (index != round_down(end, HPAGE_PMD_NR))
- start++;
- continue;
- } else if (PageTransHuge(page)) {
+ if (PageTransHuge(page)) {
+ /* Range starts in the middle of THP */
+ if (start > page->index) {
+ index += HPAGE_PMD_NR;
+ page += start - page->index;
+ while (start++ < index)
+ clear_highpage(page++);
+ unlock_page(page - 1);
+ continue;
+ }
+
+ /* Range ends in the middle of THP */
if (index == round_down(end, HPAGE_PMD_NR)) {
- /*
- * Range ends in the middle of THP:
- * zero out the page
- */
- clear_highpage(page);
+ while (index++ < end)
+ clear_highpage(page++);
unlock_page(page);
continue;
}
- index += HPAGE_PMD_NR - 1;
- i += HPAGE_PMD_NR - 1;
}
if (!unfalloc || !PageUptodate(page)) {
@@ -917,15 +896,18 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, loff_t lend,
} else {
/* Page was replaced by swap: retry */
unlock_page(page);
- index--;
- break;
+ goto retry;
}
}
unlock_page(page);
}
pagevec_remove_exceptionals(&pvec);
+ index += pvec.nr ? hpage_nr_pages(pvec.pages[pvec.nr - 1]) : 1;
+ pagevec_release(&pvec);
+ continue;
+retry:
+ pagevec_remove_exceptionals(&pvec);
pagevec_release(&pvec);
- index++;
}
spin_lock_irq(&info->lock);
@@ -1762,8 +1744,7 @@ alloc_nohuge: page = shmem_alloc_and_acct_page(gfp, info, sbinfo,
PageTransHuge(page));
if (error)
goto unacct;
- error = radix_tree_maybe_preload_order(gfp & GFP_RECLAIM_MASK,
- compound_order(page));
+ error = radix_tree_maybe_preload(gfp & GFP_RECLAIM_MASK);
if (!error) {
error = shmem_add_to_page_cache(page, mapping, hindex,
NULL);
@@ -1837,7 +1818,7 @@ alloc_nohuge: page = shmem_alloc_and_acct_page(gfp, info, sbinfo,
error = -EINVAL;
goto unlock;
}
- *pagep = page + index - hindex;
+ *pagep = find_subpage(page, index);
return 0;
/*
diff --git a/mm/truncate.c b/mm/truncate.c
index fd97f1dbce29..eb3a3a45feb6 100644
--- a/mm/truncate.c
+++ b/mm/truncate.c
@@ -479,16 +479,13 @@ unsigned long invalidate_mapping_pages(struct address_space *mapping,
WARN_ON(page_to_index(page) != index);
- /* Middle of THP: skip */
- if (PageTransTail(page)) {
+ /* Is 'start' or 'end' in the middle of THP ? */
+ if (PageTransHuge(page) &&
+ (start > index ||
+ (index == round_down(end, HPAGE_PMD_NR)))) {
+ /* skip */
unlock_page(page);
continue;
- } else if (PageTransHuge(page)) {
- index += HPAGE_PMD_NR - 1;
- i += HPAGE_PMD_NR - 1;
- /* 'end' is in the middle of THP */
- if (index == round_down(end, HPAGE_PMD_NR))
- continue;
}
ret = invalidate_inode_page(page);
@@ -502,9 +499,9 @@ unsigned long invalidate_mapping_pages(struct address_space *mapping,
count += ret;
}
pagevec_remove_exceptionals(&pvec);
+ index += pvec.nr ? hpage_nr_pages(pvec.pages[pvec.nr - 1]) : 1;
pagevec_release(&pvec);
cond_resched();
- index++;
}
return count;
}
--
2.10.2
Slab pages can be compound, but we shouldn't threat them as THP for
pupose of hpage_* helpers, otherwise it would lead to confusing results.
For instance, ext4 uses slab pages for journal pages and we shouldn't
confuse them with THPs. The easiest way is to exclude them in hpage_*
helpers.
Signed-off-by: Kirill A. Shutemov <[email protected]>
---
include/linux/huge_mm.h | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h
index e5c9c26d2439..5e6c408f5b47 100644
--- a/include/linux/huge_mm.h
+++ b/include/linux/huge_mm.h
@@ -137,21 +137,21 @@ static inline spinlock_t *pmd_trans_huge_lock(pmd_t *pmd,
}
static inline int hpage_nr_pages(struct page *page)
{
- if (unlikely(PageTransHuge(page)))
+ if (unlikely(!PageSlab(page) && PageTransHuge(page)))
return HPAGE_PMD_NR;
return 1;
}
static inline int hpage_size(struct page *page)
{
- if (unlikely(PageTransHuge(page)))
+ if (unlikely(!PageSlab(page) && PageTransHuge(page)))
return HPAGE_PMD_SIZE;
return PAGE_SIZE;
}
static inline unsigned long hpage_mask(struct page *page)
{
- if (unlikely(PageTransHuge(page)))
+ if (unlikely(!PageSlab(page) && PageTransHuge(page)))
return HPAGE_PMD_MASK;
return PAGE_MASK;
}
--
2.10.2
Introduce new helpers which return size/mask of the page:
HPAGE_PMD_SIZE/HPAGE_PMD_MASK if the page is PageTransHuge() and
PAGE_SIZE/PAGE_MASK otherwise.
Signed-off-by: Kirill A. Shutemov <[email protected]>
---
include/linux/huge_mm.h | 16 ++++++++++++++++
1 file changed, 16 insertions(+)
diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h
index 97e478d6b690..e5c9c26d2439 100644
--- a/include/linux/huge_mm.h
+++ b/include/linux/huge_mm.h
@@ -142,6 +142,20 @@ static inline int hpage_nr_pages(struct page *page)
return 1;
}
+static inline int hpage_size(struct page *page)
+{
+ if (unlikely(PageTransHuge(page)))
+ return HPAGE_PMD_SIZE;
+ return PAGE_SIZE;
+}
+
+static inline unsigned long hpage_mask(struct page *page)
+{
+ if (unlikely(PageTransHuge(page)))
+ return HPAGE_PMD_MASK;
+ return PAGE_MASK;
+}
+
extern int do_huge_pmd_numa_page(struct vm_fault *vmf, pmd_t orig_pmd);
extern struct page *huge_zero_page;
@@ -167,6 +181,8 @@ void mm_put_huge_zero_page(struct mm_struct *mm);
#define HPAGE_PMD_SIZE ({ BUILD_BUG(); 0; })
#define hpage_nr_pages(x) 1
+#define hpage_size(x) PAGE_SIZE
+#define hpage_mask(x) PAGE_MASK
#define transparent_hugepage_enabled(__vma) 0
--
2.10.2
On Tuesday, November 29, 2016 7:23 PM Kirill A. Shutemov wrote:
> @@ -607,10 +605,10 @@ static long hugetlbfs_fallocate(struct file *file, int mode, loff_t offset,
> }
>
> /* Set numa allocation policy based on index */
> - hugetlb_set_vma_policy(&pseudo_vma, inode, index);
> + hugetlb_set_vma_policy(&pseudo_vma, inode, index >> huge_page_order(h));
>
> /* addr is the offset within the file (zero based) */
> - addr = index * hpage_size;
> + addr = index << PAGE_SHIFT & ~huge_page_mask(h);
>
> /* mutex taken here, fault path and hole punch */
> hash = hugetlb_fault_mutex_hash(h, mm, &pseudo_vma, mapping,
Seems we can't use index in computing hash as long as it isn't in huge page size.
On Wed, Nov 30, 2016 at 05:48:05PM +0800, Hillf Danton wrote:
> On Tuesday, November 29, 2016 7:23 PM Kirill A. Shutemov wrote:
> > @@ -607,10 +605,10 @@ static long hugetlbfs_fallocate(struct file *file, int mode, loff_t offset,
> > }
> >
> > /* Set numa allocation policy based on index */
> > - hugetlb_set_vma_policy(&pseudo_vma, inode, index);
> > + hugetlb_set_vma_policy(&pseudo_vma, inode, index >> huge_page_order(h));
> >
> > /* addr is the offset within the file (zero based) */
> > - addr = index * hpage_size;
> > + addr = index << PAGE_SHIFT & ~huge_page_mask(h);
> >
> > /* mutex taken here, fault path and hole punch */
> > hash = hugetlb_fault_mutex_hash(h, mm, &pseudo_vma, mapping,
>
> Seems we can't use index in computing hash as long as it isn't in huge page size.
Look at changes in hugetlb_fault_mutex_hash(): we shift the index right by
huge_page_order(), before calculating the hash. I don't see a problem
here.
--
Kirill A. Shutemov
On Wednesday, November 30, 2016 9:16 PM Kirill A. Shutemov wrote:
> On Wed, Nov 30, 2016 at 05:48:05PM +0800, Hillf Danton wrote:
> > On Tuesday, November 29, 2016 7:23 PM Kirill A. Shutemov wrote:
> > > @@ -607,10 +605,10 @@ static long hugetlbfs_fallocate(struct file *file, int mode, loff_t offset,
> > > }
> > >
> > > /* Set numa allocation policy based on index */
> > > - hugetlb_set_vma_policy(&pseudo_vma, inode, index);
> > > + hugetlb_set_vma_policy(&pseudo_vma, inode, index >> huge_page_order(h));
> > >
> > > /* addr is the offset within the file (zero based) */
> > > - addr = index * hpage_size;
> > > + addr = index << PAGE_SHIFT & ~huge_page_mask(h);
> > >
> > > /* mutex taken here, fault path and hole punch */
> > > hash = hugetlb_fault_mutex_hash(h, mm, &pseudo_vma, mapping,
> >
> > Seems we can't use index in computing hash as long as it isn't in huge page size.
>
> Look at changes in hugetlb_fault_mutex_hash(): we shift the index right by
> huge_page_order(), before calculating the hash. I don't see a problem
> here.
>
You are right. I missed that critical point.
thanks
Hillf