This patch series fixes unnecessarry resource consuming
in khugepaged swapin and introduces a new function to
calculate value of specific vm event.
Ebru Akagunduz (2):
mm, vmstat: calculate particular vm event
mm, thp: avoid unnecessary swapin in khugepaged
include/linux/vmstat.h | 6 ++++++
mm/huge_memory.c | 18 +++++++++++++++---
mm/vmstat.c | 12 ++++++++++++
3 files changed, 33 insertions(+), 3 deletions(-)
--
1.9.1
Currently, vmstat can calculate specific vm event with all_vm_events()
however it allocates all vm events to stack. This patch introduces
a helper to sum value of a specific vm event over all cpu, without
loading all the events.
Signed-off-by: Ebru Akagunduz <[email protected]>
Suggested-by: Kirill A. Shutemov <[email protected]>
Acked-by: Kirill A. Shutemov <[email protected]>
Reviewed-by: Rik van Riel <[email protected]>
Acked-by: Vlastimil Babka <[email protected]>
---
Changes in v2:
- this patch newly created in this version
- create sum event function to
calculate particular vm event (Kirill A. Shutemov)
Changes in v3:
- add dummy definition of sum_vm_event
when CONFIG_VM_EVENTS is not set
(Kirill A. Shutemov)
Changes in v4:
- add Suggested-by tag (Vlastimil Babka)
Changes in v5:
- CC'ed Christoph Lameter <[email protected]> (Andrew Morton)
include/linux/vmstat.h | 6 ++++++
mm/vmstat.c | 12 ++++++++++++
2 files changed, 18 insertions(+)
diff --git a/include/linux/vmstat.h b/include/linux/vmstat.h
index 73fae8c..e5ec287 100644
--- a/include/linux/vmstat.h
+++ b/include/linux/vmstat.h
@@ -53,6 +53,8 @@ static inline void count_vm_events(enum vm_event_item item, long delta)
extern void all_vm_events(unsigned long *);
+extern unsigned long sum_vm_event(enum vm_event_item item);
+
extern void vm_events_fold_cpu(int cpu);
#else
@@ -73,6 +75,10 @@ static inline void __count_vm_events(enum vm_event_item item, long delta)
static inline void all_vm_events(unsigned long *ret)
{
}
+static inline unsigned long sum_vm_event(enum vm_event_item item)
+{
+ return 0;
+}
static inline void vm_events_fold_cpu(int cpu)
{
}
diff --git a/mm/vmstat.c b/mm/vmstat.c
index 9e07d11..1d5d89e 100644
--- a/mm/vmstat.c
+++ b/mm/vmstat.c
@@ -34,6 +34,18 @@
DEFINE_PER_CPU(struct vm_event_state, vm_event_states) = {{0}};
EXPORT_PER_CPU_SYMBOL(vm_event_states);
+unsigned long sum_vm_event(enum vm_event_item item)
+{
+ int cpu;
+ unsigned long ret = 0;
+
+ get_online_cpus();
+ for_each_online_cpu(cpu)
+ ret += per_cpu(vm_event_states, cpu).event[item];
+ put_online_cpus();
+ return ret;
+}
+
static void sum_vm_events(unsigned long *ret)
{
int cpu;
--
1.9.1
Currently khugepaged makes swapin readahead to improve
THP collapse rate. This patch checks vm statistics
to avoid workload of swapin, if unnecessary. So that
when system under pressure, khugepaged won't consume
resources to swapin and won't trigger direct reclaim
when swapin readahead.
The patch was tested with a test program that allocates
800MB of memory, writes to it, and then sleeps. The system
was forced to swap out all. Afterwards, the test program
touches the area by writing, it skips a page in each
20 pages of the area. When waiting to swapin readahead
left part of the test, the memory forced to be busy
doing page reclaim. There was enough free memory during
test, khugepaged did not swapin readahead due to business.
Test results:
After swapped out
-------------------------------------------------------------------
| Anonymous | AnonHugePages | Swap | Fraction |
-------------------------------------------------------------------
With patch | 0 kB | 0 kB | 800000 kB | %100 |
-------------------------------------------------------------------
Without patch | 0 kB | 0 kB | 800000 kB | %100 |
-------------------------------------------------------------------
After swapped in
-------------------------------------------------------------------
| Anonymous | AnonHugePages | Swap | Fraction |
-------------------------------------------------------------------
With patch | 384812 kB | 96256 kB | 415188 kB | %25 |
-------------------------------------------------------------------
Without patch | 389728 kB | 194560 kB | 410272 kB | %49 |
-------------------------------------------------------------------
Signed-off-by: Ebru Akagunduz <[email protected]>
---
Changes in v2:
- Add reference to specify which patch fixed (Ebru Akagunduz)
- Fix commit subject line (Ebru Akagunduz)
Changes in v3:
- Remove default values of allocstall (Kirill A. Shutemov)
Changes in v4:
- define unsigned long allocstall instead of unsigned long int
(Vlastimil Babka)
- compare allocstall when khugepaged goes to sleep
(Rik van Riel, Vlastimil Babka)
Changes in v5:
- Drop fixes sha part because fixed patch is not in Linus's tree
(Michal Hocko)
- Save allocstall where khugepaged exactly sleeps (Michal Hocko)
Note: I didn't add optimistic swapin and mmap_sem in this
patch series. I couldn't overcome yet.
I'll send them after the series ends up.
mm/huge_memory.c | 18 +++++++++++++++---
1 file changed, 15 insertions(+), 3 deletions(-)
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 8202141..e7d905c 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -102,6 +102,7 @@ static DECLARE_WAIT_QUEUE_HEAD(khugepaged_wait);
*/
static unsigned int khugepaged_max_ptes_none __read_mostly;
static unsigned int khugepaged_max_ptes_swap __read_mostly = HPAGE_PMD_NR/8;
+static unsigned long allocstall;
static int khugepaged(void *none);
static int khugepaged_slab_init(void);
@@ -2437,7 +2438,7 @@ static void collapse_huge_page(struct mm_struct *mm,
struct page *new_page;
spinlock_t *pmd_ptl, *pte_ptl;
int isolated = 0, result = 0;
- unsigned long hstart, hend;
+ unsigned long hstart, hend, swap, curr_allocstall;
struct mem_cgroup *memcg;
unsigned long mmun_start; /* For mmu_notifiers */
unsigned long mmun_end; /* For mmu_notifiers */
@@ -2492,7 +2493,14 @@ static void collapse_huge_page(struct mm_struct *mm,
goto out;
}
- __collapse_huge_page_swapin(mm, vma, address, pmd);
+ swap = get_mm_counter(mm, MM_SWAPENTS);
+ curr_allocstall = sum_vm_event(ALLOCSTALL);
+ /*
+ * When system under pressure, don't swapin readahead.
+ * So that avoid unnecessary resource consuming.
+ */
+ if (allocstall == curr_allocstall && swap !=)
+ __collapse_huge_page_swapin(mm, vma, address, pmd);
anon_vma_lock_write(vma->anon_vma);
@@ -2886,14 +2894,17 @@ static void khugepaged_wait_work(void)
if (!khugepaged_scan_sleep_millisecs)
return;
+ allocstall = sum_vm_event(ALLOCSTALL);
wait_event_freezable_timeout(khugepaged_wait,
kthread_should_stop(),
msecs_to_jiffies(khugepaged_scan_sleep_millisecs));
return;
}
- if (khugepaged_enabled())
+ if (khugepaged_enabled()) {
+ allocstall = sum_vm_event(ALLOCSTALL);
wait_event_freezable(khugepaged_wait, khugepaged_wait_event());
+ }
}
static int khugepaged(void *none)
@@ -2902,6 +2913,7 @@ static int khugepaged(void *none)
set_freezable();
set_user_nice(current, MAX_NICE);
+ allocstall = sum_vm_event(ALLOCSTALL);
while (!kthread_should_stop()) {
khugepaged_do_scan();
--
1.9.1
On Thu, 2016-04-07 at 20:28 +0300, Ebru Akagunduz wrote:
> Currently khugepaged makes swapin readahead to improve
> THP collapse rate. This patch checks vm statistics
> to avoid workload of swapin, if unnecessary. So that
> when system under pressure, khugepaged won't consume
> resources to swapin and won't trigger direct reclaim
> when swapin readahead.
>
> The patch was tested with a test program that allocates
> 800MB of memory, writes to it, and then sleeps. The system
> was forced to swap out all. Afterwards, the test program
> touches the area by writing, it skips a page in each
> 20 pages of the area. When waiting to swapin readahead
> left part of the test, the memory forced to be busy
> doing page reclaim. There was enough free memory during
> test, khugepaged did not swapin readahead due to business.
>
> Test results:
>
> After swapped out
> -------------------------------------------------------------------
> | Anonymous | AnonHugePages | Swap | Fraction |
> -------------------------------------------------------------------
> With patch | 0 kB | 0 kB | 800000 kB | %100 |
> -------------------------------------------------------------------
> Without patch | 0 kB | 0 kB | 800000 kB | %100 |
> -------------------------------------------------------------------
>
> After swapped in
> -------------------------------------------------------------------
> | Anonymous | AnonHugePages | Swap | Fraction |
> -------------------------------------------------------------------
> With patch | 384812 kB | 96256 kB | 415188 kB | %25 |
> -------------------------------------------------------------------
> Without patch | 389728 kB | 194560 kB | 410272 kB | %49 |
> -------------------------------------------------------------------
>
> Signed-off-by: Ebru Akagunduz <[email protected]>
Acked-by: Rik van Riel <[email protected]>
--
All Rights Reversed.
Hi Ebru,
[auto build test ERROR on next-20160407]
[cannot apply to v4.6-rc2 v4.6-rc1 v4.5-rc7 v4.6-rc2]
[if your patch is applied to the wrong git tree, please drop us a note to help improving the system]
url: https://github.com/0day-ci/linux/commits/Ebru-Akagunduz/mm-thp-Fix-unnecessarry-resource-consuming-in-swapin/20160408-013104
config: x86_64-randconfig-x000-201614 (attached as .config)
reproduce:
# save the attached .config to linux build tree
make ARCH=x86_64
All errors (new ones prefixed by >>):
mm/huge_memory.c: In function 'collapse_huge_page':
>> mm/huge_memory.c:2516:46: error: expected expression before ')' token
if (allocstall == curr_allocstall && swap !=)
^
vim +2516 mm/huge_memory.c
2510 swap = get_mm_counter(mm, MM_SWAPENTS);
2511 curr_allocstall = sum_vm_event(ALLOCSTALL);
2512 /*
2513 * When system under pressure, don't swapin readahead.
2514 * So that avoid unnecessary resource consuming.
2515 */
> 2516 if (allocstall == curr_allocstall && swap !=)
2517 __collapse_huge_page_swapin(mm, vma, address, pmd);
2518
2519 anon_vma_lock_write(vma->anon_vma);
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
Hi Ebru,
[auto build test ERROR on next-20160407]
[cannot apply to v4.6-rc2 v4.6-rc1 v4.5-rc7 v4.6-rc2]
[if your patch is applied to the wrong git tree, please drop us a note to help improving the system]
url: https://github.com/0day-ci/linux/commits/Ebru-Akagunduz/mm-thp-Fix-unnecessarry-resource-consuming-in-swapin/20160408-013104
config: i386-randconfig-x016-201614 (attached as .config)
reproduce:
# save the attached .config to linux build tree
make ARCH=i386
All error/warnings (new ones prefixed by >>):
In file included from include/asm-generic/bug.h:4:0,
from arch/x86/include/asm/bug.h:35,
from include/linux/bug.h:4,
from include/linux/mmdebug.h:4,
from include/linux/mm.h:8,
from mm/huge_memory.c:10:
mm/huge_memory.c: In function 'collapse_huge_page':
>> include/linux/compiler.h:149:57: error: expected expression before ')' token
#define if(cond, ...) __trace_if( (cond , ## __VA_ARGS__) )
^
include/linux/compiler.h:151:30: note: in definition of macro '__trace_if'
if (__builtin_constant_p(!!(cond)) ? !!(cond) : \
^
>> mm/huge_memory.c:2516:2: note: in expansion of macro 'if'
if (allocstall == curr_allocstall && swap !=)
^
>> include/linux/compiler.h:149:57: error: expected expression before ')' token
#define if(cond, ...) __trace_if( (cond , ## __VA_ARGS__) )
^
include/linux/compiler.h:162:16: note: in definition of macro '__trace_if'
______r = !!(cond); \
^
>> mm/huge_memory.c:2516:2: note: in expansion of macro 'if'
if (allocstall == curr_allocstall && swap !=)
^
vim +/if +2516 mm/huge_memory.c
2500 if (!hugepage_vma_check(vma)) {
2501 result = SCAN_VMA_CHECK;
2502 goto out;
2503 }
2504 pmd = mm_find_pmd(mm, address);
2505 if (!pmd) {
2506 result = SCAN_PMD_NULL;
2507 goto out;
2508 }
2509
2510 swap = get_mm_counter(mm, MM_SWAPENTS);
2511 curr_allocstall = sum_vm_event(ALLOCSTALL);
2512 /*
2513 * When system under pressure, don't swapin readahead.
2514 * So that avoid unnecessary resource consuming.
2515 */
> 2516 if (allocstall == curr_allocstall && swap !=)
2517 __collapse_huge_page_swapin(mm, vma, address, pmd);
2518
2519 anon_vma_lock_write(vma->anon_vma);
2520
2521 pte = pte_offset_map(pmd, address);
2522 pte_ptl = pte_lockptr(mm, pmd);
2523
2524 mmun_start = address;
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
On Thu, Apr 07, 2016 at 08:28:01PM +0300, Ebru Akagunduz wrote:
...
> + swap = get_mm_counter(mm, MM_SWAPENTS);
> + curr_allocstall = sum_vm_event(ALLOCSTALL);
> + /*
> + * When system under pressure, don't swapin readahead.
> + * So that avoid unnecessary resource consuming.
> + */
> + if (allocstall == curr_allocstall && swap !=)
> + __collapse_huge_page_swapin(mm, vma, address, pmd);
This !=) looks like someone got fun ;)
On Thu, 2016-04-07 at 21:58 +0300, Cyrill Gorcunov wrote:
> On Thu, Apr 07, 2016 at 08:28:01PM +0300, Ebru Akagunduz wrote:
> ...
> >
> > + swap = get_mm_counter(mm, MM_SWAPENTS);
> > + curr_allocstall = sum_vm_event(ALLOCSTALL);
> > + /*
> > + * When system under pressure, don't swapin readahead.
> > + * So that avoid unnecessary resource consuming.
> > + */
> > + if (allocstall == curr_allocstall && swap !=)
> > + __collapse_huge_page_swapin(mm, vma, address,
> > pmd);
> This !=) looks like someone got fun ;)
Looks like someone sent out emails before refreshing the
patch, which is a such an easy mistake to make I must have
done it a dozen times by now :)
--
All Rights Reversed.
On Thu, Apr 07, 2016 at 03:39:05PM -0400, Rik van Riel wrote:
> > This !=) looks like someone got fun ;)
>
> Looks like someone sent out emails before refreshing the
> patch, which is a such an easy mistake to make I must have
> done it a dozen times by now :)
I've been there many times as well :)
On Thu, Apr 07, 2016 at 10:47:59PM +0300, Cyrill Gorcunov wrote:
> On Thu, Apr 07, 2016 at 03:39:05PM -0400, Rik van Riel wrote:
> > > This !=) looks like someone got fun ;)
> >
> > Looks like someone sent out emails before refreshing the
> > patch, which is a such an easy mistake to make I must have
> > done it a dozen times by now :)
>
> I've been there many times as well :)
I apologize for inconvenience. When making
last checks on this patch, this happened and
I wasn't aware of it. I'll fix this, test and
send in next version.
On Thu, 7 Apr 2016, Ebru Akagunduz wrote:
> Currently, vmstat can calculate specific vm event with all_vm_events()
> however it allocates all vm events to stack. This patch introduces
> a helper to sum value of a specific vm event over all cpu, without
> loading all the events.
The first sentence is inaccurate. all_vm_events() takes a pointer to an
array of of counters and does not allocate on the stack. Fix this and
then add
Acked-by: Christoph Lameter <[email protected]>