LinuxLists.cc - mmotm 2010-04-15-14-42 uploaded

[permalink] [raw]

Subject: error at compaction (Re: mmotm 2010-04-15-14-42 uploaded

mmotm 2010-04-15-14-42

When I tried
# echo 0 > /proc/sys/vm/compaction

I see following.

My enviroment was
2.6.34-rc4-mm1+ (2010-04-15-14-42) (x86-64) CPUx8
allocating tons of hugepages and reduce free memory.

What I did was:
# echo 0 > /proc/sys/vm/compact_memory

Hmm, I see this kind of error at migation for the 1st time..
my.config is attached. Hmm... ?

(I'm sorry I'll be offline soon.)
-Kame
==

Apr 19 18:55:04 localhost kernel: BUG: unable to handle kernel paging request at ffff8806213ff000
Apr 19 18:55:04 localhost kernel: IP: [<ffffffff812ae3a5>] copy_page_c+0x5/0x10
Apr 19 18:55:04 localhost kernel: PGD 1a43063 PUD 50d5067 PMD 51df067 PTE 80000006213ff160
Apr 19 18:55:04 localhost kernel: Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Apr 19 18:55:04 localhost kernel: last sysfs file: /sys/devices/pci0000:00/0000:00:1d.3/usb5/devnum
Apr 19 18:55:04 localhost kernel: CPU 1
Apr 19 18:55:04 localhost kernel: Modules linked in: sit tunnel4 ipt_MASQUERADE iptable_nat nf_nat
bridge stp llc sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf xt_physdev ip6t_REJECT nf_conn
track_ipv6 ip6table_filter ip6_tables ipv6 uinput e1000e bnx2 shpchp i5000_edac edac_core i2c_i801
i2c_core ppdev i5k_amb parport_pc parport ioatdma dca iTCO_wdt iTCO_vendor_support pcspkr kvm_intel
kvm dm_multipath megaraid_sas [last unloaded: microcode]
Apr 19 18:55:04 localhost kernel:
Apr 19 18:55:04 localhost kernel: Pid: 2427, comm: bash Tainted: G W 2.6.34-rc4-mm1+ #1 D2
519/PRIMERGY
Apr 19 18:55:04 localhost kernel: RIP: 0010:[<ffffffff812ae3a5>] [<ffffffff812ae3a5>] copy_page_c+
0x5/0x10
Apr 19 18:55:04 localhost kernel: RSP: 0018:ffff88061c025b70 EFLAGS: 00010286
Apr 19 18:55:04 localhost kernel: RAX: ffff880000000000 RBX: ffffea0003801180 RCX: 0000000000000200
Apr 19 18:55:04 localhost kernel: RDX: 6db6db6db6db6db7 RSI: ffff880100050000 RDI: ffff8806213ff000
Apr 19 18:55:04 localhost kernel: RBP: ffff88061c025b98 R08: 0000000000000048 R09: 0000000000000001
Apr 19 18:55:04 localhost kernel: R10: 0000000000000000 R11: 0000000000000001 R12: ffffea0015745fc8
Apr 19 18:55:04 localhost kernel: R13: ffff88061c024000 R14: ffffea00038011a8 R15: 0000000000000000
Apr 19 18:55:04 localhost kernel: FS: 00007f634035f700(0000) GS:ffff880005600000(0000) knlGS:00000
00000000000
Apr 19 18:55:04 localhost kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Apr 19 18:55:04 localhost kernel: CR2: ffff8806213ff000 CR3: 0000000612a15000 CR4: 00000000000006e0
Apr 19 18:55:04 localhost kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Apr 19 18:55:04 localhost kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Apr 19 18:55:04 localhost kernel: Process bash (pid: 2427, threadinfo ffff88061c024000, task ffff880611422410)
Apr 19 18:55:04 localhost kernel: Stack:
Apr 19 18:55:04 localhost kernel: ffffffff8114ea2b ffffea0015745fc8 ffffea0003801180 ffffea0015745fc8
Apr 19 18:55:04 localhost kernel: <0> ffffea0003801180 ffff88061c025bc8 ffffffff8114ee75 0000000000000061
Apr 19 18:55:04 localhost kernel: <0> ffffc90000000000 ffffea0003801180 ffffea0015745fc8 ffff88061c025c08
Apr 19 18:55:04 localhost kernel: Call Trace:
Apr 19 18:55:04 localhost kernel: [<ffffffff8114ea2b>] ? migrate_page_copy+0x7b/0x200
Apr 19 18:55:04 localhost kernel: [<ffffffff8114ee75>] migrate_page+0x35/0x50
Apr 19 18:55:04 localhost kernel: [<ffffffff8114fe85>] buffer_migrate_page+0x125/0x150
Apr 19 18:55:04 localhost kernel: [<ffffffff8114f3f2>] migrate_pages+0x562/0x7d0
Apr 19 18:55:04 localhost kernel: [<ffffffff81111500>] ? lru_add_drain_per_cpu+0x0/0x10
Apr 19 18:55:04 localhost kernel: [<ffffffff81142630>] ? compaction_alloc+0x0/0x370
Apr 19 18:55:04 localhost kernel: [<ffffffff8107cc90>] ? schedule_on_each_cpu+0x150/0x180
Apr 19 18:55:04 localhost kernel: [<ffffffff81141f20>] ? compact_zone+0x270/0x530
Apr 19 18:55:04 localhost kernel: [<ffffffff811420ec>] compact_zone+0x43c/0x530
Apr 19 18:55:04 localhost kernel: [<ffffffff811422e9>] compact_node+0x109/0x140
Apr 19 18:55:04 localhost kernel: [<ffffffff811423cc>] sysctl_compaction_handler+0x5c/0x90
Apr 19 18:55:04 localhost kernel: [<ffffffff811c28f7>] proc_sys_call_handler+0x97/0xd0
Apr 19 18:55:04 localhost kernel: [<ffffffff811c2944>] proc_sys_write+0x14/0x20
Apr 19 18:55:04 localhost kernel: [<ffffffff8115c448>] vfs_write+0xc8/0x190
Apr 19 18:55:04 localhost kernel: [<ffffffff8115ce41>] sys_write+0x51/0x90
Apr 19 18:55:04 localhost kernel: [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
Apr 19 18:55:04 localhost kernel: Code: 0f 1f 84 00 00 00 00 00 0f 1f 84 00 00 00 00 00 0f 1f 84 00 00 00 00 00 0f 1f 80 00 00 00 00 90 90 90 90 90 90 90 b9 00 02 00 00 <f3> 48 a5 c3 0f 1f 80 00 00 00 00 eb ee 0f 1f 84 00 00 00 00 00
Apr 19 18:55:04 localhost kernel: RIP [<ffffffff812ae3a5>] copy_page_c+0x5/0x10
Apr 19 18:55:04 localhost kernel: RSP <ffff88061c025b70>
Apr 19 18:55:04 localhost kernel: CR2: ffff8806213ff000
Apr 19 18:55:04 localhost kernel: ---[ end trace a512635642ce2994 ]---

Attachments:

myconfig (94.97 kB)

2010-04-19 18:15:06

[permalink] [raw]

Subject: Re: error at compaction (Re: mmotm 2010-04-15-14-42 uploaded

On Mon, Apr 19, 2010 at 07:01:33PM +0900, KAMEZAWA Hiroyuki wrote:
>
> mmotm 2010-04-15-14-42
>
> When I tried
> # echo 0 > /proc/sys/vm/compaction
>
> I see following.
>
> My enviroment was
> 2.6.34-rc4-mm1+ (2010-04-15-14-42) (x86-64) CPUx8
> allocating tons of hugepages and reduce free memory.
>
> What I did was:
> # echo 0 > /proc/sys/vm/compact_memory
>
> Hmm, I see this kind of error at migation for the 1st time..
> my.config is attached. Hmm... ?
>
> (I'm sorry I'll be offline soon.)

That's ok, thanks you for the report. I'm afraid I made little progress
as I spent most of the day on other bugs but I do have something for
you.

First, I reproduced the problem using your .config. However, the problem does
not manifest with the .config I normally use which is derived from the distro
kernel configuration (Debian Lenny). So, there is something in your .config
that triggers the problem. I very strongly suspect this is an interaction
between migration, compaction and page allocation debug. Compaction takes
pages directly off the buddy list and I bet you a shiny penny they are still
unmapped when the copy takes place resulting in your oops.

I'll verify the theory tomorrow but it's a plausible explanation. On a
different note, where did config options like the following come out of?

CONFIG_ARCH_HWEIGHT_CFLAGS="-fcall-saved-rdi -fcall-saved-rsi -fcall-saved-rdx -fcall-saved-rcx -fcall-saved-r8 -fcall-saved-r9 -fcall-saved-r10 -fcall-saved-r11"

I don't think they are a factor but I'm curious.

--
Mel Gorman
Part-time Phd Student Linux Technology Center
University of Limerick IBM Dublin Software Lab

2010-04-19 19:39:40

[permalink] [raw]

Subject: Re: error at compaction (Re: mmotm 2010-04-15-14-42 uploaded

On Mon, Apr 19, 2010 at 07:14:42PM +0100, Mel Gorman wrote:
> On Mon, Apr 19, 2010 at 07:01:33PM +0900, KAMEZAWA Hiroyuki wrote:
> >
> > mmotm 2010-04-15-14-42
> >
> > When I tried
> > # echo 0 > /proc/sys/vm/compaction
> >
> > I see following.
> >
> > My enviroment was
> > 2.6.34-rc4-mm1+ (2010-04-15-14-42) (x86-64) CPUx8
> > allocating tons of hugepages and reduce free memory.
> >
> > What I did was:
> > # echo 0 > /proc/sys/vm/compact_memory
> >
> > Hmm, I see this kind of error at migation for the 1st time..
> > my.config is attached. Hmm... ?
> >
> > (I'm sorry I'll be offline soon.)
>
> That's ok, thanks you for the report. I'm afraid I made little progress
> as I spent most of the day on other bugs but I do have something for
> you.
>
> First, I reproduced the problem using your .config. However, the problem does
> not manifest with the .config I normally use which is derived from the distro
> kernel configuration (Debian Lenny). So, there is something in your .config
> that triggers the problem. I very strongly suspect this is an interaction
> between migration, compaction and page allocation debug.

I unexpecedly had the time to dig into this. Does the following patch fix
your problem? It Worked For Me.

==== CUT HERE ====
mm,compaction: Map free pages in the address space after they get split for compaction

split_free_page() is a helper function which takes a free page from the
buddy lists and splits it into order-0 pages. It is used by memory
compaction to build a list of destination pages. If
CONFIG_DEBUG_PAGEALLOC is set, a kernel paging request bug is triggered
because split_free_page() did not call the arch-allocation hooks or map
the page into the kernel address space.

This patch does not update split_free_page() as it is called with
interrupts held. Instead it documents that callers of split_free_page()
are responsible for calling the arch hooks and to map the page and fixes
compaction.

This is a fix to the patch mm-compaction-memory-compaction-core.patch.

Signed-off-by: Mel Gorman <[email protected]>
---
mm/compaction.c | 6 ++++++
mm/page_alloc.c | 3 +++
2 files changed, 9 insertions(+), 0 deletions(-)

diff --git a/mm/compaction.c b/mm/compaction.c
index 8f4c518..6218e03 100644
--- a/mm/compaction.c
+++ b/mm/compaction.c
@@ -184,6 +184,12 @@ static void isolate_freepages(struct zone *zone,
}
spin_unlock_irqrestore(&zone->lock, flags);

+ /* split_free_page does not map the pages */
+ list_for_each_entry(page, freelist, lru) {
+ arch_alloc_page(page, 0);
+ kernel_map_pages(page, 1, 1);
+ }
+
cc->free_pfn = high_pfn;
cc->nr_freepages = nr_freepages;
}
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 53442fd..b2af4d9 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -1211,6 +1211,9 @@ void split_page(struct page *page, unsigned int order)
/*
* Similar to split_page except the page is already free. As this is only
* being used for migration, the migratetype of the block also changes.
+ * As this is called with interrupts disabled, the caller is responsible
+ * for calling arch_alloc_page() and kernel_map_page() after interrupts
+ * are enabled.
*/
int split_free_page(struct page *page)
{

2010-04-20 02:34:29

[permalink] [raw]

Subject: Re: error at compaction (Re: mmotm 2010-04-15-14-42 uploaded

On Mon, 19 Apr 2010 20:39:19 +0100
Mel Gorman <[email protected]> wrote:

> On Mon, Apr 19, 2010 at 07:14:42PM +0100, Mel Gorman wrote:
> > On Mon, Apr 19, 2010 at 07:01:33PM +0900, KAMEZAWA Hiroyuki wrote:
> > >
> > > mmotm 2010-04-15-14-42
> > >
> > > When I tried
> > > # echo 0 > /proc/sys/vm/compaction
> > >
> > > I see following.
> > >
> > > My enviroment was
> > > 2.6.34-rc4-mm1+ (2010-04-15-14-42) (x86-64) CPUx8
> > > allocating tons of hugepages and reduce free memory.
> > >
> > > What I did was:
> > > # echo 0 > /proc/sys/vm/compact_memory
> > >
> > > Hmm, I see this kind of error at migation for the 1st time..
> > > my.config is attached. Hmm... ?
> > >
> > > (I'm sorry I'll be offline soon.)
> >
> > That's ok, thanks you for the report. I'm afraid I made little progress
> > as I spent most of the day on other bugs but I do have something for
> > you.
> >
> > First, I reproduced the problem using your .config. However, the problem does
> > not manifest with the .config I normally use which is derived from the distro
> > kernel configuration (Debian Lenny). So, there is something in your .config
> > that triggers the problem. I very strongly suspect this is an interaction
> > between migration, compaction and page allocation debug.
>
> I unexpecedly had the time to dig into this. Does the following patch fix
> your problem? It Worked For Me.
>
Ok, works for me, too.

Tested-by: KAMEZAWA Hiroyuki <[email protected]>

Thank you.
-Kame
> ==== CUT HERE ====
> mm,compaction: Map free pages in the address space after they get split for compaction
>
> split_free_page() is a helper function which takes a free page from the
> buddy lists and splits it into order-0 pages. It is used by memory
> compaction to build a list of destination pages. If
> CONFIG_DEBUG_PAGEALLOC is set, a kernel paging request bug is triggered
> because split_free_page() did not call the arch-allocation hooks or map
> the page into the kernel address space.
>
> This patch does not update split_free_page() as it is called with
> interrupts held. Instead it documents that callers of split_free_page()
> are responsible for calling the arch hooks and to map the page and fixes
> compaction.
>
> This is a fix to the patch mm-compaction-memory-compaction-core.patch.
>
> Signed-off-by: Mel Gorman <[email protected]>
> ---
> mm/compaction.c | 6 ++++++
> mm/page_alloc.c | 3 +++
> 2 files changed, 9 insertions(+), 0 deletions(-)
>
> diff --git a/mm/compaction.c b/mm/compaction.c
> index 8f4c518..6218e03 100644
> --- a/mm/compaction.c
> +++ b/mm/compaction.c
> @@ -184,6 +184,12 @@ static void isolate_freepages(struct zone *zone,
> }
> spin_unlock_irqrestore(&zone->lock, flags);
>
> + /* split_free_page does not map the pages */
> + list_for_each_entry(page, freelist, lru) {
> + arch_alloc_page(page, 0);
> + kernel_map_pages(page, 1, 1);
> + }
> +
> cc->free_pfn = high_pfn;
> cc->nr_freepages = nr_freepages;
> }
> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> index 53442fd..b2af4d9 100644
> --- a/mm/page_alloc.c
> +++ b/mm/page_alloc.c
> @@ -1211,6 +1211,9 @@ void split_page(struct page *page, unsigned int order)
> /*
> * Similar to split_page except the page is already free. As this is only
> * being used for migration, the migratetype of the block also changes.
> + * As this is called with interrupts disabled, the caller is responsible
> + * for calling arch_alloc_page() and kernel_map_page() after interrupts
> + * are enabled.
> */
> int split_free_page(struct page *page)
> {
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
>

2010-04-20 02:39:38

[permalink] [raw]

Subject: Re: error at compaction (Re: mmotm 2010-04-15-14-42 uploaded

On Mon, 19 Apr 2010 19:14:42 +0100
Mel Gorman <[email protected]> wrote:

> On Mon, Apr 19, 2010 at 07:01:33PM +0900, KAMEZAWA Hiroyuki wrote:

> I'll verify the theory tomorrow but it's a plausible explanation. On a
> different note, where did config options like the following come out of?
>
> CONFIG_ARCH_HWEIGHT_CFLAGS="-fcall-saved-rdi -fcall-saved-rsi -fcall-saved-rdx -fcall-saved-rcx -fcall-saved-r8 -fcall-saved-r9 -fcall-saved-r10 -fcall-saved-r11"
>
> I don't think they are a factor but I'm curious.
>

Hmm ? arch/x86/Kconfig.

config ARCH_HWEIGHT_CFLAGS
string
default "-fcall-saved-ecx -fcall-saved-edx" if X86_32
default "-fcall-saved-rdi -fcall-saved-rsi -fcall-saved-rdx -fcall-saved-rcx -fcall-saved-r8 -fcall-saved-r9 -fcall-saved-r10 -fcall-saved-r11" if X86_64

Seems to be from
patches/x86-add-optimized-popcnt-variants.patch

Thanks,
-Kame

2010-04-20 02:39:49

[permalink] [raw]

Subject: Re: error at compaction (Re: mmotm 2010-04-15-14-42 uploaded

On Tue, Apr 20, 2010 at 4:39 AM, Mel Gorman <[email protected]> wrote:
> On Mon, Apr 19, 2010 at 07:14:42PM +0100, Mel Gorman wrote:
>> On Mon, Apr 19, 2010 at 07:01:33PM +0900, KAMEZAWA Hiroyuki wrote:
>> >
>> > mmotm 2010-04-15-14-42
>> >
>> > When I tried
>> > # echo 0 > /proc/sys/vm/compaction
>> >
>> > I see following.
>> >
>> > My enviroment was
>> > 2.6.34-rc4-mm1+ (2010-04-15-14-42) (x86-64) CPUx8
>> > allocating tons of hugepages and reduce free memory.
>> >
>> > What I did was:
>> > # echo 0 > /proc/sys/vm/compact_memory
>> >
>> > Hmm, I see this kind of error at migation for the 1st time..
>> > my.config is attached. Hmm... ?
>> >
>> > (I'm sorry I'll be offline soon.)
>>
>> That's ok, thanks you for the report. I'm afraid I made little progress
>> as I spent most of the day on other bugs but I do have something for
>> you.
>>
>> First, I reproduced the problem using your .config. However, the problem does
>> not manifest with the .config I normally use which is derived from the distro
>> kernel configuration (Debian Lenny). So, there is something in your .config
>> that triggers the problem. I very strongly suspect this is an interaction
>> between migration, compaction and page allocation debug.
>
> I unexpecedly had the time to dig into this. Does the following patch fix
> your problem? It Worked For Me.

Nice catch during shot time. Below is comment.

>
> ==== CUT HERE ====
> mm,compaction: Map free pages in the address space after they get split for compaction
>
> split_free_page() is a helper function which takes a free page from the
> buddy lists and splits it into order-0 pages. It is used by memory
> compaction to build a list of destination pages. If
> CONFIG_DEBUG_PAGEALLOC is set, a kernel paging request bug is triggered
> because split_free_page() did not call the arch-allocation hooks or map
> the page into the kernel address space.
>
> This patch does not update split_free_page() as it is called with
> interrupts held. Instead it documents that callers of split_free_page()
> are responsible for calling the arch hooks and to map the page and fixes
> compaction.

Dumb question. Why can't we call arch_alloc_page and kernel_map_pages
as interrupt disabled? It's deadlock issue or latency issue?
I don't found any comment about it.
It should have added the comment around that functions. :)

And now compaction only uses split_free_page and it is exposed by mm.h.
I think it would be better to map pages inside split_free_page to
export others.(ie, making generic function).
If we can't do, how about making split_free_page static as static function?
And only uses it in compaction.

--
Kind regards,
Minchan Kim

2010-04-20 03:11:50

[permalink] [raw]

Subject: Re: error at compaction (Re: mmotm 2010-04-15-14-42 uploaded

On Tue, 20 Apr 2010 11:39:46 +0900
Minchan Kim <[email protected]> wrote:

> On Tue, Apr 20, 2010 at 4:39 AM, Mel Gorman <[email protected]> wrote:
> > On Mon, Apr 19, 2010 at 07:14:42PM +0100, Mel Gorman wrote:
> >> On Mon, Apr 19, 2010 at 07:01:33PM +0900, KAMEZAWA Hiroyuki wrote:
> >> >
> >> > mmotm 2010-04-15-14-42
> >> >
> >> > When I tried
> >> > # echo 0 > /proc/sys/vm/compaction
> >> >
> >> > I see following.
> >> >
> >> > My enviroment was
> >> > 2.6.34-rc4-mm1+ (2010-04-15-14-42) (x86-64) CPUx8
> >> > allocating tons of hugepages and reduce free memory.
> >> >
> >> > What I did was:
> >> > # echo 0 > /proc/sys/vm/compact_memory
> >> >
> >> > Hmm, I see this kind of error at migation for the 1st time..
> >> > my.config is attached. Hmm... ?
> >> >
> >> > (I'm sorry I'll be offline soon.)
> >>
> >> That's ok, thanks you for the report. I'm afraid I made little progress
> >> as I spent most of the day on other bugs but I do have something for
> >> you.
> >>
> >> First, I reproduced the problem using your .config. However, the problem does
> >> not manifest with the .config I normally use which is derived from the distro
> >> kernel configuration (Debian Lenny). So, there is something in your .config
> >> that triggers the problem. I very strongly suspect this is an interaction
> >> between migration, compaction and page allocation debug.
> >
> > I unexpecedly had the time to dig into this. Does the following patch fix
> > your problem? It Worked For Me.
>
> Nice catch during shot time. Below is comment.
>
> >
> > ==== CUT HERE ====
> > mm,compaction: Map free pages in the address space after they get split for compaction
> >
> > split_free_page() is a helper function which takes a free page from the
> > buddy lists and splits it into order-0 pages. It is used by memory
> > compaction to build a list of destination pages. If
> > CONFIG_DEBUG_PAGEALLOC is set, a kernel paging request bug is triggered
> > because split_free_page() did not call the arch-allocation hooks or map
> > the page into the kernel address space.
> >
> > This patch does not update split_free_page() as it is called with
> > interrupts held. Instead it documents that callers of split_free_page()
> > are responsible for calling the arch hooks and to map the page and fixes
> > compaction.
>
> Dumb question. Why can't we call arch_alloc_page and kernel_map_pages
> as interrupt disabled? It's deadlock issue or latency issue?
> I don't found any comment about it.
> It should have added the comment around that functions. :)
>

I guess it's from the same reason as vfree(), which can't be called under
irq-disabled.

Both of them has to flush TLB of all cpus. At flushing TLB (of other cpus), cpus has
to send IPI via smp_call_function. What I know from old stories is below.

At sendinf IPI, usual sequence is following. (This may be old.)

spin_lock(&ipi_lock);
set up cpu mask for getting notification from other cpu for declearing
"I received IPI and finished my own work".
spin_unlock(&ipi_lock);

Then,
CPU0 CPU1

irq_disable (somewhere) spin_lock
send IPI and wait for notification.
spin_lock()

deadlock. Seeing decription of kernel/smp.c::smp_call_function_many(), it says
this function should not be called under irq-disabled.
(Maybe the same kind of spin-wait deadlock can happen.)

Thanks,
-Kame

2010-04-20 03:58:45

[permalink] [raw]

Subject: Re: error at compaction (Re: mmotm 2010-04-15-14-42 uploaded

On Tue, Apr 20, 2010 at 12:07 PM, KAMEZAWA Hiroyuki
<[email protected]> wrote:
> On Tue, 20 Apr 2010 11:39:46 +0900
> Minchan Kim <[email protected]> wrote:
>
>> On Tue, Apr 20, 2010 at 4:39 AM, Mel Gorman <[email protected]> wrote:
>> > On Mon, Apr 19, 2010 at 07:14:42PM +0100, Mel Gorman wrote:
>> >> On Mon, Apr 19, 2010 at 07:01:33PM +0900, KAMEZAWA Hiroyuki wrote:
>> >> >
>> >> > mmotm 2010-04-15-14-42
>> >> >
>> >> > When I tried
>> >> > # echo 0 > /proc/sys/vm/compaction
>> >> >
>> >> > I see following.
>> >> >
>> >> > My enviroment was
>> >> > 2.6.34-rc4-mm1+ (2010-04-15-14-42) (x86-64) CPUx8
>> >> > allocating tons of hugepages and reduce free memory.
>> >> >
>> >> > What I did was:
>> >> > # echo 0 > /proc/sys/vm/compact_memory
>> >> >
>> >> > Hmm, I see this kind of error at migation for the 1st time..
>> >> > my.config is attached. Hmm... ?
>> >> >
>> >> > (I'm sorry I'll be offline soon.)
>> >>
>> >> That's ok, thanks you for the report. I'm afraid I made little progress
>> >> as I spent most of the day on other bugs but I do have something for
>> >> you.
>> >>
>> >> First, I reproduced the problem using your .config. However, the problem does
>> >> not manifest with the .config I normally use which is derived from the distro
>> >> kernel configuration (Debian Lenny). So, there is something in your .config
>> >> that triggers the problem. I very strongly suspect this is an interaction
>> >> between migration, compaction and page allocation debug.
>> >
>> > I unexpecedly had the time to dig into this. Does the following patch fix
>> > your problem? It Worked For Me.
>>
>> Nice catch during shot time. Below is comment.
>>
>> >
>> > ==== CUT HERE ====
>> > mm,compaction: Map free pages in the address space after they get split for compaction
>> >
>> > split_free_page() is a helper function which takes a free page from the
>> > buddy lists and splits it into order-0 pages. It is used by memory
>> > compaction to build a list of destination pages. If
>> > CONFIG_DEBUG_PAGEALLOC is set, a kernel paging request bug is triggered
>> > because split_free_page() did not call the arch-allocation hooks or map
>> > the page into the kernel address space.
>> >
>> > This patch does not update split_free_page() as it is called with
>> > interrupts held. Instead it documents that callers of split_free_page()
>> > are responsible for calling the arch hooks and to map the page and fixes
>> > compaction.
>>
>> Dumb question. Why can't we call arch_alloc_page and kernel_map_pages
>> as interrupt disabled? It's deadlock issue or latency issue?
>> I don't found any comment about it.
>> It should have added the comment around that functions. :)
>>
>
> I guess it's from the same reason as vfree(), which can't be called under
> irq-disabled.
>
> Both of them has to flush TLB of all cpus. At flushing TLB (of other cpus), cpus has
> to send IPI via smp_call_function. What I know from old stories is below.
>
> At sendinf IPI, usual sequence is following. (This may be old.)
>
> spin_lock(&ipi_lock);
> set up cpu mask for getting notification from other cpu for declearing
> "I received IPI and finished my own work".
> spin_unlock(&ipi_lock);
>
> Then,
> CPU0 CPU1
>
> irq_disable (somewhere) spin_lock
> send IPI and wait for notification.
> spin_lock()
>
> deadlock. Seeing decription of kernel/smp.c::smp_call_function_many(), it says
> this function should not be called under irq-disabled.
> (Maybe the same kind of spin-wait deadlock can happen.)
>

Thanks for kind explanation.
Actually I guessed TLB issue but I can't find any glue point which
connect tlb flush to smp_call_function_xxx. :(

Now look at the __native_flush_tlb_global.
It just read and write cr4 with just mask off X86_CR4_PGE.
So i don't know how connect this and smp_schedule_xxxx.
Hmm,, maybe APIC?

Sorry for dumb question.

--
Kind regards,
Minchan Kim

2010-04-20 04:28:33

[permalink] [raw]

Subject: Re: error at compaction (Re: mmotm 2010-04-15-14-42 uploaded

On Tue, 20 Apr 2010 12:58:43 +0900
Minchan Kim <[email protected]> wrote:

> On Tue, Apr 20, 2010 at 12:07 PM, KAMEZAWA Hiroyuki
> <[email protected]> wrote:
> > On Tue, 20 Apr 2010 11:39:46 +0900
> >> Dumb question. Why can't we call arch_alloc_page and kernel_map_pages
> >> as interrupt disabled? It's deadlock issue or latency issue?
> >> I don't found any comment about it.
> >> It should have added the comment around that functions. :)
> >>
> >
> > I guess it's from the same reason as vfree(), which can't be called under
> > irq-disabled.
> >
> > Both of them has to flush TLB of all cpus. At flushing TLB (of other cpus), cpus has
> > to send IPI via smp_call_function. What I know from old stories is below.
> >
> > At sendinf IPI, usual sequence is following. (This may be old.)
> >
> > spin_lock(&ipi_lock);
> > set up cpu mask for getting notification from other cpu for declearing
> > "I received IPI and finished my own work".
> > spin_unlock(&ipi_lock);
> >
> > Then,
> > CPU0 CPU1
> >
> > irq_disable (somewhere) spin_lock
> > send IPI and wait for notification.
> > spin_lock()
> >
> > deadlock. Seeing decription of kernel/smp.c::smp_call_function_many(), it says
> > this function should not be called under irq-disabled.
> > (Maybe the same kind of spin-wait deadlock can happen.)
> >
>
> Thanks for kind explanation.
> Actually I guessed TLB issue but I can't find any glue point which
> connect tlb flush to smp_call_function_xxx. :(
>
> Now look at the __native_flush_tlb_global.
> It just read and write cr4 with just mask off X86_CR4_PGE.
> So i don't know how connect this and smp_schedule_xxxx.
> Hmm,, maybe APIC?
>
> Sorry for dumb question.
>
Hmm...seeing again,

arch/x86/mm/pageattr.c::kernel_map_pages() says:

1293 /*
1294 * We should perform an IPI and flush all tlbs,
1295 * but that can deadlock->flush only current cpu:
1296 */

Wow. It just flush only local cpu. Then, no IPI.

Hmm...all other archs does the same thing ? If so, kernel_map_pages()
can be called under irq_disabled. The author of kernel_map_pages()
is aware that this can be called under irq-disabled.

Hmm...

Thanks,
-Kame

2010-04-20 08:21:23

[permalink] [raw]

Subject: Re: error at compaction (Re: mmotm 2010-04-15-14-42 uploaded

On Tue, Apr 20, 2010 at 11:39:46AM +0900, Minchan Kim wrote:
> On Tue, Apr 20, 2010 at 4:39 AM, Mel Gorman <[email protected]> wrote:
> > On Mon, Apr 19, 2010 at 07:14:42PM +0100, Mel Gorman wrote:
> >> On Mon, Apr 19, 2010 at 07:01:33PM +0900, KAMEZAWA Hiroyuki wrote:
> >> >
> >> > mmotm 2010-04-15-14-42
> >> >
> >> > When I tried
> >> > ?# echo 0 > /proc/sys/vm/compaction
> >> >
> >> > I see following.
> >> >
> >> > My enviroment was
> >> > ? 2.6.34-rc4-mm1+ (2010-04-15-14-42) (x86-64) CPUx8
> >> > ? allocating tons of hugepages and reduce free memory.
> >> >
> >> > What I did was:
> >> > ? # echo 0 > /proc/sys/vm/compact_memory
> >> >
> >> > Hmm, I see this kind of error at migation for the 1st time..
> >> > my.config is attached. Hmm... ?
> >> >
> >> > (I'm sorry I'll be offline soon.)
> >>
> >> That's ok, thanks you for the report. I'm afraid I made little progress
> >> as I spent most of the day on other bugs but I do have something for
> >> you.
> >>
> >> First, I reproduced the problem using your .config. However, the problem does
> >> not manifest with the .config I normally use which is derived from the distro
> >> kernel configuration (Debian Lenny). So, there is something in your .config
> >> that triggers the problem. I very strongly suspect this is an interaction
> >> between migration, compaction and page allocation debug.
> >
> > I unexpecedly had the time to dig into this. Does the following patch fix
> > your problem? It Worked For Me.
>
> Nice catch during shot time. Below is comment.
>
> >
> > ==== CUT HERE ====
> > mm,compaction: Map free pages in the address space after they get split for compaction
> >
> > split_free_page() is a helper function which takes a free page from the
> > buddy lists and splits it into order-0 pages. It is used by memory
> > compaction to build a list of destination pages. If
> > CONFIG_DEBUG_PAGEALLOC is set, a kernel paging request bug is triggered
> > because split_free_page() did not call the arch-allocation hooks or map
> > the page into the kernel address space.
> >
> > This patch does not update split_free_page() as it is called with
> > interrupts held. Instead it documents that callers of split_free_page()
> > are responsible for calling the arch hooks and to map the page and fixes
> > compaction.
>
> Dumb question. Why can't we call arch_alloc_page and kernel_map_pages
> as interrupt disabled?

In theory, it isn't known what arch_alloc_page is going to do but more
practically kernel_map_pages() is updating mappings and should be
flushing all the TLBs. It can't do that with interrupts disabled.

I checked X86 and it should be fine but only because it flushes the
local CPU and appears to just hope for the best that this doesn't cause
problems.

> It's deadlock issue or latency issue?

deadlock

> I don't found any comment about it.

I'm not aware of any. arch_alloc_page() is only used by s390 so it's not
well known. kernel_map_pages() is only active for a rarely used
debugging option.

> It should have added the comment around that functions. :)
>
> And now compaction only uses split_free_page and it is exposed by mm.h.
> I think it would be better to map pages inside split_free_page to
> export others.(ie, making generic function).

I considered that and it would not be ideal. It would have to disable and
reenable interrupts as each page is taken from the list or alternatively
require that the caller not have the zone lock taken. The latter of these
options is more reasonable but would still result in more interrupt enabling
and disabling.

split_free_page() is extremely specialised and requires knowledge of the
page allocator internals to call properly. There is little pressure to
make this easier to use at the cost of increased locking.

> If we can't do, how about making split_free_page static as static function?
> And only uses it in compaction.
>

It pretty much has to be in page_alloc.c because it uses internal
functions of the page allocator - e.g. rmv_page_order. I could move it
to mm/internal.h because whatever about split_page, I can't imagine why
anyone else would need to call split_free_page.

--
Mel Gorman
Part-time Phd Student Linux Technology Center
University of Limerick IBM Dublin Software Lab

2010-04-20 08:32:16

[permalink] [raw]

Subject: Re: error at compaction (Re: mmotm 2010-04-15-14-42 uploaded

On Tue, Apr 20, 2010 at 5:20 PM, Mel Gorman <[email protected]> wrote:
> On Tue, Apr 20, 2010 at 11:39:46AM +0900, Minchan Kim wrote:
>> On Tue, Apr 20, 2010 at 4:39 AM, Mel Gorman <[email protected]> wrote:
>> > On Mon, Apr 19, 2010 at 07:14:42PM +0100, Mel Gorman wrote:
>> >> On Mon, Apr 19, 2010 at 07:01:33PM +0900, KAMEZAWA Hiroyuki wrote:
>> >> >
>> >> > mmotm 2010-04-15-14-42
>> >> >
>> >> > When I tried
>> >> > # echo 0 > /proc/sys/vm/compaction
>> >> >
>> >> > I see following.
>> >> >
>> >> > My enviroment was
>> >> > 2.6.34-rc4-mm1+ (2010-04-15-14-42) (x86-64) CPUx8
>> >> > allocating tons of hugepages and reduce free memory.
>> >> >
>> >> > What I did was:
>> >> > # echo 0 > /proc/sys/vm/compact_memory
>> >> >
>> >> > Hmm, I see this kind of error at migation for the 1st time..
>> >> > my.config is attached. Hmm... ?
>> >> >
>> >> > (I'm sorry I'll be offline soon.)
>> >>
>> >> That's ok, thanks you for the report. I'm afraid I made little progress
>> >> as I spent most of the day on other bugs but I do have something for
>> >> you.
>> >>
>> >> First, I reproduced the problem using your .config. However, the problem does
>> >> not manifest with the .config I normally use which is derived from the distro
>> >> kernel configuration (Debian Lenny). So, there is something in your .config
>> >> that triggers the problem. I very strongly suspect this is an interaction
>> >> between migration, compaction and page allocation debug.
>> >
>> > I unexpecedly had the time to dig into this. Does the following patch fix
>> > your problem? It Worked For Me.
>>
>> Nice catch during shot time. Below is comment.
>>
>> >
>> > ==== CUT HERE ====
>> > mm,compaction: Map free pages in the address space after they get split for compaction
>> >
>> > split_free_page() is a helper function which takes a free page from the
>> > buddy lists and splits it into order-0 pages. It is used by memory
>> > compaction to build a list of destination pages. If
>> > CONFIG_DEBUG_PAGEALLOC is set, a kernel paging request bug is triggered
>> > because split_free_page() did not call the arch-allocation hooks or map
>> > the page into the kernel address space.
>> >
>> > This patch does not update split_free_page() as it is called with
>> > interrupts held. Instead it documents that callers of split_free_page()
>> > are responsible for calling the arch hooks and to map the page and fixes
>> > compaction.
>>
>> Dumb question. Why can't we call arch_alloc_page and kernel_map_pages
>> as interrupt disabled?
>
> In theory, it isn't known what arch_alloc_page is going to do but more
> practically kernel_map_pages() is updating mappings and should be
> flushing all the TLBs. It can't do that with interrupts disabled.
>
> I checked X86 and it should be fine but only because it flushes the
> local CPU and appears to just hope for the best that this doesn't cause
> problems.

Okay.

>> And now compaction only uses split_free_page and it is exposed by mm.h.
>> I think it would be better to map pages inside split_free_page to
>> export others.(ie, making generic function).
>
> I considered that and it would not be ideal. It would have to disable and
> reenable interrupts as each page is taken from the list or alternatively
> require that the caller not have the zone lock taken. The latter of these
> options is more reasonable but would still result in more interrupt enabling
> and disabling.
>
> split_free_page() is extremely specialised and requires knowledge of the
> page allocator internals to call properly. There is little pressure to
> make this easier to use at the cost of increased locking.
>
>> If we can't do, how about making split_free_page static as static function?
>> And only uses it in compaction.
>>
>
> It pretty much has to be in page_alloc.c because it uses internal
> functions of the page allocator - e.g. rmv_page_order. I could move it
> to mm/internal.h because whatever about split_page, I can't imagine why
> anyone else would need to call split_free_page.

Yes. Then, Let's add comment like split_page. :)
/*
* Note: this is probably too low level an operation for use in drivers.
* Please consult with lkml before using this in your driver.
*/

>
> --
> Mel Gorman
> Part-time Phd Student Linux Technology Center
> University of Limerick IBM Dublin Software Lab
>

--
Kind regards,
Minchan Kim

2010-04-20 08:45:16

[permalink] [raw]

Subject: Re: error at compaction (Re: mmotm 2010-04-15-14-42 uploaded

On Tue, Apr 20, 2010 at 05:32:13PM +0900, Minchan Kim wrote:
> On Tue, Apr 20, 2010 at 5:20 PM, Mel Gorman <[email protected]> wrote:
> > On Tue, Apr 20, 2010 at 11:39:46AM +0900, Minchan Kim wrote:
> >> On Tue, Apr 20, 2010 at 4:39 AM, Mel Gorman <[email protected]> wrote:
> >> > On Mon, Apr 19, 2010 at 07:14:42PM +0100, Mel Gorman wrote:
> >> >> On Mon, Apr 19, 2010 at 07:01:33PM +0900, KAMEZAWA Hiroyuki wrote:
> >> >> >
> >> >> > mmotm 2010-04-15-14-42
> >> >> >
> >> >> > When I tried
> >> >> > ?# echo 0 > /proc/sys/vm/compaction
> >> >> >
> >> >> > I see following.
> >> >> >
> >> >> > My enviroment was
> >> >> > ? 2.6.34-rc4-mm1+ (2010-04-15-14-42) (x86-64) CPUx8
> >> >> > ? allocating tons of hugepages and reduce free memory.
> >> >> >
> >> >> > What I did was:
> >> >> > ? # echo 0 > /proc/sys/vm/compact_memory
> >> >> >
> >> >> > Hmm, I see this kind of error at migation for the 1st time..
> >> >> > my.config is attached. Hmm... ?
> >> >> >
> >> >> > (I'm sorry I'll be offline soon.)
> >> >>
> >> >> That's ok, thanks you for the report. I'm afraid I made little progress
> >> >> as I spent most of the day on other bugs but I do have something for
> >> >> you.
> >> >>
> >> >> First, I reproduced the problem using your .config. However, the problem does
> >> >> not manifest with the .config I normally use which is derived from the distro
> >> >> kernel configuration (Debian Lenny). So, there is something in your .config
> >> >> that triggers the problem. I very strongly suspect this is an interaction
> >> >> between migration, compaction and page allocation debug.
> >> >
> >> > I unexpecedly had the time to dig into this. Does the following patch fix
> >> > your problem? It Worked For Me.
> >>
> >> Nice catch during shot time. Below is comment.
> >>
> >> >
> >> > ==== CUT HERE ====
> >> > mm,compaction: Map free pages in the address space after they get split for compaction
> >> >
> >> > split_free_page() is a helper function which takes a free page from the
> >> > buddy lists and splits it into order-0 pages. It is used by memory
> >> > compaction to build a list of destination pages. If
> >> > CONFIG_DEBUG_PAGEALLOC is set, a kernel paging request bug is triggered
> >> > because split_free_page() did not call the arch-allocation hooks or map
> >> > the page into the kernel address space.
> >> >
> >> > This patch does not update split_free_page() as it is called with
> >> > interrupts held. Instead it documents that callers of split_free_page()
> >> > are responsible for calling the arch hooks and to map the page and fixes
> >> > compaction.
> >>
> >> Dumb question. Why can't we call arch_alloc_page and kernel_map_pages
> >> as interrupt disabled?
> >
> > In theory, it isn't known what arch_alloc_page is going to do but more
> > practically kernel_map_pages() is updating mappings and should be
> > flushing all the TLBs. It can't do that with interrupts disabled.
> >
> > I checked X86 and it should be fine but only because it flushes the
> > local CPU and appears to just hope for the best that this doesn't cause
> > problems.
>
> Okay.
>
> >> And now compaction only uses split_free_page and it is exposed by mm.h.
> >> I think it would be better to map pages inside split_free_page to
> >> export others.(ie, making generic function).
> >
> > I considered that and it would not be ideal. It would have to disable and
> > reenable interrupts as each page is taken from the list or alternatively
> > require that the caller not have the zone lock taken. The latter of these
> > options is more reasonable but would still result in more interrupt enabling
> > and disabling.
> >
> > split_free_page() is extremely specialised and requires knowledge of the
> > page allocator internals to call properly. There is little pressure to
> > make this easier to use at the cost of increased locking.
> >
> >> If we can't do, how about making split_free_page static as static function?
> >> And only uses it in compaction.
> >>
> >
> > It pretty much has to be in page_alloc.c because it uses internal
> > functions of the page allocator - e.g. rmv_page_order. I could move it
> > to mm/internal.h because whatever about split_page, I can't imagine why
> > anyone else would need to call split_free_page.
>
> Yes. Then, Let's add comment like split_page. :)
> /*
> * Note: this is probably too low level an operation for use in drivers.
> * Please consult with lkml before using this in your driver.
> */
>

I can, but the comment that was there says it's like split_page except the
page is already free. This also covers not using it in a driver.

--
Mel Gorman
Part-time Phd Student Linux Technology Center
University of Limerick IBM Dublin Software Lab

2010-04-20 09:50:26

[permalink] [raw]

Subject: Re: error at compaction (Re: mmotm 2010-04-15-14-42 uploaded

On Tue, Apr 20, 2010 at 5:44 PM, Mel Gorman <[email protected]> wrote:
> On Tue, Apr 20, 2010 at 05:32:13PM +0900, Minchan Kim wrote:
>>
>> Yes. Then, Let's add comment like split_page. :)
>> /*
>> * Note: this is probably too low level an operation for use in drivers.
>> * Please consult with lkml before using this in your driver.
>> */
>>
>
> I can, but the comment that was there says it's like split_page except the
> page is already free. This also covers not using it in a driver.

I see. In addition, you already mentioned "As this is only being used
for migration".
I missed one.
I don't have any against one.
Will you repost v2 which move split_free_pages out of compaction.c?
Anyway, feel free to add my reviewed-by sign.
Thanks, Mel.

Reviewed-by: Minchan Kim <[email protected]>

>
> --
> Mel Gorman
> Part-time Phd Student Linux Technology Center
> University of Limerick IBM Dublin Software Lab
>

--
Kind regards,
Minchan Kim

2010-04-20 09:59:15

[permalink] [raw]

Subject: Re: error at compaction (Re: mmotm 2010-04-15-14-42 uploaded

On Tue, Apr 20, 2010 at 06:50:23PM +0900, Minchan Kim wrote:
> On Tue, Apr 20, 2010 at 5:44 PM, Mel Gorman <[email protected]> wrote:
> > On Tue, Apr 20, 2010 at 05:32:13PM +0900, Minchan Kim wrote:
> >>
> >> Yes. Then, Let's add comment like split_page. :)
> >> ?/*
> >> ?* Note: this is probably too low level an operation for use in drivers.
> >> ?* Please consult with lkml before using this in your driver.
> >> ?*/
> >>
> >
> > I can, but the comment that was there says it's like split_page except the
> > page is already free. This also covers not using it in a driver.
>
> I see. In addition, you already mentioned "As this is only being used
> for migration".
> I missed one.
> I don't have any against one.
> Will you repost v2 which move split_free_pages out of compaction.c?

I don't understand your suggestion. split_free_pages is already out of
compaction.c.

> Anyway, feel free to add my reviewed-by sign.
> Thanks, Mel.
>
> Reviewed-by: Minchan Kim <[email protected]>
>

Thanks

--
Mel Gorman
Part-time Phd Student Linux Technology Center
University of Limerick IBM Dublin Software Lab

2010-04-20 10:14:10

[permalink] [raw]

Subject: Re: error at compaction (Re: mmotm 2010-04-15-14-42 uploaded

On Tue, Apr 20, 2010 at 6:58 PM, Mel Gorman <[email protected]> wrote:
> On Tue, Apr 20, 2010 at 06:50:23PM +0900, Minchan Kim wrote:
>> On Tue, Apr 20, 2010 at 5:44 PM, Mel Gorman <[email protected]> wrote:
>> > On Tue, Apr 20, 2010 at 05:32:13PM +0900, Minchan Kim wrote:
>> >>
>> >> Yes. Then, Let's add comment like split_page. :)
>> >> /*
>> >> * Note: this is probably too low level an operation for use in drivers.
>> >> * Please consult with lkml before using this in your driver.
>> >> */
>> >>
>> >
>> > I can, but the comment that was there says it's like split_page except the
>> > page is already free. This also covers not using it in a driver.
>>
>> I see. In addition, you already mentioned "As this is only being used
>> for migration".
>> I missed one.
>> I don't have any against one.
>> Will you repost v2 which move split_free_pages out of compaction.c?
>
> I don't understand your suggestion. split_free_pages is already out of
> compaction.c.

Ahh. Sorry. It's my fault. I confused. forget it, please.
--
Kind regards,
Minchan Kim

2010-04-21 08:32:38

[permalink] [raw]

Subject: Re: error at compaction (Re: mmotm 2010-04-15-14-42 uploaded

On Mon, 19 Apr 2010 20:39:19 +0100
Mel Gorman <[email protected]> wrote:

> On Mon, Apr 19, 2010 at 07:14:42PM +0100, Mel Gorman wrote:
> ==== CUT HERE ====
> mm,compaction: Map free pages in the address space after they get split for compaction
>
> split_free_page() is a helper function which takes a free page from the
> buddy lists and splits it into order-0 pages. It is used by memory
> compaction to build a list of destination pages. If
> CONFIG_DEBUG_PAGEALLOC is set, a kernel paging request bug is triggered
> because split_free_page() did not call the arch-allocation hooks or map
> the page into the kernel address space.
>
> This patch does not update split_free_page() as it is called with
> interrupts held. Instead it documents that callers of split_free_page()
> are responsible for calling the arch hooks and to map the page and fixes
> compaction.
>
> This is a fix to the patch mm-compaction-memory-compaction-core.patch.
>
> Signed-off-by: Mel Gorman <[email protected]>

Sorry, I think I hit another? error again. (sorry, no log.)
What I did was...
Running 2 shells.
while true; do make -j 16;make cleanl;done
and
while true; do echo 0 > /proc/sys/vm/compact_memory;done

Using the same config.

Apr 21 17:27:47 localhost kernel: ------------[ cut here ]------------
Apr 21 17:27:47 localhost kernel: kernel BUG at include/linux/swapops.h:105!
Apr 21 17:27:47 localhost kernel: invalid opcode: 0000 [#1] SMP DEBUG_PAGEALLOC
Apr 21 17:27:47 localhost kernel: last sysfs file: /sys/devices/virtual/net/br0/statistics/collisions
Apr 21 17:27:47 localhost kernel: CPU 3
Apr 21 17:27:47 localhost kernel: Modules linked in: fuse sit tunnel4 ipt_MASQUERADE iptable_nat nf_nat bridge stp llc sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf xt_physdev ip6t_REJECT nf_conntrack_ipv6 ip6table_filter ip6_tables ipv6 dm_multipath uinput ioatdma ppdev parport_pc i5000_edac bnx2 iTCO_wdt edac_core iTCO_vendor_support shpchp parport e1000e kvm_intel dca kvm i2c_i801 i2c_core i5k_amb pcspkr megaraid_sas [last unloaded: microcode]
Apr 21 17:27:47 localhost kernel:
Apr 21 17:27:47 localhost kernel: Pid: 27892, comm: cc1 Tainted: G W 2.6.34-rc4-mm1+ #4 D2519/PRIMERGY
Apr 21 17:27:47 localhost kernel: RIP: 0010:[<ffffffff8114e9cf>] [<ffffffff8114e9cf>] migration_entry_wait+0x16f/0x180
Apr 21 17:27:47 localhost kernel: RSP: 0000:ffff88008d9efe08 EFLAGS: 00010246
Apr 21 17:27:47 localhost kernel: RAX: ffffea0000000000 RBX: ffffea0000241100 RCX: 0000000000000001
Apr 21 17:27:47 localhost kernel: RDX: 000000000000a4e0 RSI: ffff880621a4ab00 RDI: 000000000149c03e
Apr 21 17:27:47 localhost kernel: RBP: ffff88008d9efe38 R08: 0000000000000000 R09: 0000000000000000
Apr 21 17:27:47 localhost kernel: R10: 0000000000000000 R11: 0000000000000001 R12: ffff880621a4aae8
Apr 21 17:27:47 localhost kernel: R13: 00000000bf811000 R14: 000000000149c03e R15: 0000000000000000
Apr 21 17:27:47 localhost kernel: FS: 00007fe6abc90700(0000) GS:ffff880005a00000(0000) knlGS:0000000000000000
Apr 21 17:27:47 localhost kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Apr 21 17:27:47 localhost kernel: CR2: 00007fe6a37279a0 CR3: 000000008d942000 CR4: 00000000000006e0
Apr 21 17:27:47 localhost kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Apr 21 17:27:47 localhost kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Apr 21 17:27:47 localhost kernel: Process cc1 (pid: 27892, threadinfo ffff88008d9ee000, task ffff8800b23ec820)
Apr 21 17:27:47 localhost kernel: Stack:
Apr 21 17:27:47 localhost kernel: ffffea000101aee8 ffff880621a4aae8 ffff88008d9efe38 00007fe6a37279a0
Apr 21 17:27:47 localhost kernel: <0> ffff8805d9706d90 ffff880621a4aa00 ffff88008d9efef8 ffffffff81126d05
Apr 21 17:27:47 localhost kernel: <0> ffff88008d9efec8 0000000000000246 0000000000000000 ffffffff81586533
Apr 21 17:27:47 localhost kernel: Call Trace:
Apr 21 17:27:47 localhost kernel: [<ffffffff81126d05>] handle_mm_fault+0x995/0x9b0
Apr 21 17:27:47 localhost kernel: [<ffffffff81586533>] ? do_page_fault+0x103/0x330
Apr 21 17:27:47 localhost kernel: [<ffffffff8104bf40>] ? finish_task_switch+0x0/0xf0
Apr 21 17:27:47 localhost kernel: [<ffffffff8158659e>] do_page_fault+0x16e/0x330
Apr 21 17:27:47 localhost kernel: [<ffffffff81582f35>] page_fault+0x25/0x30
Apr 21 17:27:47 localhost kernel: Code: 53 08 85 c9 0f 84 32 ff ff ff 8d 41 01 89 4d d8 89 45 d4 8b 75 d4 8b 45 d8 f0 0f b1 32 89 45 dc 8b 45 dc 39 c8 74 aa 89 c1 eb d7 <0f> 0b eb fe 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5
Apr 21 17:27:47 localhost kernel: RIP [<ffffffff8114e9cf>] migration_entry_wait+0x16f/0x180
Apr 21 17:27:47 localhost kernel: RSP <ffff88008d9efe08>
Apr 21 17:27:47 localhost kernel: ---[ end trace 4860ab585c1fcddb ]---

Regards,
-Kame

2010-04-21 09:52:06

[permalink] [raw]

Subject: Re: error at compaction (Re: mmotm 2010-04-15-14-42 uploaded

On Wed, 21 Apr 2010 17:28:38 +0900
KAMEZAWA Hiroyuki <[email protected]> wrote:

> On Mon, 19 Apr 2010 20:39:19 +0100
> Mel Gorman <[email protected]> wrote:
>
> > On Mon, Apr 19, 2010 at 07:14:42PM +0100, Mel Gorman wrote:
> > ==== CUT HERE ====
> > mm,compaction: Map free pages in the address space after they get split for compaction
> >
> > split_free_page() is a helper function which takes a free page from the
> > buddy lists and splits it into order-0 pages. It is used by memory
> > compaction to build a list of destination pages. If
> > CONFIG_DEBUG_PAGEALLOC is set, a kernel paging request bug is triggered
> > because split_free_page() did not call the arch-allocation hooks or map
> > the page into the kernel address space.
> >
> > This patch does not update split_free_page() as it is called with
> > interrupts held. Instead it documents that callers of split_free_page()
> > are responsible for calling the arch hooks and to map the page and fixes
> > compaction.
> >
> > This is a fix to the patch mm-compaction-memory-compaction-core.patch.
> >
> > Signed-off-by: Mel Gorman <[email protected]>
>
> Sorry, I think I hit another? error again. (sorry, no log.)
> What I did was...
> Running 2 shells.
> while true; do make -j 16;make cleanl;done
> and
> while true; do echo 0 > /proc/sys/vm/compact_memory;done
>
>
> Using the same config.
>
> Apr 21 17:27:47 localhost kernel: ------------[ cut here ]------------
> Apr 21 17:27:47 localhost kernel: kernel BUG at include/linux/swapops.h:105!
> Apr 21 17:27:47 localhost kernel: invalid opcode: 0000 [#1] SMP DEBUG_PAGEALLOC
> Apr 21 17:27:47 localhost kernel: last sysfs file: /sys/devices/virtual/net/br0/statistics/collisions
> Apr 21 17:27:47 localhost kernel: CPU 3
> Apr 21 17:27:47 localhost kernel: Modules linked in: fuse sit tunnel4 ipt_MASQUERADE iptable_nat nf_nat bridge stp llc sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf xt_physdev ip6t_REJECT nf_conntrack_ipv6 ip6table_filter ip6_tables ipv6 dm_multipath uinput ioatdma ppdev parport_pc i5000_edac bnx2 iTCO_wdt edac_core iTCO_vendor_support shpchp parport e1000e kvm_intel dca kvm i2c_i801 i2c_core i5k_amb pcspkr megaraid_sas [last unloaded: microcode]
> Apr 21 17:27:47 localhost kernel:
> Apr 21 17:27:47 localhost kernel: Pid: 27892, comm: cc1 Tainted: G W 2.6.34-rc4-mm1+ #4 D2519/PRIMERGY
> Apr 21 17:27:47 localhost kernel: RIP: 0010:[<ffffffff8114e9cf>] [<ffffffff8114e9cf>] migration_entry_wait+0x16f/0x180
> Apr 21 17:27:47 localhost kernel: RSP: 0000:ffff88008d9efe08 EFLAGS: 00010246
> Apr 21 17:27:47 localhost kernel: RAX: ffffea0000000000 RBX: ffffea0000241100 RCX: 0000000000000001
> Apr 21 17:27:47 localhost kernel: RDX: 000000000000a4e0 RSI: ffff880621a4ab00 RDI: 000000000149c03e
> Apr 21 17:27:47 localhost kernel: RBP: ffff88008d9efe38 R08: 0000000000000000 R09: 0000000000000000
> Apr 21 17:27:47 localhost kernel: R10: 0000000000000000 R11: 0000000000000001 R12: ffff880621a4aae8
> Apr 21 17:27:47 localhost kernel: R13: 00000000bf811000 R14: 000000000149c03e R15: 0000000000000000
> Apr 21 17:27:47 localhost kernel: FS: 00007fe6abc90700(0000) GS:ffff880005a00000(0000) knlGS:0000000000000000
> Apr 21 17:27:47 localhost kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> Apr 21 17:27:47 localhost kernel: CR2: 00007fe6a37279a0 CR3: 000000008d942000 CR4: 00000000000006e0
> Apr 21 17:27:47 localhost kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> Apr 21 17:27:47 localhost kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Apr 21 17:27:47 localhost kernel: Process cc1 (pid: 27892, threadinfo ffff88008d9ee000, task ffff8800b23ec820)
> Apr 21 17:27:47 localhost kernel: Stack:
> Apr 21 17:27:47 localhost kernel: ffffea000101aee8 ffff880621a4aae8 ffff88008d9efe38 00007fe6a37279a0
> Apr 21 17:27:47 localhost kernel: <0> ffff8805d9706d90 ffff880621a4aa00 ffff88008d9efef8 ffffffff81126d05
> Apr 21 17:27:47 localhost kernel: <0> ffff88008d9efec8 0000000000000246 0000000000000000 ffffffff81586533
> Apr 21 17:27:47 localhost kernel: Call Trace:
> Apr 21 17:27:47 localhost kernel: [<ffffffff81126d05>] handle_mm_fault+0x995/0x9b0
> Apr 21 17:27:47 localhost kernel: [<ffffffff81586533>] ? do_page_fault+0x103/0x330
> Apr 21 17:27:47 localhost kernel: [<ffffffff8104bf40>] ? finish_task_switch+0x0/0xf0
> Apr 21 17:27:47 localhost kernel: [<ffffffff8158659e>] do_page_fault+0x16e/0x330
> Apr 21 17:27:47 localhost kernel: [<ffffffff81582f35>] page_fault+0x25/0x30
> Apr 21 17:27:47 localhost kernel: Code: 53 08 85 c9 0f 84 32 ff ff ff 8d 41 01 89 4d d8 89 45 d4 8b 75 d4 8b 45 d8 f0 0f b1 32 89 45 dc 8b 45 dc 39 c8 74 aa 89 c1 eb d7 <0f> 0b eb fe 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5
> Apr 21 17:27:47 localhost kernel: RIP [<ffffffff8114e9cf>] migration_entry_wait+0x16f/0x180
> Apr 21 17:27:47 localhost kernel: RSP <ffff88008d9efe08>
> Apr 21 17:27:47 localhost kernel: ---[ end trace 4860ab585c1fcddb ]---
>

It seems that this is a new error.

static inline struct page *migration_entry_to_page(swp_entry_t entry)
{
struct page *p = pfn_to_page(swp_offset(entry));
/*
* Any use of migration entries may only occur while the
* corresponding page is locked
*/
BUG_ON(!PageLocked(p));
return p;
}

Hits this BUG_ON()....then, the page migration_entry points to is unlocked.

But we always do

lock_page(old_page);
unamp(old_page);
remap(new_page);
unlock_page(old_page);

So....some pte wasn't updated at remap ?

Hmm.
-Kame

2010-04-21 10:21:00

[permalink] [raw]

Subject: Re: error at compaction (Re: mmotm 2010-04-15-14-42 uploaded

On Wed, Apr 21, 2010 at 06:48:06PM +0900, KAMEZAWA Hiroyuki wrote:
> On Wed, 21 Apr 2010 17:28:38 +0900
> KAMEZAWA Hiroyuki <[email protected]> wrote:
>
> > On Mon, 19 Apr 2010 20:39:19 +0100
> > Mel Gorman <[email protected]> wrote:
> >
> > > On Mon, Apr 19, 2010 at 07:14:42PM +0100, Mel Gorman wrote:
> > > ==== CUT HERE ====
> > > mm,compaction: Map free pages in the address space after they get split for compaction
> > >
> > > split_free_page() is a helper function which takes a free page from the
> > > buddy lists and splits it into order-0 pages. It is used by memory
> > > compaction to build a list of destination pages. If
> > > CONFIG_DEBUG_PAGEALLOC is set, a kernel paging request bug is triggered
> > > because split_free_page() did not call the arch-allocation hooks or map
> > > the page into the kernel address space.
> > >
> > > This patch does not update split_free_page() as it is called with
> > > interrupts held. Instead it documents that callers of split_free_page()
> > > are responsible for calling the arch hooks and to map the page and fixes
> > > compaction.
> > >
> > > This is a fix to the patch mm-compaction-memory-compaction-core.patch.
> > >
> > > Signed-off-by: Mel Gorman <[email protected]>
> >
> > Sorry, I think I hit another? error again. (sorry, no log.)
> > What I did was...
> > Running 2 shells.
> > while true; do make -j 16;make cleanl;done
> > and
> > while true; do echo 0 > /proc/sys/vm/compact_memory;done
> >
> >
> > Using the same config.
> >
> > Apr 21 17:27:47 localhost kernel: ------------[ cut here ]------------
> > Apr 21 17:27:47 localhost kernel: kernel BUG at include/linux/swapops.h:105!
> > Apr 21 17:27:47 localhost kernel: invalid opcode: 0000 [#1] SMP DEBUG_PAGEALLOC
> > Apr 21 17:27:47 localhost kernel: last sysfs file: /sys/devices/virtual/net/br0/statistics/collisions
> > Apr 21 17:27:47 localhost kernel: CPU 3
> > Apr 21 17:27:47 localhost kernel: Modules linked in: fuse sit tunnel4 ipt_MASQUERADE iptable_nat nf_nat bridge stp llc sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf xt_physdev ip6t_REJECT nf_conntrack_ipv6 ip6table_filter ip6_tables ipv6 dm_multipath uinput ioatdma ppdev parport_pc i5000_edac bnx2 iTCO_wdt edac_core iTCO_vendor_support shpchp parport e1000e kvm_intel dca kvm i2c_i801 i2c_core i5k_amb pcspkr megaraid_sas [last unloaded: microcode]
> > Apr 21 17:27:47 localhost kernel:
> > Apr 21 17:27:47 localhost kernel: Pid: 27892, comm: cc1 Tainted: G W 2.6.34-rc4-mm1+ #4 D2519/PRIMERGY
> > Apr 21 17:27:47 localhost kernel: RIP: 0010:[<ffffffff8114e9cf>] [<ffffffff8114e9cf>] migration_entry_wait+0x16f/0x180
> > Apr 21 17:27:47 localhost kernel: RSP: 0000:ffff88008d9efe08 EFLAGS: 00010246
> > Apr 21 17:27:47 localhost kernel: RAX: ffffea0000000000 RBX: ffffea0000241100 RCX: 0000000000000001
> > Apr 21 17:27:47 localhost kernel: RDX: 000000000000a4e0 RSI: ffff880621a4ab00 RDI: 000000000149c03e
> > Apr 21 17:27:47 localhost kernel: RBP: ffff88008d9efe38 R08: 0000000000000000 R09: 0000000000000000
> > Apr 21 17:27:47 localhost kernel: R10: 0000000000000000 R11: 0000000000000001 R12: ffff880621a4aae8
> > Apr 21 17:27:47 localhost kernel: R13: 00000000bf811000 R14: 000000000149c03e R15: 0000000000000000
> > Apr 21 17:27:47 localhost kernel: FS: 00007fe6abc90700(0000) GS:ffff880005a00000(0000) knlGS:0000000000000000
> > Apr 21 17:27:47 localhost kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > Apr 21 17:27:47 localhost kernel: CR2: 00007fe6a37279a0 CR3: 000000008d942000 CR4: 00000000000006e0
> > Apr 21 17:27:47 localhost kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> > Apr 21 17:27:47 localhost kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> > Apr 21 17:27:47 localhost kernel: Process cc1 (pid: 27892, threadinfo ffff88008d9ee000, task ffff8800b23ec820)
> > Apr 21 17:27:47 localhost kernel: Stack:
> > Apr 21 17:27:47 localhost kernel: ffffea000101aee8 ffff880621a4aae8 ffff88008d9efe38 00007fe6a37279a0
> > Apr 21 17:27:47 localhost kernel: <0> ffff8805d9706d90 ffff880621a4aa00 ffff88008d9efef8 ffffffff81126d05
> > Apr 21 17:27:47 localhost kernel: <0> ffff88008d9efec8 0000000000000246 0000000000000000 ffffffff81586533
> > Apr 21 17:27:47 localhost kernel: Call Trace:
> > Apr 21 17:27:47 localhost kernel: [<ffffffff81126d05>] handle_mm_fault+0x995/0x9b0
> > Apr 21 17:27:47 localhost kernel: [<ffffffff81586533>] ? do_page_fault+0x103/0x330
> > Apr 21 17:27:47 localhost kernel: [<ffffffff8104bf40>] ? finish_task_switch+0x0/0xf0
> > Apr 21 17:27:47 localhost kernel: [<ffffffff8158659e>] do_page_fault+0x16e/0x330
> > Apr 21 17:27:47 localhost kernel: [<ffffffff81582f35>] page_fault+0x25/0x30
> > Apr 21 17:27:47 localhost kernel: Code: 53 08 85 c9 0f 84 32 ff ff ff 8d 41 01 89 4d d8 89 45 d4 8b 75 d4 8b 45 d8 f0 0f b1 32 89 45 dc 8b 45 dc 39 c8 74 aa 89 c1 eb d7 <0f> 0b eb fe 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5
> > Apr 21 17:27:47 localhost kernel: RIP [<ffffffff8114e9cf>] migration_entry_wait+0x16f/0x180
> > Apr 21 17:27:47 localhost kernel: RSP <ffff88008d9efe08>
> > Apr 21 17:27:47 localhost kernel: ---[ end trace 4860ab585c1fcddb ]---
> >
>
> It seems that this is a new error.
>
>
> static inline struct page *migration_entry_to_page(swp_entry_t entry)
> {
> struct page *p = pfn_to_page(swp_offset(entry));
> /*
> * Any use of migration entries may only occur while the
> * corresponding page is locked
> */
> BUG_ON(!PageLocked(p));
> return p;
> }
>
>
> Hits this BUG_ON()....then, the page migration_entry points to is unlocked.
>
> But we always do
>
> lock_page(old_page);
> unamp(old_page);
> remap(new_page);
> unlock_page(old_page);
>
> So....some pte wasn't updated at remap ?
>

I'm working on reproducing the problem. I've hit it only once. My stress
tests were using dd instead of make like yours did and my
compilation-orientated test would not have been hitting compaction as
hard.

The theory I'm working on is that it's a PageSwapCache page that was
unmapped and not remapped (remap_swapcache == 0) in move_to_new_page().
In this case, the page would be migrated, left in place and unlocked.
Later when a swap fault occurred, the migration PTE is found and the
bug_on triggers i.e. the bug check is no longer valid because it is
possible for an unlocked migration pte to be left behind.

Trying to reproduce with some instrumentation in place documenting pages
left behind but haven't managed to trigger it a second time yet.

--
Mel Gorman
Part-time Phd Student Linux Technology Center
University of Limerick IBM Dublin Software Lab

2010-04-21 16:52:14

[permalink] [raw]

Subject: Re: error at compaction (Re: mmotm 2010-04-15-14-42 uploaded

Hi, Mel.

On Wed, 2010-04-21 at 11:20 +0100, Mel Gorman wrote:
> On Wed, Apr 21, 2010 at 06:48:06PM +0900, KAMEZAWA Hiroyuki wrote:
> > On Wed, 21 Apr 2010 17:28:38 +0900
> > KAMEZAWA Hiroyuki <[email protected]> wrote:
> >
> > > On Mon, 19 Apr 2010 20:39:19 +0100
> > > Mel Gorman <[email protected]> wrote:
> > >
> > > > On Mon, Apr 19, 2010 at 07:14:42PM +0100, Mel Gorman wrote:
> > > > ==== CUT HERE ====
> > > > mm,compaction: Map free pages in the address space after they get split for compaction
> > > >
> > > > split_free_page() is a helper function which takes a free page from the
> > > > buddy lists and splits it into order-0 pages. It is used by memory
> > > > compaction to build a list of destination pages. If
> > > > CONFIG_DEBUG_PAGEALLOC is set, a kernel paging request bug is triggered
> > > > because split_free_page() did not call the arch-allocation hooks or map
> > > > the page into the kernel address space.
> > > >
> > > > This patch does not update split_free_page() as it is called with
> > > > interrupts held. Instead it documents that callers of split_free_page()
> > > > are responsible for calling the arch hooks and to map the page and fixes
> > > > compaction.
> > > >
> > > > This is a fix to the patch mm-compaction-memory-compaction-core.patch.
> > > >
> > > > Signed-off-by: Mel Gorman <[email protected]>
> > >
> > > Sorry, I think I hit another? error again. (sorry, no log.)
> > > What I did was...
> > > Running 2 shells.
> > > while true; do make -j 16;make cleanl;done
> > > and
> > > while true; do echo 0 > /proc/sys/vm/compact_memory;done
> > >
> > >
> > > Using the same config.
> > >
> > > Apr 21 17:27:47 localhost kernel: ------------[ cut here ]------------
> > > Apr 21 17:27:47 localhost kernel: kernel BUG at include/linux/swapops.h:105!
> > > Apr 21 17:27:47 localhost kernel: invalid opcode: 0000 [#1] SMP DEBUG_PAGEALLOC
> > > Apr 21 17:27:47 localhost kernel: last sysfs file: /sys/devices/virtual/net/br0/statistics/collisions
> > > Apr 21 17:27:47 localhost kernel: CPU 3
> > > Apr 21 17:27:47 localhost kernel: Modules linked in: fuse sit tunnel4 ipt_MASQUERADE iptable_nat nf_nat bridge stp llc sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf xt_physdev ip6t_REJECT nf_conntrack_ipv6 ip6table_filter ip6_tables ipv6 dm_multipath uinput ioatdma ppdev parport_pc i5000_edac bnx2 iTCO_wdt edac_core iTCO_vendor_support shpchp parport e1000e kvm_intel dca kvm i2c_i801 i2c_core i5k_amb pcspkr megaraid_sas [last unloaded: microcode]
> > > Apr 21 17:27:47 localhost kernel:
> > > Apr 21 17:27:47 localhost kernel: Pid: 27892, comm: cc1 Tainted: G W 2.6.34-rc4-mm1+ #4 D2519/PRIMERGY
> > > Apr 21 17:27:47 localhost kernel: RIP: 0010:[<ffffffff8114e9cf>] [<ffffffff8114e9cf>] migration_entry_wait+0x16f/0x180
> > > Apr 21 17:27:47 localhost kernel: RSP: 0000:ffff88008d9efe08 EFLAGS: 00010246
> > > Apr 21 17:27:47 localhost kernel: RAX: ffffea0000000000 RBX: ffffea0000241100 RCX: 0000000000000001
> > > Apr 21 17:27:47 localhost kernel: RDX: 000000000000a4e0 RSI: ffff880621a4ab00 RDI: 000000000149c03e
> > > Apr 21 17:27:47 localhost kernel: RBP: ffff88008d9efe38 R08: 0000000000000000 R09: 0000000000000000
> > > Apr 21 17:27:47 localhost kernel: R10: 0000000000000000 R11: 0000000000000001 R12: ffff880621a4aae8
> > > Apr 21 17:27:47 localhost kernel: R13: 00000000bf811000 R14: 000000000149c03e R15: 0000000000000000
> > > Apr 21 17:27:47 localhost kernel: FS: 00007fe6abc90700(0000) GS:ffff880005a00000(0000) knlGS:0000000000000000
> > > Apr 21 17:27:47 localhost kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > > Apr 21 17:27:47 localhost kernel: CR2: 00007fe6a37279a0 CR3: 000000008d942000 CR4: 00000000000006e0
> > > Apr 21 17:27:47 localhost kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> > > Apr 21 17:27:47 localhost kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> > > Apr 21 17:27:47 localhost kernel: Process cc1 (pid: 27892, threadinfo ffff88008d9ee000, task ffff8800b23ec820)
> > > Apr 21 17:27:47 localhost kernel: Stack:
> > > Apr 21 17:27:47 localhost kernel: ffffea000101aee8 ffff880621a4aae8 ffff88008d9efe38 00007fe6a37279a0
> > > Apr 21 17:27:47 localhost kernel: <0> ffff8805d9706d90 ffff880621a4aa00 ffff88008d9efef8 ffffffff81126d05
> > > Apr 21 17:27:47 localhost kernel: <0> ffff88008d9efec8 0000000000000246 0000000000000000 ffffffff81586533
> > > Apr 21 17:27:47 localhost kernel: Call Trace:
> > > Apr 21 17:27:47 localhost kernel: [<ffffffff81126d05>] handle_mm_fault+0x995/0x9b0
> > > Apr 21 17:27:47 localhost kernel: [<ffffffff81586533>] ? do_page_fault+0x103/0x330
> > > Apr 21 17:27:47 localhost kernel: [<ffffffff8104bf40>] ? finish_task_switch+0x0/0xf0
> > > Apr 21 17:27:47 localhost kernel: [<ffffffff8158659e>] do_page_fault+0x16e/0x330
> > > Apr 21 17:27:47 localhost kernel: [<ffffffff81582f35>] page_fault+0x25/0x30
> > > Apr 21 17:27:47 localhost kernel: Code: 53 08 85 c9 0f 84 32 ff ff ff 8d 41 01 89 4d d8 89 45 d4 8b 75 d4 8b 45 d8 f0 0f b1 32 89 45 dc 8b 45 dc 39 c8 74 aa 89 c1 eb d7 <0f> 0b eb fe 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5
> > > Apr 21 17:27:47 localhost kernel: RIP [<ffffffff8114e9cf>] migration_entry_wait+0x16f/0x180
> > > Apr 21 17:27:47 localhost kernel: RSP <ffff88008d9efe08>
> > > Apr 21 17:27:47 localhost kernel: ---[ end trace 4860ab585c1fcddb ]---
> > >
> >
> > It seems that this is a new error.
> >
> >
> > static inline struct page *migration_entry_to_page(swp_entry_t entry)
> > {
> > struct page *p = pfn_to_page(swp_offset(entry));
> > /*
> > * Any use of migration entries may only occur while the
> > * corresponding page is locked
> > */
> > BUG_ON(!PageLocked(p));
> > return p;
> > }
> >
> >
> > Hits this BUG_ON()....then, the page migration_entry points to is unlocked.
> >
> > But we always do
> >
> > lock_page(old_page);
> > unamp(old_page);
> > remap(new_page);
> > unlock_page(old_page);
> >
> > So....some pte wasn't updated at remap ?
> >
>
> I'm working on reproducing the problem. I've hit it only once. My stress
> tests were using dd instead of make like yours did and my
> compilation-orientated test would not have been hitting compaction as
> hard.
>
> The theory I'm working on is that it's a PageSwapCache page that was
> unmapped and not remapped (remap_swapcache == 0) in move_to_new_page().
> In this case, the page would be migrated, left in place and unlocked.
> Later when a swap fault occurred, the migration PTE is found and the
> bug_on triggers i.e. the bug check is no longer valid because it is
> possible for an unlocked migration pte to be left behind.

Hmm. How about the situation?

CPU A CPU B

1. unmap_and_move
2. lock_page
3. PageAnon && !page_mapped && PageSwapCache 3' do_fork
4. remap_swapcache = 0 4' pte lock, page_dup_rmap <- race happens
5. try_to_unmap - make migration entry by 4'
6. move_to_newpage
7. don't call remove_migration due to 4
8. do_swap_page
9. migration_entry_wait
10. goto out
11. fault!

In this case, process of CPU B will be killed although it passes PageLocked
So I think we have to find another method.

I might be wrong since nearly falling asleep. :(

--
Kind regards,
Minchan Kim

2010-04-21 23:01:30