LinuxLists.cc - VM/VFS bug with large amount of memory and file systems?

[permalink] [raw]

Subject: Re: VM/VFS bug with large amount of memory and file systems?

On Sat, 2007-09-15 at 08:27 +0100, Anton Altaparmakov wrote:

Please, don't word wrap log-files, they're hard enough to read without
it :-(

( I see people do this more and more often, *WHY*? is that because we
like 80 char lines, in code and email? )

Anyway, looks like all of zone_normal is pinned in kernel allocations:

> Sep 13 15:31:25 escabot Normal free:3648kB min:3744kB low:4680kB high: 5616kB active:0kB inactive:3160kB present:894080kB pages_scanned:5336 all_unreclaimable? yes

Out of the 870 odd mb only 3 is on the lru.

Would be grand it you could have a look at slabinfo and the like.

Peter

---

> Sep 13 15:31:25 escabot init invoked oom-killer: gfp_mask=0xd0,
> order=0, oomkilladj=0
> Sep 13 15:31:25 escabot [<c0149ea8>] out_of_memory+0x158/0x1b0
> Sep 13 15:31:25 escabot [<c014b7b0>] __alloc_pages+0x2a0/0x300
> Sep 13 15:31:25 escabot [<c01729aa>] core_sys_select+0x1fa/0x2f0
> Sep 13 15:31:25 escabot [<c0162a21>] cache_alloc_refill+0x2e1/0x500
> Sep 13 15:31:25 escabot [<c0162736>] kmem_cache_alloc+0x46/0x50
> Sep 13 15:31:25 escabot [<c016e238>] getname+0x28/0xf0
> Sep 13 15:31:25 escabot [<c016fe5e>] __user_walk_fd+0x1e/0x60
> Sep 13 15:31:25 escabot [<c0168e29>] cp_new_stat64+0xf9/0x110
> Sep 13 15:31:25 escabot [<c0169152>] vfs_stat_fd+0x22/0x60
> Sep 13 15:31:25 escabot [<c016922f>] sys_stat64+0xf/0x30
> Sep 13 15:31:25 escabot [<c0124956>] do_gettimeofday+0x36/0xf0
> Sep 13 15:31:25 escabot [<c0120bbf>] sys_time+0xf/0x30
> Sep 13 15:31:25 escabot [<c0102b10>] sysenter_past_esp+0x5d/0x81
> Sep 13 15:31:25 escabot =======================
> Sep 13 15:31:25 escabot Mem-info:
> Sep 13 15:31:25 escabot DMA per-cpu:
> Sep 13 15:31:25 escabot CPU 0: Hot: hi: 0, btch: 1 usd: 0
> Cold: hi: 0, btch: 1 usd: 0
> Sep 13 15:31:25 escabot CPU 1: Hot: hi: 0, btch: 1 usd: 0
> Cold: hi: 0, btch: 1 usd: 0
> Sep 13 15:31:25 escabot CPU 2: Hot: hi: 0, btch: 1 usd: 0
> Cold: hi: 0, btch: 1 usd: 0
> Sep 13 15:31:25 escabot CPU 3: Hot: hi: 0, btch: 1 usd: 0
> Cold: hi: 0, btch: 1 usd: 0
> Sep 13 15:31:25 escabot Normal per-cpu:
> Sep 13 15:31:25 escabot CPU 0: Hot: hi: 186, btch: 31 usd: 143
> Cold: hi: 62, btch: 15 usd: 56
> Sep 13 15:31:25 escabot CPU 1: Hot: hi: 186, btch: 31 usd: 31
> Cold: hi: 62, btch: 15 usd: 53
> Sep 13 15:31:25 escabot CPU 2: Hot: hi: 186, btch: 31 usd: 8
> Cold: hi: 62, btch: 15 usd: 54
> Sep 13 15:31:25 escabot CPU 3: Hot: hi: 186, btch: 31 usd: 99
> Cold: hi: 62, btch: 15 usd: 58
> Sep 13 15:31:25 escabot HighMem per-cpu:
> Sep 13 15:31:25 escabot CPU 0: Hot: hi: 186, btch: 31 usd: 1
> Cold: hi: 62, btch: 15 usd: 10
> Sep 13 15:31:25 escabot CPU 1: Hot: hi: 186, btch: 31 usd: 23
> Cold: hi: 62, btch: 15 usd: 1
> Sep 13 15:31:25 escabot CPU 2: Hot: hi: 186, btch: 31 usd: 18
> Cold: hi: 62, btch: 15 usd: 1
> Sep 13 15:31:25 escabot CPU 3: Hot: hi: 186, btch: 31 usd: 9
> Cold: hi: 62, btch: 15 usd: 12
> Sep 13 15:31:25 escabot Active:8078 inactive:1776744 dirty:10338
> writeback:0 unstable:0
> Sep 13 15:31:25 escabot free:1090395 slab:198893 mapped:988
> pagetables:129 bounce:0
> Sep 13 15:31:25 escabot DMA free:3560kB min:68kB low:84kB high:100kB
> active:0kB inactive:0kB present:16256kB pages_scanned:0
> all_unreclaimable? yes
> Sep 13 15:31:25 escabot lowmem_reserve[]: 0 873 12176
> Sep 13 15:31:25 escabot Normal free:3648kB min:3744kB low:4680kB high:
> 5616kB active:0kB inactive:3160kB present:894080kB pages_scanned:5336
> all_unreclaimable? yes
> Sep 13 15:31:25 escabot lowmem_reserve[]: 0 0 90424
> Sep 13 15:31:25 escabot HighMem free:4354372kB min:512kB low:12640kB
> high:24768kB active:32312kB inactive:7103816kB present:11574272kB
> pages_scanned:0 all_unreclaimable? no
> Sep 13 15:31:25 escabot lowmem_reserve[]: 0 0 0
> Sep 13 15:31:25 escabot DMA: 4*4kB 3*8kB 0*16kB 0*32kB 1*64kB 1*128kB
> 1*256kB 0*512kB 1*1024kB 1*2048kB 0*4096kB = 3560kB
> Sep 13 15:31:25 escabot Normal: 44*4kB 18*8kB 6*16kB 1*32kB 6*64kB
> 2*128kB 0*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 3648kB
> Sep 13 15:31:25 escabot HighMem: 1*4kB 0*8kB 0*16kB 37914*32kB
> 43188*64kB 1628*128kB 5*256kB 247*512kB 30*1024kB 1*2048kB 2*4096kB =
> 4354372kB
> Sep 13 15:31:25 escabot Swap cache: add 983, delete 972, find 0/1,
> race 0+0
> Sep 13 15:31:25 escabot Free swap = 2004216kB
> Sep 13 15:31:25 escabot Total swap = 2008116kB
> Sep 13 15:31:25 escabot Free swap: 2004216kB

2007-09-15 10:50:42

by Anton Altaparmakov

[permalink] [raw]

Subject: Re: VM/VFS bug with large amount of memory and file systems?

Hi,

Thanks for looking into this.

On 15 Sep 2007, at 11:08, Peter Zijlstra wrote:
> On Sat, 2007-09-15 at 08:27 +0100, Anton Altaparmakov wrote:
>
> Please, don't word wrap log-files, they're hard enough to read without
> it :-(
>
> ( I see people do this more and more often, *WHY*? is that because we
> like 80 char lines, in code and email? )

I haven't word wrapped it at all. The lines appear as whole lines in
Apple Mail (my email client). It must be your email client that is
wrapping them...

> Anyway, looks like all of zone_normal is pinned in kernel allocations:
>
>> Sep 13 15:31:25 escabot Normal free:3648kB min:3744kB low:4680kB
>> high: 5616kB active:0kB inactive:3160kB present:894080kB
>> pages_scanned:5336 all_unreclaimable? yes
>
> Out of the 870 odd mb only 3 is on the lru.
>
> Would be grand it you could have a look at slabinfo and the like.

Ok, Marc, would you be able to do the copy to /dev/null again (booted
without the ram=2G option) and when you see that most of the memory
has gone run:

cat /proc/slabinfo > slabinfo.txt
cat /proc/meminfo > meminfo.txt
cat /proc/vmstat > vmstat.txt
cat /proc/zoneinfo > zoneinfo.txt

Then quickly abort the copy with "Ctrl+C" so hopefully you will not
get a kernel panic...

Please send the content of the generated files. Thanks.

That should hopefully show what in the kernel has eaten all the low
memory...

Best regards,

Anton
--
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK
Linux NTFS maintainer, http://www.linux-ntfs.org/

2007-09-15 10:53:18

[permalink] [raw]

Subject: Re: VM/VFS bug with large amount of memory and file systems?

On Sat, 15 Sep 2007 12:08:17 +0200 Peter Zijlstra <[email protected]> wrote:

> On Sat, 2007-09-15 at 08:27 +0100, Anton Altaparmakov wrote:
>
> Please, don't word wrap log-files, they're hard enough to read without
> it :-(
>
> ( I see people do this more and more often, *WHY*? is that because we
> like 80 char lines, in code and email? )

Isn't it?

>
> Anyway, looks like all of zone_normal is pinned in kernel allocations:
>
> > Sep 13 15:31:25 escabot Normal free:3648kB min:3744kB low:4680kB high: 5616kB active:0kB inactive:3160kB present:894080kB pages_scanned:5336 all_unreclaimable? yes
>
> Out of the 870 odd mb only 3 is on the lru.
>
> Would be grand it you could have a look at slabinfo and the like.

Definitely.

> > Sep 13 15:31:25 escabot free:1090395 slab:198893 mapped:988
> > pagetables:129 bounce:0

814,665,728 bytes of slab.

2007-09-15 11:19:17

[permalink] [raw]

Subject: Re: VM/VFS bug with large amount of memory and file systems?

On Sat, 2007-09-15 at 11:50 +0100, Anton Altaparmakov wrote:
> Hi,
>
> Thanks for looking into this.
>
> On 15 Sep 2007, at 11:08, Peter Zijlstra wrote:
> > On Sat, 2007-09-15 at 08:27 +0100, Anton Altaparmakov wrote:
> >
> > Please, don't word wrap log-files, they're hard enough to read without
> > it :-(
> >
> > ( I see people do this more and more often, *WHY*? is that because we
> > like 80 char lines, in code and email? )
>
> I haven't word wrapped it at all. The lines appear as whole lines in
> Apple Mail (my email client). It must be your email client that is
> wrapping them...
>
> > Anyway, looks like all of zone_normal is pinned in kernel allocations:
> >
> >> Sep 13 15:31:25 escabot Normal free:3648kB min:3744kB low:4680kB
> >> high: 5616kB active:0kB inactive:3160kB present:894080kB
> >> pages_scanned:5336 all_unreclaimable? yes
> >

Oddly, this line is still long in Andrew's reply but wrapped in yours.
Must be some odd mailer interaction.

/me apologises.

I guess I really have to start looking for a hackable GUI email client
that has semi proper IMAP support.

2007-09-16 07:23:23

by Kyle Moffett

[permalink] [raw]

Subject: Re: VM/VFS bug with large amount of memory and file systems?

On Sep 15, 2007, at 07:19:01, Peter Zijlstra wrote:
> On Sat, 2007-09-15 at 11:50 +0100, Anton Altaparmakov wrote:
>> I haven't word wrapped it at all. The lines appear as whole lines
>> in Apple Mail (my email client). It must be your email client
>> that is wrapping them...
>
> Oddly, this line is still long in Andrew's reply but wrapped in
> yours. Must be some odd mailer interaction.

Actually Apple Mail.app sends format=flowed wrapped to 73
characters. So a wrapped line has a single space character right
before each 'wrapping' newline. If your mail client supports
format=flowed viewing and sends without format=flowed (like AKPM's
mailer appears to), then it will properly unwrap the lines and resend
without the wrapping. Mailers which *DONT* support format=flowed
will see the wrapped version. Normally this is what you want but
it's a PITA for patches and logfiles.

I believe with Mail.app if you attach a .txt file it will be
unmangled and sent as "Content-Type: text/plain" and "Content-
Disposition: inline", so most email-clients will display it as part
of the message.

Cheers,
Kyle Moffett

2007-09-16 16:16:43

[permalink] [raw]

Subject: Re: VM/VFS bug with large amount of memory and file systems?

On Sun, 16 Sep 2007 03:22:11 -0400 Kyle Moffett <[email protected]>
wrote:

> On Sep 15, 2007, at 07:19:01, Peter Zijlstra wrote:
> > On Sat, 2007-09-15 at 11:50 +0100, Anton Altaparmakov wrote:
> >> I haven't word wrapped it at all. The lines appear as whole lines
> >> in Apple Mail (my email client). It must be your email client
> >> that is wrapping them...
> >
> > Oddly, this line is still long in Andrew's reply but wrapped in
> > yours. Must be some odd mailer interaction.
>
> Actually Apple Mail.app sends format=flowed wrapped to 73
> characters. So a wrapped line has a single space character right
> before each 'wrapping' newline. If your mail client supports
> format=flowed viewing and sends without format=flowed (like AKPM's
> mailer appears to), then it will properly unwrap the lines and resend
> without the wrapping. Mailers which *DONT* support format=flowed
> will see the wrapped version. Normally this is what you want but
> it's a PITA for patches and logfiles.
>
> I believe with Mail.app if you attach a .txt file it will be
> unmangled and sent as "Content-Type: text/plain" and "Content-
> Disposition: inline", so most email-clients will display it as part
> of the message.

Ah, thanks for the hint. I've switched mailer to claws-mail-3.0.0 and
hacked the thing a bit so its useful. One of the things I added was a
menu entry for respect_flowed_format. So I can now flip between flowed
and non-flowed with a key.

2007-09-17 14:04:24

by Anton Altaparmakov

[permalink] [raw]

Subject: Re: VM/VFS bug with large amount of memory and file systems?

Attachments:

slabinfo.txt (15.89 kB)
meminfo.txt (728.00 B)
vmstat.txt (808.00 B)
zoneinfo.txt (4.18 kB)
Download all attachments

2007-09-17 14:09:48

by Anton Altaparmakov

[permalink] [raw]

Subject: Re: VM/VFS bug with large amount of memory and file systems?

On 17 Sep 2007, at 15:04, Anton Altaparmakov wrote:
> On 15 Sep 2007, at 11:52, Andrew Morton wrote:
>> On Sat, 15 Sep 2007 12:08:17 +0200 Peter Zijlstra
>> <[email protected]> wrote:
>>> Anyway, looks like all of zone_normal is pinned in kernel
>>> allocations:
>>>
>>>> Sep 13 15:31:25 escabot Normal free:3648kB min:3744kB low:4680kB
>>>> high: 5616kB active:0kB inactive:3160kB present:894080kB
>>>> pages_scanned:5336 all_unreclaimable? yes
>>>
>>> Out of the 870 odd mb only 3 is on the lru.
>>>
>>> Would be grand it you could have a look at slabinfo and the like.
>>
>> Definitely.
>>
>>>> Sep 13 15:31:25 escabot free:1090395 slab:198893 mapped:988
>>>> pagetables:129 bounce:0
>>
>> 814,665,728 bytes of slab.
>
> Marc emailed me the contents of /proc/
> {slabinfo,meminfo,vmstat,zoneinfo} taken just a few seconds before
> the machine panic()ed due to running OOM completely... They files
> are attached this time rather than inlined so people don't complain
> about line wrapping! (No doubt people will not complain about them
> being attached! )-:)
>
> If I read it correctly it appears all of low memory is eaten up by
> buffer_heads.
>
> <quote>
> # name <active_objs> <num_objs> <objsize> <objperslab>
> <pagesperslab>
> : tunables <limit> <batchcount> <sharedfactor> : slabdata
> <active_slabs> <num_s
> labs> <sharedavail>
> buffer_head 12569528 12569535 56 67 1 : tunables
> 120 60 8 :
> slabdata 187605 187605 0
> </quote>
>
> That is 671MiB of low memory in buffer_heads.

I meant that is 732MiB of low memory in buffer_heads. (12569535
num_objs / 67 objperslab * 1 pagesperslab * 4096 PAGE_SIZE)

> But why is the kernel not reclaiming them by getting rid of the
> page cache pages they are attached to or even leaving the pages
> around but killing their buffers?
>
> I don't think I am doing anything in NTFS to cause this problem to
> happen... Other than using buffer heads for my page cache pages
> that is but that is hardly a crime! /-;

Best regards,

Anton
--
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK
Linux NTFS maintainer, http://www.linux-ntfs.org/

2007-09-17 14:33:19

[permalink] [raw]

Subject: Re: VM/VFS bug with large amount of memory and file systems?

On Mon, 17 Sep 2007 15:04:05 +0100 Anton Altaparmakov <[email protected]>
wrote:

> They files
> are attached this time rather than inlined so people don't complain
> about line wrapping! (No doubt people will not complain about them
> being attached! )-:)

I switched mailer after I learnt about flowed stuffs. Still,
appreciated.

> If I read it correctly it appears all of low memory is eaten up by
> buffer_heads.
>
> <quote>
> # name <active_objs> <num_objs> <objsize> <objperslab>
> <pagesperslab>
> : tunables <limit> <batchcount> <sharedfactor> : slabdata
> <active_slabs> <num_s
> labs> <sharedavail>
> buffer_head 12569528 12569535 56 67 1 : tunables
> 120 60 8 :
> slabdata 187605 187605 0
> </quote>
>
> That is 671MiB of low memory in buffer_heads.
>
> But why is the kernel not reclaiming them by getting rid of the page
> cache pages they are attached to or even leaving the pages around but
> killing their buffers?

Well, you see, you have this very odd configuration where:

11GB highmem
1GB normal

pagecache pages go into highmem
buggerheads go into normal

I'm guessing there is no pressure at all on zone_highmem so the
kernel will not try to reclaim pagecache. And because the pagecache
pages are happily sitting there, the buggerheads are pinned and do not
get reclaimed.

2007-09-17 14:49:49

by Nick Piggin

[permalink] [raw]

Subject: Re: VM/VFS bug with large amount of memory and file systems?

On Tuesday 18 September 2007 00:09, Anton Altaparmakov wrote:
> On 17 Sep 2007, at 15:04, Anton Altaparmakov wrote:
> > On 15 Sep 2007, at 11:52, Andrew Morton wrote:
> >> On Sat, 15 Sep 2007 12:08:17 +0200 Peter Zijlstra
> >>
> >> <[email protected]> wrote:
> >>> Anyway, looks like all of zone_normal is pinned in kernel
> >>>
> >>> allocations:
> >>>> Sep 13 15:31:25 escabot Normal free:3648kB min:3744kB low:4680kB
> >>>> high: 5616kB active:0kB inactive:3160kB present:894080kB
> >>>> pages_scanned:5336 all_unreclaimable? yes
> >>>
> >>> Out of the 870 odd mb only 3 is on the lru.
> >>>
> >>> Would be grand it you could have a look at slabinfo and the like.
> >>
> >> Definitely.
> >>
> >>>> Sep 13 15:31:25 escabot free:1090395 slab:198893 mapped:988
> >>>> pagetables:129 bounce:0
> >>
> >> 814,665,728 bytes of slab.
> >
> > Marc emailed me the contents of /proc/
> > {slabinfo,meminfo,vmstat,zoneinfo} taken just a few seconds before
> > the machine panic()ed due to running OOM completely... They files
> > are attached this time rather than inlined so people don't complain
> > about line wrapping! (No doubt people will not complain about them
> > being attached! )-:)
> >
> > If I read it correctly it appears all of low memory is eaten up by
> > buffer_heads.
> >
> > <quote>
> > # name <active_objs> <num_objs> <objsize> <objperslab>
> > <pagesperslab>
> >
> > : tunables <limit> <batchcount> <sharedfactor> : slabdata
> >
> > <active_slabs> <num_s
> > labs> <sharedavail>
> > buffer_head 12569528 12569535 56 67 1 : tunables
> > 120 60 8 :
> > slabdata 187605 187605 0
> > </quote>
> >
> > That is 671MiB of low memory in buffer_heads.
>
> I meant that is 732MiB of low memory in buffer_heads. (12569535
> num_objs / 67 objperslab * 1 pagesperslab * 4096 PAGE_SIZE)

There is a sledgehammer in there which is supposed to alleviate
this problem. vmscan.c has buffer_heads_over_limit (could you check
if that is kicking in? (add a line in mm/page_alloc.c:show_free_areas()
to check it).

However, I'd guess the logic should be pretty robust. So it looks like
highmem is not getting scanned, so the buffer heads are not ever
getting a chance to be stripped off highmem pages!

(Rik has a patch sitting in -mm I believe which would make this problem
even worse, by doing even less highmem scanning in response to lowmem
allocations). However your user isn't using that kernel, so I wonder why it
isn't scanning highmem... it's been a while since I looked at reclaim, but
AFAIR it *should* be scanning it.

2007-09-17 17:05:56

[permalink] [raw]

Subject: Re: VM/VFS bug with large amount of memory and file systems?

Nick Piggin wrote:

> (Rik has a patch sitting in -mm I believe which would make this problem
> even worse, by doing even less highmem scanning in response to lowmem
> allocations).

My patch should not make any difference here, since
balance_pgdat() already scans the zones from high to
low and sets an end_zone variable that determines the
highest zone to scan.

All my patch does is make sure that we do not try to
reclaim excessive amounts of dma or low memory when
a higher zone is full.

--
Politics is the struggle between those who want to make their country
the best in the world, and those who believe it already is. Each group
calls the other unpatriotic.

2007-09-17 17:11:43

[permalink] [raw]

Subject: Re: VM/VFS bug with large amount of memory and file systems?

Peter Zijlstra wrote:
> On Mon, 17 Sep 2007 15:04:05 +0100 Anton Altaparmakov <[email protected]>
> wrote:
>
>> They files
>> are attached this time rather than inlined so people don't complain
>> about line wrapping! (No doubt people will not complain about them
>> being attached! )-:)
>
> I switched mailer after I learnt about flowed stuffs. Still,
> appreciated.
>
>> If I read it correctly it appears all of low memory is eaten up by
>> buffer_heads.
>>
>> <quote>
>> # name <active_objs> <num_objs> <objsize> <objperslab>
>> <pagesperslab>
>> : tunables <limit> <batchcount> <sharedfactor> : slabdata
>> <active_slabs> <num_s
>> labs> <sharedavail>
>> buffer_head 12569528 12569535 56 67 1 : tunables
>> 120 60 8 :
>> slabdata 187605 187605 0
>> </quote>
>>
>> That is 671MiB of low memory in buffer_heads.
>>
>> But why is the kernel not reclaiming them by getting rid of the page
>> cache pages they are attached to or even leaving the pages around but
>> killing their buffers?
>
> Well, you see, you have this very odd configuration where:
>
> 11GB highmem
> 1GB normal
>
> pagecache pages go into highmem
> buggerheads go into normal
>
> I'm guessing there is no pressure at all on zone_highmem so the
> kernel will not try to reclaim pagecache. And because the pagecache
> pages are happily sitting there, the buggerheads are pinned and do not
> get reclaimed.

I've got code for this in RHEL 3, but never bothered to
merge it upstream since I thought people with large memory
systems would be running 64 bit kernels by now.

Obviously I was wrong. Andrew, are you interested in a
fix for this problem?

IIRC I simply kept a list of all buffer heads and walked
that to reclaim pages when the number of buffer heads is
too high (and we need memory). This list can be maintained
in places where we already hold the lock for the buffer head
freelist, so there should be no additional locking overhead
(again, IIRC).

--
Politics is the struggle between those who want to make their country
the best in the world, and those who believe it already is. Each group
calls the other unpatriotic.

2007-09-17 20:16:25

[permalink] [raw]

Subject: Re: VM/VFS bug with large amount of memory and file systems?

On Mon, 17 Sep 2007 13:11:14 -0400
Rik van Riel <[email protected]> wrote:

> > I'm guessing there is no pressure at all on zone_highmem so the
> > kernel will not try to reclaim pagecache. And because the pagecache
> > pages are happily sitting there, the buggerheads are pinned and do not
> > get reclaimed.

yeah, this got pretty unavoidably broken when we killed the global LRU
in 2.5.early. It's odd that it took this long for someone to hit it.

> I've got code for this in RHEL 3, but never bothered to
> merge it upstream since I thought people with large memory
> systems would be running 64 bit kernels by now.
>
> Obviously I was wrong. Andrew, are you interested in a
> fix for this problem?
>
> IIRC I simply kept a list of all buffer heads and walked
> that to reclaim pages when the number of buffer heads is
> too high (and we need memory). This list can be maintained
> in places where we already hold the lock for the buffer head
> freelist, so there should be no additional locking overhead
> (again, IIRC).

Christoph's slab defragmentation code should permit us to fix this:
grab a page of buffer_heads off the slab lists, trylock the page,
strip the buffer_heads. I think that would be a better approach
if we can get it going because it's more general.

2007-09-17 20:47:31

[permalink] [raw]

Subject: Re: VM/VFS bug with large amount of memory and file systems?

Andrew Morton wrote:
> On Mon, 17 Sep 2007 13:11:14 -0400
> Rik van Riel <[email protected]> wrote:

>> IIRC I simply kept a list of all buffer heads and walked
>> that to reclaim pages when the number of buffer heads is
>> too high (and we need memory). This list can be maintained
>> in places where we already hold the lock for the buffer head
>> freelist, so there should be no additional locking overhead
>> (again, IIRC).
>
> Christoph's slab defragmentation code should permit us to fix this:
> grab a page of buffer_heads off the slab lists, trylock the page,
> strip the buffer_heads. I think that would be a better approach
> if we can get it going because it's more general.

Is the slab defragmentation code in -mm or upstream already
or can I find it on the mailing list?

I've implemented code like you describe already, just give me
a few days to become familiar with the slab defragmentation
code and I'll get you a patch.

--
Politics is the struggle between those who want to make their country
the best in the world, and those who believe it already is. Each group
calls the other unpatriotic.

2007-09-17 21:17:54

[permalink] [raw]

Subject: Re: VM/VFS bug with large amount of memory and file systems?

On Mon, 17 Sep 2007 16:46:47 -0400
Rik van Riel <[email protected]> wrote:

> Andrew Morton wrote:
> > On Mon, 17 Sep 2007 13:11:14 -0400
> > Rik van Riel <[email protected]> wrote:
>
> >> IIRC I simply kept a list of all buffer heads and walked
> >> that to reclaim pages when the number of buffer heads is
> >> too high (and we need memory). This list can be maintained
> >> in places where we already hold the lock for the buffer head
> >> freelist, so there should be no additional locking overhead
> >> (again, IIRC).
> >
> > Christoph's slab defragmentation code should permit us to fix this:
> > grab a page of buffer_heads off the slab lists, trylock the page,
> > strip the buffer_heads. I think that would be a better approach
> > if we can get it going because it's more general.
>
> Is the slab defragmentation code in -mm or upstream already
> or can I find it on the mailing list?

Is on lkml and linux-mm: http://lkml.org/lkml/2007/8/31/329

> I've implemented code like you describe already, just give me
> a few days to become familiar with the slab defragmentation
> code and I'll get you a patch.

The patchset does buffer_heads: http://lkml.org/lkml/2007/8/31/348

I think the whole approach is reasonable. It's mainly a matter of going
through it all with a toothcomb and getting it all merged up, tested and
integrated. There's considerable potential for nasty and rarely-occurring
surprises in this stuff because it tends to approach locking in the
reversed order.

<checks the archives>

There were a few desultory comments, but I see no sign that the bulk of
the patches have had any serious review and testing from anyone yet.

2007-09-18 09:23:44

by Nick Piggin

[permalink] [raw]

Subject: Re: VM/VFS bug with large amount of memory and file systems?

On Tuesday 18 September 2007 03:04, Rik van Riel wrote:
> Nick Piggin wrote:
> > (Rik has a patch sitting in -mm I believe which would make this problem
> > even worse, by doing even less highmem scanning in response to lowmem
> > allocations).
>
> My patch should not make any difference here, since
> balance_pgdat() already scans the zones from high to
> low and sets an end_zone variable that determines the
> highest zone to scan.
>
> All my patch does is make sure that we do not try to
> reclaim excessive amounts of dma or low memory when
> a higher zone is full.

Sorry, yeah I had it the wrong way around. Your patch would not
increase the probability of this problem.

We could have some logic in there to scan highmem when buffer
heads are over limit. But that really kind of sucks in that it introduces
some arbitrary point where reclaim behaviour completely changes...
Adding a shrinker for buffer heads is the "logical" approach that we
take for other non-page caches. That also kind of sucks because we
normally don't want to do this out of band buffer reclaiming and just
have it work from page reclaim (it will introduce extra locking and list
scanning).

Maybe when the machine is near OOM, we can just change the scanning
to do all zones -- a change in scanning behaviour at that point is better
than oom kill.

2007-09-18 14:42:38

[permalink] [raw]

Subject: Re: VM/VFS bug with large amount of memory and file systems?

Nick Piggin wrote:
> On Tuesday 18 September 2007 03:04, Rik van Riel wrote:
>> Nick Piggin wrote:
>>> (Rik has a patch sitting in -mm I believe which would make this problem
>>> even worse, by doing even less highmem scanning in response to lowmem
>>> allocations).
>> My patch should not make any difference here, since
>> balance_pgdat() already scans the zones from high to
>> low and sets an end_zone variable that determines the
>> highest zone to scan.
>>
>> All my patch does is make sure that we do not try to
>> reclaim excessive amounts of dma or low memory when
>> a higher zone is full.
>
> Sorry, yeah I had it the wrong way around. Your patch would not
> increase the probability of this problem.
>
> We could have some logic in there to scan highmem when buffer
> heads are over limit. But that really kind of sucks in that it introduces
> some arbitrary point where reclaim behaviour completely changes...
> Adding a shrinker for buffer heads is the "logical" approach

Christoph Lameter's slab defragmenting patch set does
this. One reason Andrew has not merged that code yet
is a lack of reviewers, so I am going through it with
a fine comb and hope to have the patches reviewed by
the end of today.

Lets get this bug fixed the right way.

--
Politics is the struggle between those who want to make their country
the best in the world, and those who believe it already is. Each group
calls the other unpatriotic.

2007-09-19 22:34:24

[permalink] [raw]

Subject: Re: VM/VFS bug with large amount of memory and file systems?

Andrew Morton wrote:
> On Mon, 17 Sep 2007 16:46:47 -0400
> Rik van Riel <[email protected]> wrote:

>> Is the slab defragmentation code in -mm or upstream already
>> or can I find it on the mailing list?
>
> Is on lkml and linux-mm: http://lkml.org/lkml/2007/8/31/329

> I think the whole approach is reasonable. It's mainly a matter of going
> through it all with a toothcomb

I've spent the last two days combing through the patches.

Except for the one doubt I had (resolved in email), and
one function name comment (on patch 18/26) the code looks
good to me.

--
Politics is the struggle between those who want to make their country
the best in the world, and those who believe it already is. Each group
calls the other unpatriotic.

2007-09-19 22:47:08

[permalink] [raw]

Subject: Re: VM/VFS bug with large amount of memory and file systems?

On Wed, 19 Sep 2007 18:12:38 -0400
Rik van Riel <[email protected]> wrote:

> Andrew Morton wrote:
> > On Mon, 17 Sep 2007 16:46:47 -0400
> > Rik van Riel <[email protected]> wrote:
>
> >> Is the slab defragmentation code in -mm or upstream already
> >> or can I find it on the mailing list?
> >
> > Is on lkml and linux-mm: http://lkml.org/lkml/2007/8/31/329
>
> > I think the whole approach is reasonable. It's mainly a matter of going
> > through it all with a toothcomb
>
> I've spent the last two days combing through the patches.

Thanks. So would it be appropriate for Christoph to add the
thus-far-undefined reviewed-by tag to the next version?

> Except for the one doubt I had (resolved in email), and
> one function name comment (on patch 18/26)

yeah. Basic rule: if the reviewer had to ask a question then others will
later ask themselves the same question when reading the code. So this is a
very good indicator that there is a missing code comment.

> the code looks
> good to me.

Cool.

2007-09-19 22:50:46

[permalink] [raw]

Subject: Re: VM/VFS bug with large amount of memory and file systems?

Andrew Morton wrote:
> On Wed, 19 Sep 2007 18:12:38 -0400
> Rik van Riel <[email protected]> wrote:
>
>> Andrew Morton wrote:
>>> On Mon, 17 Sep 2007 16:46:47 -0400
>>> Rik van Riel <[email protected]> wrote:
>>>> Is the slab defragmentation code in -mm or upstream already
>>>> or can I find it on the mailing list?
>>> Is on lkml and linux-mm: http://lkml.org/lkml/2007/8/31/329
>>> I think the whole approach is reasonable. It's mainly a matter of going
>>> through it all with a toothcomb
>> I've spent the last two days combing through the patches.
>
> Thanks. So would it be appropriate for Christoph to add the
> thus-far-undefined reviewed-by tag to the next version?

Absolutely. Some of the code was not obvious at first,
but the more I looked at it the better it looked - that's
always a good sign, since most of the code seems to be
the other way around :)

--
Politics is the struggle between those who want to make their country
the best in the world, and those who believe it already is. Each group
calls the other unpatriotic.

2007-09-19 23:29:18

by Christoph Lameter

[permalink] [raw]

Subject: Re: VM/VFS bug with large amount of memory and file systems?

On Wed, 19 Sep 2007, Andrew Morton wrote:

> Cool.

Should I rediff on top of rc6-mm1 for submission? When will you be able to
take it?

2007-09-19 23:38:39