Within the last few weeks I'm observing sometimes a swap storm at my ThinkPad
while compiling/installing new packages at my Gentoo Linux - the load is often
something like :
Load avg: 13.6, 20.6, 20.9
I'm wondering whether this is related to kernel 3.2.x /Gentoo specific or
related to my system only.
--
MfG/Sincerely
Toralf Förster
pgp finger print: 7B1A 07F4 EC82 0F90 D4C2 8936 872A E508 7DB6 9DA3
On Sat, Feb 04, 2012 at 11:09:52AM +0100, Toralf F?rster wrote:
> Within the last few weeks I'm observing sometimes a swap storm at my ThinkPad
> while compiling/installing new packages at my Gentoo Linux - the load is often
> something like :
>
> Load avg: 13.6, 20.6, 20.9
>
> I'm wondering whether this is related to kernel 3.2.x /Gentoo specific or
> related to my system only.
Do you happen to have CONFIG_DEBUG_OBJECTS enabled? For me it
ate lots of memory with 3.2.2, easily visible in slabtop.
Johannes
Johannes Stezenbach wrote at 14:33:31
> On Sat, Feb 04, 2012 at 11:09:52AM +0100, Toralf F?rster wrote:
> > Within the last few weeks I'm observing sometimes a swap storm at my
> > ThinkPad while compiling/installing new packages at my Gentoo Linux -
> > the load is often something like :
> >
> > Load avg: 13.6, 20.6, 20.9
> >
> > I'm wondering whether this is related to kernel 3.2.x /Gentoo specific or
> > related to my system only.
>
> Do you happen to have CONFIG_DEBUG_OBJECTS enabled? For me it
> ate lots of memory with 3.2.2, easily visible in slabtop.
>
>
> Johannes
No, I've these settings :
tfoerste@n22 ~ $ zgrep -e OBJ -e SLAB -e SLUB /proc/config.gz | grep -v '#'
CONFIG_SLUB_DEBUG=y
CONFIG_SLUB=y
CONFIG_SLABINFO=y
--
MfG/Sincerely
Toralf F?rster
pgp finger print: 7B1A 07F4 EC82 0F90 D4C2 8936 872A E508 7DB6 9DA3
2012/2/4 Toralf Förster <[email protected]>:
>
> Johannes Stezenbach wrote at 14:33:31
>> On Sat, Feb 04, 2012 at 11:09:52AM +0100, Toralf Förster wrote:
>> > Within the last few weeks I'm observing sometimes a swap storm at my
>> > ThinkPad while compiling/installing new packages at my Gentoo Linux -
>> > the load is often something like :
>> >
>> > Load avg: 13.6, 20.6, 20.9
>> >
>> > I'm wondering whether this is related to kernel 3.2.x /Gentoo specific or
>> > related to my system only.
>>
>> Do you happen to have CONFIG_DEBUG_OBJECTS enabled? For me it
>> ate lots of memory with 3.2.2, easily visible in slabtop.
>>
>>
>> Johannes
>
> No, I've these settings :
>
> tfoerste@n22 ~ $ zgrep -e OBJ -e SLAB -e SLUB /proc/config.gz | grep -v '#'
> CONFIG_SLUB_DEBUG=y
> CONFIG_SLUB=y
> CONFIG_SLABINFO=y
>
Would you please try the patchset of Rik?
https://lkml.org/lkml/2012/1/26/374
Good weekend
Hillf
Hillf Danton wrote at 05:45:40
> Would you please try the patchset of Rik?
>
> https://lkml.org/lkml/2012/1/26/374
It doesn't applied successfully agains 3.2.3 (+patch +f 3.2.5)
:-(
--
MfG/Sincerely
Toralf Förster
pgp finger print: 7B1A 07F4 EC82 0F90 D4C2 8936 872A E508 7DB6 9DA3
2012/2/5 Toralf Förster <[email protected]>:
>
> Hillf Danton wrote at 05:45:40
>> Would you please try the patchset of Rik?
>>
>> https://lkml.org/lkml/2012/1/26/374
>
> It doesn't applied successfully agains 3.2.3 (+patch +f 3.2.5)
> :-(
>
That patchset already in -next tree, mind to try it with
CONFIG_SLUB_DEBUG first disabled, and try again with it enabled?
Hillf
Hillf Danton wrote at 12:38:31
> 2012/2/5 Toralf Förster <[email protected]>:
> > Hillf Danton wrote at 05:45:40
> >
> >> Would you please try the patchset of Rik?
> >>
> >> https://lkml.org/lkml/2012/1/26/374
> >
> > It doesn't applied successfully agains 3.2.3 (+patch +f 3.2.5)
> >
> > :-(
>
> That patchset already in -next tree, mind to try it with
> CONFIG_SLUB_DEBUG first disabled, and try again with it enabled?
>
> Hillf
I switched back to 3.0.20 in the mean while.
>From what I can tell is this:
If the system is under heavy I/O load and hasn't too much free RAM (git pull,
svn update and RAM consuming BOINC applications) then kernel 3.0.20 handle
this somehow while 3.2.x run into a swap storm like.
--
MfG/Sincerely
Toralf Förster
pgp finger print: 7B1A 07F4 EC82 0F90 D4C2 8936 872A E508 7DB6 9DA3
On Wed, Feb 08, 2012 at 09:56:15AM +0100, Toralf F?rster wrote:
>
> From what I can tell is this:
> If the system is under heavy I/O load and hasn't too much free RAM (git pull,
> svn update and RAM consuming BOINC applications) then kernel 3.0.20 handle
> this somehow while 3.2.x run into a swap storm like.
FWIW, I also saw heavy swapping with 3.2.2 with the
CONFIG_DEBUG_OBJECTS issue reported here:
http://lkml.org/lkml/2012/1/30/227
But the thing is that even though SUnreclaim was
huge there was still 1G MemFree and it swapped heavily
on idle system when just switching between e.g. Firefox and gvim.
Today I'm running 3.2.4 with CONFIG_DEBUG_OBJECTS disabled
(but otherwise the same config) and it doesn't swap even
after a fair amount of I/O:
MemTotal: 3940088 kB
MemFree: 1024920 kB
Buffers: 293328 kB
Cached: 447796 kB
SwapCached: 24 kB
Active: 847136 kB
Inactive: 567200 kB
Active(anon): 478736 kB
Inactive(anon): 246744 kB
Active(file): 368400 kB
Inactive(file): 320456 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 3903484 kB
SwapFree: 3903196 kB
Dirty: 16 kB
Writeback: 0 kB
AnonPages: 673192 kB
Mapped: 40956 kB
Shmem: 52268 kB
Slab: 1434188 kB
SReclaimable: 1367388 kB
SUnreclaim: 66800 kB
KernelStack: 1600 kB
PageTables: 4880 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 5873528 kB
Committed_AS: 1744916 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 348116 kB
VmallocChunk: 34359362739 kB
DirectMap4k: 12288 kB
DirectMap2M: 4098048 kB
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
586182 353006 60% 1.74K 32595 18 1043040K ext3_inode_cache
289062 170979 59% 0.58K 10706 27 171296K dentry
247266 107729 43% 0.42K 13737 18 109896K buffer_head
Johannes
2012/2/8 Johannes Stezenbach <[email protected]>:
> On Wed, Feb 08, 2012 at 09:56:15AM +0100, Toralf Förster wrote:
>>
>> From what I can tell is this:
>> If the system is under heavy I/O load and hasn't too much free RAM (git pull,
>> svn update and RAM consuming BOINC applications) then kernel 3.0.20 handle
>> this somehow while 3.2.x run into a swap storm like.
>
> FWIW, I also saw heavy swapping with 3.2.2 with the
> CONFIG_DEBUG_OBJECTS issue reported here:
> http://lkml.org/lkml/2012/1/30/227
>
> But the thing is that even though SUnreclaim was
> huge there was still 1G MemFree and it swapped heavily
> on idle system when just switching between e.g. Firefox and gvim.
>
> Today I'm running 3.2.4 with CONFIG_DEBUG_OBJECTS disabled
> (but otherwise the same config) and it doesn't swap even
> after a fair amount of I/O:
Hah, looks not related to kswapd directly;)
>
> MemTotal: 3940088 kB
> MemFree: 1024920 kB
> Buffers: 293328 kB
> Cached: 447796 kB
> SwapCached: 24 kB
> Active: 847136 kB
> Inactive: 567200 kB
> Active(anon): 478736 kB
> Inactive(anon): 246744 kB
> Active(file): 368400 kB
> Inactive(file): 320456 kB
> Unevictable: 0 kB
> Mlocked: 0 kB
> SwapTotal: 3903484 kB
> SwapFree: 3903196 kB
> Dirty: 16 kB
> Writeback: 0 kB
> AnonPages: 673192 kB
> Mapped: 40956 kB
> Shmem: 52268 kB
> Slab: 1434188 kB
> SReclaimable: 1367388 kB
> SUnreclaim: 66800 kB
> KernelStack: 1600 kB
> PageTables: 4880 kB
> NFS_Unstable: 0 kB
> Bounce: 0 kB
> WritebackTmp: 0 kB
> CommitLimit: 5873528 kB
> Committed_AS: 1744916 kB
> VmallocTotal: 34359738367 kB
> VmallocUsed: 348116 kB
> VmallocChunk: 34359362739 kB
> DirectMap4k: 12288 kB
> DirectMap2M: 4098048 kB
>
> OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
> 586182 353006 60% 1.74K 32595 18 1043040K ext3_inode_cache
> 289062 170979 59% 0.58K 10706 27 171296K dentry
> 247266 107729 43% 0.42K 13737 18 109896K buffer_head
>
>
And I want to ask kswapd to do less work, the attached diff is
based on 3.2.5, mind to test it with CONFIG_DEBUG_OBJECTS enabled?
Thanks
Hillf
--- a/mm/vmscan.c Wed Feb 8 20:10:14 2012
+++ b/mm/vmscan.c Wed Feb 8 20:15:22 2012
@@ -2113,8 +2113,11 @@ restart:
* with multiple processes reclaiming pages, the total
* freeing target can get unreasonably large.
*/
- if (nr_reclaimed >= nr_to_reclaim && priority < DEF_PRIORITY)
+ if (nr_reclaimed >= nr_to_reclaim) {
+ nr_to_reclaim = 0;
break;
+ }
+ nr_to_reclaim -= nr_reclaimed;
}
blk_finish_plug(&plug);
sc->nr_reclaimed += nr_reclaimed;
@@ -2683,12 +2686,12 @@ static unsigned long balance_pgdat(pg_da
* we want to put equal scanning pressure on each zone.
*/
.nr_to_reclaim = ULONG_MAX,
- .order = order,
.target_mem_cgroup = NULL,
};
struct shrink_control shrink = {
.gfp_mask = sc.gfp_mask,
};
+ sc.order = order = 0;
loop_again:
total_scanned = 0;
sc.nr_reclaimed = 0;
--
On Wed, Feb 08, 2012 at 08:34:14PM +0800, Hillf Danton wrote:
> And I want to ask kswapd to do less work, the attached diff is
> based on 3.2.5, mind to test it with CONFIG_DEBUG_OBJECTS enabled?
Sorry, for slow reply. The patch does not apply to 3.2.4
(3.2.5 only has the ASPM change which I don't want to
try atm). Is the patch below correct?
I'll let this run for a while and will report back.
Thanks
Johannes
--- mm/vmscan.c.orig 2012-02-03 21:39:51.000000000 +0100
+++ mm/vmscan.c 2012-02-09 12:30:42.000000000 +0100
@@ -2067,8 +2067,11 @@ restart:
* with multiple processes reclaiming pages, the total
* freeing target can get unreasonably large.
*/
- if (nr_reclaimed >= nr_to_reclaim && priority < DEF_PRIORITY)
+ if (nr_reclaimed >= nr_to_reclaim) {
+ nr_to_reclaim = 0;
break;
+ }
+ nr_to_reclaim -= nr_reclaimed;
}
blk_finish_plug(&plug);
sc->nr_reclaimed += nr_reclaimed;
@@ -2535,12 +2538,12 @@ static unsigned long balance_pgdat(pg_da
* we want to put equal scanning pressure on each zone.
*/
.nr_to_reclaim = ULONG_MAX,
- .order = order,
.mem_cgroup = NULL,
};
struct shrink_control shrink = {
.gfp_mask = sc.gfp_mask,
};
+ sc.order = order = 0;
loop_again:
total_scanned = 0;
sc.nr_reclaimed = 0;
On Thu, Feb 9, 2012 at 7:36 PM, Johannes Stezenbach <[email protected]> wrote:
> On Wed, Feb 08, 2012 at 08:34:14PM +0800, Hillf Danton wrote:
>> And I want to ask kswapd to do less work, the attached diff is
>> based on 3.2.5, mind to test it with CONFIG_DEBUG_OBJECTS enabled?
>
> Sorry, for slow reply. The patch does not apply to 3.2.4
> (3.2.5 only has the ASPM change which I don't want to
> try atm). Is the patch below correct?
>
It is fine;)
Thanks
Hillf
> I'll let this run for a while and will report back.
>
> Thanks
> Johannes
>
>
> --- mm/vmscan.c.orig 2012-02-03 21:39:51.000000000 +0100
> +++ mm/vmscan.c 2012-02-09 12:30:42.000000000 +0100
> @@ -2067,8 +2067,11 @@ restart:
> * with multiple processes reclaiming pages, the total
> * freeing target can get unreasonably large.
> */
> - if (nr_reclaimed >= nr_to_reclaim && priority < DEF_PRIORITY)
> + if (nr_reclaimed >= nr_to_reclaim) {
> + nr_to_reclaim = 0;
> break;
> + }
> + nr_to_reclaim -= nr_reclaimed;
> }
> blk_finish_plug(&plug);
> sc->nr_reclaimed += nr_reclaimed;
> @@ -2535,12 +2538,12 @@ static unsigned long balance_pgdat(pg_da
> * we want to put equal scanning pressure on each zone.
> */
> .nr_to_reclaim = ULONG_MAX,
> - .order = order,
> .mem_cgroup = NULL,
> };
> struct shrink_control shrink = {
> .gfp_mask = sc.gfp_mask,
> };
> + sc.order = order = 0;
> loop_again:
> total_scanned = 0;
> sc.nr_reclaimed = 0;
Johannes Stezenbach wrote at 12:36:06
> On Wed, Feb 08, 2012 at 08:34:14PM +0800, Hillf Danton wrote:
> > And I want to ask kswapd to do less work, the attached diff is
> > based on 3.2.5, mind to test it with CONFIG_DEBUG_OBJECTS enabled?
>
> Sorry, for slow reply. The patch does not apply to 3.2.4
> (3.2.5 only has the ASPM change which I don't want to
> try atm). Is the patch below correct?
>
confirmed - doesn't apply here too :-(
--
MfG/Sincerely
Toralf F?rster
pgp finger print: 7B1A 07F4 EC82 0F90 D4C2 8936 872A E508 7DB6 9DA3
On Thu, Feb 09, 2012 at 08:02:20PM +0800, Hillf Danton wrote:
> On Thu, Feb 9, 2012 at 7:36 PM, Johannes Stezenbach <[email protected]> wrote:
> > On Wed, Feb 08, 2012 at 08:34:14PM +0800, Hillf Danton wrote:
> >> And I want to ask kswapd to do less work, the attached diff is
> >> based on 3.2.5, mind to test it with CONFIG_DEBUG_OBJECTS enabled?
> >
> > Sorry, for slow reply. ?The patch does not apply to 3.2.4
> > (3.2.5 only has the ASPM change which I don't want to
> > try atm). ?Is the patch below correct?
> >
>
> It is fine;)
Hm, with 3.2.4 + patch +
CONFIG_DEBUG_OBJECTS=y
# CONFIG_DEBUG_OBJECTS_SELFTEST is not set
# CONFIG_DEBUG_OBJECTS_FREE is not set
CONFIG_DEBUG_OBJECTS_TIMERS=y
CONFIG_DEBUG_OBJECTS_WORK=y
CONFIG_DEBUG_OBJECTS_RCU_HEAD=y
CONFIG_DEBUG_OBJECTS_PERCPU_COUNTER=y
CONFIG_DEBUG_OBJECTS_ENABLE_DEFAULT=1
it looks good. Neither do I get the huge debug_objects_cache
nor does it swap, after running a crosstool-ng toolchain build.
Well, last time I also had one kvm -m 1G instance running. I'll
try if that triggers the issue. So far:
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
689249 689235 99% 0.36K 31334 22 250672K debug_objects_cache
625185 609295 97% 0.42K 34735 18 277880K buffer_head
103834 103393 99% 1.74K 7245 18 231840K ext3_inode_cache
84348 82351 97% 0.58K 3124 27 49984K dentry
MemTotal: 3938800 kB
MemFree: 77136 kB
Buffers: 68892 kB
Cached: 2686376 kB
SwapCached: 8 kB
Active: 1343464 kB
Inactive: 1584476 kB
Active(anon): 78712 kB
Inactive(anon): 145220 kB
Active(file): 1264752 kB
Inactive(file): 1439256 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 3903484 kB
SwapFree: 3903248 kB
Dirty: 64 kB
Writeback: 0 kB
AnonPages: 172676 kB
Mapped: 41868 kB
Shmem: 51260 kB
Slab: 872400 kB
SReclaimable: 549904 kB
SUnreclaim: 322496 kB
KernelStack: 1432 kB
PageTables: 3172 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 5872884 kB
Committed_AS: 474604 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 345800 kB
VmallocChunk: 34359386531 kB
DirectMap4k: 12288 kB
DirectMap2M: 4098048 kB
Thanks,
Johannes
On Thu, Feb 09, 2012 at 02:21:55PM +0100, Johannes Stezenbach wrote:
> it looks good. Neither do I get the huge debug_objects_cache
> nor does it swap, after running a crosstool-ng toolchain build.
> Well, last time I also had one kvm -m 1G instance running. I'll
> try if that triggers the issue. So far:
The kvm produced a bunch of page allocation failures
on the host:
[ 6496.165268] kvm: page allocation failure: order:1, mode:0x4020
[ 6496.165273] Pid: 15322, comm: kvm Not tainted 3.2.4 #2
[ 6496.165274] Call Trace:
[ 6496.165276] <IRQ> [<ffffffff810cec13>] warn_alloc_failed+0x115/0x12a
[ 6496.165286] [<ffffffff810d0c11>] __alloc_pages_nodemask+0x5f5/0x697
[ 6496.165291] [<ffffffff810fcc4b>] new_slab+0x96/0x1ef
[ 6496.165295] [<ffffffff8155b517>] __slab_alloc.isra.52.constprop.59+0x245/0x349
[ 6496.165299] [<ffffffff8146f7e9>] ? skb_segment+0x213/0x50e
[ 6496.165302] [<ffffffff810fc14b>] ? deactivate_slab+0x45b/0x47e
[ 6496.165306] [<ffffffff81008c1f>] ? sched_clock+0x9/0xd
[ 6496.165309] [<ffffffff810fe708>] __kmalloc_track_caller+0xb0/0x174
[ 6496.165312] [<ffffffff8146f7e9>] ? skb_segment+0x213/0x50e
[ 6496.165315] [<ffffffff8146dea5>] __alloc_skb+0x66/0x127
[ 6496.165318] [<ffffffff8146f7e9>] skb_segment+0x213/0x50e
[ 6496.165323] [<ffffffff814d6c69>] tcp_tso_segment+0x2cd/0x2e2
[ 6496.165327] [<ffffffff814f4200>] inet_gso_segment+0x162/0x283
[ 6496.165330] [<ffffffff814f417c>] ? inet_gso_segment+0xde/0x283
[ 6496.165333] [<ffffffff814769aa>] skb_gso_segment+0x24e/0x2fb
[ 6496.165335] [<ffffffff814768f7>] ? skb_gso_segment+0x19b/0x2fb
[ 6496.165338] [<ffffffff81479511>] ? dev_hard_start_xmit+0x20b/0x658
[ 6496.165341] [<ffffffff81008bd8>] ? native_sched_clock+0x35/0x73
[ 6496.165344] [<ffffffff81008c1f>] ? sched_clock+0x9/0xd
[ 6496.165348] [<ffffffff8107ae56>] ? put_lock_stats.isra.15+0xe/0x29
[ 6496.165351] [<ffffffff8107af09>] ? lock_release_holdtime.part.16+0x98/0xa1
[ 6496.165354] [<ffffffff81479511>] ? dev_hard_start_xmit+0x20b/0x658
[ 6496.165357] [<ffffffff814796be>] dev_hard_start_xmit+0x3b8/0x658
[ 6496.165360] [<ffffffff81479375>] ? dev_hard_start_xmit+0x6f/0x658
[ 6496.165364] [<ffffffff8148e0f4>] sch_direct_xmit+0x71/0x204
[ 6496.165367] [<ffffffff81479d96>] dev_queue_xmit+0x438/0x721
[ 6496.165370] [<ffffffff8147995e>] ? dev_hard_start_xmit+0x658/0x658
[ 6496.165374] [<ffffffff81517623>] br_dev_queue_push_xmit+0x7d/0x84
[ 6496.165377] [<ffffffff8151767b>] br_forward_finish+0x51/0x58
[ 6496.165380] [<ffffffff8151777c>] __br_deliver+0x54/0x5b
[ 6496.165383] [<ffffffff815177aa>] br_deliver+0x27/0x33
[ 6496.165386] [<ffffffff81515ec0>] br_dev_xmit+0x170/0x1ab
[ 6496.165389] [<ffffffff81515dd9>] ? br_dev_xmit+0x89/0x1ab
[ 6496.165392] [<ffffffff8147975f>] dev_hard_start_xmit+0x459/0x658
[ 6496.165395] [<ffffffff81479375>] ? dev_hard_start_xmit+0x6f/0x658
[ 6496.165398] [<ffffffff81479f31>] dev_queue_xmit+0x5d3/0x721
[ 6496.165400] [<ffffffff8147995e>] ? dev_hard_start_xmit+0x658/0x658
[ 6496.165403] [<ffffffff814ceee9>] ip_finish_output+0x2b5/0x378
[ 6496.165406] [<ffffffff814cee42>] ? ip_finish_output+0x20e/0x378
[ 6496.165408] [<ffffffff814d034d>] ip_output+0xaf/0xcc
[ 6496.165411] [<ffffffff814cfae8>] ip_local_out+0x4f/0x66
[ 6496.165413] [<ffffffff814cfe46>] ip_queue_xmit+0x347/0x3c7
[ 6496.165415] [<ffffffff814cfaff>] ? ip_local_out+0x66/0x66
[ 6496.165417] [<ffffffff8107340d>] ? getnstimeofday+0x61/0xb2
[ 6496.165421] [<ffffffff814e259c>] tcp_transmit_skb+0x6c4/0x6ff
[ 6496.165424] [<ffffffff814e3223>] tcp_write_xmit+0x7bb/0x8cf
[ 6496.165427] [<ffffffff814e3391>] __tcp_push_pending_frames+0x26/0xab
[ 6496.165430] [<ffffffff814e06b6>] tcp_rcv_established+0x10e/0x5d0
[ 6496.165433] [<ffffffff814e5e4d>] tcp_v4_do_rcv+0xc5/0x39f
[ 6496.165438] [<ffffffff81562bbe>] ? _raw_spin_lock_nested+0x70/0x79
[ 6496.165441] [<ffffffff814e8050>] ? tcp_v4_rcv+0x370/0x87a
[ 6496.165444] [<ffffffff81487abd>] ? sk_filter+0xb8/0xc3
[ 6496.165447] [<ffffffff814e8217>] tcp_v4_rcv+0x537/0x87a
[ 6496.165450] [<ffffffff814cb714>] ip_local_deliver_finish+0x124/0x19a
[ 6496.165453] [<ffffffff814cb628>] ? ip_local_deliver_finish+0x38/0x19a
[ 6496.165456] [<ffffffff814cb8e8>] ip_local_deliver+0x7a/0x81
[ 6496.165459] [<ffffffff814cb58e>] ip_rcv_finish+0x2ba/0x31c
[ 6496.165462] [<ffffffff814cbb28>] ip_rcv+0x239/0x261
[ 6496.165465] [<ffffffff81512d27>] ? packet_rcv_spkt+0x136/0x141
[ 6496.165468] [<ffffffff81474bfb>] __netif_receive_skb+0x2c8/0x34b
[ 6496.165470] [<ffffffff814749d0>] ? __netif_receive_skb+0x9d/0x34b
[ 6496.165473] [<ffffffff8147607e>] netif_receive_skb+0xd7/0xde
[ 6496.165476] [<ffffffff81475fdb>] ? netif_receive_skb+0x34/0xde
[ 6496.165479] [<ffffffff81518124>] ? br_handle_local_finish+0x44/0x44
[ 6496.165482] [<ffffffff81518353>] br_handle_frame_finish+0x22f/0x246
[ 6496.165486] [<ffffffff8151d87a>] br_nf_pre_routing_finish+0x2f2/0x317
[ 6496.165489] [<ffffffff8151de48>] br_nf_pre_routing+0x5a9/0x5be
[ 6496.165492] [<ffffffff814932c4>] nf_iterate+0x48/0x7d
[ 6496.165495] [<ffffffff81518124>] ? br_handle_local_finish+0x44/0x44
[ 6496.165498] [<ffffffff81493392>] nf_hook_slow+0x99/0x14b
[ 6496.165501] [<ffffffff81518124>] ? br_handle_local_finish+0x44/0x44
[ 6496.165504] [<ffffffff81518571>] br_handle_frame+0x207/0x232
[ 6496.165507] [<ffffffff8151836a>] ? br_handle_frame_finish+0x246/0x246
[ 6496.165510] [<ffffffff81474b3e>] __netif_receive_skb+0x20b/0x34b
[ 6496.165512] [<ffffffff814749d0>] ? __netif_receive_skb+0x9d/0x34b
[ 6496.165516] [<ffffffff8147846c>] process_backlog+0xbe/0x17f
[ 6496.165518] [<ffffffff814781fe>] net_rx_action+0x91/0x241
[ 6496.165522] [<ffffffff81053954>] __do_softirq+0xf7/0x228
[ 6496.165525] [<ffffffff815659bc>] call_softirq+0x1c/0x30
[ 6496.165527] <EOI> [<ffffffff81003756>] do_softirq+0x4b/0xa5
[ 6496.165531] [<ffffffff81476732>] netif_rx_ni+0x34/0x5e
[ 6496.165535] [<ffffffff813ad8de>] tun_get_user+0x3b6/0x3e4
[ 6496.165538] [<ffffffff813adda8>] tun_chr_aio_write+0x6c/0x87
[ 6496.165541] [<ffffffff813add3c>] ? tun_chr_poll+0xdb/0xdb
[ 6496.165545] [<ffffffff81106a1c>] do_sync_readv_writev+0xbc/0x101
[ 6496.165549] [<ffffffff81107bfc>] ? fget_light+0xe8/0x11d
[ 6496.165553] [<ffffffff81146685>] compat_do_readv_writev+0xf5/0x1bd
[ 6496.165556] [<ffffffff8107af09>] ? lock_release_holdtime.part.16+0x98/0xa1
[ 6496.165559] [<ffffffff81107bfc>] ? fget_light+0xe8/0x11d
[ 6496.165562] [<ffffffff81146795>] compat_writev+0x48/0x6f
[ 6496.165565] [<ffffffff81146e91>] compat_sys_writev+0x45/0x64
[ 6496.165568] [<ffffffff81565a80>] sysenter_dispatch+0x7/0x37
[ 6496.165571] [<ffffffff812857de>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[ 6496.165573] Mem-Info:
[ 6496.165574] DMA per-cpu:
[ 6496.165576] CPU 0: hi: 0, btch: 1 usd: 0
[ 6496.165578] CPU 1: hi: 0, btch: 1 usd: 0
[ 6496.165579] CPU 2: hi: 0, btch: 1 usd: 0
[ 6496.165581] CPU 3: hi: 0, btch: 1 usd: 0
[ 6496.165582] DMA32 per-cpu:
[ 6496.165584] CPU 0: hi: 186, btch: 31 usd: 0
[ 6496.165586] CPU 1: hi: 186, btch: 31 usd: 0
[ 6496.165587] CPU 2: hi: 186, btch: 31 usd: 1
[ 6496.165589] CPU 3: hi: 186, btch: 31 usd: 0
[ 6496.165590] Normal per-cpu:
[ 6496.165591] CPU 0: hi: 186, btch: 31 usd: 0
[ 6496.165593] CPU 1: hi: 186, btch: 31 usd: 23
[ 6496.165595] CPU 2: hi: 186, btch: 31 usd: 47
[ 6496.165596] CPU 3: hi: 186, btch: 31 usd: 0
[ 6496.165600] active_anon:72439 inactive_anon:49028 isolated_anon:0
[ 6496.165601] active_file:302270 inactive_file:313604 isolated_file:0
[ 6496.165602] unevictable:0 dirty:234 writeback:0 unstable:0
[ 6496.165602] free:10617 slab_reclaimable:137832 slab_unreclaimable:83412
[ 6496.165603] mapped:10536 shmem:8789 pagetables:1023 bounce:0
[ 6496.165609] DMA free:15680kB min:28kB low:32kB high:40kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB un
[ 6496.165613] lowmem_reserve[]: 0 3161 3915 3915
[ 6496.165621] DMA32 free:24592kB min:6452kB low:8064kB high:9676kB active_anon:117596kB inactive_anon:23728kB active_file:1129544
[ 6496.165626] lowmem_reserve[]: 0 0 754 754
[ 6496.165634] Normal free:2196kB min:1536kB low:1920kB high:2304kB active_anon:172160kB inactive_anon:172384kB active_file:79536k
[ 6496.165638] lowmem_reserve[]: 0 0 0 0
[ 6496.165642] DMA: 0*4kB 2*8kB 1*16kB 1*32kB 0*64kB 2*128kB 2*256kB 1*512kB 2*1024kB 2*2048kB 2*4096kB = 15680kB
[ 6496.165652] DMA32: 5856*4kB 58*8kB 25*16kB 3*32kB 0*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 24768kB
[ 6496.165661] Normal: 559*4kB 8*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2316kB
[ 6496.165671] 624680 total pagecache pages
[ 6496.165672] 0 pages in swap cache
[ 6496.165674] Swap cache stats: add 1427, delete 1427, find 732/869
[ 6496.165675] Free swap = 3903308kB
[ 6496.165677] Total swap = 3903484kB
[ 6496.175804] 1048048 pages RAM
[ 6496.175806] 63348 pages reserved
[ 6496.175808] 561973 pages shared
[ 6496.175809] 441897 pages non-shared
[ 6496.175813] SLUB: Unable to allocate memory on node -1 (gfp=0x20)
[ 6496.175815] cache: kmalloc-4096, object size: 4096, buffer size: 4424, default order: 3, min order: 1
[ 6496.175817] kmalloc-4096 debugging increased min order, use slub_debug=O to disable.
[ 6496.175819] node 0: slabs: 44, objs: 302, free: 0
(repeats a few times)
I guess that is expected with your patch?
MemTotal: 3938800 kB
MemFree: 115236 kB
Buffers: 74016 kB
Cached: 1643088 kB
SwapCached: 7532 kB
Active: 1501576 kB
Inactive: 1388668 kB
Active(anon): 841280 kB
Inactive(anon): 366456 kB
Active(file): 660296 kB
Inactive(file): 1022212 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 3903484 kB
SwapFree: 3864724 kB
Dirty: 12 kB
Writeback: 0 kB
AnonPages: 1165632 kB
Mapped: 27176 kB
Shmem: 34596 kB
Slab: 864524 kB
SReclaimable: 512968 kB
SUnreclaim: 351556 kB
KernelStack: 1568 kB
PageTables: 5868 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 5872884 kB
Committed_AS: 1588204 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 348068 kB
VmallocChunk: 34359372819 kB
DirectMap4k: 12288 kB
DirectMap2M: 4098048 kB
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
774103 774090 99% 0.36K 35198 22 281584K debug_objects_cache
358596 204935 57% 0.42K 19923 18 159384K buffer_head
148306 144221 97% 1.74K 12081 18 386592K ext3_inode_cache
75360 69344 92% 0.58K 2795 27 44720K dentry
Thanks
Johannes
On 02/09/2012 08:54 AM, Johannes Stezenbach wrote:
> On Thu, Feb 09, 2012 at 02:21:55PM +0100, Johannes Stezenbach wrote:
>> it looks good. Neither do I get the huge debug_objects_cache
>> nor does it swap, after running a crosstool-ng toolchain build.
>> Well, last time I also had one kvm -m 1G instance running. I'll
>> try if that triggers the issue. So far:
>
> The kvm produced a bunch of page allocation failures
> on the host:
> (repeats a few times)
>
> I guess that is expected with your patch?
That is indeed why my patch series was significantly larger
than Hillf's little test patch :)
--
All rights reversed
On Thu, Feb 9, 2012 at 10:06 PM, Rik van Riel <[email protected]> wrote:
> On 02/09/2012 08:54 AM, Johannes Stezenbach wrote:
>>
>> On Thu, Feb 09, 2012 at 02:21:55PM +0100, Johannes Stezenbach wrote:
>>>
>>> it looks good. Neither do I get the huge debug_objects_cache
>>> nor does it swap, after running a crosstool-ng toolchain build.
>>> Well, last time I also had one kvm -m 1G instance running. I'll
>>> try if that triggers the issue. So far:
>>
>>
>> The kvm produced a bunch of page allocation failures
>> on the host:
>
>
>> (repeats a few times)
>>
>> I guess that is expected with your patch?
>
>
> That is indeed why my patch series was significantly larger
> than Hillf's little test patch :)
>
Yes, Johannes did show that kswapd running purely in single
mode, without helps from compaction or lumpy, could not
meet all requests in VM core.
Thank you all
Hillf