Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752693Ab1EPI2V (ORCPT ); Mon, 16 May 2011 04:28:21 -0400 Received: from mail-wy0-f174.google.com ([74.125.82.174]:33298 "EHLO mail-wy0-f174.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752339Ab1EPI2T convert rfc822-to-8bit (ORCPT ); Mon, 16 May 2011 04:28:19 -0400 DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:content-transfer-encoding; b=dr5U+mK4iUhOAJVnxSn8I5x9GYv3Ezxbh1euWhxUY7RwFf/nNNXGdGORcvYsFquZ7B h0a/hGpP5c8lo7LiwUqfwYyqmzTm+go8+Y8hAxVXmRzOcBRiLPaKoWnSTPtN3hnBr/QV mTPONBg2QxA4HVGkwgei2pNloFNK5WDgmqvwU= MIME-Version: 1.0 In-Reply-To: References: Date: Mon, 16 May 2011 10:28:17 +0200 Message-ID: Subject: Re: Kernel 2.6.38.6 page allocation failure (ixgbe) From: Stefan Majer To: "e1000-devel@lists.sourceforge.net" , "netdev@vger.kernel.org" , linux-kernel@vger.kernel.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8BIT Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 40895 Lines: 828 Hi, after enlarging vm.min_free_kbytes to 524288 we survived almost a week, but today i got this again: May 16 09:18:13 os03 kernel: [331036.332001] kworker/0:1: page allocation failure. order:2, mode:0x4020 May 16 09:18:13 os03 kernel: [331036.332005] Pid: 0, comm: kworker/0:1 Tainted: P W 2.6.38.6-1.fits.3.el6.x86_64 #1 May 16 09:18:13 os03 kernel: [331036.332009] Call Trace: May 16 09:18:13 os03 kernel: [331036.332011] [] ? __alloc_pages_nodemask+0x6f7/0x8a0 May 16 09:18:13 os03 kernel: [331036.332024] [] ? kmalloc_large_node+0x62/0xb0 May 16 09:18:13 os03 kernel: [331036.332028] [] ? __kmalloc_node_track_caller+0x15b/0x1d0 May 16 09:18:13 os03 kernel: [331036.332033] [] ? ip_rcv+0x23d/0x310 May 16 09:18:13 os03 kernel: [331036.332038] [] ? __netdev_alloc_skb+0x24/0x50 May 16 09:18:13 os03 kernel: [331036.332042] [] ? __alloc_skb+0x83/0x170 May 16 09:18:13 os03 kernel: [331036.332045] [] ? __netdev_alloc_skb+0x24/0x50 May 16 09:18:13 os03 kernel: [331036.332054] [] ? ixgbe_alloc_rx_buffers+0x2b7/0x370 [ixgbe] May 16 09:18:13 os03 kernel: [331036.332059] [] ? sched_clock_cpu+0xcd/0x110 May 16 09:18:13 os03 kernel: [331036.332063] [] ? napi_skb_finish+0x50/0x70 May 16 09:18:13 os03 kernel: [331036.332069] [] ? ixgbe_clean_rx_irq+0x828/0x890 [ixgbe] May 16 09:18:13 os03 kernel: [331036.332076] [] ? ixgbe_clean_rxtx_many+0x10f/0x220 [ixgbe] May 16 09:18:13 os03 kernel: [331036.332080] [] ? net_rx_action+0x102/0x2a0 May 16 09:18:13 os03 kernel: [331036.332084] [] ? __do_softirq+0xb5/0x210 May 16 09:18:13 os03 kernel: [331036.332089] [] ? handle_IRQ_event+0x54/0x180 May 16 09:18:13 os03 kernel: [331036.332094] [] ? call_softirq+0x1c/0x30 May 16 09:18:13 os03 kernel: [331036.332097] [] ? do_softirq+0x65/0xa0 May 16 09:18:13 os03 kernel: [331036.332100] [] ? irq_exit+0x95/0xa0 May 16 09:18:13 os03 kernel: [331036.332105] [] ? do_IRQ+0x66/0xe0 May 16 09:18:13 os03 kernel: [331036.332108] [] ? ret_from_intr+0x0/0x15 May 16 09:18:13 os03 kernel: [331036.332110] [] ? intel_idle+0xc1/0x120 May 16 09:18:13 os03 kernel: [331036.332116] [] ? intel_idle+0xa4/0x120 May 16 09:18:13 os03 kernel: [331036.332121] [] ? cpuidle_idle_call+0xb5/0x240 May 16 09:18:13 os03 kernel: [331036.332125] [] ? cpu_idle+0xb7/0x110 May 16 09:18:13 os03 kernel: [331036.332129] [] ? start_secondary+0x21f/0x221 May 16 09:18:13 os03 kernel: [331036.332131] Mem-Info: May 16 09:18:13 os03 kernel: [331036.332132] Node 0 DMA per-cpu: May 16 09:18:13 os03 kernel: [331036.332135] CPU 0: hi: 0, btch: 1 usd: 0 May 16 09:18:13 os03 kernel: [331036.332137] CPU 1: hi: 0, btch: 1 usd: 0 May 16 09:18:13 os03 kernel: [331036.332140] CPU 2: hi: 0, btch: 1 usd: 0 May 16 09:18:13 os03 kernel: [331036.332142] CPU 3: hi: 0, btch: 1 usd: 0 May 16 09:18:13 os03 kernel: [331036.332144] CPU 4: hi: 0, btch: 1 usd: 0 May 16 09:18:13 os03 kernel: [331036.332146] CPU 5: hi: 0, btch: 1 usd: 0 May 16 09:18:13 os03 kernel: [331036.332148] CPU 6: hi: 0, btch: 1 usd: 0 May 16 09:18:13 os03 kernel: [331036.332150] CPU 7: hi: 0, btch: 1 usd: 0 May 16 09:18:13 os03 kernel: [331036.332152] Node 0 DMA32 per-cpu: May 16 09:18:13 os03 kernel: [331036.332155] CPU 0: hi: 186, btch: 31 usd: 163 May 16 09:18:13 os03 kernel: [331036.332157] CPU 1: hi: 186, btch: 31 usd: 31 May 16 09:18:13 os03 kernel: [331036.332159] CPU 2: hi: 186, btch: 31 usd: 182 May 16 09:18:13 os03 kernel: [331036.332162] CPU 3: hi: 186, btch: 31 usd: 37 May 16 09:18:13 os03 kernel: [331036.332164] CPU 4: hi: 186, btch: 31 usd: 13 May 16 09:18:13 os03 kernel: [331036.332166] CPU 5: hi: 186, btch: 31 usd: 180 May 16 09:18:13 os03 kernel: [331036.332168] CPU 6: hi: 186, btch: 31 usd: 159 May 16 09:18:13 os03 kernel: [331036.332170] CPU 7: hi: 186, btch: 31 usd: 180 May 16 09:18:13 os03 kernel: [331036.332172] Node 0 Normal per-cpu: May 16 09:18:13 os03 kernel: [331036.332174] CPU 0: hi: 186, btch: 31 usd: 156 May 16 09:18:13 os03 kernel: [331036.332177] CPU 1: hi: 186, btch: 31 usd: 160 May 16 09:18:13 os03 kernel: [331036.332179] CPU 2: hi: 186, btch: 31 usd: 163 May 16 09:18:13 os03 kernel: [331036.332181] CPU 3: hi: 186, btch: 31 usd: 168 May 16 09:18:13 os03 kernel: [331036.332183] CPU 4: hi: 186, btch: 31 usd: 163 May 16 09:18:13 os03 kernel: [331036.332185] CPU 5: hi: 186, btch: 31 usd: 180 May 16 09:18:13 os03 kernel: [331036.332187] CPU 6: hi: 186, btch: 31 usd: 156 May 16 09:18:13 os03 kernel: [331036.332189] CPU 7: hi: 186, btch: 31 usd: 182 May 16 09:18:13 os03 kernel: [331036.332195] active_anon:389538 inactive_anon:91572 isolated_anon:0 May 16 09:18:13 os03 kernel: [331036.332196] active_file:2597361 inactive_file:2476894 isolated_file:0 May 16 09:18:13 os03 kernel: [331036.332198] unevictable:123699 dirty:66164 writeback:11426 unstable:0 May 16 09:18:13 os03 kernel: [331036.332199] free:254614 slab_reclaimable:53393 slab_unreclaimable:15304 May 16 09:18:13 os03 kernel: [331036.332201] mapped:1251 shmem:91580 pagetables:1404 bounce:0 May 16 09:18:13 os03 kernel: [331036.332203] Node 0 DMA free:15852kB min:328kB low:408kB high:492kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15660kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes May 16 09:18:13 os03 kernel: [331036.332214] lowmem_reserve[]: 0 2991 24201 24201 May 16 09:18:13 os03 kernel: [331036.332217] Node 0 DMA32 free:426948kB min:64764kB low:80952kB high:97144kB active_anon:135484kB inactive_anon:0kB active_file:1133516kB inactive_file:1093352kB unevictable:49788kB isolated(anon):0kB isolated(file):0kB present:3063392kB mlocked:0kB dirty:110436kB writeback:284kB mapped:432kB shmem:0kB slab_reclaimable:46680kB slab_unreclaimable:5268kB kernel_stack:152kB pagetables:324kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no May 16 09:18:13 os03 kernel: [331036.332229] lowmem_reserve[]: 0 0 21210 21210 May 16 09:18:13 os03 kernel: [331036.332232] Node 0 Normal free:575656kB min:459188kB low:573984kB high:688780kB active_anon:1422668kB inactive_anon:366288kB active_file:9255928kB inactive_file:8814224kB unevictable:445008kB isolated(anon):0kB isolated(file):0kB present:21719040kB mlocked:0kB dirty:154220kB writeback:45420kB mapped:4572kB shmem:366320kB slab_reclaimable:166892kB slab_unreclaimable:55948kB kernel_stack:3928kB pagetables:5292kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no May 16 09:18:13 os03 kernel: [331036.332245] lowmem_reserve[]: 0 0 0 0 May 16 09:18:13 os03 kernel: [331036.332248] Node 0 DMA: 1*4kB 1*8kB 0*16kB 1*32kB 1*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15852kB May 16 09:18:13 os03 kernel: [331036.332256] Node 0 DMA32: 55808*4kB 24890*8kB 3*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 426496kB May 16 09:18:13 os03 kernel: [331036.332264] Node 0 Normal: 142372*4kB 49*8kB 67*16kB 48*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 576648kB May 16 09:18:13 os03 kernel: [331036.332272] 5289868 total pagecache pages May 16 09:18:13 os03 kernel: [331036.332274] 0 pages in swap cache Is there any way to further identify which is causing this bug? Any help appreciated. Greetings Stefan On Tue, May 10, 2011 at 9:06 PM, Brandeburg, Jesse wrote: > Adding e1000-devel, our list for the out-of-tree ixgbe driver (the issue is reported below to be in both upstream and out-of-tree) > > do you have jumbo frames enabled? > > -----Original Message----- > From: netdev-owner@vger.kernel.org [mailto:netdev-owner@vger.kernel.org] On Behalf Of Stefan Majer > Sent: Tuesday, May 10, 2011 9:03 AM > To: netdev@vger.kernel.org > Subject: Kernel 2.6.38.6 page allocation failure (ixgbe) > > Hi, > > im running 4 nodes with ceph on top of btrfs with a dualport Intel > X520 10Gb Ethernet Card with the latest 3.3.9 ixgbe driver. > during benchmarks i get the following stack. > I can easily reproduce this by simply running rados bench from a fast > machine using this 4 nodes as ceph cluster. > We saw this with stock ixgbe driver from 2.6.38.6 and with the latest > 3.3.9 ixgbe. > This kernel is tainted because we use fusion-io iodrives as journal > devices for btrfs. > > Any hints to nail this down are welcome. > > Greetings Stefan Majer > > May 10 15:26:40 os02 kernel: [ 3652.485219] cosd: page allocation > failure. order:2, mode:0x4020 > May 10 15:26:40 os02 kernel: [ 3652.485223] kswapd0: page allocation > failure. order:2, mode:0x4020 > May 10 15:26:40 os02 kernel: [ 3652.485228] Pid: 57, comm: kswapd0 > Tainted: P ? ? ? ?W ? 2.6.38.6-1.fits.1.el6.x86_64 #1 > May 10 15:26:40 os02 kernel: [ 3652.485230] Call Trace: > May 10 15:26:40 os02 kernel: [ 3652.485232] ? > [] ? __alloc_pages_nodemask+0x6f7/0x8a0 > May 10 15:26:40 os02 kernel: [ 3652.485247] ?[] ? > ip_local_deliver+0x80/0x90 > May 10 15:26:40 os02 kernel: [ 3652.485250] cosd: page allocation > failure. order:2, mode:0x4020 > May 10 15:26:40 os02 kernel: [ 3652.485256] ?[] ? > kmalloc_large_node+0x62/0xb0 > May 10 15:26:40 os02 kernel: [ 3652.485259] Pid: 1849, comm: cosd > Tainted: P ? ? ? ?W ? 2.6.38.6-1.fits.1.el6.x86_64 #1 > May 10 15:26:40 os02 kernel: [ 3652.485261] Call Trace: > May 10 15:26:40 os02 kernel: [ 3652.485264] ?[] ? > __kmalloc_node_track_caller+0x15b/0x1d0 > May 10 15:26:40 os02 kernel: [ 3652.485266] ? > [] ? __netdev_alloc_skb+0x24/0x50 > May 10 15:26:40 os02 kernel: [ 3652.485274] ?[] ? > __alloc_pages_nodemask+0x6f7/0x8a0 > May 10 15:26:40 os02 kernel: [ 3652.485277] ?[] ? > __alloc_skb+0x83/0x170 > May 10 15:26:40 os02 kernel: [ 3652.485281] ?[] ? > ip_local_deliver+0x80/0x90 > May 10 15:26:40 os02 kernel: [ 3652.485283] ?[] ? > __netdev_alloc_skb+0x24/0x50 > May 10 15:26:40 os02 kernel: [ 3652.485287] ?[] ? > kmalloc_large_node+0x62/0xb0 > May 10 15:26:40 os02 kernel: [ 3652.485297] ?[] ? > ixgbe_alloc_rx_buffers+0x9a/0x450 [ixgbe] > May 10 15:26:40 os02 kernel: [ 3652.485300] ?[] ? > __kmalloc_node_track_caller+0x15b/0x1d0 > May 10 15:26:40 os02 kernel: [ 3652.485305] ?[] ? > swiotlb_map_page+0x0/0x110 > May 10 15:26:40 os02 kernel: [ 3652.485308] ?[] ? > __netdev_alloc_skb+0x24/0x50 > May 10 15:26:40 os02 kernel: [ 3652.485315] ?[] ? > ixgbe_poll+0x1140/0x1670 [ixgbe] > May 10 15:26:40 os02 kernel: [ 3652.485318] ?[] ? > __alloc_skb+0x83/0x170 > May 10 15:26:40 os02 kernel: [ 3652.485323] ?[] ? > perf_pmu_enable+0x2b/0x40 > May 10 15:26:40 os02 kernel: [ 3652.485326] ?[] ? > __netdev_alloc_skb+0x24/0x50 > May 10 15:26:40 os02 kernel: [ 3652.485330] ?[] ? > net_rx_action+0x102/0x2a0 > May 10 15:26:40 os02 kernel: [ 3652.485336] ?[] ? > ixgbe_alloc_rx_buffers+0x9a/0x450 [ixgbe] > May 10 15:26:40 os02 kernel: [ 3652.485341] ?[] ? > __do_softirq+0xb5/0x210 > May 10 15:26:40 os02 kernel: [ 3652.485344] ?[] ? > napi_skb_finish+0x50/0x70 > May 10 15:26:40 os02 kernel: [ 3652.485348] ?[] ? > handle_IRQ_event+0x54/0x180 > May 10 15:26:40 os02 kernel: [ 3652.485354] ?[] ? > ixgbe_poll+0x1140/0x1670 [ixgbe] > May 10 15:26:40 os02 kernel: [ 3652.485357] ?[] ? > __do_softirq+0x12d/0x210 > May 10 15:26:40 os02 kernel: [ 3652.485360] ?[] ? > perf_pmu_enable+0x2b/0x40 > May 10 15:26:40 os02 kernel: [ 3652.485364] ?[] ? > call_softirq+0x1c/0x30 > May 10 15:26:40 os02 kernel: [ 3652.485367] ?[] ? > net_rx_action+0x102/0x2a0 > May 10 15:26:40 os02 kernel: [ 3652.485369] ?[] ? > do_softirq+0x65/0xa0 > May 10 15:26:40 os02 kernel: [ 3652.485372] ?[] ? > __do_softirq+0xb5/0x210 > May 10 15:26:40 os02 kernel: [ 3652.485375] ?[] ? > irq_exit+0x95/0xa0 > May 10 15:26:40 os02 kernel: [ 3652.485379] ?[] ? > handle_IRQ_event+0x54/0x180 > May 10 15:26:40 os02 kernel: [ 3652.485383] ?[] ? > do_IRQ+0x66/0xe0 > May 10 15:26:40 os02 kernel: [ 3652.485386] ?[] ? > __do_softirq+0x12d/0x210 > May 10 15:26:40 os02 kernel: [ 3652.485389] ?[] ? > ret_from_intr+0x0/0x15 > May 10 15:26:40 os02 kernel: [ 3652.485391] ? > [] ? call_softirq+0x1c/0x30 > May 10 15:26:40 os02 kernel: [ 3652.485397] ?[] ? > shrink_inactive_list+0x164/0x460 > May 10 15:26:40 os02 kernel: [ 3652.485400] ?[] ? > do_softirq+0x65/0xa0 > May 10 15:26:40 os02 kernel: [ 3652.485404] ?[] ? > schedule+0x44c/0xa10 > May 10 15:26:40 os02 kernel: [ 3652.485407] ?[] ? > irq_exit+0x95/0xa0 > May 10 15:26:40 os02 kernel: [ 3652.485412] ?[] ? > determine_dirtyable_memory+0x1a/0x30 > May 10 15:26:40 os02 kernel: [ 3652.485416] ?[] ? > do_IRQ+0x66/0xe0 > May 10 15:26:40 os02 kernel: [ 3652.485419] ?[] ? > shrink_zone+0x3d3/0x530 > May 10 15:26:40 os02 kernel: [ 3652.485422] ?[] ? > ret_from_intr+0x0/0x15 > May 10 15:26:40 os02 kernel: [ 3652.485423] ? > [] ? del_timer_sync+0x3a/0x60 > May 10 15:26:40 os02 kernel: [ 3652.485430] ?[] ? > copy_user_generic_string+0x2d/0x40 > May 10 15:26:40 os02 kernel: [ 3652.485435] ?[] ? > zone_watermark_ok_safe+0xb5/0xd0 > May 10 15:26:40 os02 kernel: [ 3652.485439] ?[] ? > iov_iter_copy_from_user_atomic+0x101/0x170 > May 10 15:26:40 os02 kernel: [ 3652.485442] ?[] ? > kswapd+0x889/0xb20 > May 10 15:26:40 os02 kernel: [ 3652.485457] ?[] ? > btrfs_copy_from_user+0xcd/0x130 [btrfs] > May 10 15:26:40 os02 kernel: [ 3652.485460] ?[] ? > kswapd+0x0/0xb20 > May 10 15:26:40 os02 kernel: [ 3652.485472] ?[] ? > __btrfs_buffered_write+0x1a4/0x330 [btrfs] > May 10 15:26:40 os02 kernel: [ 3652.485476] ?[] ? > kthread+0x96/0xa0 > May 10 15:26:40 os02 kernel: [ 3652.485479] ?[] ? > file_update_time+0x5f/0x170 > May 10 15:26:40 os02 kernel: [ 3652.485482] ?[] ? > kernel_thread_helper+0x4/0x10 > May 10 15:26:40 os02 kernel: [ 3652.485493] ?[] ? > btrfs_file_aio_write+0x238/0x4e0 [btrfs] > May 10 15:26:40 os02 kernel: [ 3652.485496] ?[] ? > kthread+0x0/0xa0 > May 10 15:26:40 os02 kernel: [ 3652.485507] ?[] ? > btrfs_file_aio_write+0x0/0x4e0 [btrfs] > May 10 15:26:40 os02 kernel: [ 3652.485511] ?[] ? > kernel_thread_helper+0x0/0x10 > May 10 15:26:40 os02 kernel: [ 3652.485515] ?[] ? > do_sync_readv_writev+0xd3/0x110 > May 10 15:26:40 os02 kernel: [ 3652.485516] Mem-Info: > May 10 15:26:40 os02 kernel: [ 3652.485519] ?[] ? > path_put+0x22/0x30 > May 10 15:26:40 os02 kernel: [ 3652.485521] Node 0 DMA per-cpu: > May 10 15:26:40 os02 kernel: [ 3652.485525] ?[] ? > selinux_file_permission+0xf3/0x150 > May 10 15:26:40 os02 kernel: [ 3652.485528] CPU ? ?0: hi: ? ?0, btch: > ?1 usd: ? 0 > May 10 15:26:40 os02 kernel: [ 3652.485530] CPU ? ?1: hi: ? ?0, btch: > ?1 usd: ? 0 > May 10 15:26:40 os02 kernel: [ 3652.485534] ?[] ? > security_file_permission+0x23/0x90 > May 10 15:26:40 os02 kernel: [ 3652.485535] CPU ? ?2: hi: ? ?0, btch: > ?1 usd: ? 0 > May 10 15:26:40 os02 kernel: [ 3652.485538] CPU ? ?3: hi: ? ?0, btch: > ?1 usd: ? 0 > May 10 15:26:40 os02 kernel: [ 3652.485542] ?[] ? > do_readv_writev+0xd4/0x1e0 > May 10 15:26:40 os02 kernel: [ 3652.485544] CPU ? ?4: hi: ? ?0, btch: > ?1 usd: ? 0 > May 10 15:26:40 os02 kernel: [ 3652.485547] CPU ? ?5: hi: ? ?0, btch: > ?1 usd: ? 0 > May 10 15:26:40 os02 kernel: [ 3652.485550] ?[] ? > mutex_lock+0x31/0x60 > May 10 15:26:40 os02 kernel: [ 3652.485552] CPU ? ?6: hi: ? ?0, btch: > ?1 usd: ? 0 > May 10 15:26:40 os02 kernel: [ 3652.485554] CPU ? ?7: hi: ? ?0, btch: > ?1 usd: ? 0 > May 10 15:26:40 os02 kernel: [ 3652.485557] ?[] ? > vfs_writev+0x46/0x60 > May 10 15:26:40 os02 kernel: [ 3652.485558] Node 0 DMA32 per-cpu: > May 10 15:26:40 os02 kernel: [ 3652.485562] ?[] ? > sys_writev+0x51/0xc0 > May 10 15:26:40 os02 kernel: [ 3652.485564] CPU ? ?0: hi: ?186, btch: > 31 usd: 144 > May 10 15:26:40 os02 kernel: [ 3652.485567] CPU ? ?1: hi: ?186, btch: > 31 usd: 198 > May 10 15:26:40 os02 kernel: [ 3652.485571] ?[] ? > system_call_fastpath+0x16/0x1b > May 10 15:26:40 os02 kernel: [ 3652.485573] CPU ? ?2: hi: ?186, btch: > 31 usd: 180 > May 10 15:26:40 os02 kernel: [ 3652.485574] Mem-Info: > May 10 15:26:40 os02 kernel: [ 3652.485576] CPU ? ?3: hi: ?186, btch: > 31 usd: 171 > May 10 15:26:40 os02 kernel: [ 3652.485578] Node 0 CPU ? ?4: hi: ?186, > btch: ?31 usd: 159 > May 10 15:26:40 os02 kernel: [ 3652.485581] DMA per-cpu: > May 10 15:26:40 os02 kernel: [ 3652.485582] CPU ? ?5: hi: ?186, btch: > 31 usd: ?69 > May 10 15:26:40 os02 kernel: [ 3652.485585] CPU ? ?0: hi: ? ?0, btch: > ?1 usd: ? 0 > May 10 15:26:40 os02 kernel: [ 3652.485587] CPU ? ?6: hi: ?186, btch: > 31 usd: 180 > May 10 15:26:40 os02 kernel: [ 3652.485589] CPU ? ?1: hi: ? ?0, btch: > ?1 usd: ? 0 > May 10 15:26:40 os02 kernel: [ 3652.485591] CPU ? ?7: hi: ?186, btch: > 31 usd: 184 > May 10 15:26:40 os02 kernel: [ 3652.485593] CPU ? ?2: hi: ? ?0, btch: > ?1 usd: ? 0 > May 10 15:26:40 os02 kernel: [ 3652.485594] Node 0 CPU ? ?3: hi: ? ?0, > btch: ? 1 usd: ? 0 > May 10 15:26:40 os02 kernel: [ 3652.485597] Normal per-cpu: > May 10 15:26:40 os02 kernel: [ 3652.485598] CPU ? ?4: hi: ? ?0, btch: > ?1 usd: ? 0 > May 10 15:26:40 os02 kernel: [ 3652.485600] CPU ? ?0: hi: ?186, btch: > 31 usd: 100 > May 10 15:26:40 os02 kernel: [ 3652.485602] CPU ? ?5: hi: ? ?0, btch: > ?1 usd: ? 0 > May 10 15:26:40 os02 kernel: [ 3652.485604] CPU ? ?1: hi: ?186, btch: > 31 usd: ?47 > May 10 15:26:40 os02 kernel: [ 3652.485606] CPU ? ?6: hi: ? ?0, btch: > ?1 usd: ? 0 > May 10 15:26:40 os02 kernel: [ 3652.485608] CPU ? ?2: hi: ?186, btch: > 31 usd: 168 > May 10 15:26:40 os02 kernel: [ 3652.485610] CPU ? ?7: hi: ? ?0, btch: > ?1 usd: ? 0 > May 10 15:26:40 os02 kernel: [ 3652.485612] CPU ? ?3: hi: ?186, btch: > 31 usd: 140 > May 10 15:26:40 os02 kernel: [ 3652.485614] Node 0 CPU ? ?4: hi: ?186, > btch: ?31 usd: 177 > May 10 15:26:40 os02 kernel: [ 3652.485617] DMA32 per-cpu: > May 10 15:26:40 os02 kernel: [ 3652.485618] CPU ? ?5: hi: ?186, btch: > 31 usd: ?77 > May 10 15:26:40 os02 kernel: [ 3652.485621] CPU ? ?0: hi: ?186, btch: > 31 usd: 144 > May 10 15:26:40 os02 kernel: [ 3652.485623] CPU ? ?6: hi: ?186, btch: > 31 usd: 168 > May 10 15:26:40 os02 kernel: [ 3652.485625] CPU ? ?1: hi: ?186, btch: > 31 usd: 198 > May 10 15:26:40 os02 kernel: [ 3652.485627] CPU ? ?7: hi: ?186, btch: > 31 usd: ?68 > May 10 15:26:40 os02 kernel: [ 3652.485629] CPU ? ?2: hi: ?186, btch: > 31 usd: 180 > May 10 15:26:40 os02 kernel: [ 3652.485634] active_anon:255806 > inactive_anon:19454 isolated_anon:0 > May 10 15:26:40 os02 kernel: [ 3652.485636] ?active_file:420093 > inactive_file:5180559 isolated_file:0 > May 10 15:26:40 os02 kernel: [ 3652.485637] ?unevictable:50582 > dirty:314034 writeback:8484 unstable:0 > May 10 15:26:40 os02 kernel: [ 3652.485639] ?free:30074 > slab_reclaimable:35739 slab_unreclaimable:13526 > May 10 15:26:40 os02 kernel: [ 3652.485641] ?mapped:3440 shmem:51 > pagetables:1342 bounce:0 > May 10 15:26:40 os02 kernel: [ 3652.485643] CPU ? ?3: hi: ?186, btch: > 31 usd: 171 > May 10 15:26:40 os02 kernel: [ 3652.485644] Node 0 CPU ? ?4: hi: ?186, > btch: ?31 usd: 159 > May 10 15:26:40 os02 kernel: [ 3652.485652] DMA free:15852kB min:12kB > low:12kB high:16kB active_anon:0kB inactive_anon:0kB active_file:0kB > inactive_file:0kB unevictable:0kB isolated(anon):0kB > isolated(file):0kB present:15660kB mlocked:0kB dirty:0kB writeback:0kB > mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB > kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB > writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes > May 10 15:26:40 os02 kernel: [ 3652.485659] CPU ? ?5: hi: ?186, btch: > 31 usd: ?69 > May 10 15:26:40 os02 kernel: [ 3652.485661] lowmem_reserve[]:CPU ? ?6: > hi: ?186, btch: ?31 usd: 180 > May 10 15:26:40 os02 kernel: [ 3652.485663] ?0CPU ? ?7: hi: ?186, > btch: ?31 usd: 184 > May 10 15:26:40 os02 kernel: [ 3652.485665] ?2991Node 0 ?24201Normal per-cpu: > May 10 15:26:40 os02 kernel: [ 3652.485668] ?24201CPU ? ?0: hi: ?186, > btch: ?31 usd: 100 > May 10 15:26:40 os02 kernel: [ 3652.485671] > May 10 15:26:40 os02 kernel: [ 3652.485672] CPU ? ?1: hi: ?186, btch: > 31 usd: ?47 > May 10 15:26:40 os02 kernel: [ 3652.485674] Node 0 CPU ? ?2: hi: ?186, > btch: ?31 usd: 168 > May 10 15:26:40 os02 kernel: [ 3652.485682] DMA32 free:85748kB > min:2460kB low:3072kB high:3688kB active_anon:20480kB > inactive_anon:5268kB active_file:151588kB inactive_file:2645188kB > unevictable:72kB isolated(anon):0kB isolated(file):0kB > present:3063392kB mlocked:0kB dirty:210820kB writeback:0kB > mapped:648kB shmem:0kB slab_reclaimable:28400kB > slab_unreclaimable:2152kB kernel_stack:520kB pagetables:100kB > unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 > all_unreclaimable? no > May 10 15:26:40 os02 kernel: [ 3652.485690] CPU ? ?3: hi: ?186, btch: > 31 usd: 140 > May 10 15:26:40 os02 kernel: [ 3652.485691] lowmem_reserve[]:CPU ? ?4: > hi: ?186, btch: ?31 usd: 177 > May 10 15:26:40 os02 kernel: [ 3652.485693] ?0CPU ? ?5: hi: ?186, > btch: ?31 usd: ?77 > May 10 15:26:40 os02 kernel: [ 3652.485696] ?0CPU ? ?6: hi: ?186, > btch: ?31 usd: 168 > May 10 15:26:40 os02 kernel: [ 3652.485698] ?21210CPU ? ?7: hi: ?186, > btch: ?31 usd: ?68 > May 10 15:26:40 os02 kernel: [ 3652.485701] ?21210active_anon:255806 > inactive_anon:19454 isolated_anon:0 > May 10 15:26:40 os02 kernel: [ 3652.485705] ?active_file:420093 > inactive_file:5180559 isolated_file:0 > May 10 15:26:40 os02 kernel: [ 3652.485706] ?unevictable:50582 > dirty:314034 writeback:8484 unstable:0 > May 10 15:26:40 os02 kernel: [ 3652.485707] ?free:30074 > slab_reclaimable:35739 slab_unreclaimable:13526 > May 10 15:26:40 os02 kernel: [ 3652.485708] ?mapped:3440 shmem:51 > pagetables:1342 bounce:0 > May 10 15:26:40 os02 kernel: [ 3652.485709] > May 10 15:26:40 os02 kernel: [ 3652.485710] Node 0 Node 0 DMA > free:15852kB min:12kB low:12kB high:16kB active_anon:0kB > inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB > isolated(anon):0kB isolated(file):0kB present:15660kB mlocked:0kB > dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB > slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB > bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes > May 10 15:26:40 os02 kernel: [ 3652.485724] Normal free:18696kB > min:17440kB low:21800kB high:26160kB active_anon:1002744kB > inactive_anon:72548kB active_file:1528784kB inactive_file:18077048kB > unevictable:202256kB isolated(anon):0kB isolated(file):0kB > present:21719040kB mlocked:0kB dirty:1045316kB writeback:33936kB > mapped:13112kB shmem:204kB slab_reclaimable:114556kB > slab_unreclaimable:51952kB kernel_stack:3768kB pagetables:5268kB > unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:32 > all_unreclaimable? no > May 10 15:26:40 os02 kernel: [ 3652.485731] > lowmem_reserve[]:lowmem_reserve[]: 0 0 2991 0 24201 0 24201 0 > May 10 15:26:40 os02 kernel: [ 3652.485737] > May 10 15:26:40 os02 kernel: [ 3652.485738] Node 0 Node 0 DMA32 > free:85748kB min:2460kB low:3072kB high:3688kB active_anon:20480kB > inactive_anon:5268kB active_file:151588kB inactive_file:2645188kB > unevictable:72kB isolated(anon):0kB isolated(file):0kB > present:3063392kB mlocked:0kB dirty:210820kB writeback:0kB > mapped:648kB shmem:0kB slab_reclaimable:28400kB > slab_unreclaimable:2152kB kernel_stack:520kB pagetables:100kB > unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 > all_unreclaimable? no > May 10 15:26:40 os02 kernel: [ 3652.485747] DMA: > lowmem_reserve[]:1*4kB ?01*8kB ?00*16kB ?212101*32kB ?212101*64kB > May 10 15:26:40 os02 kernel: [ 3652.485754] 1*128kB Node 0 1*256kB > Normal free:18696kB min:17440kB low:21800kB high:26160kB > active_anon:1002744kB inactive_anon:72548kB active_file:1528784kB > inactive_file:18077048kB unevictable:202256kB isolated(anon):0kB > isolated(file):0kB present:21719040kB mlocked:0kB dirty:1045316kB > writeback:33936kB mapped:13112kB shmem:204kB slab_reclaimable:114556kB > slab_unreclaimable:51952kB kernel_stack:3768kB pagetables:5268kB > unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:32 > all_unreclaimable? no > May 10 15:26:40 os02 kernel: [ 3652.485764] 0*512kB > lowmem_reserve[]:1*1024kB ?01*2048kB ?03*4096kB ?0= 15852kB > May 10 15:26:40 os02 kernel: [ 3652.485771] ?0Node 0 > May 10 15:26:40 os02 kernel: [ 3652.485773] DMA32: Node 0 59*4kB DMA: > 125*8kB 1*4kB 66*16kB 1*8kB 80*32kB 0*16kB 188*64kB 1*32kB 51*128kB > 1*64kB 15*256kB 1*128kB 40*512kB 1*256kB 31*1024kB 0*512kB 1*2048kB > 1*1024kB 1*4096kB 1*2048kB = 85620kB > May 10 15:26:40 os02 kernel: [ 3652.485789] 3*4096kB Node 0 = 15852kB > May 10 15:26:40 os02 kernel: [ 3652.485791] Normal: Node 0 3930*4kB > DMA32: 0*8kB 59*4kB 1*16kB 125*8kB 0*32kB 66*16kB 0*64kB 80*32kB > 0*128kB 188*64kB 1*256kB 51*128kB 1*512kB 15*256kB 0*1024kB 40*512kB > 1*2048kB 31*1024kB 0*4096kB 1*2048kB = 18552kB > May 10 15:26:40 os02 kernel: [ 3652.485807] 1*4096kB 5651289 total > pagecache pages > May 10 15:26:40 os02 kernel: [ 3652.485809] = 85620kB > May 10 15:26:40 os02 kernel: [ 3652.485810] 0 pages in swap cache > May 10 15:26:40 os02 kernel: [ 3652.485811] Node 0 Swap cache stats: > add 0, delete 0, find 0/0 > May 10 15:26:40 os02 kernel: [ 3652.485814] Normal: Free swap ?= 1048572kB > May 10 15:26:40 os02 kernel: [ 3652.485815] 3930*4kB Total swap = 1048572kB > May 10 15:26:40 os02 kernel: [ 3652.485817] 0*8kB 1*16kB 0*32kB 0*64kB > 0*128kB 1*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 18552kB > May 10 15:26:40 os02 kernel: [ 3652.485822] 5651289 total pagecache pages > May 10 15:26:40 os02 kernel: [ 3652.485823] 0 pages in swap cache > May 10 15:26:40 os02 kernel: [ 3652.485824] Swap cache stats: add 0, > delete 0, find 0/0 > May 10 15:26:40 os02 kernel: [ 3652.485825] Free swap ?= 1048572kB > May 10 15:26:40 os02 kernel: [ 3652.485826] Total swap = 1048572kB > May 10 15:26:40 os02 kernel: [ 3652.486439] kworker/0:1: page > allocation failure. order:2, mode:0x4020 > May 10 15:26:40 os02 kernel: [ 3652.486443] Pid: 0, comm: kworker/0:1 > Tainted: P ? ? ? ?W ? 2.6.38.6-1.fits.1.el6.x86_64 #1 > May 10 15:26:40 os02 kernel: [ 3652.486446] Call Trace: > May 10 15:26:40 os02 kernel: [ 3652.486448] ? > [] ? __alloc_pages_nodemask+0x6f7/0x8a0 > May 10 15:26:40 os02 kernel: [ 3652.486459] ?[] ? > ip_local_deliver+0x80/0x90 > May 10 15:26:40 os02 kernel: [ 3652.486464] ?[] ? > kmalloc_large_node+0x62/0xb0 > May 10 15:26:40 os02 kernel: [ 3652.486468] ?[] ? > __kmalloc_node_track_caller+0x15b/0x1d0 > May 10 15:26:40 os02 kernel: [ 3652.486473] ?[] ? > __netdev_alloc_skb+0x24/0x50 > May 10 15:26:40 os02 kernel: [ 3652.486476] ?[] ? > __alloc_skb+0x83/0x170 > May 10 15:26:40 os02 kernel: [ 3652.486479] ?[] ? > __netdev_alloc_skb+0x24/0x50 > May 10 15:26:40 os02 kernel: [ 3652.486489] ?[] ? > ixgbe_alloc_rx_buffers+0x9a/0x450 [ixgbe] > May 10 15:26:40 os02 kernel: [ 3652.486494] ?[] ? > napi_skb_finish+0x50/0x70 > May 10 15:26:40 os02 kernel: [ 3652.486501] ?[] ? > ixgbe_poll+0x1140/0x1670 [ixgbe] > May 10 15:26:40 os02 kernel: [ 3652.486506] ?[] ? > sched_clock+0x9/0x10 > May 10 15:26:40 os02 kernel: [ 3652.486510] ?[] ? > net_rx_action+0x102/0x2a0 > May 10 15:26:40 os02 kernel: [ 3652.486514] ?[] ? > __do_softirq+0xb5/0x210 > May 10 15:26:40 os02 kernel: [ 3652.486520] ?[] ? > hrtimer_interrupt+0x134/0x240 > May 10 15:26:40 os02 kernel: [ 3652.486523] ?[] ? > call_softirq+0x1c/0x30 > May 10 15:26:40 os02 kernel: [ 3652.486526] ?[] ? > do_softirq+0x65/0xa0 > May 10 15:26:40 os02 kernel: [ 3652.486529] ?[] ? > irq_exit+0x95/0xa0 > May 10 15:26:40 os02 kernel: [ 3652.486533] ?[] ? > smp_apic_timer_interrupt+0x70/0x9b > May 10 15:26:40 os02 kernel: [ 3652.486536] ?[] ? > apic_timer_interrupt+0x13/0x20 > May 10 15:26:40 os02 kernel: [ 3652.486538] ? > [] ? intel_idle+0xc1/0x120 > May 10 15:26:40 os02 kernel: [ 3652.486544] ?[] ? > intel_idle+0xa4/0x120 > May 10 15:26:40 os02 kernel: [ 3652.486549] ?[] ? > cpuidle_idle_call+0xb5/0x240 > May 10 15:26:40 os02 kernel: [ 3652.486554] ?[] ? > cpu_idle+0xb7/0x110 > May 10 15:26:40 os02 kernel: [ 3652.486558] ?[] ? > start_secondary+0x21f/0x221 > May 10 15:26:40 os02 kernel: [ 3652.486561] Mem-Info: > May 10 15:26:40 os02 kernel: [ 3652.486562] Node 0 DMA per-cpu: > May 10 15:26:40 os02 kernel: [ 3652.486564] CPU ? ?0: hi: ? ?0, btch: > ?1 usd: ? 0 > May 10 15:26:40 os02 kernel: [ 3652.486567] CPU ? ?1: hi: ? ?0, btch: > ?1 usd: ? 0 > May 10 15:26:40 os02 kernel: [ 3652.486569] CPU ? ?2: hi: ? ?0, btch: > ?1 usd: ? 0 > May 10 15:26:40 os02 kernel: [ 3652.486571] CPU ? ?3: hi: ? ?0, btch: > ?1 usd: ? 0 > May 10 15:26:40 os02 kernel: [ 3652.486573] CPU ? ?4: hi: ? ?0, btch: > ?1 usd: ? 0 > May 10 15:26:40 os02 kernel: [ 3652.486575] CPU ? ?5: hi: ? ?0, btch: > ?1 usd: ? 0 > May 10 15:26:40 os02 kernel: [ 3652.486578] CPU ? ?6: hi: ? ?0, btch: > ?1 usd: ? 0 > May 10 15:26:40 os02 kernel: [ 3652.486580] CPU ? ?7: hi: ? ?0, btch: > ?1 usd: ? 0 > May 10 15:26:40 os02 kernel: [ 3652.486581] Node 0 DMA32 per-cpu: > May 10 15:26:40 os02 kernel: [ 3652.486584] CPU ? ?0: hi: ?186, btch: > 31 usd: 144 > May 10 15:26:40 os02 kernel: [ 3652.486586] CPU ? ?1: hi: ?186, btch: > 31 usd: 198 > May 10 15:26:40 os02 kernel: [ 3652.486588] CPU ? ?2: hi: ?186, btch: > 31 usd: 180 > May 10 15:26:40 os02 kernel: [ 3652.486590] CPU ? ?3: hi: ?186, btch: > 31 usd: 172 > May 10 15:26:40 os02 kernel: [ 3652.486593] CPU ? ?4: hi: ?186, btch: > 31 usd: 159 > May 10 15:26:40 os02 kernel: [ 3652.486595] CPU ? ?5: hi: ?186, btch: > 31 usd: ?69 > May 10 15:26:40 os02 kernel: [ 3652.486597] CPU ? ?6: hi: ?186, btch: > 31 usd: 180 > May 10 15:26:40 os02 kernel: [ 3652.486599] CPU ? ?7: hi: ?186, btch: > 31 usd: 184 > May 10 15:26:40 os02 kernel: [ 3652.486601] Node 0 Normal per-cpu: > May 10 15:26:40 os02 kernel: [ 3652.486603] CPU ? ?0: hi: ?186, btch: > 31 usd: 162 > May 10 15:26:40 os02 kernel: [ 3652.486605] CPU ? ?1: hi: ?186, btch: > 31 usd: ?47 > May 10 15:26:40 os02 kernel: [ 3652.486608] CPU ? ?2: hi: ?186, btch: > 31 usd: 168 > May 10 15:26:40 os02 kernel: [ 3652.486610] CPU ? ?3: hi: ?186, btch: > 31 usd: 141 > May 10 15:26:40 os02 kernel: [ 3652.486612] CPU ? ?4: hi: ?186, btch: > 31 usd: 177 > May 10 15:26:40 os02 kernel: [ 3652.486614] CPU ? ?5: hi: ?186, btch: > 31 usd: ?77 > May 10 15:26:40 os02 kernel: [ 3652.486616] CPU ? ?6: hi: ?186, btch: > 31 usd: 168 > May 10 15:26:40 os02 kernel: [ 3652.486618] CPU ? ?7: hi: ?186, btch: > 31 usd: 174 > May 10 15:26:40 os02 kernel: [ 3652.486624] active_anon:255806 > inactive_anon:19454 isolated_anon:0 > May 10 15:26:40 os02 kernel: [ 3652.486625] ?active_file:420093 > inactive_file:5180745 isolated_file:0 > May 10 15:26:40 os02 kernel: [ 3652.486627] ?unevictable:50582 > dirty:314470 writeback:8484 unstable:0 > May 10 15:26:40 os02 kernel: [ 3652.486628] ?free:29795 > slab_reclaimable:35739 slab_unreclaimable:13526 > May 10 15:26:40 os02 kernel: [ 3652.486629] ?mapped:3440 shmem:51 > pagetables:1342 bounce:0 > May 10 15:26:40 os02 kernel: [ 3652.486631] Node 0 DMA free:15852kB > min:12kB low:12kB high:16kB active_anon:0kB inactive_anon:0kB > active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB > isolated(file):0kB present:15660kB mlocked:0kB dirty:0kB writeback:0kB > mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB > kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB > writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes > May 10 15:26:40 os02 kernel: [ 3652.486642] lowmem_reserve[]: 0 2991 24201 24201 > May 10 15:26:40 os02 kernel: [ 3652.486645] Node 0 DMA32 free:85748kB > min:2460kB low:3072kB high:3688kB active_anon:20480kB > inactive_anon:5268kB active_file:151588kB inactive_file:2645188kB > unevictable:72kB isolated(anon):0kB isolated(file):0kB > present:3063392kB mlocked:0kB dirty:210820kB writeback:0kB > mapped:648kB shmem:0kB slab_reclaimable:28400kB > slab_unreclaimable:2152kB kernel_stack:520kB pagetables:100kB > unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 > all_unreclaimable? no > May 10 15:26:40 os02 kernel: [ 3652.486657] lowmem_reserve[]: 0 0 21210 21210 > May 10 15:26:40 os02 kernel: [ 3652.486660] Node 0 Normal free:17580kB > min:17440kB low:21800kB high:26160kB active_anon:1002744kB > inactive_anon:72548kB active_file:1528784kB inactive_file:18077792kB > unevictable:202256kB isolated(anon):0kB isolated(file):0kB > present:21719040kB mlocked:0kB dirty:1047060kB writeback:33936kB > mapped:13112kB shmem:204kB slab_reclaimable:114556kB > slab_unreclaimable:51952kB kernel_stack:3768kB pagetables:5268kB > unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:64 > all_unreclaimable? no > May 10 15:26:40 os02 kernel: [ 3652.486673] lowmem_reserve[]: 0 0 0 0 > May 10 15:26:40 os02 kernel: [ 3652.486675] Node 0 DMA: 1*4kB 1*8kB > 0*16kB 1*32kB 1*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB > 3*4096kB = 15852kB > May 10 15:26:40 os02 kernel: [ 3652.486684] Node 0 DMA32: 59*4kB > 125*8kB 66*16kB 80*32kB 188*64kB 51*128kB 15*256kB 40*512kB 31*1024kB > 1*2048kB 1*4096kB = 85620kB > May 10 15:26:40 os02 kernel: [ 3652.486692] Node 0 Normal: 3705*4kB > 12*8kB 16*16kB 4*32kB 1*64kB 0*128kB 1*256kB 1*512kB 0*1024kB 1*2048kB > 0*4096kB = 18180kB > May 10 15:26:40 os02 kernel: [ 3652.486700] 5651289 total pagecache pages > May 10 15:26:40 os02 kernel: [ 3652.486702] 0 pages in swap cache > May 10 15:26:40 os02 kernel: [ 3652.486704] Swap cache stats: add 0, > delete 0, find 0/0 > May 10 15:26:40 os02 kernel: [ 3652.486705] Free swap ?= 1048572kB > May 10 15:26:40 os02 kernel: [ 3652.486707] Total swap = 1048572kB > May 10 15:26:40 os02 kernel: [ 3652.562795] 6291440 pages RAM > May 10 15:26:40 os02 kernel: [ 3652.562798] 108688 pages reserved > May 10 15:26:40 os02 kernel: [ 3652.562799] 5429575 pages shared > May 10 15:26:40 os02 kernel: [ 3652.562801] 783596 pages non-shared > May 10 15:26:40 os02 kernel: [ 3652.651570] 6291440 pages RAM > May 10 15:26:40 os02 kernel: [ 3652.651572] 108688 pages reserved > May 10 15:26:40 os02 kernel: [ 3652.651573] 5430055 pages shared > May 10 15:26:40 os02 kernel: [ 3652.651575] 782974 pages non-shared > May 10 15:26:40 os02 kernel: [ 3652.721553] 6291440 pages RAM > May 10 15:26:40 os02 kernel: [ 3652.721555] 108688 pages reserved > May 10 15:26:40 os02 kernel: [ 3652.721556] 5430961 pages shared > May 10 15:26:40 os02 kernel: [ 3652.721557] 781496 pages non-shared > May 10 15:26:40 os02 kernel: [ 3654.349865] Pid: 1846, comm: cosd > Tainted: P ? ? ? ?W ? 2.6.38.6-1.fits.1.el6.x86_64 #1 > May 10 15:26:40 os02 kernel: [ 3654.358792] Call Trace: > May 10 15:26:40 os02 kernel: [ 3654.361519] ? > [] ? __alloc_pages_nodemask+0x6f7/0x8a0 > May 10 15:26:40 os02 kernel: [ 3654.369495] ?[] ? > ip_local_deliver+0x80/0x90 > May 10 15:26:40 os02 kernel: [ 3654.376005] ?[] ? > kmalloc_large_node+0x62/0xb0 > May 10 15:26:40 os02 kernel: [ 3654.382703] ?[] ? > __kmalloc_node_track_caller+0x15b/0x1d0 > May 10 15:26:40 os02 kernel: [ 3654.390464] ?[] ? > __netdev_alloc_skb+0x24/0x50 > May 10 15:26:40 os02 kernel: [ 3654.397163] ?[] ? > __alloc_skb+0x83/0x170 > May 10 15:26:40 os02 kernel: [ 3654.403277] ?[] ? > __netdev_alloc_skb+0x24/0x50 > May 10 15:26:40 os02 kernel: [ 3654.409970] ?[] ? > ixgbe_alloc_rx_buffers+0x9a/0x450 [ixgbe] > May 10 15:26:40 os02 kernel: [ 3654.417926] ?[] ? > swiotlb_map_page+0x0/0x110 > May 10 15:26:40 os02 kernel: [ 3654.424432] ?[] ? > ixgbe_poll+0x1140/0x1670 [ixgbe] > May 10 15:26:40 os02 kernel: [ 3654.431518] ?[] ? > perf_pmu_enable+0x2b/0x40 > May 10 15:26:40 os02 kernel: [ 3654.437924] ?[] ? > net_rx_action+0x102/0x2a0 > May 10 15:26:40 os02 kernel: [ 3654.444329] ?[] ? > __do_softirq+0xb5/0x210 > May 10 15:26:40 os02 kernel: [ 3654.450541] ?[] ? > handle_IRQ_event+0x54/0x180 > May 10 15:26:40 os02 kernel: [ 3654.457138] ?[] ? > __do_softirq+0x12d/0x210 > May 10 15:26:40 os02 kernel: [ 3654.463446] ?[] ? > call_softirq+0x1c/0x30 > May 10 15:26:40 os02 kernel: [ 3654.469562] ?[] ? > do_softirq+0x65/0xa0 > May 10 15:26:40 os02 kernel: [ 3654.475484] ?[] ? > irq_exit+0x95/0xa0 > May 10 15:26:40 os02 kernel: [ 3654.481218] ?[] ? > do_IRQ+0x66/0xe0 > May 10 15:26:40 os02 kernel: [ 3654.486754] ?[] ? > ret_from_intr+0x0/0x15 > May 10 15:26:40 os02 kernel: [ 3654.492867] ? > [] ? __make_request+0x149/0x4c0 > May 10 15:26:40 os02 kernel: [ 3654.500061] ?[] ? > __make_request+0x114/0x4c0 > May 10 15:26:41 os02 kernel: [ 3654.506565] ?[] ? > generic_make_request+0x2fd/0x5e0 > May 10 15:26:41 os02 kernel: [ 3654.513649] ?[] ? > dm_get_live_table+0x4b/0x60 > May 10 15:26:41 os02 kernel: [ 3654.520248] ?[] ? > dm_merge_bvec+0xc1/0x140 > May 10 15:26:41 os02 kernel: [ 3654.526555] ?[] ? > submit_bio+0x86/0x110 > May 10 15:26:41 os02 kernel: [ 3654.532574] ?[] ? > dio_bio_submit+0xbc/0xc0 > May 10 15:26:41 os02 kernel: [ 3654.538881] ?[] ? > dio_send_cur_page+0x90/0xc0 > May 10 15:26:41 os02 kernel: [ 3654.545478] ?[] ? > submit_page_section+0x65/0x180 > May 10 15:26:41 os02 kernel: [ 3654.552370] ?[] ? > __blockdev_direct_IO+0x678/0xb30 > May 10 15:26:41 os02 kernel: [ 3654.559454] ?[] ? > security_inode_getsecurity+0x1f/0x30 > May 10 15:26:41 os02 kernel: [ 3654.566924] ?[] ? > blkdev_direct_IO+0x57/0x60 > May 10 15:26:41 os02 kernel: [ 3654.573414] ?[] ? > blkdev_get_blocks+0x0/0xc0 > May 10 15:26:41 os02 kernel: [ 3654.579954] ?[] ? > generic_file_direct_write+0xc2/0x190 > May 10 15:26:41 os02 kernel: [ 3654.587424] ?[] ? > file_update_time+0xf6/0x170 > May 10 15:26:41 os02 kernel: [ 3654.594025] ?[] ? > __generic_file_aio_write+0x32b/0x460 > May 10 15:26:41 os02 kernel: [ 3654.601494] ?[] ? > wake_up_state+0x10/0x20 > > > > and so on. > > -- > Stefan Majer > > > > -- > Stefan Majer > -- > To unsubscribe from this list: send the line "unsubscribe netdev" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at ?http://vger.kernel.org/majordomo-info.html > -- Stefan Majer -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/