Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752515AbaLCM1w (ORCPT ); Wed, 3 Dec 2014 07:27:52 -0500 Received: from forward-corp1g.mail.yandex.net ([95.108.253.251]:55732 "EHLO forward-corp1g.mail.yandex.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751509AbaLCM1u convert rfc822-to-8bit (ORCPT ); Wed, 3 Dec 2014 07:27:50 -0500 X-Yandex-Uniq: b06469f1-73db-478b-9f35-ea596f30d7c1 Authentication-Results: smtpcorp4.mail.yandex.net; dkim=pass header.i=@yandex-team.ru Date: Wed, 03 Dec 2014 15:27:46 +0300 Message-ID: <874mtcx6il.wl%klamm@yandex-team.ru> From: Roman Gushchin To: Tejun Heo Cc: Roman Gushchin , "linux-kernel@vger.kernel.org" , Li Zefan Subject: Re: [PATCH 3.10] cgroup: break infinite loop in __css_tryget() In-Reply-To: <20141202193538.GO10918@htj.dyndns.org> References: <1417193274-20679-1-git-send-email-klamm@yandex-team.ru> <20141202165553.GB10918@htj.dyndns.org> <119181417545645@webcorp02h.yandex-team.ru> <20141202193538.GO10918@htj.dyndns.org> User-Agent: Wanderlust/2.15.9 MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8BIT Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org At Tue, 2 Dec 2014 14:35:38 -0500, Tejun Heo wrote: > > On Tue, Dec 02, 2014 at 09:40:45PM +0300, Roman Gushchin wrote: > > Hi, Tejun! > > > > 02.12.2014, 19:56, "Tejun Heo" : > > > Hello, Roman. > > > > > > On Fri, Nov 28, 2014 at 07:47:54PM +0300, Roman Gushchin wrote: > > >> ?If cgroup_destroy_locked() sets the css refcount to a negative value, > > >> ?we get an infinite loop in __css_tryget(). > > >> > > >> ?In this case css_refcnt() returns modified (non-negative value), so > > >> ?both (t == v) and (t < 0) conditions are always false. > > > > > > I don't follow. ?The count is biased and modified by unbiasing iff the > > > value is negative. ?Here, @v is the unbiased value and @t is the > > > verbatim value. ?If @v is different from @t due to unbiasing, @t must > > > be negative satisfying the second condition and returning NULL, no? > > > > Yep. I missed that we compare t with 0 (not v). > > Hmm... what's up with the rcu stall message then? It should be something else) Actually, I have a bunch of stacktraces like the one below. I have them at least on 3.10.53 and 3.10.58. It seems like it's a rare race: <0.1% machines are stalling per day. -- [1190308.537478] INFO: rcu_sched self-detected stall on CPU { 10} (t=15001 jiffies g=30275281 c=30275280 q=133479) [1190308.538833] sending NMI to all CPUs: [1190308.538839] NMI backtrace for cpu 0 [1190308.538844] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.10.58-18 #14 [1190308.538846] Hardware name: Aquarius Aquarius Server/X9DRFF(-7), BIOS 1.0b 10/26/2012 [1190308.538848] task: ffffffff81a13440 ti: ffffffff81a00000 task.ti: ffffffff81a00000 [1190308.538861] RIP: 0010:[] [] intel_idle+0xcc/0x140 [1190308.538868] RSP: 0018:ffffffff81a01d98 EFLAGS: 00000046 [1190308.538870] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001 [1190308.538872] RDX: 0000000000000000 RSI: ffffffff81a01fd8 RDI: 0000000000000000 [1190308.538873] RBP: ffffffff81a01dc8 R08: 00000000000000b1 R09: 0000000000002be4 [1190308.538875] R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000004 [1190308.538876] R13: 0000000000000020 R14: 0000000000000003 R15: 0000000000000004 [1190308.538879] FS: 0000000000000000(0000) GS:ffff88085fc00000(0000) knlGS:0000000000000000 [1190308.538881] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [1190308.538883] CR2: 00007f1d4d27aae0 CR3: 0000000001a0c000 CR4: 00000000000407f0 [1190308.538885] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [1190308.538887] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [1190308.538888] Stack: [1190308.538889] ffffffff81a01dc8 000000008109b864 ffffe8f7dfc00400 00043b9213aa19b8 [1190308.538893] ffffffff81a82318 ffffffff81a821a0 ffffffff81a01e28 ffffffff814ebf0f [1190308.538896] 0000000000000000 00000000003cb8b9 0000000000000000 00000000003cb8b9 [1190308.538899] Call Trace: [1190308.538906] [] cpuidle_enter_state+0x4f/0xe0 [1190308.538909] [] cpuidle_idle_call+0xc0/0x220 [1190308.538914] [] arch_cpu_idle+0xe/0x30 [1190308.538919] [] cpu_startup_entry+0xe8/0x2b0 [1190308.538923] [] rest_init+0x77/0x80 [1190308.538928] [] start_kernel+0x402/0x40f [1190308.538931] [] ? repair_env_string+0x5a/0x5a [1190308.538934] [] ? early_idt_handlers+0x120/0x120 [1190308.538938] [] x86_64_start_reservations+0x2a/0x2c [1190308.538941] [] x86_64_start_kernel+0xf3/0x102 [1190308.538942] Code: 48 8b 34 25 f0 c6 00 00 48 89 d1 48 8d 86 38 e0 ff ff 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <85> 1d ee 3f 72 00 75 0e 48 8d 75 dc bf 05 00 00 00 e8 fe 33 d4 [1190308.538976] NMI backtrace for cpu 16 [1190308.538980] CPU: 16 PID: 0 Comm: swapper/16 Not tainted 3.10.58-18 #14 [1190308.538981] Hardware name: Aquarius Aquarius Server/X9DRFF(-7), BIOS 1.0b 10/26/2012 [1190308.538983] task: ffff88085462ba00 ti: ffff88085463c000 task.ti: ffff88085463c000 [1190308.538984] RIP: 0010:[] [] intel_idle+0xcc/0x140 [1190308.538987] RSP: 0018:ffff88085463dda8 EFLAGS: 00000046 [1190308.538988] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001 [1190308.538989] RDX: 0000000000000000 RSI: ffff88085463dfd8 RDI: 0000000000000010 [1190308.538990] RBP: ffff88085463ddd8 R08: 0000000000000005 R09: 0000000000000009 [1190308.538992] R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000004 [1190308.538993] R13: 0000000000000020 R14: 0000000000000003 R15: 0000000000000004 [1190308.538994] FS: 0000000000000000(0000) GS:ffff88085fd00000(0000) knlGS:0000000000000000 [1190308.538996] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [1190308.538997] CR2: 00007f5883684a70 CR3: 0000000001a0c000 CR4: 00000000000407e0 [1190308.538998] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [1190308.538999] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [1190308.539000] Stack: [1190308.539001] ffff88085463ddd8 000000108109b864 ffffe8f7dfd00400 00043b9213a9dc35 [1190308.539003] ffffffff81a82318 ffffffff81a821a0 ffff88085463de38 ffffffff814ebf0f [1190308.539005] 0000000000000000 00000000003cf622 0000000000000000 00000000003cf622 [1190308.539007] Call Trace: [1190308.539010] [] cpuidle_enter_state+0x4f/0xe0 [1190308.539013] [] cpuidle_idle_call+0xc0/0x220 [1190308.539015] [] arch_cpu_idle+0xe/0x30 [1190308.539017] [] cpu_startup_entry+0xe8/0x2b0 [1190308.539021] [] start_secondary+0x204/0x20b [1190308.539023] Code: 48 8b 34 25 f0 c6 00 00 48 89 d1 48 8d 86 38 e0 ff ff 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <85> 1d ee 3f 72 00 75 0e 48 8d 75 dc bf 05 00 00 00 e8 fe 33 d4 [1190308.539045] NMI backtrace for cpu 21 [1190308.539050] CPU: 21 PID: 9465 Comm: vmtouch Not tainted 3.10.58-18 #14 [1190308.539052] Hardware name: Aquarius Aquarius Server/X9DRFF(-7), BIOS 1.0b 10/26/2012 [1190308.539054] task: ffff880695666cc0 ti: ffff8808525e2000 task.ti: ffff8808525e2000 [1190308.539056] RIP: 0010:[] [] __css_tryget+0x24/0x50 [1190308.539061] RSP: 0000:ffff8808525e37e8 EFLAGS: 00000a87 [1190308.539063] RAX: 0000000080000000 RBX: ffff8810531f6800 RCX: ffff8810531f6808 [1190308.539065] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8810531f6800 [1190308.539067] RBP: ffff8808525e37e8 R08: 0000000000000001 R09: 00000000000001ff [1190308.539069] R10: ffff8806946f6d80 R11: 0000000000000400 R12: 0000000000000001 [1190308.539070] R13: 0000000000000020 R14: 0000000000000001 R15: 00000000000000d0 [1190308.539073] FS: 00007fc619bea700(0000) GS:ffff88085fda0000(0000) knlGS:0000000000000000 [1190308.539075] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [1190308.539077] CR2: 00007fc60e5c8000 CR3: 000000085365b000 CR4: 00000000000407e0 [1190308.539079] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [1190308.539081] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [1190308.539082] Stack: [1190308.539083] ffff8808525e3908 ffffffff811705e8 ffff8808525e3858 ffff880600011280 [1190308.539087] ffff8808525e3818 ffffffff81115385 ffff8808525e3898 ffffffff811156be [1190308.539090] ffff8808525e2000 ffff8808525e3898 000000d000000002 00000000000000d0 [1190308.539093] Call Trace: [1190308.539099] [] __mem_cgroup_try_charge+0x2c8/0xb20 [1190308.539103] [] ? mempool_alloc_slab+0x15/0x20 [1190308.539106] [] ? mempool_alloc+0x5e/0x160 [1190308.539111] [] ? __bio_add_page.part.15+0x101/0x240 [1190308.539114] [] mem_cgroup_charge_common+0x57/0xa0 [1190308.539118] [] mem_cgroup_cache_charge+0xb0/0xf0 [1190308.539121] [] add_to_page_cache_locked+0x4a/0x190 [1190308.539126] [] ? __inc_zone_state+0x5f/0x70 [1190308.539129] [] add_to_page_cache_lru+0x21/0x50 [1190308.539132] [] mpage_readpages+0xba/0x130 [1190308.539136] [] ? ext4_get_block_write+0x20/0x20 [1190308.539138] [] ? ext4_get_block_write+0x20/0x20 [1190308.539142] [] ? alloc_pages_current+0xba/0x170 [1190308.539145] [] ext4_readpages+0x45/0x60 [1190308.539149] [] __do_page_cache_readahead+0x1af/0x260 [1190308.539153] [] ra_submit+0x21/0x30 [1190308.539155] [] filemap_fault+0x297/0x500 [1190308.539159] [] __do_fault+0x72/0x520 [1190308.539162] [] handle_pte_fault+0xf6/0xa60 [1190308.539165] [] ? alloc_pages_current+0xba/0x170 [1190308.539169] [] ? pte_alloc_one+0x36/0x40 [1190308.539172] [] ? _raw_spin_lock+0xe/0x20 [1190308.539175] [] ? __pte_alloc+0xa9/0x160 [1190308.539179] [] handle_mm_fault+0x271/0x390 [1190308.539181] [] ? find_get_page+0x37/0x90 [1190308.539186] [] __do_page_fault+0x19f/0x540 [1190308.539189] [] ? __free_pages+0x47/0x70 [1190308.539192] [] ? free_pages+0x45/0x50 [1190308.539195] [] ? SyS_mincore+0x6b8/0x740 [1190308.539198] [] do_page_fault+0xe/0x10 [1190308.539201] [] page_fault+0x28/0x30 [1190308.539203] Code: 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 8d 4f 08 48 89 e5 8b 57 08 8d 82 00 00 00 80 85 d2 0f 48 d0 8d 72 01 89 d0 f0 0f b1 31 <39> c2 75 07 b8 01 00 00 00 5d c3 85 c0 78 04 f3 90 eb d6 31 c0 [1190308.539237] NMI backtrace for cpu 5 [1190308.539240] CPU: 5 PID: 0 Comm: swapper/5 Not tainted 3.10.58-18 #14 [1190308.539242] Hardware name: Aquarius Aquarius Server/X9DRFF(-7), BIOS 1.0b 10/26/2012 [1190308.539244] task: ffff88085449e580 ti: ffff880854616000 task.ti: ffff880854616000 [1190308.539246] RIP: 0010:[] [] intel_idle+0xcc/0x140 [1190308.539251] RSP: 0018:ffff880854617da8 EFLAGS: 00000046 [1190308.539253] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001 [1190308.539255] RDX: 0000000000000000 RSI: ffff880854617fd8 RDI: 0000000000000005 [1190308.539257] RBP: ffff880854617dd8 R08: 000000000000098d R09: 000000000009d6c9 [1190308.539258] R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000004 [1190308.539260] R13: 0000000000000020 R14: 0000000000000003 R15: 0000000000000004 [1190308.539263] FS: 0000000000000000(0000) GS:ffff88085fca0000(0000) knlGS:0000000000000000 [1190308.539265] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [1190308.539267] CR2: 00007fb8fc195000 CR3: 0000000001a0c000 CR4: 00000000000407e0 [1190308.539269] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [1190308.539271] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [1190308.539272] Stack: [1190308.539274] ffff880854617dd8 000000058109b864 ffffe8f7dfca0400 00043b9213a9eb82 [1190308.539277] ffffffff81a82318 ffffffff81a821a0 ffff880854617e38 ffffffff814ebf0f [1190308.539280] 0000000000000000 00000000003cea7e 0000000000000000 00000000003cea7e [1190308.539284] Call Trace: [1190308.539288] [] cpuidle_enter_state+0x4f/0xe0 [1190308.539292] [] cpuidle_idle_call+0xc0/0x220 [1190308.539296] [] arch_cpu_idle+0xe/0x30 [1190308.539300] [] cpu_startup_entry+0xe8/0x2b0 [1190308.539305] [] start_secondary+0x204/0x20b [1190308.539306] Code: 48 8b 34 25 f0 c6 00 00 48 89 d1 48 8d 86 38 e0 ff ff 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <85> 1d ee 3f 72 00 75 0e 48 8d 75 dc bf 05 00 00 00 e8 fe 33 d4 [1190308.539343] NMI backtrace for cpu 27 [1190308.539349] CPU: 27 PID: 0 Comm: swapper/27 Not tainted 3.10.58-18 #14 [1190308.539352] Hardware name: Aquarius Aquarius Server/X9DRFF(-7), BIOS 1.0b 10/26/2012 [1190308.539354] task: ffff880854650e80 ti: ffff88085465a000 task.ti: ffff88085465a000 [1190308.539356] RIP: 0010:[] [] intel_idle+0xcc/0x140 [1190308.539363] RSP: 0018:ffff88085465bda8 EFLAGS: 00000046 [1190308.539365] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001 [1190308.539367] RDX: 0000000000000000 RSI: ffff88085465bfd8 RDI: 000000000000001b [1190308.539369] RBP: ffff88085465bdd8 R08: 0000000000ad01f5 R09: 000004f3ad35ef89 [1190308.539371] R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000004 [1190308.539373] R13: 0000000000000020 R14: 0000000000000003 R15: 0000000000000004 [1190308.539375] FS: 0000000000000000(0000) GS:ffff88107fd60000(0000) knlGS:0000000000000000 [1190308.539377] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [1190308.539379] CR2: 00007fbfc2926420 CR3: 0000000001a0c000 CR4: 00000000000407e0 [1190308.539381] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [1190308.539383] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [1190308.539384] Stack: [1190308.539386] ffff88085465bdd8 0000001b8109b864 ffffe8ffffd60400 00043b90957bc0f4 [1190308.539389] ffffffff81a82318 ffffffff81a821a0 ffff88085465be38 ffffffff814ebf0f [1190308.539393] 00000000000000d0 00000000188d53e3 00000000000000d0 00000000188d53e3 [1190308.539396] Call Trace: [1190308.539402] [] cpuidle_enter_state+0x4f/0xe0 [1190308.539406] [] cpuidle_idle_call+0xc0/0x220 [1190308.539411] [] arch_cpu_idle+0xe/0x30 [1190308.539416] [] cpu_startup_entry+0xe8/0x2b0 [1190308.539422] [] start_secondary+0x204/0x20b [1190308.539423] Code: 48 8b 34 25 f0 c6 00 00 48 89 d1 48 8d 86 38 e0 ff ff 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <85> 1d ee 3f 72 00 75 0e 48 8d 75 dc bf 05 00 00 00 e8 fe 33 d4 [1190308.539457] NMI backtrace for cpu 11 [1190308.539460] CPU: 11 PID: 9466 Comm: vmtouch Not tainted 3.10.58-18 #14 [1190308.539462] Hardware name: Aquarius Aquarius Server/X9DRFF(-7), BIOS 1.0b 10/26/2012 [1190308.539464] task: ffff880031e42b80 ti: ffff880853768000 task.ti: ffff880853768000 [1190308.539466] RIP: 0010:[] [] __css_tryget+0x37/0x50 [1190308.539471] RSP: 0000:ffff8808537697e8 EFLAGS: 00000286 [1190308.539475] RAX: 0000000080000000 RBX: ffff8810531f6800 RCX: ffff8810531f6808 [1190308.539479] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8810531f6800 [1190308.539483] RBP: ffff8808537697e8 R08: 0000000000000001 R09: 00000000000001ff [1190308.539487] R10: ffff8810519b4600 R11: 0000000000000400 R12: 0000000000000001 [1190308.539491] R13: 0000000000000020 R14: 0000000000000001 R15: 00000000000000d0 [1190308.539497] FS: 00007f20e541f700(0000) GS:ffff88107fc60000(0000) knlGS:0000000000000000 [1190308.539502] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [1190308.539507] CR2: 00007f20e4cb7000 CR3: 0000000850f21000 CR4: 00000000000407e0 [1190308.539513] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [1190308.539518] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [1190308.539522] Stack: [1190308.539526] ffff880853769908 ffffffff811705e8 ffff880853769858 ffff880031e42b80 [1190308.539541] ffff880853769818 ffffffff81115385 ffff880853769898 ffffffff811ae7d8 [1190308.539545] ffff880853768000 ffff880853769898 000000d0519b49c0 00000000000000d0 [1190308.539548] Call Trace: [1190308.539553] [] __mem_cgroup_try_charge+0x2c8/0xb20 [1190308.539557] [] ? mempool_alloc_slab+0x15/0x20 [1190308.539562] [] ? bvec_alloc+0x68/0x100 [1190308.539566] [] ? __bio_add_page.part.15+0x101/0x240 [1190308.539569] [] mem_cgroup_charge_common+0x57/0xa0 [1190308.539572] [] mem_cgroup_cache_charge+0xb0/0xf0 [1190308.539575] [] add_to_page_cache_locked+0x4a/0x190 [1190308.539578] [] add_to_page_cache_lru+0x21/0x50 [1190308.539582] [] mpage_readpages+0xba/0x130 [1190308.539586] [] ? ext4_get_block_write+0x20/0x20 [1190308.539589] [] ? ext4_get_block_write+0x20/0x20 [1190308.539594] [] ? alloc_pages_current+0xba/0x170 [1190308.539597] [] ext4_readpages+0x45/0x60 [1190308.539602] [] __do_page_cache_readahead+0x1af/0x260 [1190308.539606] [] ra_submit+0x21/0x30 [1190308.539608] [] filemap_fault+0x297/0x500 [1190308.539612] [] __do_fault+0x72/0x520 [1190308.539615] [] handle_pte_fault+0xf6/0xa60 [1190308.539618] [] ? alloc_pages_current+0xba/0x170 [1190308.539623] [] ? inc_zone_page_state+0x59/0xa0 [1190308.539627] [] ? pte_alloc_one+0x36/0x40 [1190308.539631] [] ? _raw_spin_lock+0xe/0x20 [1190308.539635] [] ? __pte_alloc+0xa9/0x160 [1190308.539639] [] ? __mem_cgroup_count_vm_event+0x44/0x60 [1190308.539642] [] handle_mm_fault+0x271/0x390 [1190308.539647] [] __do_page_fault+0x19f/0x540 [1190308.539650] [] ? set_next_entity+0x3d/0x80 [1190308.539654] [] ? __switch_to+0x16d/0x4a0 [1190308.539659] [] ? finish_task_switch+0x4a/0xf0 [1190308.539665] [] ? __schedule+0x2cd/0x770 [1190308.539669] [] do_page_fault+0xe/0x10 [1190308.539672] [] page_fault+0x28/0x30 [1190308.539674] Code: e5 8b 57 08 8d 82 00 00 00 80 85 d2 0f 48 d0 8d 72 01 89 d0 f0 0f b1 31 39 c2 75 07 b8 01 00 00 00 5d c3 85 c0 78 04 f3 90 eb d6 <31> c0 5d 66 0f 1f 44 00 00 c3 66 66 66 66 66 66 2e 0f 1f 84 00 [1190308.539707] NMI backtrace for cpu 10 [1190308.539710] CPU: 10 PID: 9464 Comm: vmtouch Not tainted 3.10.58-18 #14 [1190308.539712] Hardware name: Aquarius Aquarius Server/X9DRFF(-7), BIOS 1.0b 10/26/2012 [1190308.539715] task: ffff8800641cde40 ti: ffff880852ec8000 task.ti: ffff880852ec8000 [1190308.539716] RIP: 0010:[] [] __bitmap_empty+0x6d/0x80 [1190308.539723] RSP: 0000:ffff88107fc43db8 EFLAGS: 00000006 [1190308.539725] RAX: 0000000000000000 RBX: 0000000000002710 RCX: 0000000000000002 [1190308.539727] RDX: 0000000000000009 RSI: 0000000000000080 RDI: ffffffff81ad06e0 [1190308.539729] RBP: ffff88107fc43db8 R08: ffff88107fc4dc50 R09: 0000000000000080 [1190308.539731] R10: 000000000000b788 R11: 000000000000b787 R12: ffffffff81a4bfc0 [1190308.539732] R13: ffffffff81a4c080 R14: 000000000000000a R15: ffffffff81ad0c00 [1190308.539735] FS: 00007fe5c385b700(0000) GS:ffff88107fc40000(0000) knlGS:0000000000000000 [1190308.539737] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [1190308.539739] CR2: 00007fe5a8fcc000 CR3: 0000001051cc8000 CR4: 00000000000407e0 [1190308.539741] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [1190308.539743] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [1190308.539744] Stack: [1190308.539745] ffff88107fc43dd8 ffffffff810392e0 0000000000020967 ffff88107fc4e760 [1190308.539749] ffff88107fc43e48 ffffffff810d6e92 000000000000000a ffff88107fc53d80 [1190308.539752] 0000000000000096 0000000000000000 000000000000000a ffff880852ec8000 [1190308.539755] Call Trace: [1190308.539757] [1190308.539758] [] arch_trigger_all_cpu_backtrace+0x90/0xb0 [1190308.539774] [] rcu_check_callbacks+0x212/0x680 [1190308.539785] [] update_process_times+0x48/0x80 [1190308.539797] [] tick_sched_timer+0x6e/0xa0 [1190308.539808] [] __run_hrtimer+0x68/0x1c0 [1190308.539819] [] ? tick_nohz_handler+0xd0/0xd0 [1190308.539827] [] ? __do_softirq+0x149/0x270 [1190308.539836] [] hrtimer_interrupt+0x107/0x270 [1190308.539846] [] ? call_softirq+0x1c/0x30 [1190308.539850] [] smp_apic_timer_interrupt+0x69/0x99 [1190308.539853] [] apic_timer_interrupt+0x6d/0x80 [1190308.539854] [1190308.539855] [] ? __css_tryget+0x24/0x50 [1190308.539861] [] __mem_cgroup_try_charge+0x2c8/0xb20 [1190308.539864] [] ? mempool_alloc_slab+0x15/0x20 [1190308.539867] [] ? bvec_alloc+0x68/0x100 [1190308.539871] [] ? __bio_add_page.part.15+0x101/0x240 [1190308.539874] [] mem_cgroup_charge_common+0x57/0xa0 [1190308.539878] [] mem_cgroup_cache_charge+0xb0/0xf0 [1190308.539881] [] add_to_page_cache_locked+0x4a/0x190 [1190308.539884] [] add_to_page_cache_lru+0x21/0x50 [1190308.539887] [] mpage_readpages+0xba/0x130 [1190308.539890] [] ? ext4_get_block_write+0x20/0x20 [1190308.539893] [] ? ext4_get_block_write+0x20/0x20 [1190308.539896] [] ? alloc_pages_current+0xba/0x170 [1190308.539899] [] ext4_readpages+0x45/0x60 [1190308.539902] [] __do_page_cache_readahead+0x1af/0x260 [1190308.539906] [] ra_submit+0x21/0x30 [1190308.539908] [] filemap_fault+0x297/0x500 [1190308.539911] [] __do_fault+0x72/0x520 [1190308.539915] [] handle_pte_fault+0xf6/0xa60 [1190308.539917] [] ? alloc_pages_current+0xba/0x170 [1190308.539921] [] ? pte_alloc_one+0x36/0x40 [1190308.539923] [] ? _raw_spin_lock+0xe/0x20 [1190308.539926] [] ? __pte_alloc+0xa9/0x160 [1190308.539929] [] handle_mm_fault+0x271/0x390 [1190308.539933] [] __do_page_fault+0x19f/0x540 [1190308.539936] [] ? __free_pages+0x47/0x70 [1190308.539939] [] ? free_pages+0x45/0x50 [1190308.539942] [] ? SyS_mincore+0x6b8/0x740 [1190308.539946] [] do_page_fault+0xe/0x10 [1190308.539948] [] page_fault+0x28/0x30 [1190308.539950] Code: 74 29 89 f0 48 63 d2 c1 f8 1f c1 e8 1a 8d 0c 06 83 e1 3f 29 c1 b8 01 00 00 00 48 d3 e0 48 83 e8 01 48 85 04 d7 0f 94 c0 0f b6 c0 <5d> c3 90 31 c0 5d c3 31 d2 eb c1 0f 1f 84 00 00 00 00 00 8d 4e [1190308.539984] NMI backtrace for cpu 26 [1190308.539987] CPU: 26 PID: 0 Comm: swapper/26 Not tainted 3.10.58-18 #14 [1190308.539988] Hardware name: Aquarius Aquarius Server/X9DRFF(-7), BIOS 1.0b 10/26/2012 [1190308.539991] task: ffff880854650740 ti: ffff880854658000 task.ti: ffff880854658000 [1190308.539992] RIP: 0010:[] [] intel_idle+0xcc/0x140 [1190308.539996] RSP: 0018:ffff880854659da8 EFLAGS: 00000046 [1190308.539998] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001 [1190308.539999] RDX: 0000000000000000 RSI: ffff880854659fd8 RDI: 000000000000001a [1190308.540001] RBP: ffff880854659dd8 R08: 0000000000000009 R09: 0000000000000010 [1190308.540003] R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000004 [1190308.540005] R13: 0000000000000020 R14: 0000000000000003 R15: 0000000000000004 [1190308.540007] FS: 0000000000000000(0000) GS:ffff88107fd40000(0000) knlGS:0000000000000000 [1190308.540009] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [1190308.540011] CR2: 00007f058131f420 CR3: 0000000001a0c000 CR4: 00000000000407e0 [1190308.540012] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [1190308.540014] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [1190308.540015] Stack: [1190308.540017] ffff880854659dd8 0000001a8109b864 ffffe8ffffd40400 00043b9213aa04a4 [1190308.540020] ffffffff81a82318 ffffffff81a821a0 ffff880854659e38 ffffffff814ebf0f [1190308.540023] 000000000000006a 00000000162c0a55 000000000000006a 00000000162c0a55 [1190308.540026] Call Trace: [1190308.540036] [] cpuidle_enter_state+0x4f/0xe0 [1190308.540047] [] cpuidle_idle_call+0xc0/0x220 [1190308.540057] [] arch_cpu_idle+0xe/0x30 [1190308.540068] [] cpu_startup_entry+0xe8/0x2b0 [1190308.540079] [] start_secondary+0x204/0x20b [1190308.540085] Code: 48 8b 34 25 f0 c6 00 00 48 89 d1 48 8d 86 38 e0 ff ff 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <85> 1d ee 3f 72 00 75 0e 48 8d 75 dc bf 05 00 00 00 e8 fe 33 d4 [1190308.540123] NMI backtrace for cpu 12 [1190308.540128] CPU: 12 PID: 0 Comm: swapper/12 Not tainted 3.10.58-18 #14 [1190308.540130] Hardware name: Aquarius Aquarius Server/X9DRFF(-7), BIOS 1.0b 10/26/2012 [1190308.540132] task: ffff880854629d00 ti: ffff880854634000 task.ti: ffff880854634000 [1190308.540134] RIP: 0010:[] [] intel_idle+0xcc/0x140 [1190308.540139] RSP: 0018:ffff880854635da8 EFLAGS: 00000046 [1190308.540141] RAX: 0000000000000001 RBX: 0000000000000002 RCX: 0000000000000001 [1190308.540142] RDX: 0000000000000000 RSI: ffff880854635fd8 RDI: 000000000000000c [1190308.540144] RBP: ffff880854635dd8 R08: 00000000000000ad R09: 0000000000001900 [1190308.540146] R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000002 [1190308.540148] R13: 0000000000000001 R14: 0000000000000001 R15: 0000000000000002 [1190308.540150] FS: 0000000000000000(0000) GS:ffff88107fc80000(0000) knlGS:0000000000000000 [1190308.540152] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [1190308.540154] CR2: 00007f1e5fe60000 CR3: 0000000001a0c000 CR4: 00000000000407e0 [1190308.540156] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [1190308.540158] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [1190308.540159] Stack: [1190308.540161] ffff880854635dd8 0000000c8109b864 ffffe8ffffc80400 00043b9213aa3dbd [1190308.540164] ffffffff81a82268 ffffffff81a821a0 ffff880854635e38 ffffffff814ebf0f [1190308.540167] 0000000000000000 00000000003c958a 0000000000000000 00000000003c958a [1190308.540170] Call Trace: [1190308.540175] [] cpuidle_enter_state+0x4f/0xe0 [1190308.540178] [] cpuidle_idle_call+0xc0/0x220 [1190308.540182] [] arch_cpu_idle+0xe/0x30 [1190308.540186] [] cpu_startup_entry+0xe8/0x2b0 [1190308.540189] [] start_secondary+0x204/0x20b [1190308.540191] Code: 48 8b 34 25 f0 c6 00 00 48 89 d1 48 8d 86 38 e0 ff ff 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <85> 1d ee 3f 72 00 75 0e 48 8d 75 dc bf 05 00 00 00 e8 fe 33 d4 [1190308.540225] NMI backtrace for cpu 28 [1190308.540228] CPU: 28 PID: 0 Comm: swapper/28 Not tainted 3.10.58-18 #14 [1190308.540229] Hardware name: Aquarius Aquarius Server/X9DRFF(-7), BIOS 1.0b 10/26/2012 [1190308.540230] task: ffff8808546515c0 ti: ffff88085465c000 task.ti: ffff88085465c000 [1190308.540232] RIP: 0010:[] [] intel_idle+0xcc/0x140 [1190308.540234] RSP: 0018:ffff88085465dda8 EFLAGS: 00000046 [1190308.540236] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001 [1190308.540237] RDX: 0000000000000000 RSI: ffff88085465dfd8 RDI: 000000000000001c [1190308.540238] RBP: ffff88085465ddd8 R08: 000000000000000d R09: 0000000000000004 [1190308.540239] R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000004 [1190308.540240] R13: 0000000000000020 R14: 0000000000000003 R15: 0000000000000004 [1190308.540242] FS: 0000000000000000(0000) GS:ffff88107fd80000(0000) knlGS:0000000000000000 [1190308.540243] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [1190308.540244] CR2: 00007f28a020ae60 CR3: 0000000001a0c000 CR4: 00000000000407e0 [1190308.540245] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [1190308.540247] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [1190308.540247] Stack: [1190308.540248] ffff88085465ddd8 0000001c8109b864 ffffe8ffffd80400 00043b9213a9e5d1 [1190308.540250] ffffffff81a82318 ffffffff81a821a0 ffff88085465de38 ffffffff814ebf0f [1190308.540252] 0000000000000000 00000000003cedb6 0000000000000000 00000000003cedb6 [1190308.540254] Call Trace: [1190308.540257] [] cpuidle_enter_state+0x4f/0xe0 [1190308.540260] [] cpuidle_idle_call+0xc0/0x220 [1190308.540262] [] arch_cpu_idle+0xe/0x30 [1190308.540264] [] cpu_startup_entry+0xe8/0x2b0 [1190308.540273] [] start_secondary+0x204/0x20b [1190308.540277] Code: 48 8b 34 25 f0 c6 00 00 48 89 d1 48 8d 86 38 e0 ff ff 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <85> 1d ee 3f 72 00 75 0e 48 8d 75 dc bf 05 00 00 00 e8 fe 33 d4 [1190308.540341] NMI backtrace for cpu 8 [1190308.540345] CPU: 8 PID: 0 Comm: swapper/8 Not tainted 3.10.58-18 #14 [1190308.540347] Hardware name: Aquarius Aquarius Server/X9DRFF(-7), BIOS 1.0b 10/26/2012 [1190308.540349] task: ffff880854628000 ti: ffff88085461c000 task.ti: ffff88085461c000 [1190308.540351] RIP: 0010:[] [] intel_idle+0xcc/0x140 [1190308.540356] RSP: 0018:ffff88085461dda8 EFLAGS: 00000046 [1190308.540357] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001 [1190308.540359] RDX: 0000000000000000 RSI: ffff88085461dfd8 RDI: 0000000000000008 [1190308.540361] RBP: ffff88085461ddd8 R08: 0000000000000005 R09: 0000000000000000 [1190308.540363] R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000004 [1190308.540364] R13: 0000000000000020 R14: 0000000000000003 R15: 0000000000000004 [1190308.540367] FS: 0000000000000000(0000) GS:ffff88107fc00000(0000) knlGS:0000000000000000 [1190308.540369] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [1190308.540371] CR2: 0000000000f4f190 CR3: 0000000001a0c000 CR4: 00000000000407e0 [1190308.540373] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [1190308.540374] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [1190308.540376] Stack: [1190308.540377] ffff88085461ddd8 000000088109b864 ffffe8ffffc00400 00043b9213aa205f [1190308.540381] ffffffff81a82318 ffffffff81a821a0 ffff88085461de38 ffffffff814ebf0f [1190308.540384] 0000000000000000 00000000003cb29e 0000000000000000 00000000003cb29e [1190308.540387] Call Trace: [1190308.540391] [] cpuidle_enter_state+0x4f/0xe0 [1190308.540395] [] cpuidle_idle_call+0xc0/0x220 [1190308.540398] [] arch_cpu_idle+0xe/0x30 [1190308.540402] [] cpu_startup_entry+0xe8/0x2b0 [1190308.540406] [] start_secondary+0x204/0x20b [1190308.540407] Code: 48 8b 34 25 f0 c6 00 00 48 89 d1 48 8d 86 38 e0 ff ff 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <85> 1d ee 3f 72 00 75 0e 48 8d 75 dc bf 05 00 00 00 e8 fe 33 d4 [1190308.540441] NMI backtrace for cpu 24 [1190308.540444] CPU: 24 PID: 0 Comm: swapper/24 Not tainted 3.10.58-18 #14 [1190308.540446] Hardware name: Aquarius Aquarius Server/X9DRFF(-7), BIOS 1.0b 10/26/2012 [1190308.540448] task: ffff88085462f400 ti: ffff88085464c000 task.ti: ffff88085464c000 [1190308.540449] RIP: 0010:[] [] intel_idle+0xcc/0x140 [1190308.540451] RSP: 0018:ffff88085464dda8 EFLAGS: 00000046 [1190308.540453] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001 [1190308.540454] RDX: 0000000000000000 RSI: ffff88085464dfd8 RDI: 0000000000000018 [1190308.540455] RBP: ffff88085464ddd8 R08: 0000000000000005 R09: 0000000000000009 [1190308.540456] R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000004 [1190308.540457] R13: 0000000000000020 R14: 0000000000000003 R15: 0000000000000004 [1190308.540459] FS: 0000000000000000(0000) GS:ffff88107fd00000(0000) knlGS:0000000000000000 [1190308.540460] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [1190308.540461] CR2: 00007fd28b8f1420 CR3: 0000000001a0c000 CR4: 00000000000407e0 [1190308.540462] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [1190308.540464] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [1190308.540464] Stack: [1190308.540465] ffff88085464ddd8 000000188109b864 ffffe8ffffd00400 00043b9213aa133c [1190308.540467] ffffffff81a82318 ffffffff81a821a0 ffff88085464de38 ffffffff814ebf0f [1190308.540469] 0000000000000000 00000000003cc497 0000000000000000 00000000003cc497 [1190308.540471] Call Trace: [1190308.540474] [] cpuidle_enter_state+0x4f/0xe0 [1190308.540477] [] cpuidle_idle_call+0xc0/0x220 [1190308.540479] [] arch_cpu_idle+0xe/0x30 [1190308.540481] [] cpu_startup_entry+0xe8/0x2b0 [1190308.540484] [] start_secondary+0x204/0x20b [1190308.540485] Code: 48 8b 34 25 f0 c6 00 00 48 89 d1 48 8d 86 38 e0 ff ff 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <85> 1d ee 3f 72 00 75 0e 48 8d 75 dc bf 05 00 00 00 e8 fe 33 d4 [1190308.540552] NMI backtrace for cpu 1 [1190308.540563] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 3.10.58-18 #14 [1190308.540572] Hardware name: Aquarius Aquarius Server/X9DRFF(-7), BIOS 1.0b 10/26/2012 [1190308.540575] task: ffff88085449c880 ti: ffff88085460e000 task.ti: ffff88085460e000 [1190308.540576] RIP: 0010:[] [] intel_idle+0xcc/0x140 [1190308.540588] RSP: 0018:ffff88085460fda8 EFLAGS: 00000046 [1190308.540590] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001 [1190308.540592] RDX: 0000000000000000 RSI: ffff88085460ffd8 RDI: 0000000000000001 [1190308.540594] RBP: ffff88085460fdd8 R08: 000000000000090d R09: 000000000010fbc1 [1190308.540596] R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000004 [1190308.540597] R13: 0000000000000020 R14: 0000000000000003 R15: 0000000000000004 [1190308.540600] FS: 0000000000000000(0000) GS:ffff88085fc20000(0000) knlGS:0000000000000000 [1190308.540602] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [1190308.540604] CR2: 0000000000ef8708 CR3: 0000000001a0c000 CR4: 00000000000407e0 [1190308.540606] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [1190308.540608] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [1190308.540609] Stack: [1190308.540610] ffff88085460fdd8 000000018109b864 ffffe8f7dfc20400 00043b9213aa1c05 [1190308.540614] ffffffff81a82318 ffffffff81a821a0 ffff88085460fe38 ffffffff814ebf0f [1190308.540617] 0000000000000000 00000000003cb960 0000000000000000 00000000003cb960 [1190308.540620] Call Trace: [1190308.540625] [] cpuidle_enter_state+0x4f/0xe0 [1190308.540628] [] cpuidle_idle_call+0xc0/0x220 [1190308.540632] [] arch_cpu_idle+0xe/0x30 [1190308.540635] [] cpu_startup_entry+0xe8/0x2b0 [1190308.540639] [] start_secondary+0x204/0x20b [1190308.540641] Code: 48 8b 34 25 f0 c6 00 00 48 89 d1 48 8d 86 38 e0 ff ff 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <85> 1d ee 3f 72 00 75 0e 48 8d 75 dc bf 05 00 00 00 e8 fe 33 d4 -- Regards, Roman -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/