Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753609AbZIPP3a (ORCPT ); Wed, 16 Sep 2009 11:29:30 -0400 Received: (majordomo@vger.kernel.org) by vger.kernel.org id S1750757AbZIPP33 (ORCPT ); Wed, 16 Sep 2009 11:29:29 -0400 Received: from e8.ny.us.ibm.com ([32.97.182.138]:45041 "EHLO e8.ny.us.ibm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750736AbZIPP32 (ORCPT ); Wed, 16 Sep 2009 11:29:28 -0400 Date: Wed, 16 Sep 2009 08:29:29 -0700 From: "Paul E. McKenney" To: Catalin Marinas Cc: Eric Sesterhenn , linux-kernel Subject: Re: RCU callbacks and TREE_PREEMPT_RCU Message-ID: <20090916152929.GA6737@linux.vnet.ibm.com> Reply-To: paulmck@linux.vnet.ibm.com References: <1253110641.31751.92.camel@pc1117.cambridge.arm.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1253110641.31751.92.camel@pc1117.cambridge.arm.com> User-Agent: Mutt/1.5.15+20070412 (2007-04-11) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 7064 Lines: 143 On Wed, Sep 16, 2009 at 03:17:21PM +0100, Catalin Marinas wrote: > Hi Paul, > > Eric was reporting some issues with kmemleak on 2.6.31 accessing freed > memory under heavy stress (using the "stress" application). Basically, > the system gets into an oom state (because of "stress -m 1000") and > kmemleak fails to allocate its metadata (correct behaviour so far). At > that point, it disables itself and schedules the clean-up work which > does this (among other locking, the kmemleak_do_cleanup function the > latest mainline): > > rcu_read_lock(); > list_for_each_entry_rcu(object, &object_list, object_list) > delete_object_full(object->pointer); > rcu_read_unlock(); > > The kmemleak objects are freed via put_object() with: > > call_rcu(&object->rcu, free_object_rcu); > > (the free_object_rcu calls kmem_cache_free). > > When TREE_PREEMPT_RCU is enabled, the rcu list traversing above fails > with access to 0x6b6b6b6b but it is fine with TREE_PREEMPT_RCU=n and > TREE_RCU=y. During clean-up, kmemleak objects should no longer be freed > by other means since kmemleak was disabled and all callbacks are > ignored. The system is a 900Mhz P3, 256MB RAM, CONFIG_SMP=n. > > Is there something I'm doing wrong in kmemleak or a bug with RCU > preemption? The kernel oops looks like this: >From your description and the code above, I must suspect a bug with RCU preemption. A new one, as the only bugs I am currently chasing involve NR_CPUS>32 (>64 on 64-bit systems). CONFIG_SMP=n implies NR_CPUS==1 in your build, correct? Thanx, Paul > [ 5346.582119] kmemleak: Cannot allocate a kmemleak_object structure > [ 5346.582208] Pid: 31302, comm: stress Not tainted 2.6.31-01335-g86d7101 #5 > [ 5346.582313] Call Trace: > [ 5346.582414] [] create_object+0x215/0x220 > [ 5346.582529] [] ? alloc_arch_preferred_bootmem+0x30/0x50 > [ 5346.582628] [] ? mark_held_locks+0x52/0x70 > [ 5346.582734] [] ? alloc_arch_preferred_bootmem+0x30/0x50 > [ 5346.582823] [] ? __free+0x38/0x90 > [ 5346.582941] [] kmemleak_alloc+0x2b/0x60 > [ 5346.705312] [] kmem_cache_alloc+0x11c/0x1a0 > [ 5346.705453] [] ? cfq_set_request+0xf3/0x310 > [ 5346.705573] [] ? alloc_arch_preferred_bootmem+0x30/0x50 > [ 5346.705660] [] ? get_io_context+0x13/0x40 > [ 5346.705765] [] ? cfq_set_request+0x0/0x310 > [ 5346.705850] [] cfq_set_request+0xf3/0x310 > [ 5346.705968] [] ? trace_hardirqs_on_caller+0x12c/0x180 > [ 5346.706133] [] ? cfq_set_request+0x0/0x310 > [ 5346.706230] [] elv_set_request+0x1f/0x50 > [ 5346.706342] [] get_request+0x27c/0x2f0 > [ 5346.706426] [] get_request_wait+0xe2/0x140 > [ 5346.706545] [] ? autoremove_wake_function+0x0/0x40 > [ 5346.706638] [] __make_request+0x89/0x3e0 > [ 5346.706744] [] generic_make_request+0x192/0x400 > [ 5346.706835] [] submit_bio+0x71/0x110 > [ 5346.706939] [] ? trace_hardirqs_on_caller+0x12c/0x180 > [ 5346.797327] [] ? trace_hardirqs_on+0xb/0x10 > [ 5346.797478] [] ? _spin_unlock_irqrestore+0x39/0x70 > [ 5346.797597] [] ? test_set_page_writeback+0x6d/0x140 > [ 5346.797699] [] swap_writepage+0x9a/0xd0 > [ 5346.797804] [] ? end_swap_bio_write+0x0/0x80 > [ 5346.797895] [] shrink_page_list+0x316/0x700 > [ 5346.798003] [] ? __lock_acquire+0x40f/0xab0 > [ 5346.798170] [] ? validate_chain+0xe9/0x1030 > [ 5346.798260] [] shrink_list+0x1da/0x4e0 > [ 5346.798370] [] shrink_zone+0x297/0x310 > [ 5346.798454] [] ? shrink_slab+0x161/0x1a0 > [ 5346.798563] [] try_to_free_pages+0x1e1/0x2e0 > [ 5346.798650] [] ? isolate_pages_global+0x0/0x1e0 > [ 5346.798774] [] __alloc_pages_nodemask+0x35e/0x5d0 > [ 5346.798864] [] do_wp_page+0xb7/0x690 > [ 5346.798968] [] ? handle_mm_fault+0x263/0x600 > [ 5346.929240] [] ? _spin_lock+0x65/0x70 > [ 5346.929378] [] handle_mm_fault+0x465/0x600 > [ 5346.929496] [] ? do_page_fault+0x14b/0x390 > [ 5346.929589] [] ? down_read_trylock+0x5c/0x70 > [ 5346.929696] [] do_page_fault+0x1b0/0x390 > [ 5346.929780] [] ? do_page_fault+0x0/0x390 > [ 5346.929884] [] error_code+0x70/0x78 > [ 5347.889442] BUG: unable to handle kernel paging request at 6b6b6b6b > [ 5347.889626] IP: [] kmemleak_do_cleanup+0x60/0xa0 > [ 5347.889835] *pde = 00000000 > [ 5347.889933] Oops: 0000 [#1] PREEMPT > [ 5347.890038] last sysfs file: /sys/class/vc/vcsa9/dev > [ 5347.890038] Modules linked in: [last unloaded: rcutorture] > [ 5347.890038] > [ 5347.890038] Pid: 5, comm: events/0 Not tainted (2.6.31-01335-g86d7101 #5) > System Name > [ 5347.890038] EIP: 0060:[] EFLAGS: 00010286 CPU: 0 > [ 5347.890038] EIP is at kmemleak_do_cleanup+0x60/0xa0 > [ 5347.890038] EAX: 002ed661 EBX: 6b6b6b43 ECX: 00000007 EDX: 6b6b6b6b > [ 5347.890038] ESI: cf8b40b0 EDI: 00000002 EBP: cf8b8f3c ESP: cf8b8f28 > [ 5347.890038] DS: 007b ES: 007b FS: 0000 GS: 00e0 SS: 0068 > [ 5347.890038] Process events/0 (pid: 5, ti=cf8b8000 task=cf8c3500 > task.ti=cf8b8000) > [ 5347.890038] Stack: > [ 5347.890038] 00000002 00000001 00000000 c01c3180 c0cd6640 cf8b8f98 c0142857 > 00000000 > [ 5347.890038] <0> 00000002 00000000 c01427f6 cf8b40d4 cf8b40dc cf8c3500 > c01c3180 c0cd6640 > [ 5347.890038] <0> c0f938b0 c0a89514 00000000 00000000 00000000 cf8c3500 > c0146290 cf8b8f84 > [ 5347.890038] Call Trace: > [ 5347.890038] [] ? kmemleak_do_cleanup+0x0/0xa0 > [ 5347.890038] [] ? worker_thread+0x1d7/0x300 > [ 5347.890038] [] ? worker_thread+0x176/0x300 > [ 5347.890038] [] ? kmemleak_do_cleanup+0x0/0xa0 > [ 5347.890038] [] ? autoremove_wake_function+0x0/0x40 > [ 5347.890038] [] ? worker_thread+0x0/0x300 > [ 5347.890038] [] ? kthread+0x77/0x80 > [ 5347.890038] [] ? kthread+0x0/0x80 > [ 5347.890038] [] ? kernel_thread_helper+0x7/0x1c > [ 5347.890038] Code: 89 44 24 04 b8 e0 2c cd c0 c7 04 24 02 00 00 00 e8 76 7f > f9 ff 8b 15 d0 66 cd c0 eb 0b 8b 43 58 e8 76 ff ff ff 8b 53 28 8d 5a d8 <8b> > 43 28 0f 18 00 90 81 fa d0 66 cd c0 75 e3 b9 ef 31 1c c0 ba > [ 5347.890038] EIP: [] kmemleak_do_cleanup+0x60/0xa0 SS:ESP > 0068:cf8b8f28 > [ 5347.890038] CR2: 000000006b6b6b6b > > > Thanks. > > -- > Catalin > > -- > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > Please read the FAQ at http://www.tux.org/lkml/ -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/