Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753782AbbFQUx0 (ORCPT ); Wed, 17 Jun 2015 16:53:26 -0400 Received: from mail-pa0-f42.google.com ([209.85.220.42]:35019 "EHLO mail-pa0-f42.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750954AbbFQUxT (ORCPT ); Wed, 17 Jun 2015 16:53:19 -0400 Message-ID: <5581DE3D.1010609@plumgrid.com> Date: Wed, 17 Jun 2015 13:53:17 -0700 From: Alexei Starovoitov User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: paulmck@linux.vnet.ibm.com CC: Daniel Wagner , LKML , rostedt@goodmis.org Subject: Re: call_rcu from trace_preempt References: <20150615230702.GB3913@linux.vnet.ibm.com> <557F7764.5060707@plumgrid.com> <20150616021458.GE3913@linux.vnet.ibm.com> <557FB7E1.6080004@plumgrid.com> <20150616122733.GG3913@linux.vnet.ibm.com> <558018DD.1080701@monom.org> <55805AC5.8020507@plumgrid.com> <55812BC1.4010604@bmw-carit.de> <5581385D.9060608@bmw-carit.de> <5581BEE1.5060302@plumgrid.com> <20150617203745.GR3913@linux.vnet.ibm.com> In-Reply-To: <20150617203745.GR3913@linux.vnet.ibm.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 3995 Lines: 77 On 6/17/15 1:37 PM, Paul E. McKenney wrote: > On Wed, Jun 17, 2015 at 11:39:29AM -0700, Alexei Starovoitov wrote: >> On 6/17/15 2:05 AM, Daniel Wagner wrote: >>>>> Steven's suggestion deferring the work via irq_work results in the same >>>>> stack trace. (Now I get cold feets, without the nice heat from the CPU >>>>> busy looping...) >>> That one still not working. It also makes the system really really slow. >>> I guess I still do something completely wrong. >> >> tried your irq_work patch. It indeed makes the whole system >> unresponsive. Ctrl-C of hwlathist no longer works and >> it runs out of memory in 20 sec or so of running hwlathist >> on idle system (without parallel hackbench). >> It looks that free_pending flag is racy, so I removed it, >> but it didn't help. >> >> Also I've tried all sort of other things in rcu including >> add rcu_bpf similar to rcu_sched to make sure that recursive >> call into call_rcu will not be messing rcu_preempt or rcu_sched >> states and instead will be operating on rcu_bpf per-cpu states. >> In theory that should have worked flawlessly and it sort-of did. >> But multiple hackbench runs still managed to crash it. >> So far I think the temp workaround is to stick with array maps >> for probing such low level things like trace_preempt. >> Note that pre-allocation of all elements in hash map also won't >> help, since the problem here is some collision of call_rcu and >> rcu_process_callbacks. I'm pretty sure that kfree_rcu with >> rcu_is_watching patch is ready for this type of abuse. >> The rcu_process_callbacks() path - no yet. I'm still analyzing it. > > How about if I just gave you a hook in __call_rcu() itself, just before > it returns, just after the local_irq_restore()? You could maintain > recursion flags and test the environment, at some point handling any > memory that needed freeing. > > The idea would be to use an atomic queue to accumulate the to-be-freed > data, then kfree_rcu() it in the hook if it was safe to do so. I'm not yet seeing how it will look. You mean I'll just locklessly enqueue into some global llist and it will get kfree-d from rcu_process_callbacks() ? Something like kfree_rcu_lockless(ptr_to_be_freed) that llist_adds and let rcu core know that something has to be freed? I think such feature would be very useful in general. Or may be kfree_rcu_this_cpu(ptr_to_be_freed) that uses per-cpu llist of 'to-be-kfreed' objects? Performance will be great and not need to embed rcu_head in every datastructure. btw, irq_work suffers the same re-entrancy problem: [ 19.914910] [] free_work_cb+0x2c/0x50 [ 19.914910] [] irq_work_run_list+0x44/0x70 [ 19.914910] [] irq_work_run+0x2e/0x50 [ 19.914910] [] smp_irq_work_interrupt+0x2e/0x40 [ 19.914910] [] irq_work_interrupt+0x70/0x80 [ 19.914910] [] ? debug_object_active_state+0xfc/0x150 [ 19.914910] [] ? debug_object_active_state+0xfc/0x150 [ 19.914910] [] ? _raw_spin_unlock_irqrestore+0x4b/0x80 [ 19.914910] [] ? trace_preempt_on+0x7/0x100 [ 19.914910] [] ? preempt_count_sub+0x73/0xf0 [ 19.914910] [] _raw_spin_unlock_irqrestore+0x4b/0x80 [ 19.914910] [] debug_object_active_state+0xfc/0x150 [ 19.914910] [] ? get_max_files+0x20/0x20 [ 19.914910] [] __call_rcu.constprop.67+0x5f/0x350 [ 19.914910] [] call_rcu+0x17/0x20 [ 19.914910] [] __fput+0x183/0x200 so if I do call_rcu from free_work_cb, it's equally bad as calling call_rcu from trace_preempt_on -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/