Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S966205AbWKTQ6M (ORCPT ); Mon, 20 Nov 2006 11:58:12 -0500 Received: (majordomo@vger.kernel.org) by vger.kernel.org id S966197AbWKTQ6M (ORCPT ); Mon, 20 Nov 2006 11:58:12 -0500 Received: from e35.co.us.ibm.com ([32.97.110.153]:23478 "EHLO e35.co.us.ibm.com") by vger.kernel.org with ESMTP id S966205AbWKTQ6K (ORCPT ); Mon, 20 Nov 2006 11:58:10 -0500 Date: Mon, 20 Nov 2006 08:59:23 -0800 From: "Paul E. McKenney" To: Jens Axboe Cc: Oleg Nesterov , "Paul E. McKenney" , Alan Stern , Linus Torvalds , Thomas Gleixner , Ingo Molnar , LKML , john stultz , David Miller , Arjan van de Ven , Andrew Morton , Andi Kleen , manfred@colorfullife.com Subject: Re: [patch] cpufreq: mark cpufreq_tsc() as core_initcall_sync Message-ID: <20061120165923.GD8033@us.ibm.com> Reply-To: paulmck@linux.vnet.ibm.com References: <20061117065128.GA5452@us.ibm.com> <20061117092925.GT7164@kernel.dk> <20061117183945.GA367@oleg> <20061118002845.GF2632@us.ibm.com> <20061118184624.GA163@oleg> <20061119210746.GD4427@us.ibm.com> <20061120071514.GB4077@kernel.dk> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20061120071514.GB4077@kernel.dk> User-Agent: Mutt/1.4.1i Sender: linux-kernel-owner@vger.kernel.org X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 4231 Lines: 104 On Mon, Nov 20, 2006 at 08:15:14AM +0100, Jens Axboe wrote: > On Sun, Nov 19 2006, Paul E. McKenney wrote: > > On Sat, Nov 18, 2006 at 09:46:24PM +0300, Oleg Nesterov wrote: > > > On 11/17, Paul E. McKenney wrote: > > > > > > > > Oleg, any thoughts about Jens's optimization? He would code something > > > > like: > > > > > > > > if (srcu_readers_active(&my_srcu)) > > > > synchronize_srcu(); > > > > else > > > > smp_mb(); > > > > > > Well, this is clearly racy, no? I am not sure, but may be we can do > > > > > > smp_mb(); > > > if (srcu_readers_active(&my_srcu)) > > > synchronize_srcu(); > > > > > > in this case we also need to add 'smp_mb()' into srcu_read_lock() after > > > 'atomic_inc(&sp->hardluckref)'. > > > > > > > However, he is doing ordered I/O requests rather than protecting data > > > > structures. > > > > > > Probably this makes a difference, but I don't understand this. > > > > OK, one hypothesis here... > > > > The I/Os must be somehow explicitly ordered to qualify > > for I/O-barrier separation. If two independent processes > > issue I/Os concurrently with a third process doing an > > I/O barrier, the I/O barrier is free to separate the > > two concurrent I/Os or not, on its whim. > > > > Jens, is the above correct? If so, what would the two processes > > That's completely correct, hence my somewhat relaxed approach with SRCU. OK, less scary in that case. ;-) > > need to do in order to ensure that their I/O was considered to be > > ordered with respect to the I/O barrier? Here are some possibilities: > > If we consider the barrier a barrier in a certain stream of requests, > it is the responsibility of the issuer of that barrier to ensure that > the queueing is ordered. So if two "unrelated" streams of requests with > barriers hit __make_request() at the same time, we don't go to great > lengths to ensure who gets there firt. So the "preceding" requests have to have completed their I/O system calls? If this is the case, does this include normal (non-direct/raw) writes and asynchronous reads? My guess is that it would include asynchronous I/O, but not buffered writes. > > 1. I/O barriers apply only to preceding and following I/Os from > > the process issuing the I/O barrier. > > > > 2. As for #1 above, but restricted to task rather than process. > > > > 3. I/O system calls that have completed are ordered by the > > barrier to precede I/O system calls that have not yet > > started, but I/O system calls still in flight could legally > > land on either side of the concurrently executing I/O > > barrier. > > > > 4. Something else entirely? > > > > Given some restriction like one of the above, it is entirely possible > > that we don't even need the memory barrier... > > 3 is the closest. The request queue doesn't really know the scope of the > barrier, it has to rely on the issuer getting it right. If you have two > competing processes issuing io and process A relies on process B issuing > a barrier, they have to synchronize that between them. Normally that is > not a problem, since that's how the file systems always did io before > barriers on items that need to be on disk (it was a serialization point > anyway, it's just a stronger one now). So something like a user-level mutex or atomic instructions must be used by the tasks doing the pre-barrier I/Os to announce that these I/Os have been started in the kernel. > That said, I think the > > smp_mb(); > if (srcu_readers_active(sp)) > synchronize_srcu(); > > makes the most sense. If the user-level tasks/threads/processes must explicitly synchronize, and if the pre-barrier I/O-initation syscalls have to have completed, then I am not sure that the smp_mb() is needed. Seems like the queuing mechanisms in the syscall and the user-level synchronization would have supplied the needed memory barriers. Or are you using some extremely lightweight user-level synchronization? Thanx, Paul - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/