From: Jan Kara Subject: Re: jbd2: don't wake kjournald unnecessarily Date: Wed, 23 Jan 2013 20:29:11 +0100 Message-ID: <20130123192911.GB21112@quack.suse.cz> References: <20130121104733.GE5588@quack.suse.cz> <20130121140738.GI5588@quack.suse.cz> <20130121231130.GB12410@thunk.org> <20130122235041.GA7497@quack.suse.cz> <50FF3EEA.2030408@redhat.com> <20130123094421.GB9821@quack.suse.cz> <50FFFFC6.8070308@redhat.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Cc: Jan Kara , Theodore Ts'o , Sedat Dilek , linux-fsdevel , Ext4 Developers List , LKML , linux-next , mszeredi@suse.cz To: Eric Sandeen Return-path: Received: from cantor2.suse.de ([195.135.220.15]:34944 "EHLO mx2.suse.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750813Ab3AWT3N (ORCPT ); Wed, 23 Jan 2013 14:29:13 -0500 Content-Disposition: inline In-Reply-To: <50FFFFC6.8070308@redhat.com> Sender: linux-ext4-owner@vger.kernel.org List-ID: On Wed 23-01-13 09:20:38, Eric Sandeen wrote: > On 1/23/13 3:44 AM, Jan Kara wrote: > > On Tue 22-01-13 19:37:46, Eric Sandeen wrote: > >> On 1/22/13 5:50 PM, Jan Kara wrote: > >>> On Mon 21-01-13 18:11:30, Ted Tso wrote: > >>>> On Tue, Jan 22, 2013 at 12:04:32AM +0100, Sedat Dilek wrote: > >>>>> > >>>>> Beyond the FUSE/LOOP fun, will you apply this patch to your linux-next GIT tree? > >>>>> > >>>>> Feel free to add... > >>>>> > >>>>> Tested-by: Sedat Dilek > >>>>> > >>>>> A similiar patch for JBD went through your tree into mainline (see [1] and [2]). > >>>> > >>>> I'm not at all convinced that this patch has anything to do with your > >>>> problem. I don't see how it could affect things, and I believe you > >>>> mentioned that you saw the problem even with this patch applied? (I'm > >>>> not sure; some of your messages which you sent were hard to > >>>> understand, and you mentioned something about trying to send messages > >>>> when low on sleep :-). > >>>> > >>>> In any case, the reason why I haven't pulled this patch into the ext4 > >>>> tree is because I was waiting for Eric and some of the performance > >>>> team folks at Red Hat to supply some additional information about why > >>>> this commit was making a difference in performance for a particular > >>>> proprietary, closed source benchmark. > >>> Just a small correction - it was aim7 AFAIK which isn't closed source > >>> (anymore). You can download it from SourceForge > >>> (http://sourceforge.net/projects/aimbench/files/aim-suite7/Initial%20release/). > >>> Now I have some reservations about what the benchmark does but historically > >>> it has found quite a few issues for us as well. > >>> > >>>> I'm very suspicious about applying patches under the "cargo cult" > >>>> school of programming. ("We don't understand why it makes a > >>>> difference, but it seems to be good, so bombs away!" :-) > >>> Well, neither am I ;) But it is obvious the patch speeds up > >>> log_start_commit() by 'a bit' (taking spinlock, disabling irqs, ...). And > >>> apparently 'a bit' is noticeable for particular workload on a particular > >>> machine - commit statistics Eric provided showed that clearly. I'd still be > >>> happier if Eric also told us how much log_start_commit() calls there were > >>> so that one could verify that 'a bit' could indeed multiply to a measurable > >>> difference. But given how simple the patch is, I gave away after a while > >>> and just merged it... > >> > >> I am still trying to get our perf guys to collect that data, FWIW... > >> I will send it when I get it. I bugged them again today. :) > >> > >> (Just to be sure: I was going to measure the wakeups the old way, and the > >> avoided wakeups with the new change; sound ok?) > > Yes, that would be what I'm interested in. > > Holy cow, this is much more than I expected, but here's what they report: > > old JBD: AIM7 jobs/min 97624.39; got 78193 jbd wakeups > new JBD: AIM7 jobs/min 85929.43; got 6306999 jbd wakeups, 6264684 extra wakeups Yeah, that's a lot. My guess would be *a lot* of processes are hanging in start_this_handle() and waiting for transaction commit. Each of them calls __log_start_commit() and things add up. Thanks for getting these numbers. Honza -- Jan Kara SUSE Labs, CR