Date: Tue, 23 Jul 2013 11:45:13 +0200
From: Ingo Molnar <mingo@kernel.org>
To: Tim Chen <tim.c.chen@linux.intel.com>
Cc: Ingo Molnar <mingo@elte.hu>, Andrea Arcangeli <aarcange@redhat.com>,
        Mel Gorman <mgorman@suse.de>, "Shi, Alex" <alex.shi@intel.com>,
        Andi Kleen <andi@firstfloor.org>,
        Andrew Morton <akpm@linux-foundation.org>,
        Michel Lespinasse <walken@google.com>,
        Davidlohr Bueso <davidlohr.bueso@hp.com>,
        "Wilcox, Matthew R" <matthew.r.wilcox@intel.com>,
        Dave Hansen <dave.hansen@intel.com>,
        Peter Zijlstra <a.p.zijlstra@chello.nl>,
        Rik van Riel <riel@redhat.com>, linux-kernel@vger.kernel.org,
        linux-mm <linux-mm@kvack.org>
Subject: Re: Performance regression from switching lock to rw-sem for
 anon-vma tree
Message-ID: <20130723094513.GA24522@gmail.com>
References: <1372292701.22432.152.camel@schen9-DESK>
 <20130627083651.GA3730@gmail.com>
 <1372366385.22432.185.camel@schen9-DESK>
 <1372375873.22432.200.camel@schen9-DESK>
 <20130628093809.GB29205@gmail.com>
 <1372453461.22432.216.camel@schen9-DESK>
 <20130629071245.GA5084@gmail.com>
 <1372710497.22432.224.camel@schen9-DESK>
 <20130702064538.GB3143@gmail.com>
 <1373997195.22432.297.camel@schen9-DESK>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <1373997195.22432.297.camel@schen9-DESK>
User-Agent: Mutt/1.5.21 (2010-09-15)
Sender: linux-kernel-owner@vger.kernel.org
Content-Length: 1825
Lines: 44


* Tim Chen <tim.c.chen@linux.intel.com> wrote:

> Ingo,
> 
> I tried MCS locking to order the writers but it didn't make much 
> difference on my particular workload. After thinking about this some 
> more, a likely explanation of the performance difference between mutex 
> and rwsem performance is:
> 
> 1) Jobs acquiring mutex put itself on the wait list only after 
> optimistic spinning.  That's only 2% of the time on my test workload so 
> they access the wait list rarely.
> 
> 2) Jobs acquiring rw-sem for write *always* put itself on the wait list 
> first before trying lock stealing and optimistic spinning.  This creates 
> a bottleneck at the wait list, and also more cache bouncing.

Indeed ...

> One possible optimization is to delay putting the writer on the wait 
> list till after optimistic spinning, but we may need to keep track of 
> the number of writers waiting.  We could add a WAIT_BIAS to count for 
> each write waiter and remove the WAIT_BIAS each time a writer job 
> completes.  This is tricky as I'm changing the semantics of the count 
> field and likely will require a number of changes to rwsem code.  Your 
> thoughts on a better way to do this?

Why not just try the delayed addition approach first? The spinning is time 
limited AFAICS, so we don't _have to_ recognize those as writers per se, 
only if the spinning fails and it wants to go on the waitlist. Am I 
missing something?

It will change patterns, it might even change the fairness balance - but 
is a legit change otherwise, especially if it helps performance.

Thanks,

	Ingo
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/