Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Subject: Re: [RFC PATCH 5/5] locking/rwsem: Remove reader optimistic spinning
To:     David Laight <David.Laight@ACULAB.COM>,
        Davidlohr Bueso <dave@stgolabs.net>
Cc:     Peter Zijlstra <peterz@infradead.org>,
        Ingo Molnar <mingo@redhat.com>, Will Deacon <will@kernel.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        Phil Auld <pauld@redhat.com>
References: <20201118030429.23017-1-longman@redhat.com>
 <20201118030429.23017-6-longman@redhat.com>
 <20201118053556.3fmmtat7upv6dtvd@linux-p48b.lan>
 <ee34bc01-9fef-23ff-ada1-1ec2d39533c9@redhat.com>
 <5fe76531782f4a8492b341d5f381147b@AcuMS.aculab.com>
From:   Waiman Long <longman@redhat.com>
Organization: Red Hat
Message-ID: <68d07068-ff31-26b5-f90d-7ea8b4ee2389@redhat.com>
Date:   Fri, 20 Nov 2020 12:04:24 -0500
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101
 Thunderbird/68.12.0
MIME-Version: 1.0
In-Reply-To: <5fe76531782f4a8492b341d5f381147b@AcuMS.aculab.com>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit
Content-Language: en-US
Precedence: bulk

On 11/20/20 8:11 AM, David Laight wrote:
> From: Waiman Long
>> Sent: 19 November 2020 18:40
> ...
>> My own testing also show not too much performance difference when
>> removing reader spinning except in the lightly loaded cases.
> I'm confused.
>
> I got massive performance improvements from changing a driver
> we have to use mutex instead of the old semaphores (the driver
> was written a long time ago).
>
> While these weren't 'rw' the same issue will apply.
>
> The problem was that the semaphore/mutex was typically only held over
> a few instructions (eg to add an item to a list).
> But with semaphore if you got contention the process always slept.
> OTOH mutex spin 'for a while' before sleeping so the code rarely slept.
>
> So I really expect that readers need to spin (for a while) if
> a rwsem (etc) is locked for writing.
>
> Clearly you really need a CBU (Crystal Ball Unit) to work out
> whether to spin or not.

That is the hard part. For short critical section and not many readers 
around, making the readers spin will likely improve performance. On the 
other hand, if the critical section is long with many readers, make 
readers sleep and then wake them all up at once can have better 
performance. There is no one-size-fit-all solution here.

Cheers,
Longman