The following set of not-thoroughly-tested patches are based on the
discussion of holding the ipc lock unnecessarily, such as for permissions
and security checks:
https://lkml.org/lkml/2013/2/28/540
Patch 0/1: Introduces new functions, analogous to ipc_lock and ipc_lock_check
in the ipc utility code, allowing to obtain the ipc object without holding the lock.
Patch 0/2: Use the new functions and only acquire the ipc lock when needed.
With Rik's semop-multi.c microbenchmark we can see the following
results:
256 sems without patches:
+ 59.40% a.out [kernel.kallsyms] [k] _raw_spin_lock
+ 6.14% a.out [kernel.kallsyms] [k] sys_semtimedop
+ 3.84% a.out [kernel.kallsyms] [k] avc_has_perm_flags
+ 3.64% a.out [kernel.kallsyms] [k] __audit_syscall_exit
+ 2.06% a.out [kernel.kallsyms] [k] copy_user_enhanced_fast_string
+ 1.86% a.out [kernel.kallsyms] [k] ipc_lock
+ 1.75% a.out [kernel.kallsyms] [k] __audit_syscall_entry
+ 1.69% a.out [kernel.kallsyms] [k] ipc_has_perm.isra.21
+ 1.47% a.out [kernel.kallsyms] [k] do_smart_update
+ 1.43% a.out [kernel.kallsyms] [k] pid_vnr
+ 1.39% a.out [kernel.kallsyms] [k] try_atomic_semop.isra.5
total operations: 151452270, ops/sec 5048409
256 sems with patches:
+ 17.47% a.out [kernel.kallsyms] [k] _raw_spin_lock
+ 11.08% a.out [kernel.kallsyms] [k] sys_semtimedop
+ 8.81% a.out [kernel.kallsyms] [k] avc_has_perm_flags
+ 7.96% a.out [kernel.kallsyms] [k] ipc_has_perm.isra.21
+ 6.50% a.out [kernel.kallsyms] [k] __audit_syscall_exit
+ 4.67% a.out [kernel.kallsyms] [k] ipc_obtain_object_check
+ 4.19% a.out [kernel.kallsyms] [k] ipcperms
+ 3.75% a.out [kernel.kallsyms] [k] copy_user_enhanced_fast_string
+ 3.38% a.out [kernel.kallsyms] [k] system_call
+ 3.05% a.out [kernel.kallsyms] [k] try_atomic_semop.isra.5
+ 2.70% a.out [kernel.kallsyms] [k] do_smart_update
+ 2.60% a.out [kernel.kallsyms] [k] __audit_syscall_entry
total operations: 266502912, ops/sec 8883430
While the _raw_spin_lock time is drastically reduced, others do increase.
This results in an overall speedup of ~1.7x regarding ops/sec.
On Fri, Mar 1, 2013 at 4:16 PM, Davidlohr Bueso <[email protected]> wrote:
>
> With Rik's semop-multi.c microbenchmark we can see the following
> results:
Ok, that certainly looks very good.
> + 59.40% a.out [kernel.kallsyms] [k] _raw_spin_lock
> + 17.47% a.out [kernel.kallsyms] [k] _raw_spin_lock
I had somewhat high expectations, but that's just better than I really
hoped for. Not only is the percentage down, it's down for the case of
a much smaller number of overall cycle cost, so it's a really big
reduction in contention spinning.
Of course, contention will come back and overwhelm you at *some*
point, but it seems the patches certainly moved the really bad
contention point out some way..
> + 6.14% a.out [kernel.kallsyms] [k] sys_semtimedop
> + 11.08% a.out [kernel.kallsyms] [k] sys_semtimedop
> While the _raw_spin_lock time is drastically reduced, others do increase.
> This results in an overall speedup of ~1.7x regarding ops/sec.
Actually, the others don't really increase. Sure, the *percentages* go
up, but that's just because it has to add up to 100% in the end. So
it's not that you're moving costs from one place to another - the 1.7x
speedup is the real reduction in costs, and then that 6.14% -> 11.08%
"growth" is really nothing but that (and yes, 1.7 x 6.14 really does
get pretty close).
So nothing really got slower, despite the percentages going up.
Looks good to me. Of course, the *real* issue is if this is a win on
real code too. And I bet it is, it just won't be quite as noticeable.
But if anything, real code is likely to have less contention to begin
with, because it has more things going on outside of the spinlocks. So
it should see an improvement, but not nearly the kind of improvement
you quote here.
Although your 800-user swingbench numbers were pretty horrible, so
maybe that case can improve by comparable amounts in the bad cases.
Linus
On Fri, Mar 1, 2013 at 5:32 PM, Linus Torvalds
<[email protected]> wrote:
> On Fri, Mar 1, 2013 at 4:16 PM, Davidlohr Bueso <[email protected]> wrote:
>>
>> With Rik's semop-multi.c microbenchmark we can see the following
>> results:
>
> Ok, that certainly looks very good.
Side note: it's fairly late in the merge window, and I don't feel
comfy merging this without more testing and more people looking at it,
so I think this is a 3.10 thing. But it would be great to see ipc
people really look at the patches, and perhaps run a few real loads to
see if it's even noticeable in practice..
Linus
Hi,
On Sat, Mar 2, 2013 at 7:16 AM, Davidlohr Bueso <[email protected]> wrote:
> The following set of not-thoroughly-tested patches are based on the
> discussion of holding the ipc lock unnecessarily, such as for permissions
> and security checks:
>
> https://lkml.org/lkml/2013/2/28/540
>
> Patch 0/1: Introduces new functions, analogous to ipc_lock and ipc_lock_check
> in the ipc utility code, allowing to obtain the ipc object without holding the lock.
>
> Patch 0/2: Use the new functions and only acquire the ipc lock when needed.
Not sure how much a work in progress this is but my machine dies
immediately when I start chromium, crappy mobile phone picture here:
http://i.imgur.com/S0hfPz3.jpg
Thanks.
On Sat, Mar 2, 2013 at 12:43 PM, Emmanuel Benisty <[email protected]> wrote:
> Hi,
>
> On Sat, Mar 2, 2013 at 7:16 AM, Davidlohr Bueso <[email protected]> wrote:
>> The following set of not-thoroughly-tested patches are based on the
>> discussion of holding the ipc lock unnecessarily, such as for permissions
>> and security checks:
>>
>> https://lkml.org/lkml/2013/2/28/540
>>
>> Patch 0/1: Introduces new functions, analogous to ipc_lock and ipc_lock_check
>> in the ipc utility code, allowing to obtain the ipc object without holding the lock.
>>
>> Patch 0/2: Use the new functions and only acquire the ipc lock when needed.
>
> Not sure how much a work in progress this is but my machine dies
> immediately when I start chromium, crappy mobile phone picture here:
> http://i.imgur.com/S0hfPz3.jpg
We are missing the top of the trace there, so it's hard to be sure -
however, this could well be caused by the if (!out) check (instead of
if (IS_ERR(out)) that I noticed in patch 1/2.
--
Michel "Walken" Lespinasse
A program is never fully debugged until the last user dies.
On Sat, Mar 2, 2013 at 2:08 PM, Michel Lespinasse <[email protected]> wrote:
> On Sat, Mar 2, 2013 at 12:43 PM, Emmanuel Benisty <[email protected]> wrote:
>> Hi,
>>
>> On Sat, Mar 2, 2013 at 7:16 AM, Davidlohr Bueso <[email protected]> wrote:
>>> The following set of not-thoroughly-tested patches are based on the
>>> discussion of holding the ipc lock unnecessarily, such as for permissions
>>> and security checks:
>>>
>>> https://lkml.org/lkml/2013/2/28/540
>>>
>>> Patch 0/1: Introduces new functions, analogous to ipc_lock and ipc_lock_check
>>> in the ipc utility code, allowing to obtain the ipc object without holding the lock.
>>>
>>> Patch 0/2: Use the new functions and only acquire the ipc lock when needed.
>>
>> Not sure how much a work in progress this is but my machine dies
>> immediately when I start chromium, crappy mobile phone picture here:
>> http://i.imgur.com/S0hfPz3.jpg
>
> We are missing the top of the trace there, so it's hard to be sure -
> however, this could well be caused by the if (!out) check (instead of
> if (IS_ERR(out)) that I noticed in patch 1/2.
Merci Michel but unfortunately, I'm still getting the same issue.
On Sat, 2013-03-02 at 15:35 +0700, Emmanuel Benisty wrote:
> On Sat, Mar 2, 2013 at 2:08 PM, Michel Lespinasse <[email protected]> wrote:
> > On Sat, Mar 2, 2013 at 12:43 PM, Emmanuel Benisty <[email protected]> wrote:
> >> Hi,
> >>
> >> On Sat, Mar 2, 2013 at 7:16 AM, Davidlohr Bueso <[email protected]> wrote:
> >>> The following set of not-thoroughly-tested patches are based on the
> >>> discussion of holding the ipc lock unnecessarily, such as for permissions
> >>> and security checks:
> >>>
> >>> https://lkml.org/lkml/2013/2/28/540
> >>>
> >>> Patch 0/1: Introduces new functions, analogous to ipc_lock and ipc_lock_check
> >>> in the ipc utility code, allowing to obtain the ipc object without holding the lock.
> >>>
> >>> Patch 0/2: Use the new functions and only acquire the ipc lock when needed.
> >>
> >> Not sure how much a work in progress this is but my machine dies
> >> immediately when I start chromium, crappy mobile phone picture here:
> >> http://i.imgur.com/S0hfPz3.jpg
> >
> > We are missing the top of the trace there, so it's hard to be sure -
> > however, this could well be caused by the if (!out) check (instead of
> > if (IS_ERR(out)) that I noticed in patch 1/2.
>
> Merci Michel but unfortunately, I'm still getting the same issue.
Will try to reproduce (and further testing on other machines) and debug
later today.
Thanks for testing,
Davidlohr
On Fri, 2013-03-01 at 17:32 -0800, Linus Torvalds wrote:
> On Fri, Mar 1, 2013 at 4:16 PM, Davidlohr Bueso <[email protected]> wrote:
> >
> > With Rik's semop-multi.c microbenchmark we can see the following
> > results:
>
> Ok, that certainly looks very good.
>
> > + 59.40% a.out [kernel.kallsyms] [k] _raw_spin_lock
> > + 17.47% a.out [kernel.kallsyms] [k] _raw_spin_lock
>
> I had somewhat high expectations, but that's just better than I really
> hoped for. Not only is the percentage down, it's down for the case of
> a much smaller number of overall cycle cost, so it's a really big
> reduction in contention spinning.
>
> Of course, contention will come back and overwhelm you at *some*
> point, but it seems the patches certainly moved the really bad
> contention point out some way..
>
> > + 6.14% a.out [kernel.kallsyms] [k] sys_semtimedop
> > + 11.08% a.out [kernel.kallsyms] [k] sys_semtimedop
> > While the _raw_spin_lock time is drastically reduced, others do increase.
> > This results in an overall speedup of ~1.7x regarding ops/sec.
>
> Actually, the others don't really increase. Sure, the *percentages* go
> up, but that's just because it has to add up to 100% in the end. So
> it's not that you're moving costs from one place to another - the 1.7x
> speedup is the real reduction in costs, and then that 6.14% -> 11.08%
> "growth" is really nothing but that (and yes, 1.7 x 6.14 really does
> get pretty close).
>
> So nothing really got slower, despite the percentages going up.
>
> Looks good to me. Of course, the *real* issue is if this is a win on
> real code too. And I bet it is, it just won't be quite as noticeable.
> But if anything, real code is likely to have less contention to begin
> with, because it has more things going on outside of the spinlocks. So
> it should see an improvement, but not nearly the kind of improvement
> you quote here.
>
> Although your 800-user swingbench numbers were pretty horrible, so
> maybe that case can improve by comparable amounts in the bad cases.
>
Absolutely, I'll be sure to try these changes with my Oracle workloads
and report with some numbers. This obviously still needs a lot of
testing.
Thanks,
Davidlohr
