2013-10-01 19:01:14

by Srivatsa S. Bhat

[permalink] [raw]

2013-10-02 15:18:01

by Peter Zijlstra

[permalink] [raw]

Subject: Re: [PATCH] hotplug: Optimize {get,put}_online_cpus()

On Wed, Oct 02, 2013 at 04:00:20PM +0200, Oleg Nesterov wrote:
> And again, even
>
> for (;;) {
> percpu_down_write();
> percpu_up_write();
> }
>
> should not completely block the readers.

Sure there's a tiny window, but don't forget that a reader will have to
wait for the gp_state cacheline to transfer to shared state and the
per-cpu refcount cachelines to be brought back into exclusive mode and
the above can be aggressive enough that by that time we'll observe
state == blocked again.

So I don't think that in practise a reader will get in.

Also, since the write side is exposed to userspace; you've got an
effective DoS.

So I'll stick to waitcount -- as you can see in the patches I've just
posted.

2013-10-02 16:39:03

by Oleg Nesterov

[permalink] [raw]

Subject: Re: [PATCH] hotplug: Optimize {get,put}_online_cpus()

On 10/02, Peter Zijlstra wrote:
>
> On Wed, Oct 02, 2013 at 04:00:20PM +0200, Oleg Nesterov wrote:
> > And again, even
> >
> > for (;;) {
> > percpu_down_write();
> > percpu_up_write();
> > }
> >
> > should not completely block the readers.
>
> Sure there's a tiny window, but don't forget that a reader will have to
> wait for the gp_state cacheline to transfer to shared state and the
> per-cpu refcount cachelines to be brought back into exclusive mode and
> the above can be aggressive enough that by that time we'll observe
> state == blocked again.

Sure, but don't forget that other callers of cpu_down() do a lot more
work before/after they actually call cpu_hotplug_begin/end().

> So I'll stick to waitcount -- as you can see in the patches I've just
> posted.

I still do not believe we need this waitcount "in practice" ;)

But even if I am right this is minor and we can reconsider this later,
so please forget.

Oleg.

2013-10-02 17:52:42

by Paul E. McKenney

[permalink] [raw]

Subject: Re: [PATCH] hotplug: Optimize {get,put}_online_cpus()

On Wed, Oct 02, 2013 at 04:00:20PM +0200, Oleg Nesterov wrote:
> On 10/02, Peter Zijlstra wrote:
> >
> > On Wed, Oct 02, 2013 at 02:13:56PM +0200, Oleg Nesterov wrote:
> > > In short: unless a gp elapses between _exit() and _enter(), the next
> > > _enter() does nothing and avoids synchronize_sched().
> >
> > That does however make the entire scheme entirely writer biased;
>
> Well, this makes the scheme "a bit more" writer biased, but this is
> exactly what we want in this case.
>
> We do not block the readers after xxx_exit() entirely, but we do want
> to keep them in SLOW state and avoid the costly SLOW -> FAST -> SLOW
> transitions.

Yes -- should help -a- -lot- for bulk write-side operations, such as
onlining all CPUs at boot time. ;-)

Thanx, Paul

> Lets even forget about disable_nonboot_cpus(), lets consider
> percpu_rwsem-like logic "in general".
>
> Yes, it is heavily optimizied for readers. But if the writers come in
> a batch, or the same writer does down_write + up_write twice or more,
> I think state == FAST is pointless in between (if we can avoid it).
> This is the rare case (the writers should be rare), but if it happens
> it makes sense to optimize the writers too. And again, even
>
> for (;;) {
> percpu_down_write();
> percpu_up_write();
> }
>
> should not completely block the readers.
>
> IOW. "turn sync_sched() into call_rcu_sched() in up_write()" is obviously
> a win. If the next down_write/xxx_enter "knows" that the readers are
> still in SLOW mode because gp was not completed yet, why should we
> add the artificial delay?
>
> As for disable_nonboot_cpus(). You are going to move cpu_hotplug_begin()
> outside of the loop, this is the same thing.
>
> Oleg.
>

2013-10-03 07:05:06

by Ingo Molnar

[permalink] [raw]

Subject: Re: [RFC] introduce synchronize_sched_{enter,exit}()

* Peter Zijlstra <[email protected]> wrote:

>

Fully agreed! :-)

Ingo

2013-10-03 07:43:28

by Peter Zijlstra

[permalink] [raw]

Subject: Re: [RFC] introduce synchronize_sched_{enter,exit}()

On Thu, Oct 03, 2013 at 09:04:59AM +0200, Ingo Molnar wrote:
>
> * Peter Zijlstra <[email protected]> wrote:
>
> >
>
> Fully agreed! :-)

haha.. never realized I send that email completely empty. It was
supposed to contain the patch I later send as 2/3.