2006-03-29 22:08:43

by Pavel Machek

[permalink] [raw]
Subject: [rfc] fix Kconfig, hotplug_cpu is needed for swsusp

HOTPLUG_CPU is needed on normal PCs, too -- it is neccessary for
software suspend.

Signed-off-by: Pavel Machek <[email protected]>

---
commit 2206dab43d50723d6b15caa8821e8d97c6b5ef28
tree cbfc324e15d216aa91ce6a51927668076de5b7db
parent dd76aabd03933b80c61fa5b0c0c995950246c603
author <[email protected]> Thu, 30 Mar 2006 00:06:31 +0200
committer <[email protected]> Thu, 30 Mar 2006 00:06:31 +0200

arch/i386/Kconfig | 8 ++++----
kernel/power/process.c | 3 +--
kernel/signal.c | 1 +
3 files changed, 6 insertions(+), 6 deletions(-)

diff --git a/arch/i386/Kconfig b/arch/i386/Kconfig
index f7db71d..955dc08 100644
--- a/arch/i386/Kconfig
+++ b/arch/i386/Kconfig
@@ -741,12 +741,12 @@ config PHYSICAL_START

config HOTPLUG_CPU
bool "Support for hot-pluggable CPUs (EXPERIMENTAL)"
- depends on SMP && HOTPLUG && EXPERIMENTAL && !X86_VOYAGER && !X86_PC
+ depends on SMP && HOTPLUG && EXPERIMENTAL && !X86_VOYAGER
---help---
- Say Y here to experiment with turning CPUs off and on. CPUs
- can be controlled through /sys/devices/system/cpu.
+ Say Y here to experiment with turning CPUs off and on, and to
+ enable suspend on SMP systems. CPUs can be controlled through
+ /sys/devices/system/cpu.

- Say N.

config DOUBLEFAULT
default y

--
Picture of sleeping (Linux) penguin wanted...


2006-03-29 22:45:28

by Andrew Morton

[permalink] [raw]
Subject: Re: [rfc] fix Kconfig, hotplug_cpu is needed for swsusp

Pavel Machek <[email protected]> wrote:
>
> HOTPLUG_CPU is needed on normal PCs, too -- it is neccessary for
> software suspend.
>
> Signed-off-by: Pavel Machek <[email protected]>
>
> ---
> commit 2206dab43d50723d6b15caa8821e8d97c6b5ef28
> tree cbfc324e15d216aa91ce6a51927668076de5b7db
> parent dd76aabd03933b80c61fa5b0c0c995950246c603
> author <[email protected]> Thu, 30 Mar 2006 00:06:31 +0200
> committer <[email protected]> Thu, 30 Mar 2006 00:06:31 +0200
>
> arch/i386/Kconfig | 8 ++++----
> kernel/power/process.c | 3 +--
> kernel/signal.c | 1 +
> 3 files changed, 6 insertions(+), 6 deletions(-)
>
> diff --git a/arch/i386/Kconfig b/arch/i386/Kconfig
> index f7db71d..955dc08 100644
> --- a/arch/i386/Kconfig
> +++ b/arch/i386/Kconfig
> @@ -741,12 +741,12 @@ config PHYSICAL_START
>
> config HOTPLUG_CPU
> bool "Support for hot-pluggable CPUs (EXPERIMENTAL)"
> - depends on SMP && HOTPLUG && EXPERIMENTAL && !X86_VOYAGER && !X86_PC
> + depends on SMP && HOTPLUG && EXPERIMENTAL && !X86_VOYAGER
> ---help---
> - Say Y here to experiment with turning CPUs off and on. CPUs
> - can be controlled through /sys/devices/system/cpu.
> + Say Y here to experiment with turning CPUs off and on, and to
> + enable suspend on SMP systems. CPUs can be controlled through
> + /sys/devices/system/cpu.
>
> - Say N.
>

OK, this will get ugly. APICs are involved.

That patch simply undoes this one:

> [PATCH] x86: make CONFIG_HOTPLUG_CPU depend on !X86_PC
>
> Make CONFIG_HOTPLUG_CPU depend on !X86_PC, so we need to turn on either
> CONFIG_GENERICARCH, CONFIG_BIGSMP or any other subarch except X86_PC when
> CONFIG_HOTPLUG_CPU=y
>
> With 2.6.15+ kernels when CONFIG_HOTPLUG_CPU is turned on we switch to
> bigsmp mode for sending IPI's and ioapic configurations that caused the
> following error message.
>
> >> More than 8 CPUs detected and CONFIG_X86_PC cannot handle it.
> >> Use CONFIG_X86_GENERICARCH or CONFIG_X86_BIGSMP.
>
> Originally bigsmp was added just to handle >8 cpus, but now with hotplug
> cpu support we need to use bigsmp mode (why? see below), that cause the
> above error message even if there were less than 8 cpus in the system.
>
> The message is bogus, but we are cannot use logical flat mode due to issues
> with broadcast IPI can confuse a CPU just comming up. We use flat physical
> mode just like x86_64 case. More details on why bigsmp now uses flat
> physical mode (vs. cluster mode) in following link.
>
> http://marc.theaimsgroup.com/?l=linux-kernel&m=113261865814107&w=2
>

Which I was distinctly uncomfortable with. (And glad I nixed it for 2.6.16 ;))

Ashok, we need to do something better, but what?

2006-03-29 23:10:29

by Ashok Raj

[permalink] [raw]
Subject: Re: [rfc] fix Kconfig, hotplug_cpu is needed for swsusp

On Wed, Mar 29, 2006 at 02:47:46PM -0800, Andrew Morton wrote:
> Pavel Machek <[email protected]> wrote:
> >
> > HOTPLUG_CPU is needed on normal PCs, too -- it is neccessary for
> > software suspend.
> >
>
> OK, this will get ugly. APICs are involved.

I guess you need only on systems that support >1 cpu right? I doubt you will need
it on a system that cannot run with the config-generic-arch on. although we use bigsmp
when hotplug is turned on, all we really end up is using flat physical mode instead
of using logical mode.

I still havent understood why this wont work. Choosing CONFIG_X86_GENERICARCH shouldnt
break anything AFAICT.

Pavel, you could use CONFIG_HOTPLUG_CPU, just need to enable X86_GENERICARCH now.
Is there a reason you think that wont work? I wish we would revert it for a strong
reason that we know will not make hotplug work on certain systems because of this choise
not that we currently have X86_PC now, and are unwiling to change the config.

(PS: the word bigsmp although sounds like some large NR_CPUS, its just using a mode that
permits the system to work from 1 .. >8 cpus. So there is really nothing determental
to selecting this.)


--
Cheers,
Ashok Raj
- Open Source Technology Center

2006-03-29 23:37:44

by Nigel Cunningham

[permalink] [raw]
Subject: Re: [rfc] fix Kconfig, hotplug_cpu is needed for swsusp

Hi.

On Thursday 30 March 2006 09:09, Ashok Raj wrote:
> On Wed, Mar 29, 2006 at 02:47:46PM -0800, Andrew Morton wrote:
> > Pavel Machek <[email protected]> wrote:
> > > HOTPLUG_CPU is needed on normal PCs, too -- it is neccessary for
> > > software suspend.
> >
> > OK, this will get ugly. APICs are involved.
>
> I guess you need only on systems that support >1 cpu right? I doubt you
> will need it on a system that cannot run with the config-generic-arch on.
> although we use bigsmp when hotplug is turned on, all we really end up is
> using flat physical mode instead of using logical mode.
>
> I still havent understood why this wont work. Choosing
> CONFIG_X86_GENERICARCH shouldnt break anything AFAICT.
>
> Pavel, you could use CONFIG_HOTPLUG_CPU, just need to enable
> X86_GENERICARCH now. Is there a reason you think that wont work? I wish we
> would revert it for a strong reason that we know will not make hotplug work
> on certain systems because of this choise not that we currently have X86_PC
> now, and are unwiling to change the config.
>
> (PS: the word bigsmp although sounds like some large NR_CPUS, its just
> using a mode that permits the system to work from 1 .. >8 cpus. So there is
> really nothing determental to selecting this.)

So if you have a single core x86, you want X86_PC, and if you have HT or SMP,
you want GENERICARCH? If so, could this be done via selects or depends or at
least defaults in Kconfig?

Regards,

Nigel


Attachments:
(No filename) (1.42 kB)
(No filename) (189.00 B)
Download all attachments

2006-03-29 23:47:51

by Ashok Raj

[permalink] [raw]
Subject: Re: [rfc] fix Kconfig, hotplug_cpu is needed for swsusp

On Thu, Mar 30, 2006 at 09:36:16AM +1000, Nigel Cunningham wrote:
> Hi.
>
>
> So if you have a single core x86, you want X86_PC, and if you have HT or SMP,
> you want GENERICARCH? If so, could this be done via selects or depends or at
> least defaults in Kconfig?

Yes, i think only SUSPEND_SMP is affect by this. I thought Rafael cced Pavel during
that exchange, maybe i missed.

>
> Regards,
>
> Nigel

How about this patch.

Make SUSPEND_SMP depend on X86_GENERICARCH, since hotplug cpu requires !X86_PC
due to some race in IPI handling. See more discussion here

http://marc.theaimsgroup.com/?l=linux-kernel&m=114303306032338&w=2

Signed-off-by: Ashok Raj <[email protected]>
--------------------------------------------------------------

Index: linux-2.6.16-git16/kernel/power/Kconfig
===================================================================
--- linux-2.6.16-git16.orig/kernel/power/Kconfig
+++ linux-2.6.16-git16/kernel/power/Kconfig
@@ -96,5 +96,5 @@ config SWSUSP_ENCRYPT

config SUSPEND_SMP
bool
- depends on HOTPLUG_CPU && X86 && PM
+ depends on HOTPLUG_CPU && X86 && PM && X86_GENERICARCH
default y


--
Cheers,
Ashok Raj
- Open Source Technology Center

2006-03-29 23:54:50

by Nigel Cunningham

[permalink] [raw]
Subject: Re: [rfc] fix Kconfig, hotplug_cpu is needed for swsusp

Hi.

On Thursday 30 March 2006 09:47, Ashok Raj wrote:
> On Thu, Mar 30, 2006 at 09:36:16AM +1000, Nigel Cunningham wrote:
> > Hi.
> >
> >
> > So if you have a single core x86, you want X86_PC, and if you have HT or
> > SMP, you want GENERICARCH? If so, could this be done via selects or
> > depends or at least defaults in Kconfig?
>
> Yes, i think only SUSPEND_SMP is affect by this. I thought Rafael cced
> Pavel during that exchange, maybe i missed.
>
> > Regards,
> >
> > Nigel
>
> How about this patch.
>
> Make SUSPEND_SMP depend on X86_GENERICARCH, since hotplug cpu requires
> !X86_PC due to some race in IPI handling. See more discussion here
>
> http://marc.theaimsgroup.com/?l=linux-kernel&m=114303306032338&w=2
>
> Signed-off-by: Ashok Raj <[email protected]>
> --------------------------------------------------------------
>
> Index: linux-2.6.16-git16/kernel/power/Kconfig
> ===================================================================
> --- linux-2.6.16-git16.orig/kernel/power/Kconfig
> +++ linux-2.6.16-git16/kernel/power/Kconfig
> @@ -96,5 +96,5 @@ config SWSUSP_ENCRYPT
>
> config SUSPEND_SMP
> bool
> - depends on HOTPLUG_CPU && X86 && PM
> + depends on HOTPLUG_CPU && X86 && PM && X86_GENERICARCH
> default y

Sounds like the right approach to me, but I think it's better to use selects.
I reckon that if the user selects SMP and then selects suspend support,
everything else required should be automatic. If we do too many 'depends
on's, they have to mess about figuring out what they haven't selected yet and
why they can't find the option to suspend. Most people don't seem to know
about '/' in make menuconfig.

Regards,

Nigel


Attachments:
(No filename) (1.66 kB)
(No filename) (189.00 B)
Download all attachments

2006-03-30 00:12:30

by Andrew Morton

[permalink] [raw]
Subject: Re: [rfc] fix Kconfig, hotplug_cpu is needed for swsusp

Nigel Cunningham <[email protected]> wrote:
>
> I think it's better to use selects.
> I reckon that if the user selects SMP and then selects suspend support,
> everything else required should be automatic.

Yes. `select' is end-user-friendly but kernel-developer-hostile.
Sometimes it's infuriating trying to work out why a symbol keeps on getting
turned on.

<checks>

hm, menuconfig's "/" command does show "Selected by:". That helps.

2006-03-30 00:13:01

by Ashok Raj

[permalink] [raw]
Subject: Re: [rfc] fix Kconfig, hotplug_cpu is needed for swsusp

On Thu, Mar 30, 2006 at 09:53:26AM +1000, Nigel Cunningham wrote:
>
> > config SUSPEND_SMP
> > bool
> > - depends on HOTPLUG_CPU && X86 && PM
> > + depends on HOTPLUG_CPU && X86 && PM && X86_GENERICARCH
> > default y
>
> Sounds like the right approach to me, but I think it's better to use selects.
> I reckon that if the user selects SMP and then selects suspend support,
> everything else required should be automatic. If we do too many 'depends
> on's, they have to mess about figuring out what they haven't selected yet and
> why they can't find the option to suspend. Most people don't seem to know
> about '/' in make menuconfig.
>

I tried the same with HOTPLUG_CPU as well, to just say

select X86_GENERICARCH

but problem was this didnt enforce the selection, i.e user still could go and revert
the selection made automatic for him, i.e go ahead and select X86_PC, and it would
still leave the HOTPLUG_CPU=y around. I thought "depends" sort of forces the selection.

Maybe i didnt try correctly, if you have alternatives please do, actually even for
HOTPLUG_CPU if this could be made automatic select, and at the same time enforced
strictly, thats great.

(for e.g i shoud;t be able to select X86_PC=y and leave CONFIG_HOTPLUG_CPU=y around)





--
Cheers,
Ashok Raj
- Open Source Technology Center

2006-03-30 00:20:03

by Nigel Cunningham

[permalink] [raw]
Subject: Re: [rfc] fix Kconfig, hotplug_cpu is needed for swsusp

Hi.

On Thursday 30 March 2006 10:13, Andrew Morton wrote:
> Nigel Cunningham <[email protected]> wrote:
> > I think it's better to use selects.
> > I reckon that if the user selects SMP and then selects suspend support,
> > everything else required should be automatic.
>
> Yes. `select' is end-user-friendly but kernel-developer-hostile.
> Sometimes it's infuriating trying to work out why a symbol keeps on getting
> turned on.
>
> <checks>
>
> hm, menuconfig's "/" command does show "Selected by:". That helps.

This might give the Kconfig guys a headache, but maybe a middle road would be
to make selects turn on options rather than force them on. That is, make it
so that if SUSPEND depends on SWAP, and selects it, after enabling SUSPEND,
SWAP is also enabled, but you can still go to SWAP and turn it off, thereby
also disabling SUSPEND again?

Regards,

Nigel


Attachments:
(No filename) (882.00 B)
(No filename) (189.00 B)
Download all attachments

2006-03-30 00:33:52

by Nigel Cunningham

[permalink] [raw]
Subject: Re: [rfc] fix Kconfig, hotplug_cpu is needed for swsusp

Hi.

On Thursday 30 March 2006 10:12, Ashok Raj wrote:
> On Thu, Mar 30, 2006 at 09:53:26AM +1000, Nigel Cunningham wrote:
> > > config SUSPEND_SMP
> > > bool
> > > - depends on HOTPLUG_CPU && X86 && PM
> > > + depends on HOTPLUG_CPU && X86 && PM && X86_GENERICARCH
> > > default y
> >
> > Sounds like the right approach to me, but I think it's better to use
> > selects. I reckon that if the user selects SMP and then selects suspend
> > support, everything else required should be automatic. If we do too many
> > 'depends on's, they have to mess about figuring out what they haven't
> > selected yet and why they can't find the option to suspend. Most people
> > don't seem to know about '/' in make menuconfig.
>
> I tried the same with HOTPLUG_CPU as well, to just say
>
> select X86_GENERICARCH
>
> but problem was this didnt enforce the selection, i.e user still could go
> and revert the selection made automatic for him, i.e go ahead and select
> X86_PC, and it would still leave the HOTPLUG_CPU=y around. I thought
> "depends" sort of forces the selection.
>
> Maybe i didnt try correctly, if you have alternatives please do, actually
> even for HOTPLUG_CPU if this could be made automatic select, and at the
> same time enforced strictly, thats great.
>
> (for e.g i shoud;t be able to select X86_PC=y and leave
> CONFIG_HOTPLUG_CPU=y around)

I tried too, with the same results. It seems to me that the problem is that
Kconfig doesn't enforce select for choices in the same way that it does for
bools. Should this be filed as a bug with the Kconfig guys?

Regards,

Nigel


Attachments:
(No filename) (1.58 kB)
(No filename) (189.00 B)
Download all attachments

2006-03-30 03:03:04

by Nigel Cunningham

[permalink] [raw]
Subject: Re: [rfc] fix Kconfig, hotplug_cpu is needed for swsusp

Hi Sam.

A bunch of us were discussing an issue this morning, and came across the
problem that selects don't seem to be enforced in choice menus. To give a
concrete example, a couple of us tried to make CONFIG_SOFTWARE_SUSPEND select
CONFIG_X86_GENERICARCH. After enabling SOFTWARE_SUSPEND, we want back to the
subarchitecture menu, and could still select other subarches. Is this by
design?

Regards,

Nigel


Attachments:
(No filename) (414.00 B)
(No filename) (189.00 B)
Download all attachments

2006-03-30 03:25:11

by Andrew Morton

[permalink] [raw]
Subject: Re: [rfc] fix Kconfig, hotplug_cpu is needed for swsusp

Ashok Raj <[email protected]> wrote:
>
> On Wed, Mar 29, 2006 at 02:47:46PM -0800, Andrew Morton wrote:
> > Pavel Machek <[email protected]> wrote:
> > >
> > > HOTPLUG_CPU is needed on normal PCs, too -- it is neccessary for
> > > software suspend.
> > >
> >
> > OK, this will get ugly. APICs are involved.
>
> I guess you need only on systems that support >1 cpu right? I doubt you will need
> it on a system that cannot run with the config-generic-arch on. although we use bigsmp
> when hotplug is turned on, all we really end up is using flat physical mode instead
> of using logical mode.
>
> I still havent understood why this wont work. Choosing CONFIG_X86_GENERICARCH shouldnt
> break anything AFAICT.
>
> Pavel, you could use CONFIG_HOTPLUG_CPU, just need to enable X86_GENERICARCH now.
> Is there a reason you think that wont work? I wish we would revert it for a strong
> reason that we know will not make hotplug work on certain systems because of this choise
> not that we currently have X86_PC now, and are unwiling to change the config.
>
> (PS: the word bigsmp although sounds like some large NR_CPUS, its just using a mode that
> permits the system to work from 1 .. >8 cpus. So there is really nothing determental
> to selecting this.)

Something which remains to be beaten into my head: *why* does HOTPLUG_CPU
require flat pyhsical mode? What necessitated that change, and cannot we
make it work OK in logical mode as well as flat mode?

2006-03-30 08:25:12

by Jan Engelhardt

[permalink] [raw]
Subject: Re: [rfc] fix Kconfig, hotplug_cpu is needed for swsusp


>Most people don't seem to know
>about '/' in make menuconfig.

I find it very confusing to use, since it returns too verbose text to skim
through. Probably the search function should be split into "only search
titles[*]" and "search description too".


config UNIX
bool "Unix socket" <-- that's a "title", as in menuconfig



Jan Engelhardt
--

2006-03-30 08:31:17

by Pavel Machek

[permalink] [raw]
Subject: Re: [rfc] fix Kconfig, hotplug_cpu is needed for swsusp

On St 29-03-06 15:09:51, Ashok Raj wrote:
> On Wed, Mar 29, 2006 at 02:47:46PM -0800, Andrew Morton wrote:
> > Pavel Machek <[email protected]> wrote:
> > >
> > > HOTPLUG_CPU is needed on normal PCs, too -- it is neccessary for
> > > software suspend.
> > >
> >
> > OK, this will get ugly. APICs are involved.
>
> I guess you need only on systems that support >1 cpu right? I doubt you will need
> it on a system that cannot run with the config-generic-arch on. although we use bigsmp
> when hotplug is turned on, all we really end up is using flat physical mode instead
> of using logical mode.
>
> I still havent understood why this wont work. Choosing CONFIG_X86_GENERICARCH shouldnt
> break anything AFAICT.

If X86_GENERICARCH is not breaking anything, why does X86_PC exist? If
you say X86_GENERICARCH is okay for everyone, at least help texts need
to be fixed.

config X86_PC
bool "PC-compatible"
help
Choose this option if your computer is a standard PC or compatible.

config X86_BIGSMP
bool "Support for other sub-arch SMP systems with more than 8 CPUs"
depends on SMP
help
This option is needed for the systems that have more than 8 CPUs
and if the system is not of any sub-arch type above.

If you don't have such a system, you should say N here.

I certainly have PCtype machine here, and definitely not more than 8 CPUs.

> Pavel, you could use CONFIG_HOTPLUG_CPU, just need to enable X86_GENERICARCH now.
> Is there a reason you think that wont work? I wish we would revert it for a strong
> reason that we know will not make hotplug work on certain systems because of this choise
> not that we currently have X86_PC now, and are unwiling to change the config.
>
> (PS: the word bigsmp although sounds like some large NR_CPUS, its just using a mode that
> permits the system to work from 1 .. >8 cpus. So there is really nothing determental
> to selecting this.)

At least config texts need to be fixed if you want people to use
X86_BIGSMP. And why keep X86_PC when X86_BIGSMP works as well?

Pavel
--
Picture of sleeping (Linux) penguin wanted...

2006-03-30 08:42:13

by Pavel Machek

[permalink] [raw]
Subject: Re: [rfc] fix Kconfig, hotplug_cpu is needed for swsusp

Hi!

> > So if you have a single core x86, you want X86_PC, and if you have HT or SMP,
> > you want GENERICARCH? If so, could this be done via selects or depends or at
> > least defaults in Kconfig?
>
> Yes, i think only SUSPEND_SMP is affect by this. I thought Rafael cced Pavel during
> that exchange, maybe i missed.
>
> >
> > Regards,
> >
> > Nigel
>
> How about this patch.
>
> Make SUSPEND_SMP depend on X86_GENERICARCH, since hotplug cpu requires !X86_PC
> due to some race in IPI handling. See more discussion here
>
> http://marc.theaimsgroup.com/?l=linux-kernel&m=114303306032338&w=2

I can't see useful discussion there.

> Index: linux-2.6.16-git16/kernel/power/Kconfig
> ===================================================================
> --- linux-2.6.16-git16.orig/kernel/power/Kconfig
> +++ linux-2.6.16-git16/kernel/power/Kconfig
> @@ -96,5 +96,5 @@ config SWSUSP_ENCRYPT
>
> config SUSPEND_SMP
> bool
> - depends on HOTPLUG_CPU && X86 && PM
> + depends on HOTPLUG_CPU && X86 && PM && X86_GENERICARCH
> default y


Heh, great, so one more magic option that is required.

Plus GENERICARCH does not sound like something normal users would
enable:

config X86_GENERICARCH
bool "Generic architecture (Summit, bigsmp, ES7000, default)"
depends on SMP
help
This option compiles in the Summit, bigsmp, ES7000, default subarchitectures.
It is intended for a generic binary kernel.

(What does "default" mean there, anyway? X86_PC?)

Pavel
--
Picture of sleeping (Linux) penguin wanted...

2006-03-30 10:12:24

by Roman Zippel

[permalink] [raw]
Subject: Re: [rfc] fix Kconfig, hotplug_cpu is needed for swsusp

Hi,

On Thu, 30 Mar 2006, Sam Ravnborg wrote:

> Roman is the one to addrress this.
> Roman?
>
> On Thu, Mar 30, 2006 at 01:01:37PM +1000, Nigel Cunningham wrote:
> > Hi Sam.
> >
> > A bunch of us were discussing an issue this morning, and came across the
> > problem that selects don't seem to be enforced in choice menus. To give a
> > concrete example, a couple of us tried to make CONFIG_SOFTWARE_SUSPEND select
> > CONFIG_X86_GENERICARCH. After enabling SOFTWARE_SUSPEND, we want back to the
> > subarchitecture menu, and could still select other subarches. Is this by
> > design?

Yes, kconfig is supposed to be deterministic. Imagine two options each
select a choice option, which would create an inconsistent configuration,
so it's not allowed in first place.

bye, Roman

2006-03-30 13:00:08

by Ashok Raj

[permalink] [raw]
Subject: Re: [rfc] fix Kconfig, hotplug_cpu is needed for swsusp

On Wed, Mar 29, 2006 at 07:24:53PM -0800, Andrew Morton wrote:
Hi Andrew,

here is an attempt to explain why...

>
> Something which remains to be beaten into my head: *why* does HOTPLUG_CPU
> require flat pyhsical mode? What necessitated that change, and cannot we
> make it work OK in logical mode as well as flat mode?

Short answer

1. Will hotplug work today in logical flat mode without selecting bigsmp?

Yes, since we avoid broadcast on i386 IPI's with no_broadcast option.

2. Why use bigsmp then?
There is no reason to do the same thing in two different ways. i.e using
logical flat with no_broadcast=1, or use the bigsmp by default when hotplug cpu is
enabled. IOW we wanted the handling consistent with what we do for x86_64, hence the
decision to use bigsmp automatically when hotplug is enabled or we notice >8 CPUS.


Long and detailed ...

1. broadcast IPI's are not hotplug friendly since hw has no control to *not* deliver
the message to offline cpus. Nice explanation from Vatsa on what the corner cases are
in link below.

http://marc.theaimsgroup.com/?l=linux-kernel&m=111695485610434&w=2

2. The discussion above was to support removing shortcuts, and using a send_IPI_mask()
instead of send_IPI_allbutself() via a shortcut msg. This would not deliver an IPI
to offline cpus. Andi Kleen had some doubts about tlb flush performance when we use
send_IPI_mask() since it does one cpu at a time.

So we added a no_broadcast option, that permits us to continue to use the normal
shortcut method for send_IPI_allbutself() when cpu-hotplug is disabled, and use the
send_IPI_mask() version when cpu-hotplug is enabled.

[ set usage of no_broadcast in arch/i386/mach-default/setup.c and used in
include/asm-i386/mach-default/mach_ipi.h]

In addition we also posted some perf testing patch for testing, and it proved we
didnt have any perf degradation for using the mask or the shortcut. Based on those
results, Andi removed the no_broadcast option in X86_64 and we used the physflat
option since it does the unicast method anyway.

[Physflat mode is equivalent of bigsmp in x86_64]


3. So today using logical flat mode will work with hotlug without any races due to the
no_broadcast=1 is automatically selected when hotplug cpu is enabled, even without
bigsmp apic mode.

So Why we moved to automatically selecting bigsmp when hotlug was enabled?
we really didnt want 2 ways to do the same thing without clear benefit for one
over other. (i.e using logical flat mode with no_broadcast v.s using bigsmp that
already does send_IPI_sequence() which doesnt do the shortcut msg )

4. Why bigsmp was converted to physical flat mode instead of logical cluster mode?

We discovered a problem when platform/chipset is working in logical cluster
mode, and a new CPU is brought up that causes bad interaction. More
details below in the patch description.

http://marc.theaimsgroup.com/?l=linux-kernel&m=113261865814107&w=2

[In i386 bigsmp also means flat physical mode after the above patch was accepted]

5. Ideally the no_broadcast option will be removed soon once things settle down.

Hope this helps.
--
Cheers,
Ashok Raj
- Open Source Technology Center

2006-03-30 13:06:35

by Eric W. Biederman

[permalink] [raw]
Subject: Re: [rfc] fix Kconfig, hotplug_cpu is needed for swsusp

Andrew Morton <[email protected]> writes:

> Something which remains to be beaten into my head: *why* does HOTPLUG_CPU
> require flat pyhsical mode? What necessitated that change, and cannot we
> make it work OK in logical mode as well as flat mode?

Good question.

There is a window after a cpu starts and before we initialize it that
we must be very careful and not deliver it any interrupts because
we don't know what the cpu will do.

Which means in the presence of cpu hotplug we can never use broadcast
interrupts.

I don't know how the that disallows logical mode though.

Eric

2006-03-30 13:18:09

by Pavel Machek

[permalink] [raw]
Subject: Re: [rfc] fix Kconfig, hotplug_cpu is needed for swsusp

Hi!

> here is an attempt to explain why...
>
> >
> > Something which remains to be beaten into my head: *why* does HOTPLUG_CPU
> > require flat pyhsical mode? What necessitated that change, and cannot we
> > make it work OK in logical mode as well as flat mode?
>
> Short answer
>
> 1. Will hotplug work today in logical flat mode without selecting bigsmp?
>
> Yes, since we avoid broadcast on i386 IPI's with no_broadcast option.
>
> 2. Why use bigsmp then?
> There is no reason to do the same thing in two different ways. i.e using
> logical flat with no_broadcast=1, or use the bigsmp by default when hotplug cpu is
> enabled. IOW we wanted the handling consistent with what we do for x86_64, hence the
> decision to use bigsmp automatically when hotplug is enabled or
> we notice >8 CPUS.

Why not make it even more consistent and just use bigsmp mode by
default, even for X86_PC? X86_BIGSMP would be reduntant, then...
Pavel

--
Picture of sleeping (Linux) penguin wanted...

2006-03-30 16:18:46

by Rafael J. Wysocki

[permalink] [raw]
Subject: Re: [rfc] fix Kconfig, hotplug_cpu is needed for swsusp

Hi,

On Thursday 30 March 2006 10:41, Pavel Machek wrote:
> > > So if you have a single core x86, you want X86_PC, and if you have HT or SMP,
> > > you want GENERICARCH? If so, could this be done via selects or depends or at
> > > least defaults in Kconfig?
> >
> > Yes, i think only SUSPEND_SMP is affect by this.

That's correct.

> > I thought Rafael cced Pavel during that exchange, maybe i missed.

I did, but Pavel was travelling at that time (I think).

[Well, I had the feeling it would cause problems but unfortunately I couldn't
show that.]

> > How about this patch.
> >
> > Make SUSPEND_SMP depend on X86_GENERICARCH, since hotplug cpu requires !X86_PC
> > due to some race in IPI handling. See more discussion here
> >
> > http://marc.theaimsgroup.com/?l=linux-kernel&m=114303306032338&w=2
>
> I can't see useful discussion there.
>
> > Index: linux-2.6.16-git16/kernel/power/Kconfig
> > ===================================================================
> > --- linux-2.6.16-git16.orig/kernel/power/Kconfig
> > +++ linux-2.6.16-git16/kernel/power/Kconfig
> > @@ -96,5 +96,5 @@ config SWSUSP_ENCRYPT
> >
> > config SUSPEND_SMP
> > bool
> > - depends on HOTPLUG_CPU && X86 && PM
> > + depends on HOTPLUG_CPU && X86 && PM && X86_GENERICARCH
> > default y
>
>
> Heh, great, so one more magic option that is required.

BTW it's i386-centric, as X86_GENERICARCH is not defined on x86_64, for
example.

FUrther, the whole problem, as far as I can understand it, is i386-specific,
so it should be resolved in the i386 architecture config rather than anywhere
else.

Greetings,
Rafael

2006-03-30 18:24:50

by Pavel Machek

[permalink] [raw]
Subject: Re: [rfc] fix Kconfig, hotplug_cpu is needed for swsusp

Hi!

> > > I thought Rafael cced Pavel during that exchange, maybe i missed.
>
> I did, but Pavel was travelling at that time (I think).

Yes, I was cc-ed, but only watching discussion by single eye...

> FUrther, the whole problem, as far as I can understand it, is i386-specific,
> so it should be resolved in the i386 architecture config rather than anywhere
> else.

Or maybe in i386 .c files :-). Could we just switch to BIGSMP mode by
default? Intel claims it has no performance disadvatage, and distros
want suspend, anyway...
Pavel
--
Thanks, Sharp!