The XEN config option enables the Xen paravirt_ops interface, which is
installed when the kernel finds itself running under Xen. (By some
as-yet fully defined mechanism, implemented in a future patch.)
Xen is no longer a sub-architecture, so the X86_XEN subarch config
option has gone.
The disabled config options are:
- PREEMPT: Xen doesn't support it
- HZ: set to 100Hz for now, to cut down on VCPU context switch rate.
This will be adapted to use tickless later.
- kexec: not yet supported
Signed-off-by: Jeremy Fitzhardinge <[email protected]>
Signed-off-by: Ian Pratt <[email protected]>
Signed-off-by: Christian Limpach <[email protected]>
Signed-off-by: Chris Wright <[email protected]>
---
arch/i386/Kconfig | 7 +++++--
arch/i386/Kconfig.debug | 1 +
arch/i386/xen/Kconfig | 10 ++++++++++
kernel/Kconfig.hz | 4 ++--
kernel/Kconfig.preempt | 1 +
5 files changed, 19 insertions(+), 4 deletions(-)
===================================================================
--- a/arch/i386/Kconfig
+++ b/arch/i386/Kconfig
@@ -192,6 +192,8 @@ config PARAVIRT
under a hypervisor, improving performance significantly.
However, when run without a hypervisor the kernel is
theoretically slower. If in doubt, say N.
+
+source "arch/i386/xen/Kconfig"
config ACPI_SRAT
bool
@@ -298,12 +300,12 @@ config X86_UP_IOAPIC
config X86_LOCAL_APIC
bool
- depends on X86_UP_APIC || ((X86_VISWS || SMP) && !X86_VOYAGER) || X86_GENERICARCH
+ depends on X86_UP_APIC || (((X86_VISWS || SMP) && !X86_VOYAGER) || X86_GENERICARCH)
default y
config X86_IO_APIC
bool
- depends on X86_UP_IOAPIC || (SMP && !(X86_VISWS || X86_VOYAGER)) || X86_GENERICARCH
+ depends on X86_UP_IOAPIC || ((SMP && !(X86_VISWS || X86_VOYAGER)) || X86_GENERICARCH)
default y
config X86_VISWS_APIC
@@ -743,6 +745,7 @@ source kernel/Kconfig.hz
config KEXEC
bool "kexec system call"
+ depends on !XEN
help
kexec is a system call that implements the ability to shutdown your
current kernel, and to start another kernel. It is like a reboot
===================================================================
--- a/arch/i386/Kconfig.debug
+++ b/arch/i386/Kconfig.debug
@@ -79,6 +79,7 @@ config DOUBLEFAULT
config DOUBLEFAULT
default y
bool "Enable doublefault exception handler" if EMBEDDED
+ depends on !XEN
help
This option allows trapping of rare doublefault exceptions that
would otherwise cause a system to silently reboot. Disabling this
===================================================================
--- /dev/null
+++ b/arch/i386/xen/Kconfig
@@ -0,0 +1,10 @@
+#
+# This Kconfig describes xen options
+#
+
+config XEN
+ bool "Enable support for Xen hypervisor"
+ depends PARAVIRT
+ default y
+ help
+ This is the Linux Xen port.
===================================================================
--- a/kernel/Kconfig.hz
+++ b/kernel/Kconfig.hz
@@ -3,7 +3,7 @@
#
choice
- prompt "Timer frequency"
+ prompt "Timer frequency" if !XEN
default HZ_250
help
Allows the configuration of the timer frequency. It is customary
@@ -49,7 +49,7 @@ endchoice
config HZ
int
- default 100 if HZ_100
+ default 100 if HZ_100 || XEN
default 250 if HZ_250
default 300 if HZ_300
default 1000 if HZ_1000
===================================================================
--- a/kernel/Kconfig.preempt
+++ b/kernel/Kconfig.preempt
@@ -35,6 +35,7 @@ config PREEMPT_VOLUNTARY
config PREEMPT
bool "Preemptible Kernel (Low-Latency Desktop)"
+ depends on !XEN
help
This option reduces the latency of the kernel by making
all kernel code (that is not executing in a critical section)
--
On 02/15/2007 06:25 PM, Jeremy Fitzhardinge wrote:
> The XEN config option enables the Xen paravirt_ops interface, which is
> installed when the kernel finds itself running under Xen. (By some
> as-yet fully defined mechanism, implemented in a future patch.)
>
> Xen is no longer a sub-architecture, so the X86_XEN subarch config
> option has gone.
>
> The disabled config options are:
> - PREEMPT: Xen doesn't support it
> - HZ: set to 100Hz for now, to cut down on VCPU context switch rate.
> This will be adapted to use tickless later.
> - kexec: not yet supported
>
>
> config KEXEC
> bool "kexec system call"
> + depends on !XEN
> help
> kexec is a system call that implements the ability to shutdown your
> current kernel, and to start another kernel. It is like a reboot
> config DOUBLEFAULT
> default y
> bool "Enable doublefault exception handler" if EMBEDDED
> + depends on !XEN
> help
> This option allows trapping of rare doublefault exceptions that
> would otherwise cause a system to silently reboot. Disabling this
> +config XEN
> + bool "Enable support for Xen hypervisor"
> + depends PARAVIRT
> + default y
> + help
> + This is the Linux Xen port.
> choice
> - prompt "Timer frequency"
> + prompt "Timer frequency" if !XEN
> default HZ_250
> help
> Allows the configuration of the timer frequency. It is customary
> @@ -49,7 +49,7 @@ endchoice
>
> config HZ
> int
> - default 100 if HZ_100
> + default 100 if HZ_100 || XEN
> default 250 if HZ_250
> default 300 if HZ_300
> default 1000 if HZ_1000
> config PREEMPT
> bool "Preemptible Kernel (Low-Latency Desktop)"
> + depends on !XEN
> help
> This option reduces the latency of the kernel by making
> all kernel code (that is not executing in a critical section)
>
I hate to sound like a broken record, but this really isn't the right
way to do this. If you are going to inhibit config settings when Xen
support is compiled, then it should be:
config XEN
depends on !KEXEC && !DOUBLEFAULT && HZ_100 && !PREEMPT
I'm really not trying to make things more difficult for you or for Xen
users.
One of the primary goals of paravirt-ops is to allow the same kernel
binary to bind to multiple hypervisor interfaces (and native), at
runtime. So, we should assume that kernels which are built with
CONFIG_PARAVIRT will also enable all the paravirt-ops backends; and this
is a good thing.
But, the problem is, then, when someone enables paravirt (and all the
backends, including Xen), you are silently forcing all these options
off. Compiling in paravirt support should not change force you into
using some set of config option; when a paravirt-ops kernel is run on
native, it should be as fully featured as a non-paravirtops kernel.
So, this should really be solved (in order of preference) by:
1) Complete the xen paravirt-ops backend so it can handle these
"incompatible" options. I realize this is just a matter of time (at
least for most of them, what is your plan for PREEMPT?).
2) Disable the option at runtime only if the kernel is booted on Xen.
When the kernel is booted on native, lhype, paravirt kvm, vmware, etc it
should be not be inhibited. This may not be feasible for all of these
options, but as Zach pointed out, is easy enough for DOUBLEFAULT. Maybe
it can be done for KEXEC? And HZ is easy to allow too, even though Xen
will still give interrupts at 100hz -- we do this when you boot on VMI.
PREEMPT is probably the only real compile time incompatible options
with Xen. You just simply have to change the loop decrementors in your
timer interrupt.
You basically chose #2 for SMP: while your backend doesn't support it
yet, it's not harmful to have the config option enabled; you just don't
allow a second vcpu to startup when running on Xen.
3) Use the config XEN line I suggest above. The net effect is the same
as your proposed change (it prevents users from compiling the
incompatible options together), but at least the user understands what
is going on, instead of options just silently changing on them.
Dan
Jeremy Fitzhardinge wrote:
> The XEN config option enables the Xen paravirt_ops interface, which is
> installed when the kernel finds itself running under Xen. (By some
> as-yet fully defined mechanism, implemented in a future patch.)
>
> Xen is no longer a sub-architecture, so the X86_XEN subarch config
> option has gone.
>
> The disabled config options are:
> - PREEMPT: Xen doesn't support it
> - HZ: set to 100Hz for now, to cut down on VCPU context switch rate.
> This will be adapted to use tickless later.
> - kexec: not yet supported
>
> Signed-off-by: Jeremy Fitzhardinge <[email protected]>
> Signed-off-by: Ian Pratt <[email protected]>
> Signed-off-by: Christian Limpach <[email protected]>
> Signed-off-by: Chris Wright <[email protected]>
>
We do support different HZ values, although 100HZ is actually preferable
for us, so I don't object to that. PREEMPT is supported by us, but not
as tested as I would like, so I also don't object to dropping it for
generic paravirt guests - Rusty - Avi any objections to dropping preempt
in terms of lguest/KVM paravirtualization?
Paravirt-ops definitely needs a hook for kexec, although we should not
disable kexec for the natively booted paravirt-ops. Eric - is there a
way to disable it at runtime?
We do support the doublefault task gate, and it would be good to keep
it, but I can't complain so much if it is gone from generic paravirt
kernels for now, because it is non-essential, and generally fatal
anyway. We do need it for native boots of paravirt-ops kernels,
however, so turning off the config option still needs to be revisited.
Zach
Dan Hecht wrote:
> 1) Complete the xen paravirt-ops backend so it can handle these
> "incompatible" options. I realize this is just a matter of time (at
> least for most of them, what is your plan for PREEMPT?).
Hope it will go away? There's some relatively deep reason to do with
migration which makes preempt very difficult to support, but I've
forgotten the details. I've been wondering if there's some way to make
it runtime selectable without massive performance cost.
> 2) Disable the option at runtime only if the kernel is booted on Xen.
> When the kernel is booted on native, lhype, paravirt kvm, vmware, etc
> it should be not be inhibited. This may not be feasible for all of
> these options, but as Zach pointed out, is easy enough for
> DOUBLEFAULT. Maybe it can be done for KEXEC? And HZ is easy to allow
> too, even though Xen will still give interrupts at 100hz -- we do this
> when you boot on VMI. PREEMPT is probably the only real compile time
> incompatible options with Xen. You just simply have to change the
> loop decrementors in your timer interrupt.
>
> You basically chose #2 for SMP: while your backend doesn't support it
> yet, it's not harmful to have the config option enabled; you just
> don't allow a second vcpu to startup when running on Xen.
Yes, this has been my preferred approach. So for each of the restrictions:
PREEMPT - hard, will try to do something at runtime
HZ - I'm assuming dynticks will appear in the short term, and this will
become moot
KEXEC - there's been some work on making KEXEC and Xen play nicely
together; I was waiting to see if that matures, and/or make it runtime
switchable in the meantime
DOUBLEFAULT - not really an issue; guests won't get them under Xen (I
think) and we can ignore setting the gate pretty easily
J
On Thu, 15 Feb 2007 22:14:45 -0800 Dan Hecht <[email protected]> wrote:
> > config PREEMPT
> > bool "Preemptible Kernel (Low-Latency Desktop)"
> > + depends on !XEN
> > help
> > This option reduces the latency of the kernel by making
> > all kernel code (that is not executing in a critical section)
> >
Oh, so that's why it doesn't break when CONFIG_PREEMPT=y. In which case
that preempt_disable() I spotted is wrong-and-unneeded.
Why doesn't Xen work with preemption??
> I hate to sound like a broken record, but this really isn't the right
> way to do this. If you are going to inhibit config settings when Xen
> support is compiled, then it should be:
>
> config XEN
> depends on !KEXEC && !DOUBLEFAULT && HZ_100 && !PREEMPT
>
Agree.
Andrew Morton wrote:
> On Thu, 15 Feb 2007 22:14:45 -0800 Dan Hecht <[email protected]> wrote:
>
>
>>> config PREEMPT
>>> bool "Preemptible Kernel (Low-Latency Desktop)"
>>> + depends on !XEN
>>> help
>>> This option reduces the latency of the kernel by making
>>> all kernel code (that is not executing in a critical section)
>>>
>>>
>
> Oh, so that's why it doesn't break when CONFIG_PREEMPT=y. In which case
> that preempt_disable() I spotted is wrong-and-unneeded.
>
> Why doesn't Xen work with preemption??
>
I've forgotten the details. Ian? Keir? Steven? Maybe it can be done.
J
Zachary Amsden <[email protected]> writes:
>
> We do support different HZ values, although 100HZ is actually preferable for us,
> so I don't object to that. PREEMPT is supported by us, but not as tested as I
> would like, so I also don't object to dropping it for generic paravirt guests -
> Rusty - Avi any objections to dropping preempt in terms of lguest/KVM
> paravirtualization?
>
> Paravirt-ops definitely needs a hook for kexec, although we should not disable
> kexec for the natively booted paravirt-ops. Eric - is there a way to disable it
> at runtime?
>
> We do support the doublefault task gate, and it would be good to keep it, but I
> can't complain so much if it is gone from generic paravirt kernels for now,
> because it is non-essential, and generally fatal anyway. We do need it for
> native boots of paravirt-ops kernels, however, so turning off the config option
> still needs to be revisited
Have machine_kexec_prepare fail.
I think your machine description or paravirt_ops or whatever it is needs
to hook both machine_kexec_prepare and machine_kexec.
I know there actually has been some work to get kexec actually working under
Xen but I don't know where that has gone.
Eric
On 02/15/2007 11:04 PM, Jeremy Fitzhardinge wrote:
> HZ - I'm assuming dynticks will appear in the short term, and this will
> become moot
Doesn't Xen send any non-blocked domain a 100hz alarm implicitly,
without anyway for the guest to disable it? I guess you'll have to
break kernel/hypervisor compatibility if you want dynticks?
Dan Hecht wrote:
> On 02/15/2007 11:04 PM, Jeremy Fitzhardinge wrote:
>> HZ - I'm assuming dynticks will appear in the short term, and this will
>> become moot
>
> Doesn't Xen send any non-blocked domain a 100hz alarm implicitly,
> without anyway for the guest to disable it? I guess you'll have to
> break kernel/hypervisor compatibility if you want dynticks?
The timer VIRQ will generate events at a fixed 100Hz (which can be
masked, like any other event), but you can also ask for a timer event at
a particular time. We do this already for tickless idle.
J
Ian Campbell <[email protected]> writes:
> On Fri, 2007-02-16 at 00:33 -0700, Eric W. Biederman wrote:
>> I know there actually has been some work to get kexec actually working under
>> Xen but I don't know where that has gone.
>
> kexec/kdump works with Xen 3.0.4 but it's a dom0 only thing so it
> doesn't appear in this patchset.
Ok but we still need to get the failure to user space for domU instead
of trying what works on real hardware and failing.
Eric
On Fri, 2007-02-16 at 00:33 -0700, Eric W. Biederman wrote:
> I know there actually has been some work to get kexec actually working under
> Xen but I don't know where that has gone.
kexec/kdump works with Xen 3.0.4 but it's a dom0 only thing so it
doesn't appear in this patchset.
Ian.
Eric W. Biederman wrote:
> Ian Campbell <[email protected]> writes:
>
>
>> On Fri, 2007-02-16 at 00:33 -0700, Eric W. Biederman wrote:
>>
>>> I know there actually has been some work to get kexec actually working under
>>> Xen but I don't know where that has gone.
>>>
>> kexec/kdump works with Xen 3.0.4 but it's a dom0 only thing so it
>> doesn't appear in this patchset.
>>
>
> Ok but we still need to get the failure to user space for domU instead
> of trying what works on real hardware and failing.
>
> Eric
Acked-by: Zachary Amsden <[email protected]>
On 02/16/2007 12:05 AM, Jeremy Fitzhardinge wrote:
> Dan Hecht wrote:
>> On 02/15/2007 11:04 PM, Jeremy Fitzhardinge wrote:
>>> HZ - I'm assuming dynticks will appear in the short term, and this will
>>> become moot
>> Doesn't Xen send any non-blocked domain a 100hz alarm implicitly,
>> without anyway for the guest to disable it? I guess you'll have to
>> break kernel/hypervisor compatibility if you want dynticks?
>
> The timer VIRQ will generate events at a fixed 100Hz (which can be
> masked, like any other event), but you can also ask for a timer event at
> a particular time. We do this already for tickless idle.
>
> J
>
Hmm? I thought the periodic timer and one-shot timer both generate the
same VIRQ. So, how can you mask one without masking the other?
The tickless idle works since the block hypercall disables the periodic
timer. But for dynticks (aka NO_HZ), you'll need to mask the periodic
timer (even for a running vcpu), while keeping the one-shot timer
unmasked. I don't think Xen provides an interface to do that.
Dan
Keir Fraser wrote:
> On 16/2/07 07:25, "Jeremy Fitzhardinge" <[email protected]> wrote:
>
>
>>> Oh, so that's why it doesn't break when CONFIG_PREEMPT=y. In which case
>>> that preempt_disable() I spotted is wrong-and-unneeded.
>>>
>>> Why doesn't Xen work with preemption??
>>>
>> I've forgotten the details. Ian? Keir? Steven? Maybe it can be done.
>>
>
> It breaks guest save/restore for us currently because threads can be
> sleeping with machine addresses in local storage (registers, stack). There
> are a few ways to achieve an acceptable solution:
>
> 1. Put processes in the freezer when we suspend. This should avoid any
> thread being in a critical section with machine addresses in its hand. We
> haven't yet investigated the performance impact of freezing processes,
> particularly on the downtime of live relocation.
>
> 2. Allow CONFIG_PREEMPT to be compiled in, but disable it at runtime. We
> could do this by, for example, reserving a bit in preempt_count() so that
> most preemption checks do not touch any more cache lines. I guess it would
> need a bit of fixing up (e.g., so that in_atomic() would not be always
> asserted). Even better for us would be to allow switching between
> involuntary and voluntary preemption at runtime. It looks as though the hook
> points for these two techniques are not usually compiled in at the same
> time, however.
>
>
Doesn't stop_machine_run already take care of getting you out of all
kernel threads? So you can only be sleeping, not preempted, in which
case, this might not be an issue?
Zach
On 16/2/07 07:25, "Jeremy Fitzhardinge" <[email protected]> wrote:
>> Oh, so that's why it doesn't break when CONFIG_PREEMPT=y. In which case
>> that preempt_disable() I spotted is wrong-and-unneeded.
>>
>> Why doesn't Xen work with preemption??
>
> I've forgotten the details. Ian? Keir? Steven? Maybe it can be done.
It breaks guest save/restore for us currently because threads can be
sleeping with machine addresses in local storage (registers, stack). There
are a few ways to achieve an acceptable solution:
1. Put processes in the freezer when we suspend. This should avoid any
thread being in a critical section with machine addresses in its hand. We
haven't yet investigated the performance impact of freezing processes,
particularly on the downtime of live relocation.
2. Allow CONFIG_PREEMPT to be compiled in, but disable it at runtime. We
could do this by, for example, reserving a bit in preempt_count() so that
most preemption checks do not touch any more cache lines. I guess it would
need a bit of fixing up (e.g., so that in_atomic() would not be always
asserted). Even better for us would be to allow switching between
involuntary and voluntary preemption at runtime. It looks as though the hook
points for these two techniques are not usually compiled in at the same
time, however.
-- Keir
On 16/2/07 10:19, "Zachary Amsden" <[email protected]> wrote:
> Doesn't stop_machine_run already take care of getting you out of all
> kernel threads? So you can only be sleeping, not preempted, in which
> case, this might not be an issue?
It ensures that no (non-stopmachine) threads are running on any of the other
CPUs, but it doesn't guarantee that all threads are at any form of quiescent
point. They can certainly have private references to machine addresses.
Otherwise, what would be the point of the process freezer? :-)
-- Keir
On 16/2/07 08:37, "Dan Hecht" <[email protected]> wrote:
> Hmm? I thought the periodic timer and one-shot timer both generate the
> same VIRQ. So, how can you mask one without masking the other?
>
> The tickless idle works since the block hypercall disables the periodic
> timer. But for dynticks (aka NO_HZ), you'll need to mask the periodic
> timer (even for a running vcpu), while keeping the one-shot timer
> unmasked. I don't think Xen provides an interface to do that.
There's no interface for this right now, but we can easily add one in a
backward compatible way. This also will not prevent a dyntick kernel from
running on an older Xen -- the guest will then receive to many timer
interrupts, but that should only be a performance concern fixable by
upgrading the hypervisor component of the system.
-- Keir
Keir Fraser wrote:
> On 16/2/07 10:19, "Zachary Amsden" <[email protected]> wrote:
>
>
>> Doesn't stop_machine_run already take care of getting you out of all
>> kernel threads? So you can only be sleeping, not preempted, in which
>> case, this might not be an issue?
>>
>
> It ensures that no (non-stopmachine) threads are running on any of the other
> CPUs, but it doesn't guarantee that all threads are at any form of quiescent
> point. They can certainly have private references to machine addresses.
>
> Otherwise, what would be the point of the process freezer? :-)
>
Yeah, I was over-hoping on that one. At least thought it was worth
suggesting as a starting point. I had similar issues considering
loading VMI as a module at runtime.
Zach
Eric W. Biederman wrote:
> Have machine_kexec_prepare fail.
>
> I think your machine description or paravirt_ops or whatever it is needs
> to hook both machine_kexec_prepare and machine_kexec.
>
> I know there actually has been some work to get kexec actually working under
> Xen but I don't know where that has gone.
Works, modulo bugs. Shouldn't be hard to get work on top of
paravirt_ops. The current 2.6.18 code uses function pointers already
because the dom0 code looks very different from domU code. It is one of
the items on my todo list. I'll probably look into it once I'm done
with the earlyprintk and console mess ...
cheers,
Gerd
--
Gerd Hoffmann <[email protected]>
On Fri, 2007-02-16 at 02:19 -0800, Zachary Amsden wrote:
> Doesn't stop_machine_run already take care of getting you out of all
> kernel threads? So you can only be sleeping, not preempted, in which
> case, this might not be an issue?
No, stop_machine only protects against preempt-disabled regions.
I thought about making that particularly hammer bigger, but noone had
needed it yet. It would simply require a runtime way of disabling
further preemption and flushing preempted threads (AFAICT there can be
no deadlock issues with this).
Rusty.
Zachary Amsden wrote:
> Jeremy Fitzhardinge wrote:
>> The XEN config option enables the Xen paravirt_ops interface, which is
>> installed when the kernel finds itself running under Xen. (By some
>> as-yet fully defined mechanism, implemented in a future patch.)
>>
>> Xen is no longer a sub-architecture, so the X86_XEN subarch config
>> option has gone.
>>
>> The disabled config options are:
>> - PREEMPT: Xen doesn't support it
>> - HZ: set to 100Hz for now, to cut down on VCPU context switch rate.
>> This will be adapted to use tickless later.
>> - kexec: not yet supported
>>
>> Signed-off-by: Jeremy Fitzhardinge <[email protected]>
>> Signed-off-by: Ian Pratt <[email protected]>
>> Signed-off-by: Christian Limpach <[email protected]>
>> Signed-off-by: Chris Wright <[email protected]>
>>
>
> We do support different HZ values, although 100HZ is actually
> preferable for us, so I don't object to that.
Ditto for kvm.
> PREEMPT is supported by us, but not as tested as I would like, so I
> also don't object to dropping it for generic paravirt guests - Rusty -
> Avi any objections to dropping preempt in terms of lguest/KVM
> paravirtualization?
I don't have any objections myself, but Ingo (who has done the bulk of
the kvm paravirt work; cc'ed) uses PREEMPT_RT, so he will certainly object.
>
> Paravirt-ops definitely needs a hook for kexec, although we should not
> disable kexec for the natively booted paravirt-ops. Eric - is there a
> way to disable it at runtime?
kvm paravirt should work correctly with kexec.
>
> We do support the doublefault task gate, and it would be good to keep
> it, but I can't complain so much if it is gone from generic paravirt
> kernels for now, because it is non-essential, and generally fatal
> anyway. We do need it for native boots of paravirt-ops kernels,
> however, so turning off the config option still needs to be revisited.
kvm doesn't support task gates (a task switch will immediately kill the
guest).
--
error compiling committee.c: too many arguments to function