Now crashkernel=X will fail out if there's not enough memory at
low (below 896M). What makes sense for crashkernel=X would be:
- First try to reserve X below 896M (for being compatible with old
kexec-tools).
- If fails, try to reserve X below 4G (swiotlb need to stay below 4G).
- If fails, try to reserve X from MAXMEM top down.
So that user can easily reserve large memory with crashkernel=X instead
of crashkernel=X,high. It's more transparent and user-friendly.
If crashkernel is large and the reserved is beyond 896M, old kexec-tools
won't be compatible with new kernel for most of time.
kexec will fail out immediately in this case. But the failure could be
expected, because old kexec users should not try to reserve that large
amount of memory at the first place.
On the other hand, old kexec also will fail on old kernel when there's
not enough low memory to reserve a large crash kernel area. So the
failure of old kexec is consistent between old kernel and new kernel.
Signed-off-by: WANG Chao <[email protected]>
---
arch/x86/kernel/setup.c | 14 ++++++++++++++
1 file changed, 14 insertions(+)
diff --git a/arch/x86/kernel/setup.c b/arch/x86/kernel/setup.c
index f0de629..38e6c1f 100644
--- a/arch/x86/kernel/setup.c
+++ b/arch/x86/kernel/setup.c
@@ -593,6 +593,20 @@ static void __init reserve_crashkernel(void)
high ? CRASH_KERNEL_ADDR_HIGH_MAX :
CRASH_KERNEL_ADDR_LOW_MAX,
crash_size, alignment);
+ /*
+ * crashkernel=X reserve below 896M fails? Try below 4G
+ */
+ if (!high && !crash_base)
+ crash_base = memblock_find_in_range(alignment,
+ (1ULL << 32),
+ crash_size, alignment);
+ /*
+ * crashkernel=X reserve below 4G fails? Try MAXMEM
+ */
+ if (!high && !crash_base)
+ crash_base = memblock_find_in_range(alignment,
+ CRASH_KERNEL_ADDR_HIGH_MAX,
+ crash_size, alignment);
if (!crash_base) {
pr_info("crashkernel reservation failed - No suitable area found.\n");
--
1.8.3.1
On Mon, Oct 14, 2013 at 4:46 AM, WANG Chao <[email protected]> wrote:
> Now crashkernel=X will fail out if there's not enough memory at
> low (below 896M). What makes sense for crashkernel=X would be:
>
> - First try to reserve X below 896M (for being compatible with old
> kexec-tools).
> - If fails, try to reserve X below 4G (swiotlb need to stay below 4G).
> - If fails, try to reserve X from MAXMEM top down.
>
> So that user can easily reserve large memory with crashkernel=X instead
> of crashkernel=X,high. It's more transparent and user-friendly.
>
> If crashkernel is large and the reserved is beyond 896M, old kexec-tools
> won't be compatible with new kernel for most of time.
>
> kexec will fail out immediately in this case. But the failure could be
> expected, because old kexec users should not try to reserve that large
> amount of memory at the first place.
>
> On the other hand, old kexec also will fail on old kernel when there's
> not enough low memory to reserve a large crash kernel area. So the
> failure of old kexec is consistent between old kernel and new kernel.
>
> Signed-off-by: WANG Chao <[email protected]>
> ---
> arch/x86/kernel/setup.c | 14 ++++++++++++++
> 1 file changed, 14 insertions(+)
>
> diff --git a/arch/x86/kernel/setup.c b/arch/x86/kernel/setup.c
> index f0de629..38e6c1f 100644
> --- a/arch/x86/kernel/setup.c
> +++ b/arch/x86/kernel/setup.c
> @@ -593,6 +593,20 @@ static void __init reserve_crashkernel(void)
> high ? CRASH_KERNEL_ADDR_HIGH_MAX :
> CRASH_KERNEL_ADDR_LOW_MAX,
> crash_size, alignment);
> + /*
> + * crashkernel=X reserve below 896M fails? Try below 4G
> + */
> + if (!high && !crash_base)
> + crash_base = memblock_find_in_range(alignment,
> + (1ULL << 32),
> + crash_size, alignment);
> + /*
> + * crashkernel=X reserve below 4G fails? Try MAXMEM
> + */
> + if (!high && !crash_base)
> + crash_base = memblock_find_in_range(alignment,
> + CRASH_KERNEL_ADDR_HIGH_MAX,
> + crash_size, alignment);
>
> if (!crash_base) {
> pr_info("crashkernel reservation failed - No suitable area found.\n");
> --
User should change crashkernel=X to crashkernel=X,high.
As user could forget to update kexec-tools to utilize ",high" feature, and get
kdump later fail later.
Yinghai
On 10/14/13 at 11:54am, Yinghai Lu wrote:
> On Mon, Oct 14, 2013 at 4:46 AM, WANG Chao <[email protected]> wrote:
> > Now crashkernel=X will fail out if there's not enough memory at
> > low (below 896M). What makes sense for crashkernel=X would be:
> >
> > - First try to reserve X below 896M (for being compatible with old
> > kexec-tools).
> > - If fails, try to reserve X below 4G (swiotlb need to stay below 4G).
> > - If fails, try to reserve X from MAXMEM top down.
> >
> > So that user can easily reserve large memory with crashkernel=X instead
> > of crashkernel=X,high. It's more transparent and user-friendly.
> >
> > If crashkernel is large and the reserved is beyond 896M, old kexec-tools
> > won't be compatible with new kernel for most of time.
> >
> > kexec will fail out immediately in this case. But the failure could be
> > expected, because old kexec users should not try to reserve that large
> > amount of memory at the first place.
> >
> > On the other hand, old kexec also will fail on old kernel when there's
> > not enough low memory to reserve a large crash kernel area. So the
> > failure of old kexec is consistent between old kernel and new kernel.
> >
> > Signed-off-by: WANG Chao <[email protected]>
> > ---
> > arch/x86/kernel/setup.c | 14 ++++++++++++++
> > 1 file changed, 14 insertions(+)
> >
> > diff --git a/arch/x86/kernel/setup.c b/arch/x86/kernel/setup.c
> > index f0de629..38e6c1f 100644
> > --- a/arch/x86/kernel/setup.c
> > +++ b/arch/x86/kernel/setup.c
> > @@ -593,6 +593,20 @@ static void __init reserve_crashkernel(void)
> > high ? CRASH_KERNEL_ADDR_HIGH_MAX :
> > CRASH_KERNEL_ADDR_LOW_MAX,
> > crash_size, alignment);
> > + /*
> > + * crashkernel=X reserve below 896M fails? Try below 4G
> > + */
> > + if (!high && !crash_base)
> > + crash_base = memblock_find_in_range(alignment,
> > + (1ULL << 32),
> > + crash_size, alignment);
> > + /*
> > + * crashkernel=X reserve below 4G fails? Try MAXMEM
> > + */
> > + if (!high && !crash_base)
> > + crash_base = memblock_find_in_range(alignment,
> > + CRASH_KERNEL_ADDR_HIGH_MAX,
> > + crash_size, alignment);
> >
> > if (!crash_base) {
> > pr_info("crashkernel reservation failed - No suitable area found.\n");
> > --
>
> User should change crashkernel=X to crashkernel=X,high.
crashkernel=X is more straightforward. IMHO, It should be a more general
crash kernel reservation parameter and it shouldn't reserve at low only.
>
> As user could forget to update kexec-tools to utilize ",high" feature, and get
> kdump later fail later.
old kexec-tools should fail to load kernel high anyway, right?
Thanks
WANG Chao
On Mon, Oct 14, 2013 at 11:54:22AM -0700, Yinghai Lu wrote:
> On Mon, Oct 14, 2013 at 4:46 AM, WANG Chao <[email protected]> wrote:
> > Now crashkernel=X will fail out if there's not enough memory at
> > low (below 896M). What makes sense for crashkernel=X would be:
> >
> > - First try to reserve X below 896M (for being compatible with old
> > kexec-tools).
> > - If fails, try to reserve X below 4G (swiotlb need to stay below 4G).
> > - If fails, try to reserve X from MAXMEM top down.
> >
> > So that user can easily reserve large memory with crashkernel=X instead
> > of crashkernel=X,high. It's more transparent and user-friendly.
> >
> > If crashkernel is large and the reserved is beyond 896M, old kexec-tools
> > won't be compatible with new kernel for most of time.
> >
> > kexec will fail out immediately in this case. But the failure could be
> > expected, because old kexec users should not try to reserve that large
> > amount of memory at the first place.
> >
> > On the other hand, old kexec also will fail on old kernel when there's
> > not enough low memory to reserve a large crash kernel area. So the
> > failure of old kexec is consistent between old kernel and new kernel.
> >
> > Signed-off-by: WANG Chao <[email protected]>
> > ---
> > arch/x86/kernel/setup.c | 14 ++++++++++++++
> > 1 file changed, 14 insertions(+)
> >
> > diff --git a/arch/x86/kernel/setup.c b/arch/x86/kernel/setup.c
> > index f0de629..38e6c1f 100644
> > --- a/arch/x86/kernel/setup.c
> > +++ b/arch/x86/kernel/setup.c
> > @@ -593,6 +593,20 @@ static void __init reserve_crashkernel(void)
> > high ? CRASH_KERNEL_ADDR_HIGH_MAX :
> > CRASH_KERNEL_ADDR_LOW_MAX,
> > crash_size, alignment);
> > + /*
> > + * crashkernel=X reserve below 896M fails? Try below 4G
> > + */
> > + if (!high && !crash_base)
> > + crash_base = memblock_find_in_range(alignment,
> > + (1ULL << 32),
> > + crash_size, alignment);
> > + /*
> > + * crashkernel=X reserve below 4G fails? Try MAXMEM
> > + */
> > + if (!high && !crash_base)
> > + crash_base = memblock_find_in_range(alignment,
> > + CRASH_KERNEL_ADDR_HIGH_MAX,
> > + crash_size, alignment);
> >
> > if (!crash_base) {
> > pr_info("crashkernel reservation failed - No suitable area found.\n");
> > --
>
> User should change crashkernel=X to crashkernel=X,high.
I think if we can extend old syntax of crashkernel=X, then it makes life
really easy for users.
>
> As user could forget to update kexec-tools to utilize ",high" feature, and get
> kdump later fail later.
Previously high reservation (reservation above 896M) will anyway fail. So
instead of failing, if we try reservation in higher memory areas why that
would break old kexec-tools?
IOW, previously anyway kexec-tools will not work as no memory will be
reserved in higher memory area. Now memory will be reserved but old
kexec-tools should fail as it can't load in that area.
If that works, then one would use crashkernel=X,high only if he is
particualr that memory reservation comes from area above 4G (despite
the fact that same memory could have been reserved below 4G too).
Thanks
Vivek
On Tue, Oct 15, 2013 at 7:48 AM, Vivek Goyal <[email protected]> wrote:
> On Mon, Oct 14, 2013 at 11:54:22AM -0700, Yinghai Lu wrote:
>> On Mon, Oct 14, 2013 at 4:46 AM, WANG Chao <[email protected]> wrote:
>> User should change crashkernel=X to crashkernel=X,high.
>
> I think if we can extend old syntax of crashkernel=X, then it makes life
> really easy for users.
>
>>
>> As user could forget to update kexec-tools to utilize ",high" feature, and get
>> kdump later fail later.
>
> Previously high reservation (reservation above 896M) will anyway fail. So
> instead of failing, if we try reservation in higher memory areas why that
> would break old kexec-tools?
If thel old kexec-tools would fail, we should let them fail early as possible,
do not reserve at first point.
>
> IOW, previously anyway kexec-tools will not work as no memory will be
> reserved in higher memory area. Now memory will be reserved but old
> kexec-tools should fail as it can't load in that area.
>
> If that works, then one would use crashkernel=X,high only if he is
> particualr that memory reservation comes from area above 4G (despite
> the fact that same memory could have been reserved below 4G too).
My point:
Push user to use ",high" as more as possible, so we only to handle one
path eventually.
for old kernel, leave them to use old grammer. do not need to change it.
Also boot loader should always have different entry for old kernel and
new kernel.
Thanks
Yinghai
On Thu, Oct 17, 2013 at 08:50:07PM -0700, Yinghai Lu wrote:
[..]
> > Previously high reservation (reservation above 896M) will anyway fail. So
> > instead of failing, if we try reservation in higher memory areas why that
> > would break old kexec-tools?
>
> If thel old kexec-tools would fail, we should let them fail early as possible,
> do not reserve at first point.
A user does not care if they get the message "memory nor reserved" or if
they get the message that "could not find a suitable memory hole at
address X".
>
> >
> > IOW, previously anyway kexec-tools will not work as no memory will be
> > reserved in higher memory area. Now memory will be reserved but old
> > kexec-tools should fail as it can't load in that area.
> >
> > If that works, then one would use crashkernel=X,high only if he is
> > particualr that memory reservation comes from area above 4G (despite
> > the fact that same memory could have been reserved below 4G too).
>
> My point:
> Push user to use ",high" as more as possible, so we only to handle one
> path eventually.
> for old kernel, leave them to use old grammer. do not need to change it.
I don't understand this. Why we should push users to use ",high" syntax.
That is an option. Those who want to use it, should use it.
We have been using crashkernel=XM for a long time now. And it makes sense
to extend this option to be able to reserve memory at higher addresses
if asked memory is not available at lower addresses.
You are not thinking about ease of use here for existing users.
>
> Also boot loader should always have different entry for old kernel and
> new kernel.
What does memory reservation location has to do with kernel entry point?
If it is a 64bit bzImage, we will use 64bit entry point by default, isn't
it? Does not matter whether memory is reserved above 4G or not.
I think it makes sense that existing crashkernel=XM usres be able to
reserve memory in higher memory area if sufficient memory is not available
below 896M or below 4G. Those who always want memory reservation above 4G
they should use ",high" syntax and enforce memory allocation above 4G.
Thanks
Vivek
On Fri, Oct 18, 2013 at 5:38 AM, Vivek Goyal <[email protected]> wrote:
> On Thu, Oct 17, 2013 at 08:50:07PM -0700, Yinghai Lu wrote:
>
> [..]
>> > Previously high reservation (reservation above 896M) will anyway fail. So
>> > instead of failing, if we try reservation in higher memory areas why that
>> > would break old kexec-tools?
>>
>> If thel old kexec-tools would fail, we should let them fail early as possible,
>> do not reserve at first point.
>
> A user does not care if they get the message "memory nor reserved" or if
> they get the message that "could not find a suitable memory hole at
> address X".
you are asking for trouble.
Now we have two paths:
1. old kernel with old kexec tools. crashkernel=XM, work,
the new kernel with old kexec tools still working with crashkernel=XM
2. old kernel with old kernel tools, crashkernel=XM, not working.
as X is too big.
then user update to new kernel AND new kexec-tools, and crashkernel=XM,high
with this patch, you will need to test new kernel all old kexec tools
to make sure
it will fail later instead of fail early to remind them to update kexec tools.
Also would make user to guess and try to make new kernel to work with old
kexec-tools
>
>>
>> >
>> > IOW, previously anyway kexec-tools will not work as no memory will be
>> > reserved in higher memory area. Now memory will be reserved but old
>> > kexec-tools should fail as it can't load in that area.
>> >
>> > If that works, then one would use crashkernel=X,high only if he is
>> > particualr that memory reservation comes from area above 4G (despite
>> > the fact that same memory could have been reserved below 4G too).
>>
>> My point:
>> Push user to use ",high" as more as possible, so we only to handle one
>> path eventually.
>> for old kernel, leave them to use old grammer. do not need to change it.
>
> I don't understand this. Why we should push users to use ",high" syntax.
> That is an option. Those who want to use it, should use it.
>
> We have been using crashkernel=XM for a long time now. And it makes sense
> to extend this option to be able to reserve memory at higher addresses
> if asked memory is not available at lower addresses.
>
> You are not thinking about ease of use here for existing users.
most existing user don't need to do anything. just with new kernel and
old kexec tools.
those system that did not work kexec before because XM is too big, they have to
update kexec tools, and use ",high"
Make it simple, less error.
>
>>
>> Also boot loader should always have different entry for old kernel and
>> new kernel.
>
> What does memory reservation location has to do with kernel entry point?
> If it is a 64bit bzImage, we will use 64bit entry point by default, isn't
> it? Does not matter whether memory is reserved above 4G or not.
>
> I think it makes sense that existing crashkernel=XM usres be able to
> reserve memory in higher memory area if sufficient memory is not available
> below 896M or below 4G. Those who always want memory reservation above 4G
> they should use ",high" syntax and enforce memory allocation above 4G.
We already support above 4G, what is point for trying below 4G?
You could get more bug report about new kernel with old kexec-tools, as
old kexec-tools could work with range between 896M and 4G in some case,
but not all.
Thanks
Yinghai
On Fri, Oct 18, 2013 at 10:45:43PM -0700, Yinghai Lu wrote:
[..]
> > A user does not care if they get the message "memory nor reserved" or if
> > they get the message that "could not find a suitable memory hole at
> > address X".
>
> you are asking for trouble.
>
> Now we have two paths:
> 1. old kernel with old kexec tools. crashkernel=XM, work,
> the new kernel with old kexec tools still working with crashkernel=XM
> 2. old kernel with old kernel tools, crashkernel=XM, not working.
> as X is too big.
> then user update to new kernel AND new kexec-tools, and crashkernel=XM,high
>
> with this patch, you will need to test new kernel all old kexec tools
> to make sure
> it will fail later instead of fail early to remind them to update kexec tools.
> Also would make user to guess and try to make new kernel to work with old
> kexec-tools
Seriously, I don't understand what are you trying to say above. You will
need to explain your concern more clearly.
IIUC, you are trying to say that with new kernel old kexec-tools will fail
at a different failure point. I don't see why that is a problem. It still
fails.
[..]
> > You are not thinking about ease of use here for existing users.
>
> most existing user don't need to do anything. just with new kernel and
> old kexec tools.
>
> those system that did not work kexec before because XM is too big, they have to
> update kexec tools, and use ",high"
>
> Make it simple, less error.
No, it is not that simple. Think from a distribution's perspective also.
We have the logic to scale reserved memory based on physical memory
present in the system. Now we are seeing bigger memory systems (which
would not have worked in the past). We still want to retain the existing
logic and not switch to crashkernel=x,high. One does not have to. It
makes life simpler.
Same logic working both with smaller memory systems as well as large memory
systems. One should not have to choose a different command line because
there is more physical RAM present in the system.
>
> >
> >>
> >> Also boot loader should always have different entry for old kernel and
> >> new kernel.
> >
> > What does memory reservation location has to do with kernel entry point?
> > If it is a 64bit bzImage, we will use 64bit entry point by default, isn't
> > it? Does not matter whether memory is reserved above 4G or not.
> >
> > I think it makes sense that existing crashkernel=XM usres be able to
> > reserve memory in higher memory area if sufficient memory is not available
> > below 896M or below 4G. Those who always want memory reservation above 4G
> > they should use ",high" syntax and enforce memory allocation above 4G.
>
> We already support above 4G, what is point for trying below 4G?
Because it is not *required* to reserve memory above 4G. Because we want
same command line to work with both small memory systems as well as
large memory systems and we don't care whether memory is reserved below
4G or above 4G. What does matter though that we don't have to worry about
switching command line option if it is large memory system.
>
> You could get more bug report about new kernel with old kexec-tools, as
> old kexec-tools could work with range between 896M and 4G in some case,
> but not all.
So you are worried that if we lift the artificial cap, and old kexec-tools
start working that will be a problem. I don't think that is an issue.
People worry about something existing broken. It does not break existing
kexec-tools. If in some cases they work with memory reserved at higher
addresses, where is the problem. Even in the past one could do
crashkernel=X@Y and if Y is above 896M and if kexec-tools works with that
what's wrong. I have not seen people complaining about it.
Thanks
Vivek
On Mon, Oct 21, 2013 at 8:16 AM, Vivek Goyal <[email protected]> wrote:
> On Fri, Oct 18, 2013 at 10:45:43PM -0700, Yinghai Lu wrote:
>
>
> IIUC, you are trying to say that with new kernel old kexec-tools will fail
> at a different failure point. I don't see why that is a problem. It still
> fails.
Yes, that could cause confusion.
We already knew it would fail possible at most later, we should make
it skip allocation during first kernel booting.
>
> [..]
>> > You are not thinking about ease of use here for existing users.
>>
>> most existing user don't need to do anything. just with new kernel and
>> old kexec tools.
>>
>> those system that did not work kexec before because XM is too big, they have to
>> update kexec tools, and use ",high"
>>
>> Make it simple, less error.
>
> No, it is not that simple. Think from a distribution's perspective also.
> We have the logic to scale reserved memory based on physical memory
> present in the system. Now we are seeing bigger memory systems (which
> would not have worked in the past). We still want to retain the existing
> logic and not switch to crashkernel=x,high. One does not have to. It
> makes life simpler.
distribution should go with ",high" for 64 bit kernel and new kexec-tools.
for 32bit kernel, you still can have ",high" or not, as ",high" is ignored.
>
> Same logic working both with smaller memory systems as well as large memory
> systems. One should not have to choose a different command line because
> there is more physical RAM present in the system.
",high" is working even on smaller memory sytem.
>
>>
>> We already support above 4G, what is point for trying below 4G?
>
> Because it is not *required* to reserve memory above 4G. Because we want
> same command line to work with both small memory systems as well as
> large memory systems and we don't care whether memory is reserved below
> 4G or above 4G. What does matter though that we don't have to worry about
> switching command line option if it is large memory system.
",high" will work smaller or large memory system after you install new
kexec tools.
Again, for distribution, when new kernel is added, new kernel will all
have ",high"
and new kexec-tools get installed.
Even we want to extend crashkernel=XM, then i would like to have
it identical to crashkernel=XM,high instead.
Thanks
Yinghai
On 10/23/13 at 11:11pm, Yinghai Lu wrote:
> On Mon, Oct 21, 2013 at 8:16 AM, Vivek Goyal <[email protected]> wrote:
> > On Fri, Oct 18, 2013 at 10:45:43PM -0700, Yinghai Lu wrote:
> >
> >
> > IIUC, you are trying to say that with new kernel old kexec-tools will fail
> > at a different failure point. I don't see why that is a problem. It still
> > fails.
>
> Yes, that could cause confusion.
>
> We already knew it would fail possible at most later, we should make
> it skip allocation during first kernel booting.
It's said:
crashkernel=size[KMG][@offset[KMG]]
[KNL] Using kexec, Linux can switch to a 'crash kernel'
upon panic. This parameter reserves the physical
memory region [offset, offset + size] for that kernel
image. If '@offset' is omitted, then a suitable offset
is selected automatically. Check
Documentation/kdump/kdump.txt for further details.
Nothing mentions that crashkernel=X is restricted to reserve at low.
It could be more confusing that crashkernel=X (say 1G) can not find a
suitable area while actually there's enough memory out there.
Current situation is the combination of crashkernel X/high/low is quite
complicated, it's more like supposed to be used by the pros.
Truth is, most user, after upgrading to new kernel, would want reserve
large by just changing to crashkernel=1G, not be forced to use a new
format craskernel=1G,high.
I think crashkernel=XM,high is really supposed to be used when user indeed
want to reserve from high.
Like Vivek said, failing at different point shouldn't be a problem.
That's an incorrect configuration. When crashkernel=1G,high, old
kexec-tools still fails the same way. That could cause confusion, in
your word.
Let me put it in an example, a user want to utilize this new kernel
feature to reserve 1G for crash kernel but not upgrade kexec-tools,
- W/o this patch:
First he would try crashkernel=1G, but failed to reserve. Second time,
he goes with crashkernel=1G,high, reservation is fine but kexec fails
to load. Upgrading kexec-tools is essential to him.
- W/ this patch:
First he would try crashkernel=1G, reservation is ok but kexec fails
to load the same way as the case of crashkernel=1G,high. Upgrading
kexec-tools is essential to him.
The point is old kexec-tools can't load high, no matter by what kind of
crashkernel cmdline to reserve at high.
>
> >
> > [..]
> >> > You are not thinking about ease of use here for existing users.
> >>
> >> most existing user don't need to do anything. just with new kernel and
> >> old kexec tools.
> >>
> >> those system that did not work kexec before because XM is too big, they have to
> >> update kexec tools, and use ",high"
> >>
> >> Make it simple, less error.
> >
> > No, it is not that simple. Think from a distribution's perspective also.
> > We have the logic to scale reserved memory based on physical memory
> > present in the system. Now we are seeing bigger memory systems (which
> > would not have worked in the past). We still want to retain the existing
> > logic and not switch to crashkernel=x,high. One does not have to. It
> > makes life simpler.
>
> distribution should go with ",high" for 64 bit kernel and new kexec-tools.
> for 32bit kernel, you still can have ",high" or not, as ",high" is ignored.
Why ",high" is becoming a default one instead of "XM"?
Isn't that the kernel's job to find a most suitable area at best guess
in this case? By this case, I mean kernel have no idea kexec-tools is
new or old.
>
>
> >
> > Same logic working both with smaller memory systems as well as large memory
> > systems. One should not have to choose a different command line because
> > there is more physical RAM present in the system.
>
> ",high" is working even on smaller memory sytem.
Yeah, you're right.
>
> >
> >>
> >> We already support above 4G, what is point for trying below 4G?
> >
> > Because it is not *required* to reserve memory above 4G. Because we want
> > same command line to work with both small memory systems as well as
> > large memory systems and we don't care whether memory is reserved below
> > 4G or above 4G. What does matter though that we don't have to worry about
> > switching command line option if it is large memory system.
>
> ",high" will work smaller or large memory system after you install new
> kexec tools.
>
> Again, for distribution, when new kernel is added, new kernel will all
> have ",high"
> and new kexec-tools get installed.
>
> Even we want to extend crashkernel=XM, then i would like to have
> it identical to crashkernel=XM,high instead.
Why would you like crashkernel=XM is identical to crashkernel=XM,high?
That's not an option if you want new kernel to be compatible with old
kexec-tools.
Thanks
WANG Chao
On Wed, Oct 23, 2013 at 11:11:51PM -0700, Yinghai Lu wrote:
> On Mon, Oct 21, 2013 at 8:16 AM, Vivek Goyal <[email protected]> wrote:
> > On Fri, Oct 18, 2013 at 10:45:43PM -0700, Yinghai Lu wrote:
> >
> >
> > IIUC, you are trying to say that with new kernel old kexec-tools will fail
> > at a different failure point. I don't see why that is a problem. It still
> > fails.
>
> Yes, that could cause confusion.
>
> We already knew it would fail possible at most later, we should make
> it skip allocation during first kernel booting.
One could upgrade kexec-tools later. So Kernel does not know whether
user space is able to handle higher memory regions or not. So forcing
kernel to skip memory allocation does not sound right to me.
Also even with memory reserved high, old kexec-tools which can't deal
with it will fail anyway. So I don't agree that it is a problem.
[..]
> > No, it is not that simple. Think from a distribution's perspective also.
> > We have the logic to scale reserved memory based on physical memory
> > present in the system. Now we are seeing bigger memory systems (which
> > would not have worked in the past). We still want to retain the existing
> > logic and not switch to crashkernel=x,high. One does not have to. It
> > makes life simpler.
>
> distribution should go with ",high" for 64 bit kernel and new kexec-tools.
> for 32bit kernel, you still can have ",high" or not, as ",high" is ignored.
Just because we have the facility to allocate memory starting from top, does
not mean that we should kill other options and force user to use this
option.
With crashkernel=X,high, a low memory will be reserved for swiotlb. And
I don't want that extra memory reservation. I am more than happy to
reserve memory below 4G and not do low memory reservation.
[..]
> Even we want to extend crashkernel=XM, then i would like to have
> it identical to crashkernel=XM,high instead.
You are assuming that crashkernel=xM,high is always good. I am arguing
that because it requires low memory reservation for swiotlb, when IOMMU
is not around, it will end up reserving more memory. So it is not
necessarily better than reserving memory below 4G.
Hence both crashkernel=xM and crashkernel=XM,high have their own usage.
We have been using crashkernel=xM and we know it works. So extending it
to be able to allocate memory from higher regions, if sufficient memory
is not available in lower regions makes sense. Memory reservation below
4G is more efficient due to not requiring swiotlb. And crashkernel=xM
has been working for us and users are familiar with it.
So I don't see a point that why would you try to block any move to
extend crashkernel=xM semantics.
Thanks
Vivek
On Mon, Oct 14, 2013 at 4:46 AM, WANG Chao <[email protected]> wrote:
> Now crashkernel=X will fail out if there's not enough memory at
> low (below 896M). What makes sense for crashkernel=X would be:
>
> - First try to reserve X below 896M (for being compatible with old
> kexec-tools).
> - If fails, try to reserve X below 4G (swiotlb need to stay below 4G).
> - If fails, try to reserve X from MAXMEM top down.
>
> diff --git a/arch/x86/kernel/setup.c b/arch/x86/kernel/setup.c
> index f0de629..38e6c1f 100644
> --- a/arch/x86/kernel/setup.c
> +++ b/arch/x86/kernel/setup.c
> @@ -593,6 +593,20 @@ static void __init reserve_crashkernel(void)
> high ? CRASH_KERNEL_ADDR_HIGH_MAX :
> CRASH_KERNEL_ADDR_LOW_MAX,
> crash_size, alignment);
> + /*
> + * crashkernel=X reserve below 896M fails? Try below 4G
> + */
> + if (!high && !crash_base)
> + crash_base = memblock_find_in_range(alignment,
> + (1ULL << 32),
> + crash_size, alignment);
Another problem, it would allocate range in [0,4g) for 32bit,
if the user have crashkernel=512M or plus.
> + /*
> + * crashkernel=X reserve below 4G fails? Try MAXMEM
> + */
> + if (!high && !crash_base)
> + crash_base = memblock_find_in_range(alignment,
> + CRASH_KERNEL_ADDR_HIGH_MAX,
> + crash_size, alignment);
>
> if (!crash_base) {
> pr_info("crashkernel reservation failed - No suitable area found.\n");
Yinghai
On Thu, Oct 24, 2013 at 4:01 AM, WANG Chao <[email protected]> wrote:
>
> I think crashkernel=XM,high is really supposed to be used when user indeed
> want to reserve from high.
No. Keep the all 64bit to stay high, make thing simple.
instead of some low and some use high.
>
> Like Vivek said, failing at different point shouldn't be a problem.
> That's an incorrect configuration. When crashkernel=1G,high, old
> kexec-tools still fails the same way. That could cause confusion, in
> your word.
If it would fail later, we should let it fail early as possible.
>
> Let me put it in an example, a user want to utilize this new kernel
> feature to reserve 1G for crash kernel but not upgrade kexec-tools,
>
> - W/o this patch:
> First he would try crashkernel=1G, but failed to reserve. Second time,
> he goes with crashkernel=1G,high, reservation is fine but kexec fails
> to load. Upgrading kexec-tools is essential to him.
>
> - W/ this patch:
> First he would try crashkernel=1G, reservation is ok but kexec fails
> to load the same way as the case of crashkernel=1G,high. Upgrading
> kexec-tools is essential to him.
>
> The point is old kexec-tools can't load high, no matter by what kind of
> crashkernel cmdline to reserve at high.
old kexec-tools could work cross 892M in some case.
That will confuse the user, as it works some time on some setup, but does
not work on other setup.
Thanks
Yinghai
On Thu, Oct 24, 2013 at 7:02 AM, Vivek Goyal <[email protected]> wrote:
> On Wed, Oct 23, 2013 at 11:11:51PM -0700, Yinghai Lu wrote:
>
> Hence both crashkernel=xM and crashkernel=XM,high have their own usage.
> We have been using crashkernel=xM and we know it works. So extending it
> to be able to allocate memory from higher regions, if sufficient memory
> is not available in lower regions makes sense. Memory reservation below
> 4G is more efficient due to not requiring swiotlb. And crashkernel=xM
> has been working for us and users are familiar with it.
>
> So I don't see a point that why would you try to block any move to
> extend crashkernel=xM semantics.
Make the thing simple.
Keep them separately, leave crashkernel=xM to old kexec-tools mostly
and keep crashkernel=xM,high to newer kexec-tools as needed.
Thanks
Yinghai
On Thu, Oct 24, 2013 at 12:15:25PM -0700, Yinghai Lu wrote:
> On Thu, Oct 24, 2013 at 7:02 AM, Vivek Goyal <[email protected]> wrote:
> > On Wed, Oct 23, 2013 at 11:11:51PM -0700, Yinghai Lu wrote:
> >
> > Hence both crashkernel=xM and crashkernel=XM,high have their own usage.
> > We have been using crashkernel=xM and we know it works. So extending it
> > to be able to allocate memory from higher regions, if sufficient memory
> > is not available in lower regions makes sense. Memory reservation below
> > 4G is more efficient due to not requiring swiotlb. And crashkernel=xM
> > has been working for us and users are familiar with it.
> >
> > So I don't see a point that why would you try to block any move to
> > extend crashkernel=xM semantics.
>
> Make the thing simple.
> Keep them separately, leave crashkernel=xM to old kexec-tools mostly
> and keep crashkernel=xM,high to newer kexec-tools as needed.
I am keeping things simple by making sure that both old kexec-tools and
new kexec-tools can use crashkernel=xM and one does not have to choose
between two based on what kexec-tools version you are using.
Also keeping things simple by not trying to *impose* a new crashkernel=
syntax on existing crashkernel=xM users.
Hence extending the semantics of crashkernel=xM makes sense to me.
Thanks
Vivek
On Thu, Oct 24, 2013 at 12:18 PM, Vivek Goyal <[email protected]> wrote:
> On Thu, Oct 24, 2013 at 12:15:25PM -0700, Yinghai Lu wrote:
>
> Also keeping things simple by not trying to *impose* a new crashkernel=
> syntax on existing crashkernel=xM users.
Existing user that have crashkernel=xM working with their old kernel
and old kexec-tools, they still could keep their old command line and
old kexec-tools
with new updated kernel.
We should not change semantics to surprise them.
Thanks
Yinghai
On Thu, Oct 24, 2013 at 12:13:41PM -0700, Yinghai Lu wrote:
[..]
> old kexec-tools could work cross 892M in some case.
> That will confuse the user,
If it could work beyong 892MB, then why are you limiting user to 896MB
only. That sounds wrong.
> as it works some time on some setup, but does
> not work on other setup.
If it can work above 896MB, then those other setup should be fixed.
All the existing users of crashkernel=xM *will not* be affected by this
change. Memory reservation will still be below 892MB.
Memory reservation will gove above 896MB, only if user chooses to reserve
more memory. If kexec works in that case, its a win-win situation. If it
does not and fails, then it is status quo is maintained.
Thanks
Vivek
On Thu, Oct 24, 2013 at 12:24:57PM -0700, Yinghai Lu wrote:
> On Thu, Oct 24, 2013 at 12:18 PM, Vivek Goyal <[email protected]> wrote:
> > On Thu, Oct 24, 2013 at 12:15:25PM -0700, Yinghai Lu wrote:
> >
> > Also keeping things simple by not trying to *impose* a new crashkernel=
> > syntax on existing crashkernel=xM users.
>
> Existing user that have crashkernel=xM working with their old kernel
> and old kexec-tools, they still could keep their old command line and
> old kexec-tools
> with new updated kernel.
> We should not change semantics to surprise them.
Old users will get reservation still below 896MB.
It will go above 896MB only if memory could not be allocated below 896MB.
In the past reservation will fail and kexec-tools will fail.
Now reservation will succeed but kexec-tools will fail.
So end result a user sees is that kexec-tools fails. So I don't see how
we are breaking existing installations or user setups.
Thanks
Vivek
On Thu, Oct 24, 2013 at 03:27:52PM -0400, Vivek Goyal wrote:
> On Thu, Oct 24, 2013 at 12:24:57PM -0700, Yinghai Lu wrote:
> > On Thu, Oct 24, 2013 at 12:18 PM, Vivek Goyal <[email protected]> wrote:
> > > On Thu, Oct 24, 2013 at 12:15:25PM -0700, Yinghai Lu wrote:
> > >
> > > Also keeping things simple by not trying to *impose* a new crashkernel=
> > > syntax on existing crashkernel=xM users.
> >
> > Existing user that have crashkernel=xM working with their old kernel
> > and old kexec-tools, they still could keep their old command line and
> > old kexec-tools
> > with new updated kernel.
> > We should not change semantics to surprise them.
>
> Old users will get reservation still below 896MB.
>
> It will go above 896MB only if memory could not be allocated below 896MB.
>
> In the past reservation will fail and kexec-tools will fail.
> Now reservation will succeed but kexec-tools will fail.
Only corner case I can see is where memory is reserved higher, old kexec-tools
is not smart enough to figure out in advance that kernel will not boot
at this high address and still goes ahead and loads the kernel and crash
dump fails at run time.
But this will happen only if memory reservation below 896MB fails. And
in most of the cases after kernel upgrade, memory should still come
from memory region below 896MB.
Thanks
Vivek
On Thu, Oct 24, 2013 at 12:27 PM, Vivek Goyal <[email protected]> wrote:
> On Thu, Oct 24, 2013 at 12:24:57PM -0700, Yinghai Lu wrote:
>> On Thu, Oct 24, 2013 at 12:18 PM, Vivek Goyal <[email protected]> wrote:
>> > On Thu, Oct 24, 2013 at 12:15:25PM -0700, Yinghai Lu wrote:
>> >
>> > Also keeping things simple by not trying to *impose* a new crashkernel=
>> > syntax on existing crashkernel=xM users.
>>
>> Existing user that have crashkernel=xM working with their old kernel
>> and old kexec-tools, they still could keep their old command line and
>> old kexec-tools
>> with new updated kernel.
>> We should not change semantics to surprise them.
>
> Old users will get reservation still below 896MB.
>
> It will go above 896MB only if memory could not be allocated below 896MB.
>
> In the past reservation will fail and kexec-tools will fail.
> Now reservation will succeed but kexec-tools will fail.
>
> So end result a user sees is that kexec-tools fails. So I don't see how
> we are breaking existing installations or user setups.
case could be: if user add more memory and put more pcie cards, and
second kernel will need more ram and OOM there.
so user could just increase crashkernel=512M to crashkernel=1G.
without Cong's patch, kernel will fail to reserve, and user would dig
to change it
to crashkernel=1G,high, and update kexec-tools.
with Cong's patch, kernel will reserve other range like between 896
and 4G, old kexec-tools either
fail to load second kernel or hang in purgatory or early stage of
second kernel, or other unknown behavior.
I would think first path is much clear and predicted.
If my memory is right, HPA did not like idea that we try below 896M,
and then under 4G and then above 4G.
and want us to have ",high" solution. Not sure if he would change his mind.
Also you will need and #ifdef CONFIG_X86_64 for 896M/4G searching code
so it will not confuse the
32bit arch.
Thanks
Yinghai
> Again, for distribution, when new kernel is added, new kernel will all
> have ",high"
> and new kexec-tools get installed.
>
> Even we want to extend crashkernel=XM, then i would like to have
> it identical to crashkernel=XM,high instead.
My points: I can accept the approach of extending crashkernel=XM to
use high mem firstly if we could remove the extra 72M in low mem, but
apparently it's not doable now. So I would vote for Chao's patch.
Also newer kernel use more complex ",high" then older kernel looks not good.
Thanks
Dave
On 10/28/13 at 01:49pm, Dave Young wrote:
> > without Cong's patch, kernel will fail to reserve, and user would dig
>
> s/Chao/Cong ;)
s/Cong/Chao in fact :(
> without Cong's patch, kernel will fail to reserve, and user would dig
s/Chao/Cong ;)
On 10/24/13 at 12:04pm, Yinghai Lu wrote:
> On Mon, Oct 14, 2013 at 4:46 AM, WANG Chao <[email protected]> wrote:
> > Now crashkernel=X will fail out if there's not enough memory at
> > low (below 896M). What makes sense for crashkernel=X would be:
> >
> > - First try to reserve X below 896M (for being compatible with old
> > kexec-tools).
> > - If fails, try to reserve X below 4G (swiotlb need to stay below 4G).
> > - If fails, try to reserve X from MAXMEM top down.
> >
> > diff --git a/arch/x86/kernel/setup.c b/arch/x86/kernel/setup.c
> > index f0de629..38e6c1f 100644
> > --- a/arch/x86/kernel/setup.c
> > +++ b/arch/x86/kernel/setup.c
> > @@ -593,6 +593,20 @@ static void __init reserve_crashkernel(void)
> > high ? CRASH_KERNEL_ADDR_HIGH_MAX :
> > CRASH_KERNEL_ADDR_LOW_MAX,
> > crash_size, alignment);
> > + /*
> > + * crashkernel=X reserve below 896M fails? Try below 4G
> > + */
> > + if (!high && !crash_base)
> > + crash_base = memblock_find_in_range(alignment,
> > + (1ULL << 32),
> > + crash_size, alignment);
>
> Another problem, it would allocate range in [0,4g) for 32bit,
> if the user have crashkernel=512M or plus.
Yes, should use #ifdef CONFIG_X86_64 .. #endif.
Thanks for pointing this out.
WANG Chao
>
> > + /*
> > + * crashkernel=X reserve below 4G fails? Try MAXMEM
> > + */
> > + if (!high && !crash_base)
> > + crash_base = memblock_find_in_range(alignment,
> > + CRASH_KERNEL_ADDR_HIGH_MAX,
> > + crash_size, alignment);
> >
> > if (!crash_base) {
> > pr_info("crashkernel reservation failed - No suitable area found.\n");
>
>
> Yinghai
On Thu, Oct 24, 2013 at 10:48:29PM -0700, Yinghai Lu wrote:
> On Thu, Oct 24, 2013 at 12:27 PM, Vivek Goyal <[email protected]> wrote:
> > On Thu, Oct 24, 2013 at 12:24:57PM -0700, Yinghai Lu wrote:
> >> On Thu, Oct 24, 2013 at 12:18 PM, Vivek Goyal <[email protected]> wrote:
> >> > On Thu, Oct 24, 2013 at 12:15:25PM -0700, Yinghai Lu wrote:
> >> >
> >> > Also keeping things simple by not trying to *impose* a new crashkernel=
> >> > syntax on existing crashkernel=xM users.
> >>
> >> Existing user that have crashkernel=xM working with their old kernel
> >> and old kexec-tools, they still could keep their old command line and
> >> old kexec-tools
> >> with new updated kernel.
> >> We should not change semantics to surprise them.
> >
> > Old users will get reservation still below 896MB.
> >
> > It will go above 896MB only if memory could not be allocated below 896MB.
> >
> > In the past reservation will fail and kexec-tools will fail.
> > Now reservation will succeed but kexec-tools will fail.
> >
> > So end result a user sees is that kexec-tools fails. So I don't see how
> > we are breaking existing installations or user setups.
>
> case could be: if user add more memory and put more pcie cards, and
> second kernel will need more ram and OOM there.
Now makedumpfile supports cyclic mode by default. So one does not have
to necessarily linearly scale reserved memory based on physical memory
present in the box.
> so user could just increase crashkernel=512M to crashkernel=1G.
If user has new makedumpfile, OOM should not happen and one should not
have to increase memory reservation.
>
> without Cong's patch, kernel will fail to reserve, and user would dig
> to change it
> to crashkernel=1G,high, and update kexec-tools.
>
> with Cong's patch, kernel will reserve other range like between 896
> and 4G, old kexec-tools either
> fail to load second kernel or hang in purgatory or early stage of
> second kernel, or other unknown behavior.
I understand your concern about memory being reserved high and kexec
just hanging. Only thing I am arguing is that the number of cases where
it will happen is small.
- First of all for all old existing kexec-tools memory will stil come
from 896MB.
- Old kexec-tools enforced that purgatory is loaded below 2G. So if
memory is reserved above 2G, kexec-tools will fail.
So only problematic case seems to be that if user increased crashkernel=X
value and memory got reserved between 896M and 2G. But this is not same
as breaking old setups as old setups anyway never worked with this
configuration.
You argument that user will research and upgrade kexec-tools and use
crashkernel=X,high, then it holds true for the case where memory is
reserved between 896M and 2G.
I personally think that it is easier for a user to not change any
kernel parameters with kernel and kexec-tools upgrade and still be
able to work with large memory systems. So the benefit of extending
the semantics of existing parameter seems to be outweighing the downside
of side, IMHO.
>
> I would think first path is much clear and predicted.
>
> If my memory is right, HPA did not like idea that we try below 896M,
> and then under 4G and then above 4G.
hpa, I know you did not like the idea in the past. Is it still the case.
IMHO, I like the fact that existing users still be able to work with
crashkernel=X and not forced to switch to crashkernel=x,high and also incur
the penalty of reserving extra memory for software iotlb.
Thanks
Vivek