From: Nate Stahl <[email protected]>
A full task stack dump of all tasks on a machine can generate more than
4MB of output to dmesg. Dumping this data to the serial console causes
the machine to hang for a number of minutes (an unacceptable impact),
but dumping the same data to memory is feasible if the dmesg buffer is
sized large enough to hold the output. Set to 16MB which will hopefully
be large enough to handle a dump from any of our servers at this time.
Signed-off-by: Nate Stahl <[email protected]>
Signed-off-by: Vinson Lee <[email protected]>
---
init/Kconfig | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/init/Kconfig b/init/Kconfig
index 903505e..266a3e8 100644
--- a/init/Kconfig
+++ b/init/Kconfig
@@ -812,7 +812,7 @@ config IKCONFIG_PROC
config LOG_BUF_SHIFT
int "Kernel log buffer size (16 => 64KB, 17 => 128KB)"
- range 12 21
+ range 12 24
default 17
depends on PRINTK
help
--
1.9.1
On Fri, Dec 12, 2014 at 02:38:26PM -0800, Vinson Lee wrote:
> From: Nate Stahl <[email protected]>
>
> A full task stack dump of all tasks on a machine can generate more than
> 4MB of output to dmesg. Dumping this data to the serial console causes
> the machine to hang for a number of minutes (an unacceptable impact),
> but dumping the same data to memory is feasible if the dmesg buffer is
> sized large enough to hold the output. Set to 16MB which will hopefully
> be large enough to handle a dump from any of our servers at this time.
>
> Signed-off-by: Nate Stahl <[email protected]>
> Signed-off-by: Vinson Lee <[email protected]>
Isn't this the perpetual issue of having large number of CPUs? If so
consider use of LOG_CPU_MAX_BUF_SHIFT instead, otherwise clarifying how
this would be a different issue would be good. LOG_CPU_MAX_BUF_SHIFT
should scale nicely but you can increase it as well, is it being used?
Luis
On Fri, Dec 12, 2014 at 2:45 PM, Luis R. Rodriguez <[email protected]> wrote:
> On Fri, Dec 12, 2014 at 02:38:26PM -0800, Vinson Lee wrote:
>> From: Nate Stahl <[email protected]>
>>
>> A full task stack dump of all tasks on a machine can generate more than
>> 4MB of output to dmesg. Dumping this data to the serial console causes
>> the machine to hang for a number of minutes (an unacceptable impact),
>> but dumping the same data to memory is feasible if the dmesg buffer is
>> sized large enough to hold the output. Set to 16MB which will hopefully
>> be large enough to handle a dump from any of our servers at this time.
>>
>> Signed-off-by: Nate Stahl <[email protected]>
>> Signed-off-by: Vinson Lee <[email protected]>
>
> Isn't this the perpetual issue of having large number of CPUs? If so
> consider use of LOG_CPU_MAX_BUF_SHIFT instead, otherwise clarifying how
> this would be a different issue would be good. LOG_CPU_MAX_BUF_SHIFT
> should scale nicely but you can increase it as well, is it being used?
>
> Luis
No, it is not being used.
LOG_CPU_MAX_BUF_SHIFT is a 3.17+ config and we do not have any of the
above mentioned production machines running 3.17 or later. This patch
did help us with debugging when running on older stable kernels
though.
Thanks for the quick reply.
On Fri, Dec 12, 2014 at 03:07:23PM -0800, Vinson Lee wrote:
> On Fri, Dec 12, 2014 at 2:45 PM, Luis R. Rodriguez <[email protected]> wrote:
> > On Fri, Dec 12, 2014 at 02:38:26PM -0800, Vinson Lee wrote:
> >> From: Nate Stahl <[email protected]>
> >>
> >> A full task stack dump of all tasks on a machine can generate more than
> >> 4MB of output to dmesg. Dumping this data to the serial console causes
> >> the machine to hang for a number of minutes (an unacceptable impact),
> >> but dumping the same data to memory is feasible if the dmesg buffer is
> >> sized large enough to hold the output. Set to 16MB which will hopefully
> >> be large enough to handle a dump from any of our servers at this time.
> >>
> >> Signed-off-by: Nate Stahl <[email protected]>
> >> Signed-off-by: Vinson Lee <[email protected]>
> >
> > Isn't this the perpetual issue of having large number of CPUs? If so
> > consider use of LOG_CPU_MAX_BUF_SHIFT instead, otherwise clarifying how
> > this would be a different issue would be good. LOG_CPU_MAX_BUF_SHIFT
> > should scale nicely but you can increase it as well, is it being used?
> >
> > Luis
>
>
> No, it is not being used.
>
> LOG_CPU_MAX_BUF_SHIFT is a 3.17+ config and we do not have any of the
> above mentioned production machines running 3.17 or later.
Then consider backporting it.
> This patch did help us with debugging when running on older stable kernels
> though.
Understood, give the other stuff a spin.
Luis