Hi,
this is a patch which give you a better overview of the userland
application stack usage, especially for embedded linux.
Currently you are only able to dump the main process/thread stack usage
which is showed in proc/pid/status by the "VmStk" Value. But you get no
information about the consumed stack memory of the the threads.
For some reasons it becomes important to know how much memory is
consumed by each thread:
- Get out of virtual memory by creating a lot of threads
(f.e. the developer did assign each of them the default size)
- Misuse the thread stack for big temporary data buffers
- Thread stack overruns
So this patch gives the developer an important tool to figure out if there
is one of this issues.
The patch is splitted in three parts.
Part 1 :
--------
fs/exec.c | 4 ++++
fs/proc/array.c | 22 ++++++++++++++++++++++
fs/proc/task_mmu.c | 12 ++++++++++++
include/linux/sched.h | 3 +++
init/Kconfig | 12 ++++++++++++
kernel/fork.c | 5 +++++
6 files changed, 58 insertions(+)
This is an enhancement in the /proc/<pid>/tasks/<tid>/maps and
smaps which marks the mapping where the thread stack pointer reside with
"[thread stack]".
Also there is a new entry "stack usage" in proc/pid/status, which will
you give the current stack usage.
This feature will be enabled the which "enable /proc/<pid> stack
monitoring" under "General setup-->"
Part 2:
-------
fs/proc/Makefile | 1
fs/proc/stackmon.c | 254 +++++++++++++++++++++++++++++++++++++++++++++++++++++
init/Kconfig | 10 ++
3 files changed, 265 insertions(+)
This enable a new /proc/stackmon file entry. A cat /proc/stackmon
will produce a output like:
bytes pages maxpages vm_start vm_end processid threadid name
436 1 1 afdbf000-afdd4000 pid: 409 tid: 409 syslogd
1168 1 1 afd12000-afd27000 pid: 411 tid: 411 sh
516 1 1 afe6c000-afe81000 pid: 412 tid: 412 getty
4580 2 2 af918000-af92d000 pid: 419 tid: 419 cat
The first value is the current effektive stack usage in bytes.
The second value is the current real stack usage in pages. This means
how many pages are really occupied by the stack.
The thrird value is the maximum real stack usage in pages. This value
will be determinate by the difference of the highest used page in the
mapping and the page of stack start address.
The fourth value is the start- and end- address of the mapping where the
stack currently reside.
The fifth value process id.
The sixth value is thread id.
And the seventh and last entry is the name of the process.
This feature will be enabled the which "enable /proc/<pid> stack
monitoring" under "General setup-->".
Part 3:
-------
init/Kconfig | 13 ++
mm/Makefile | 1
mm/stackdbg.c | 319 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
3 files changed, 332 insertions(+)
There is also an additional stack monitor which can be enabled by boot
time or by the /sys filesystem. Which this you are able to detect if a
application use more stack than a given value. If the application
exceeds this value, it will receive a SIGTRAP signal.
If there is a debugger attached at this time, there is the ability to
examinate the stack usage. Otherwise the application will be terminated.
In both cases a kernel log entry "pid:%d (%s) tid:%d stack size %lu
exceeds max stack size." will be written.
There are following entries under /sys/kernel/stackmon to control the
monitor.
mode:
Setting this to an value not equal zero will start the stack monitoring
thread. Default is 0. Setting it to zero will stop the kernel thread.
stacksize:
This value is the stack size in kb which triggers a SIGTRAP to the
application, if the stack usage is equal or more. Default is 256 kb.
ticks:
Number of ticks between the monitoring invocation. A higher value will
give a less change to trigger a stack over usage, but will also result
in a less CPU usage. Default is 1.
All this parameters can also set at boot time with the kernel parameter
"stackmon=<stacksize>:<mode>:<ticks>".
The stack monitor can also be compiled as a module.
This patch is against 2.6.29. The patch is cpu independent, so it
should work on all linux supported architectures, it was tested under
x86 and powerpc. There is no dependency to a library: glibc, uclibc
and all other should work.
I hope you like it and want ask for inclusion into the kernel or linux-next?
Please give it a try.
If you have ideas how to do things in a better way, please let me know.
Have a nice day,
Stefani
Stefani Seibold <[email protected]> writes:
>
> - Get out of virtual memory by creating a lot of threads
> (f.e. the developer did assign each of them the default size)
The application just fails then? I don't think that needs
a new monitoring tool.
> - Misuse the thread stack for big temporary data buffers
That would be better checked for at compile time
(except for alloca, but that is quite rare)
> - Thread stack overruns
Your method would be racy at best to determine this because
you don't keep track of the worst case, only the current case.
So e.g. if you monitoring app checks once per second the stack
could overflow between your monitoring intervals, but already
have bounced back before the checker comes in.
gcc has support to generate stack overflow checking code,
that would be more reliable. Alternatively you could keep
track of consumption in the VMA that has the stack, but
that can't handle very large jumps (like f() { char x[1<<30]; } )
The later can only be handled well by the compiler.
-Andi
--
[email protected] -- Speaking for myself only.
Hi Andi,
Am Dienstag, den 31.03.2009, 17:49 +0200 schrieb Andi Kleen:
> Stefani Seibold <[email protected]> writes:
> >
> > - Get out of virtual memory by creating a lot of threads
> > (f.e. the developer did assign each of them the default size)
>
> The application just fails then? I don't think that needs
> a new monitoring tool.
>
First, this patch is not only a monitoring tool. Only the last part 3/3
is the monitoring tool.
Patch 1/3 enhance the the proc/<pid>/task/<tid>/maps by the marking the
thread stack.
Patch 2/3 gives you an overview of the current process/thread stack
usage with the /proc/stackmon entry.
> > - Misuse the thread stack for big temporary data buffers
>
> That would be better checked for at compile time
> (except for alloca, but that is quite rare)
Fine but it did not work for functions like:
void foo(int n)
{
char buf[n*1024];
}
This is valid with gcc.
>
> > - Thread stack overruns
>
> Your method would be racy at best to determine this because
> you don't keep track of the worst case, only the current case.
>
> So e.g. if you monitoring app checks once per second the stack
> could overflow between your monitoring intervals, but already
> have bounced back before the checker comes in.
>
The Monitor is part 3/3. And you are right it is not a complete rock
solid solution. But it works in many cases and thats is what counts.
> Alternatively you could keep
> track of consumption in the VMA that has the stack, but
> that can't handle very large jumps (like f() { char x[1<<30]; } )
> The later can only be handled well by the compiler.
Thats is exactly what i am doing, i walk through the pages of the thread
stack mapped memory and keep track of the highest access page. So i have
the high water mark of the used stack.
The patches are not intrusive, especially part 1.
>
Stefani
On Tue, 31 March 2009 20:22:15 +0200, Stefani Seibold wrote:
> Am Dienstag, den 31.03.2009, 17:49 +0200 schrieb Andi Kleen:
> > Stefani Seibold <[email protected]> writes:
>
> > > - Misuse the thread stack for big temporary data buffers
> >
> > That would be better checked for at compile time
> > (except for alloca, but that is quite rare)
>
> Fine but it did not work for functions like:
>
> void foo(int n)
> {
> char buf[n*1024];
>
> }
>
> This is valid with gcc.
Good call. checkstack should look for those as well. It is certainly
possible to detect statically and warn about:
10: 29 c4 sub %eax,%esp
Runaway recursions are a different matter, though. The code I once had
to detect them depends on an old version of smatch, which in turn
depends on gcc 3.1. And even assuming this was in a reasonable shape, I
still don't know what to do about it. The kernel has thousands of
recursions and trying to work out how deep each one may stack is a
never-ending project.
Jörn
--
A quarrel is quickly settled when deserted by one party; there is
no battle unless there be two.
-- Seneca
On Tue, Mar 31, 2009 at 08:22:15PM +0200, Stefani Seibold wrote:
> Hi Andi,
>
> Am Dienstag, den 31.03.2009, 17:49 +0200 schrieb Andi Kleen:
> > Stefani Seibold <[email protected]> writes:
> > >
> > > - Get out of virtual memory by creating a lot of threads
> > > (f.e. the developer did assign each of them the default size)
> >
> > The application just fails then? I don't think that needs
> > a new monitoring tool.
> >
>
> First, this patch is not only a monitoring tool. Only the last part 3/3
> is the monitoring tool.
>
> Patch 1/3 enhance the the proc/<pid>/task/<tid>/maps by the marking the
> thread stack.
Well some implementation of it. There are certainly runtimes that
switch stacks. For example what happens when someone uses sigaltstack()?
You'll probably need some more sanity checks, otherwise a monitor
will occasionally just report crap in legitimate cases. Unfortunately
it's difficult to distingush the legitimate cases from the bugs.
> > > - Misuse the thread stack for big temporary data buffers
> >
> > That would be better checked for at compile time
> > (except for alloca, but that is quite rare)
>
> Fine but it did not work for functions like:
That's the alloca() case, but you can disable both with the right options.
There's still the "recursive function" case.
> > > - Thread stack overruns
> >
> > Your method would be racy at best to determine this because
> > you don't keep track of the worst case, only the current case.
> >
> > So e.g. if you monitoring app checks once per second the stack
> > could overflow between your monitoring intervals, but already
> > have bounced back before the checker comes in.
> >
>
> The Monitor is part 3/3. And you are right it is not a complete rock
> solid solution. But it works in many cases and thats is what counts.
For stack overflow one would think a rock solid solution
is needed? After all you'll crash if you miss a case.
> > track of consumption in the VMA that has the stack, but
> > that can't handle very large jumps (like f() { char x[1<<30]; } )
> > The later can only be handled well by the compiler.
>
> Thats is exactly what i am doing, i walk through the pages of the thread
> stack mapped memory and keep track of the highest access page. So i have
> the high water mark of the used stack.
Ok. Of course it doesn't work for really large allocations. there used
to be special patches around to add a special gap to catch those,
but it's problematic with the tight address space on 32bit and also
again not fully bullet proof. On 64bit the solution is typical
to just use a large stack.
> The patches are not intrusive, especially part 1.
To be honest it seems too much like a special case hack to me
to include by default. It could be probably done with a systemtap
script in the same way, but I would really recommend to just
build with gcc's stack overflow checker while testing together
with static checking.
-Andi
--
[email protected] -- Speaking for myself only.
Hi Andi,
stop complaining about the monitor. This is only an additional
functionality.
The main purpose are part 1 and 2.
>
> Well some implementation of it. There are certainly runtimes that
> switch stacks. For example what happens when someone uses sigaltstack()?
>
What should happen with sigaltstack? This is complete independent from
the process and thread stack. So it works.
> That's the alloca() case, but you can disable both with the right options.
> There's still the "recursive function" case.
>
And no idea ;-)
> >
> > The Monitor is part 3/3. And you are right it is not a complete rock
> > solid solution. But it works in many cases and thats is what counts.
>
> For stack overflow one would think a rock solid solution
> is needed? After all you'll crash if you miss a case.
>
Again, the monitor is the only a part of the patch and i know that this
is a issue.
The first two patches will also work without the monitor and if you
don't like the monitor, no problem. It is a CONFIG_... parameter.
> To be honest it seems too much like a special case hack to me
> to include by default. It could be probably done with a systemtap
> script in the same way, but I would really recommend to just
> build with gcc's stack overflow checker while testing together
> with static checking.
>
Thanks for the hack - I am not sure if you really had a look at my first
posting nor had a look into my code.
We discus about complete different things. You have from user land no
possibility to figure out where is the thread stack locate nor what was
the highest used thread stack address.
That is a simple debug information which can provide very easily which
the first two patches.
Stefani