2007-05-17 16:24:43

by Andrea Righi

[permalink] [raw]
Subject: [RFC] log out-of-virtual-memory events

I'm looking for a way to keep track of the processes that fail to allocate new
virtual memory. What do you think about the following approach (untested)?

--

Print informations about the processes that fail to allocate virtual memory.

Signed-off-by: Andrea Righi <[email protected]>

diff -urpN linux-2.6.21/mm/mmap.c linux-2.6.21-vm-log-enomem/mm/mmap.c
--- linux-2.6.21/mm/mmap.c 2007-04-26 05:08:32.000000000 +0200
+++ linux-2.6.21-vm-log-enomem/mm/mmap.c 2007-05-17 18:05:39.000000000 +0200
@@ -77,6 +77,26 @@ int sysctl_max_map_count __read_mostly =
atomic_t vm_committed_space = ATOMIC_INIT(0);

/*
+ * Print current process informations when it fails to allocate new virtual
+ * memory.
+ */
+static inline void log_vm_enomem(void)
+{
+ unsigned long total_vm = 0;
+ struct mm_struct *mm;
+
+ task_lock(current);
+ mm = current->mm;
+ if (mm)
+ total_vm = mm->total_vm;
+ task_unlock(current);
+
+ printk(KERN_INFO
+ "out of virtual memory for process %d (%s): total_vm=%lu, uid=%d\n",
+ current->pid, current->comm, total_vm, current->uid);
+}
+
+/*
* Check that a process has enough memory to allocate a new virtual
* mapping. 0 means there is enough memory for the allocation to
* succeed and -ENOMEM implies there is not.
@@ -175,6 +195,7 @@ int __vm_enough_memory(long pages, int c
return 0;
error:
vm_unacct_memory(pages);
+ log_vm_enomem();

return -ENOMEM;
}


2007-05-17 18:23:30

by Rik van Riel

[permalink] [raw]
Subject: Re: [RFC] log out-of-virtual-memory events

Andrea Righi wrote:
> I'm looking for a way to keep track of the processes that fail to allocate new
> virtual memory. What do you think about the following approach (untested)?

Looks like an easy way for users to spam syslogd over and
over and over again.

At the very least, shouldn't this be dependant on print_fatal_signals?

--
Politics is the struggle between those who want to make their country
the best in the world, and those who believe it already is. Each group
calls the other unpatriotic.

2007-05-18 06:31:16

by Jan Engelhardt

[permalink] [raw]
Subject: Re: signals logged / [RFC] log out-of-virtual-memory events


On May 17 2007 14:22, Rik van Riel wrote:
> Andrea Righi wrote:
>> I'm looking for a way to keep track of the processes that fail to allocate
>> new
>> virtual memory. What do you think about the following approach (untested)?
>
> Looks like an easy way for users to spam syslogd over and
> over and over again.
>
> At the very least, shouldn't this be dependant on print_fatal_signals?

Speaking of signals, everytime I get a segfault (or force one with a test
program) on x86_64, the kernel prints to dmesg:

fail[22278]: segfault at 0000000000000000 rip 00000000004004b8 rsp
00007ffff7ecda50 error 6

I do not see such on i386, so why for x86_64?


Jan
--

2007-05-18 07:50:18

by Andrea Righi

[permalink] [raw]
Subject: Re: [RFC] log out-of-virtual-memory events

Rik van Riel wrote:
> Andrea Righi wrote:
>> I'm looking for a way to keep track of the processes that fail to
>> allocate new
>> virtual memory. What do you think about the following approach
>> (untested)?
>
> Looks like an easy way for users to spam syslogd over and
> over and over again.
>
> At the very least, shouldn't this be dependant on print_fatal_signals?
>

Anyway, with print-fatal-signals enabled a user could spam syslogd too, simply
with a (char *)0 = 0 program, but we could always identify the spam attempts
logging the process uid...

In any case, I agree, it should depend on that patch...

What about adding a simple msleep_interruptible(SOME_MSECS) at the end of
log_vm_enomem() or, at least, a might_sleep() to limit the potential spam/second
rate?

-Andrea

2007-05-18 09:16:17

by Robin Holt

[permalink] [raw]
Subject: Re: [RFC] log out-of-virtual-memory events

On Fri, May 18, 2007 at 09:50:03AM +0200, Andrea Righi wrote:
> Rik van Riel wrote:
> > Andrea Righi wrote:
> >> I'm looking for a way to keep track of the processes that fail to
> >> allocate new
> >> virtual memory. What do you think about the following approach
> >> (untested)?
> >
> > Looks like an easy way for users to spam syslogd over and
> > over and over again.
> >
> > At the very least, shouldn't this be dependant on print_fatal_signals?
> >
>
> Anyway, with print-fatal-signals enabled a user could spam syslogd too, simply
> with a (char *)0 = 0 program, but we could always identify the spam attempts
> logging the process uid...
>
> In any case, I agree, it should depend on that patch...
>
> What about adding a simple msleep_interruptible(SOME_MSECS) at the end of
> log_vm_enomem() or, at least, a might_sleep() to limit the potential spam/second
> rate?

An msleep will slow down this process, but do nothing about slowing
down the amount of logging. Simply fork a few more processes and all
you are doing with msleep is polluting the pid space.

What about a throttling similar to what ia64 does for floating point
assist faults (handle_fpu_swa()). There is a thread flag to not log
the events at all. It is rate throttled globally, but uses per cpu
variables for early exits. This algorithm scaled well to a thousand
cpus.

I think this may be a good fit.

Good Luck,
Robin

2007-05-18 11:47:53

by Andi Kleen

[permalink] [raw]
Subject: Re: signals logged / [RFC] log out-of-virtual-memory events


> I do not see such on i386, so why for x86_64?

So that you know that one of your programs crashed. That's a feature.

-Andi

2007-05-18 15:55:26

by Andrea Righi

[permalink] [raw]
Subject: Re: [RFC] log out-of-virtual-memory events

Robin Holt wrote:
> On Fri, May 18, 2007 at 09:50:03AM +0200, Andrea Righi wrote:
>> Rik van Riel wrote:
>>> Andrea Righi wrote:
>>>> I'm looking for a way to keep track of the processes that fail to
>>>> allocate new
>>>> virtual memory. What do you think about the following approach
>>>> (untested)?
>>> Looks like an easy way for users to spam syslogd over and
>>> over and over again.
>>>
>>> At the very least, shouldn't this be dependant on print_fatal_signals?
>>>
>> Anyway, with print-fatal-signals enabled a user could spam syslogd too, simply
>> with a (char *)0 = 0 program, but we could always identify the spam attempts
>> logging the process uid...
>>
>> In any case, I agree, it should depend on that patch...
>>
>> What about adding a simple msleep_interruptible(SOME_MSECS) at the end of
>> log_vm_enomem() or, at least, a might_sleep() to limit the potential spam/second
>> rate?
>
> An msleep will slow down this process, but do nothing about slowing
> down the amount of logging. Simply fork a few more processes and all
> you are doing with msleep is polluting the pid space.
>

Very true.

> What about a throttling similar to what ia64 does for floating point
> assist faults (handle_fpu_swa()). There is a thread flag to not log
> the events at all. It is rate throttled globally, but uses per cpu
> variables for early exits. This algorithm scaled well to a thousand
> cpus.

Actually using printk_ratelimit() should be enough... BTW print_fatal_signals()
should use it too.

-Andrea

---

Signed-off-by: Andrea Righi <[email protected]>

diff -urpN linux-2.6.21/mm/mmap.c linux-2.6.21-vm-log-enomem/mm/mmap.c
--- linux-2.6.21/mm/mmap.c 2007-04-26 05:08:32.000000000 +0200
+++ linux-2.6.21-vm-log-enomem/mm/mmap.c 2007-05-18 17:17:32.000000000 +0200
@@ -77,6 +77,29 @@ int sysctl_max_map_count __read_mostly =
atomic_t vm_committed_space = ATOMIC_INIT(0);

/*
+ * Print current process informations when it fails to allocate new virtual
+ * memory.
+ */
+static inline void log_vm_enomem(void)
+{
+ unsigned long total_vm = 0;
+ struct mm_struct *mm;
+
+ if (unlikely(!printk_ratelimit()))
+ return;
+
+ task_lock(current);
+ mm = current->mm;
+ if (mm)
+ total_vm = mm->total_vm;
+ task_unlock(current);
+
+ printk(KERN_INFO
+ "out of virtual memory for process %d (%s): total_vm=%lu, uid=%d\n",
+ current->pid, current->comm, total_vm, current->uid);
+}
+
+/*
* Check that a process has enough memory to allocate a new virtual
* mapping. 0 means there is enough memory for the allocation to
* succeed and -ENOMEM implies there is not.
@@ -175,6 +198,7 @@ int __vm_enough_memory(long pages, int c
return 0;
error:
vm_unacct_memory(pages);
+ log_vm_enomem();

return -ENOMEM;
}

2007-05-18 16:05:25

by Andrea Righi

[permalink] [raw]
Subject: Re: [RFC] log out-of-virtual-memory events

Andrea Righi wrote:
> Robin Holt wrote:
>> On Fri, May 18, 2007 at 09:50:03AM +0200, Andrea Righi wrote:
>>> Rik van Riel wrote:
>>>> Andrea Righi wrote:
>>>>> I'm looking for a way to keep track of the processes that fail to
>>>>> allocate new
>>>>> virtual memory. What do you think about the following approach
>>>>> (untested)?
>>>> Looks like an easy way for users to spam syslogd over and
>>>> over and over again.
>>>>
>>>> At the very least, shouldn't this be dependant on print_fatal_signals?
>>>>
>>> Anyway, with print-fatal-signals enabled a user could spam syslogd too, simply
>>> with a (char *)0 = 0 program, but we could always identify the spam attempts
>>> logging the process uid...
>>>
>>> In any case, I agree, it should depend on that patch...
>>>
>>> What about adding a simple msleep_interruptible(SOME_MSECS) at the end of
>>> log_vm_enomem() or, at least, a might_sleep() to limit the potential spam/second
>>> rate?
>> An msleep will slow down this process, but do nothing about slowing
>> down the amount of logging. Simply fork a few more processes and all
>> you are doing with msleep is polluting the pid space.
>>
>
> Very true.
>
>> What about a throttling similar to what ia64 does for floating point
>> assist faults (handle_fpu_swa()). There is a thread flag to not log
>> the events at all. It is rate throttled globally, but uses per cpu
>> variables for early exits. This algorithm scaled well to a thousand
>> cpus.
>
> Actually using printk_ratelimit() should be enough... BTW print_fatal_signals()
> should use it too.
>

I mean, something like this...

---

Limit the rate of the printk()s in print_fatal_signal() to avoid potential DoS
problems.

Signed-off-by: Andrea Righi <[email protected]>

diff -urpN linux-2.6.22-rc1-mm1/kernel/signal.c linux-2.6.22-rc1-mm1-limit-print_fatal_signals-rate/kernel/signal.c
--- linux-2.6.22-rc1-mm1/kernel/signal.c 2007-05-18 17:48:55.000000000 +0200
+++ linux-2.6.22-rc1-mm1-limit-print_fatal_signals-rate/kernel/signal.c 2007-05-18 17:58:13.000000000 +0200
@@ -790,6 +790,9 @@ static void print_vmas(void)

static void print_fatal_signal(struct pt_regs *regs, int signr)
{
+ if (unlikely(!printk_ratelimit()))
+ return;
+
printk("%s/%d: potentially unexpected fatal signal %d.\n",
current->comm, current->pid, signr);

2007-05-18 16:34:42

by Bernd Eckenfels

[permalink] [raw]
Subject: Re: [RFC] log out-of-virtual-memory events

In article <[email protected]> you wrote:
> printk("%s/%d: potentially unexpected fatal signal %d.\n",
> current->comm, current->pid, signr);

can we have both KERN_WARNING please?

Gruss
Bernd

2007-05-18 16:36:49

by Bernd Eckenfels

[permalink] [raw]
Subject: Re: [RFC] log out-of-virtual-memory events

In article <[email protected]> you wrote:
> + printk(KERN_INFO
> + "out of virtual memory for process %d (%s): total_vm=%lu, uid=%d\n",
> + current->pid, current->comm, total_vm, current->uid);

And align this one with the print_fatal layout:

printk(KERN_WARNING
"%s/%d process cannot request more virtual memory: total_vm=%lu, uid=%d\n",
current->comm, current->pid, total_vm, current->uid);

Greetings
Bernd

2007-05-19 07:49:43

by Jan Engelhardt

[permalink] [raw]
Subject: Re: signals logged / [RFC] log out-of-virtual-memory events


On May 18 2007 13:47, Andi Kleen wrote:
>
>> I do not see such on i386, so why for x86_64?
>
>So that you know that one of your programs crashed. That's a feature.

This feature could be handy for i386 too.


Jan
--

2007-05-19 09:35:53

by Andrea Righi

[permalink] [raw]
Subject: Re: signals logged / [RFC] log out-of-virtual-memory events

Jan Engelhardt wrote:
> On May 18 2007 13:47, Andi Kleen wrote:
>>> I do not see such on i386, so why for x86_64?
>> So that you know that one of your programs crashed. That's a feature.
>
> This feature could be handy for i386 too.
>

What about your /proc/sys/kernel/print-fatal-signals? it must be set to 1 to
enable that feature.

-Andrea

2007-05-19 10:09:36

by Jan Engelhardt

[permalink] [raw]
Subject: Re: signals logged / [RFC] log out-of-virtual-memory events


On May 19 2007 11:35, Andrea Righi wrote:
>Jan Engelhardt wrote:
>> On May 18 2007 13:47, Andi Kleen wrote:
>>>> I do not see such on i386, so why for x86_64?
>>> So that you know that one of your programs crashed. That's a feature.
>>
>> This feature could be handy for i386 too.
>>
>
>What about your /proc/sys/kernel/print-fatal-signals? it must be set to 1 to
>enable that feature.

That file does not exist on versions
2.6.18 <= version <= 2.6.20


Jan
--

2007-05-19 10:16:27

by Andrea Righi

[permalink] [raw]
Subject: Re: signals logged / [RFC] log out-of-virtual-memory events

Jan Engelhardt wrote:
> On May 19 2007 11:35, Andrea Righi wrote:
>> Jan Engelhardt wrote:
>>> On May 18 2007 13:47, Andi Kleen wrote:
>>>>> I do not see such on i386, so why for x86_64?
>>>> So that you know that one of your programs crashed. That's a feature.
>>> This feature could be handy for i386 too.
>>>
>> What about your /proc/sys/kernel/print-fatal-signals? it must be set to 1 to
>> enable that feature.
>
> That file does not exist on versions
> 2.6.18 <= version <= 2.6.20
>

This means that you must apply the print_fatal_signals patch...

-Andrea

2007-05-19 10:33:18

by Andrea Righi

[permalink] [raw]
Subject: [PATCH 1/2] limit print_fatal_signal() rate (was: [RFC] log out-of-virtual-memory events)

Bernd Eckenfels wrote:
> In article <[email protected]> you wrote:
>> printk("%s/%d: potentially unexpected fatal signal %d.\n",
>> current->comm, current->pid, signr);
>
> can we have both KERN_WARNING please?
>
> Gruss
> Bernd

Depends on print_fatal_signals patch.

---

Limit the rate of print_fatal_signal() to avoid potential denial-of-service
attacks.

Signed-off-by: Andrea Righi <[email protected]>

diff -urpN linux-2.6.22-rc1-mm1/kernel/signal.c linux-2.6.22-rc1-mm1-vm-log-enomem/kernel/signal.c
--- linux-2.6.22-rc1-mm1/kernel/signal.c 2007-05-19 11:25:24.000000000 +0200
+++ linux-2.6.22-rc1-mm1-vm-log-enomem/kernel/signal.c 2007-05-19 11:30:00.000000000 +0200
@@ -790,7 +790,10 @@ static void print_vmas(void)

static void print_fatal_signal(struct pt_regs *regs, int signr)
{
- printk("%s/%d: potentially unexpected fatal signal %d.\n",
+ if (unlikely(!printk_ratelimit()))
+ return;
+
+ printk(KERN_WARNING "%s/%d: potentially unexpected fatal signal %d.\n",
current->comm, current->pid, signr);

#ifdef __i386__

2007-05-19 10:34:19

by Andrea Righi

[permalink] [raw]
Subject: [PATCH 2/2] log out-of-virtual-memory events (was: [RFC] log out-of-virtual-memory events)

Bernd Eckenfels wrote:
> In article <[email protected]> you wrote:
>> + printk(KERN_INFO
>> + "out of virtual memory for process %d (%s): total_vm=%lu, uid=%d\n",
>> + current->pid, current->comm, total_vm, current->uid);
>
> And align this one with the print_fatal layout:
>
> printk(KERN_WARNING
> "%s/%d process cannot request more virtual memory: total_vm=%lu, uid=%d\n",
> current->comm, current->pid, total_vm, current->uid);
>

Depends on print_fatal_signals patch.

---

Print informations about userspace processes that fail to allocate new virtual
memory.

Signed-off-by: Andrea Righi <[email protected]>

diff -urpN linux-2.6.22-rc1-mm1/mm/mmap.c linux-2.6.22-rc1-mm1-vm-log-enomem/mm/mmap.c
--- linux-2.6.22-rc1-mm1/mm/mmap.c 2007-05-19 11:25:24.000000000 +0200
+++ linux-2.6.22-rc1-mm1-vm-log-enomem/mm/mmap.c 2007-05-19 11:55:05.000000000 +0200
@@ -77,6 +77,31 @@ int sysctl_overcommit_ratio = 50; /* def
int sysctl_max_map_count __read_mostly = DEFAULT_MAX_MAP_COUNT;
atomic_t vm_committed_space = ATOMIC_INIT(0);

+extern int print_fatal_signals;
+
+/*
+ * Print current process informations when it fails to allocate new virtual
+ * memory.
+ */
+static inline void log_vm_enomem(void)
+{
+ unsigned long total_vm = 0;
+ struct mm_struct *mm;
+
+ if (unlikely(!printk_ratelimit()))
+ return;
+
+ task_lock(current);
+ mm = current->mm;
+ if (mm)
+ total_vm = mm->total_vm;
+ task_unlock(current);
+
+ printk(KERN_WARNING
+ "%s/%d process cannot request more virtual memory: total_vm=%lu, uid=%d\n",
+ current->comm, current->pid, total_vm, current->uid);
+}
+
/*
* Check that a process has enough memory to allocate a new virtual
* mapping. 0 means there is enough memory for the allocation to
@@ -177,6 +202,9 @@ int __vm_enough_memory(long pages, int c
error:
vm_unacct_memory(pages);

+ if (print_fatal_signals)
+ log_vm_enomem();
+
return -ENOMEM;
}

2007-05-20 00:14:33

by folkert

[permalink] [raw]
Subject: Re: signals logged / [RFC] log out-of-virtual-memory events

> >> I do not see such on i386, so why for x86_64?
> >So that you know that one of your programs crashed. That's a feature.
> This feature could be handy for i386 too.

Since 2.6.18.2 I use this patch. With 2.6.21.1 it still applies altough
with a small offsets. Works like a charm.


Signed-off by: Folkert van Heusden <[email protected]>

--- linux-2.6.18.2/kernel/signal.c 2006-11-04 02:33:58.000000000 +0100
+++ linux-2.6.18.2.new/kernel/signal.c 2006-11-17 15:59:13.000000000 +0100
@@ -706,6 +706,15 @@
struct sigqueue * q = NULL;
int ret = 0;

+ if (sig == SIGQUIT || sig == SIGILL || sig == SIGTRAP ||
+ sig == SIGABRT || sig == SIGBUS || sig == SIGFPE ||
+ sig == SIGSEGV || sig == SIGXCPU || sig == SIGXFSZ ||
+ sig == SIGSYS || sig == SIGSTKFLT)
+ {
+ printk(KERN_WARNING "Sig %d send to %d owned by %d.%d (%s)\n",
+ sig, t -> pid, t -> uid, t -> gid, t -> comm);
+ }
+
/*
* fast-pathed signals for kernel-internal things like SIGSTOP
* or SIGKILL.



Folkert van Heusden

--
http://www.biglumber.com <- site where one can exchange PGP key signatures
----------------------------------------------------------------------
Phone: +31-6-41278122, PGP-key: 1F28D8AE, http://www.vanheusden.com

2007-05-20 00:15:58

by folkert

[permalink] [raw]
Subject: Re: [RFC] log out-of-virtual-memory events

> >> I'm looking for a way to keep track of the processes that fail to
> >> allocate new
> >> virtual memory. What do you think about the following approach
> >> (untested)?
> > Looks like an easy way for users to spam syslogd over and
> > over and over again.
> > At the very least, shouldn't this be dependant on print_fatal_signals?
>
> Anyway, with print-fatal-signals enabled a user could spam syslogd too, simply
> with a (char *)0 = 0 program, but we could always identify the spam attempts
> logging the process uid...

Yeah well it's all captured by syslogd/klogd and written to a file and
diskspace is cheap.


Folkert van Heusden

--
Feeling generous? -> http://www.vanheusden.com/wishlist.php
----------------------------------------------------------------------
Phone: +31-6-41278122, PGP-key: 1F28D8AE, http://www.vanheusden.com

2007-05-20 03:58:19

by Eric Dumazet

[permalink] [raw]
Subject: Re: signals logged / [RFC] log out-of-virtual-memory events

Folkert van Heusden a ?crit :
>>>> I do not see such on i386, so why for x86_64?
>>> So that you know that one of your programs crashed. That's a feature.
>> This feature could be handy for i386 too.
>
> Since 2.6.18.2 I use this patch. With 2.6.21.1 it still applies altough
> with a small offsets. Works like a charm.
>
>
> Signed-off by: Folkert van Heusden <[email protected]>
>
> --- linux-2.6.18.2/kernel/signal.c 2006-11-04 02:33:58.000000000 +0100
> +++ linux-2.6.18.2.new/kernel/signal.c 2006-11-17 15:59:13.000000000 +0100
> @@ -706,6 +706,15 @@
> struct sigqueue * q = NULL;
> int ret = 0;
>
> + if (sig == SIGQUIT || sig == SIGILL || sig == SIGTRAP ||
> + sig == SIGABRT || sig == SIGBUS || sig == SIGFPE ||
> + sig == SIGSEGV || sig == SIGXCPU || sig == SIGXFSZ ||
> + sig == SIGSYS || sig == SIGSTKFLT)
> + {
> + printk(KERN_WARNING "Sig %d send to %d owned by %d.%d (%s)\n",
> + sig, t -> pid, t -> uid, t -> gid, t -> comm);
> + }
> +

Please check line 219 of Documentation/CodingStyle, Section 3.1: Spaces

and no space around the '.' and "->" structure member operators.

Thank you

2007-05-20 11:21:22

by folkert

[permalink] [raw]
Subject: Re: signals logged / [RFC] log out-of-virtual-memory events

> >>>>I do not see such on i386, so why for x86_64?
> >>>So that you know that one of your programs crashed. That's a feature.
> >>This feature could be handy for i386 too.
> >Since 2.6.18.2 I use this patch. With 2.6.21.1 it still applies altough
> >with a small offsets. Works like a charm.
> >
> >Signed-off by: Folkert van Heusden <[email protected]>
> >--- linux-2.6.18.2/kernel/signal.c 2006-11-04 02:33:58.000000000 +0100
> >+++ linux-2.6.18.2.new/kernel/signal.c 2006-11-17 15:59:13.000000000 +0100
...
> >+ sig, t -> pid, t -> uid, t -> gid, t -> comm);
>
> Please check line 219 of Documentation/CodingStyle, Section 3.1: Spaces
> and no space around the '.' and "->" structure member operators.

New version without the spaces around '->' and a nice 'unlikely' added.

Signed-off by: Folkert van Heusden <[email protected]>

--- linux-2.6.18.2/kernel/signal.c 2006-11-04 02:33:58.000000000 +0100
+++ linux-2.6.18.2.new/kernel/signal.c 2006-11-17 15:59:13.000000000 +0100
@@ -706,6 +706,15 @@
struct sigqueue * q = NULL;
int ret = 0;

+ if (unlikely(sig == SIGQUIT || sig == SIGILL || sig == SIGTRAP ||
+ sig == SIGABRT || sig == SIGBUS || sig == SIGFPE ||
+ sig == SIGSEGV || sig == SIGXCPU || sig == SIGXFSZ ||
+ sig == SIGSYS || sig == SIGSTKFLT))
+ {
+ printk(KERN_WARNING "Sig %d send to %d owned by %d.%d (%s)\n",
+ sig, t->pid, t->uid, t->gid, t->comm);
+ }
+
/*
* fast-pathed signals for kernel-internal things like SIGSTOP
* or SIGKILL.


Folkert van Heusden

--
MultiTail cok yonlu kullanimli bir program, loglari okumak, verilen
kommandolari yerine getirebilen. Filter, renk verme, merge, 'diff-
view', vs. http://www.vanheusden.com/multitail/
----------------------------------------------------------------------
Phone: +31-6-41278122, PGP-key: 1F28D8AE, http://www.vanheusden.com

2007-05-20 16:10:26

by Stephen Hemminger

[permalink] [raw]
Subject: Re: signals logged / [RFC] log out-of-virtual-memory events

On Sun, 20 May 2007 13:21:11 +0200
Folkert van Heusden <[email protected]> wrote:

> > >>>>I do not see such on i386, so why for x86_64?
> > >>>So that you know that one of your programs crashed. That's a feature.
> > >>This feature could be handy for i386 too.
> > >Since 2.6.18.2 I use this patch. With 2.6.21.1 it still applies altough
> > >with a small offsets. Works like a charm.
> > >
> > >Signed-off by: Folkert van Heusden <[email protected]>
> > >--- linux-2.6.18.2/kernel/signal.c 2006-11-04 02:33:58.000000000 +0100
> > >+++ linux-2.6.18.2.new/kernel/signal.c 2006-11-17 15:59:13.000000000 +0100
> ...
> > >+ sig, t -> pid, t -> uid, t -> gid, t -> comm);
> >
> > Please check line 219 of Documentation/CodingStyle, Section 3.1: Spaces
> > and no space around the '.' and "->" structure member operators.
>
> New version without the spaces around '->' and a nice 'unlikely' added.
>
> Signed-off by: Folkert van Heusden <[email protected]>
>
> --- linux-2.6.18.2/kernel/signal.c 2006-11-04 02:33:58.000000000 +0100
> +++ linux-2.6.18.2.new/kernel/signal.c 2006-11-17 15:59:13.000000000 +0100
> @@ -706,6 +706,15 @@
> struct sigqueue * q = NULL;
> int ret = 0;
>
> + if (unlikely(sig == SIGQUIT || sig == SIGILL || sig == SIGTRAP ||
> + sig == SIGABRT || sig == SIGBUS || sig == SIGFPE ||
> + sig == SIGSEGV || sig == SIGXCPU || sig == SIGXFSZ ||
> + sig == SIGSYS || sig == SIGSTKFLT))
> + {
> + printk(KERN_WARNING "Sig %d send to %d owned by %d.%d (%s)\n",
> + sig, t->pid, t->uid, t->gid, t->comm);
> + }
> +
> /*
> * fast-pathed signals for kernel-internal things like SIGSTOP
> * or SIGKILL.
>
>
> Folkert van Heusden
>

Would turning that into a switch() generate better code.

--
Stephen Hemminger <[email protected]>

2007-05-20 16:12:15

by folkert

[permalink] [raw]
Subject: Re: signals logged / [RFC] log out-of-virtual-memory events

> > > >>>>I do not see such on i386, so why for x86_64?
> > > >>>So that you know that one of your programs crashed. That's a feature.
> > > >>This feature could be handy for i386 too.
> > > >Since 2.6.18.2 I use this patch. With 2.6.21.1 it still applies altough
> > > >with a small offsets. Works like a charm.
> > > >
> > > >Signed-off by: Folkert van Heusden <[email protected]>
> > > >--- linux-2.6.18.2/kernel/signal.c 2006-11-04 02:33:58.000000000 +0100
> > > >+++ linux-2.6.18.2.new/kernel/signal.c 2006-11-17 15:59:13.000000000 +0100
> > ...
> > > >+ sig, t -> pid, t -> uid, t -> gid, t -> comm);
> > >
> > > Please check line 219 of Documentation/CodingStyle, Section 3.1: Spaces
> > > and no space around the '.' and "->" structure member operators.
> > New version without the spaces around '->' and a nice 'unlikely' added.
> > Signed-off by: Folkert van Heusden <[email protected]>
> > --- linux-2.6.18.2/kernel/signal.c 2006-11-04 02:33:58.000000000 +0100
> > +++ linux-2.6.18.2.new/kernel/signal.c 2006-11-17 15:59:13.000000000 +0100
> > @@ -706,6 +706,15 @@
> > struct sigqueue * q = NULL;
> > int ret = 0;
> >
> > + if (unlikely(sig == SIGQUIT || sig == SIGILL || sig == SIGTRAP ||
> > + sig == SIGABRT || sig == SIGBUS || sig == SIGFPE ||
> > + sig == SIGSEGV || sig == SIGXCPU || sig == SIGXFSZ ||
> > + sig == SIGSYS || sig == SIGSTKFLT))
> > + {
> > + printk(KERN_WARNING "Sig %d send to %d owned by %d.%d (%s)\n",
> > + sig, t->pid, t->uid, t->gid, t->comm);
> > + }
> > +
> > /*
> > * fast-pathed signals for kernel-internal things like SIGSTOP
> > * or SIGKILL.
>
> Would turning that into a switch() generate better code.

Doubt it: in the worst case you still nee to check for each possibility.
Furthermore a.f.a.i.k. with switch you cannot do 'unlinkely()'.


Folkert van Heusden

--
MultiTail er et flexible tool for ? kontrolere Logfiles og commandoer.
Med filtrer, farger, sammenf?ringer, forskeliger ansikter etc.
http://www.vanheusden.com/multitail/
----------------------------------------------------------------------
Phone: +31-6-41278122, PGP-key: 1F28D8AE, http://www.vanheusden.com

2007-05-20 20:47:33

by Jan Engelhardt

[permalink] [raw]
Subject: Re: signals logged / [RFC] log out-of-virtual-memory events


On May 20 2007 18:12, Folkert van Heusden wrote:
>> >
>> > + if (unlikely(sig == SIGQUIT || sig == SIGILL || sig == SIGTRAP ||
>> > + sig == SIGABRT || sig == SIGBUS || sig == SIGFPE ||
>> > + sig == SIGSEGV || sig == SIGXCPU || sig == SIGXFSZ ||
>> > + sig == SIGSYS || sig == SIGSTKFLT))
>> > + {
>> > + printk(KERN_WARNING "Sig %d send to %d owned by %d.%d (%s)\n",
>> > + sig, t->pid, t->uid, t->gid, t->comm);
>> > + }
>> > +
>> > /*
>> > * fast-pathed signals for kernel-internal things like SIGSTOP
>> > * or SIGKILL.
>>
>> Would turning that into a switch() generate better code.

Yes, this time.

>Doubt it: in the worst case you still nee to check for each possibility.
>Furthermore a.f.a.i.k. with switch you cannot do 'unlinkely()'.

With if(), it generates a ton of "CMP, JE" instructions.
With switch(), I would assume gcc transforms it into using
a jump table (aka "JMP [table+sig]")

I tried it: with switch(), gcc transforms this into a
bitmap comparison ("MOV eax, 1; SHL eax, sig; TEST eax, 0x830109f8"),
which seems even cheaper than a jump table.


Jan
--

2007-05-20 20:55:16

by folkert

[permalink] [raw]
Subject: Re: signals logged / [RFC] log out-of-virtual-memory events

> >> >
> >> > + if (unlikely(sig == SIGQUIT || sig == SIGILL || sig == SIGTRAP ||
> >> > + sig == SIGABRT || sig == SIGBUS || sig == SIGFPE ||
> >> > + sig == SIGSEGV || sig == SIGXCPU || sig == SIGXFSZ ||
> >> > + sig == SIGSYS || sig == SIGSTKFLT))
> >> > + {
> >> > + printk(KERN_WARNING "Sig %d send to %d owned by %d.%d (%s)\n",
> >> > + sig, t->pid, t->uid, t->gid, t->comm);
> >> > + }
> >> > +
> >> > /*
> >> > * fast-pathed signals for kernel-internal things like SIGSTOP
> >> > * or SIGKILL.
> >> Would turning that into a switch() generate better code.
> Yes, this time.
>
> >Doubt it: in the worst case you still nee to check for each possibility.
> >Furthermore a.f.a.i.k. with switch you cannot do 'unlinkely()'.
>
> With if(), it generates a ton of "CMP, JE" instructions.
> With switch(), I would assume gcc transforms it into using
> a jump table (aka "JMP [table+sig]")
> I tried it: with switch(), gcc transforms this into a
> bitmap comparison ("MOV eax, 1; SHL eax, sig; TEST eax, 0x830109f8"),
> which seems even cheaper than a jump table.

Ok, here's the new patch against 2.6.21.1:

Signed-off by Folkert van Heusden <[email protected]>

--- kernel/signal.c.org 2007-05-20 22:47:13.000000000 +0200
+++ kernel/signal.c 2007-05-20 22:54:17.000000000 +0200
@@ -739,6 +739,25 @@
struct sigqueue * q = NULL;
int ret = 0;

+ /* emit some logging for nasty signals
+ * especially SIGSEGV and friends aught to be looked at when happening
+ */
+ switch(sig) {
+ case SIGQUIT:
+ case SIGILL:
+ case SIGTRAP:
+ case SIGABRT:
+ case SIGBUS:
+ case SIGFPE:
+ case SIGSEGV:
+ case SIGXCPU:
+ case SIGXFSZ:
+ case SIGSYS:
+ case SIGSTKFLT:
+ printk(KERN_WARNING "Sig %d send to %d owned by %d.%d (%s)\n",
+ sig, t -> pid, t -> uid, t -> gid, t -> comm);
+ }
+
/*
* fast-pathed signals for kernel-internal things like SIGSTOP
* or SIGKILL.

Folkert van Heusden

--
MultiTail ? uno flexible tool per seguire di logfiles e effettuazione
di commissioni. Feltrare, provedere da colore, merge, 'diff-view',
etc. http://www.vanheusden.com/multitail/
----------------------------------------------------------------------
Phone: +31-6-41278122, PGP-key: 1F28D8AE, http://www.vanheusden.com

2007-05-20 21:15:25

by Andi Kleen

[permalink] [raw]
Subject: Re: signals logged / [RFC] log out-of-virtual-memory events


> + switch(sig) {
> + case SIGQUIT:
> + case SIGILL:
> + case SIGTRAP:
> + case SIGABRT:
> + case SIGBUS:
> + case SIGFPE:
> + case SIGSEGV:
> + case SIGXCPU:
> + case SIGXFSZ:
> + case SIGSYS:
> + case SIGSTKFLT:

Unconditional? That's definitely a very bad idea. If anything only unhandled
signals should be printed this way because some programs use them internally.
But I think your list is far too long anyways.

-Andi

2007-05-20 21:20:45

by folkert

[permalink] [raw]
Subject: Re: signals logged / [RFC] log out-of-virtual-memory events

> > + switch(sig) {
> > + case SIGQUIT:
> > + case SIGILL:
> > + case SIGTRAP:
> > + case SIGABRT:
> > + case SIGBUS:
> > + case SIGFPE:
> > + case SIGSEGV:
> > + case SIGXCPU:
> > + case SIGXFSZ:
> > + case SIGSYS:
> > + case SIGSTKFLT:
>
> Unconditional? That's definitely a very bad idea. If anything only unhandled
> signals should be printed this way because some programs use them internally.

Use these signals internally? Afaik these are fatal, stopping the
process. So using them internally would be a little tricky.

> But I think your list is far too long anyways.

So, which ones would you like to have removed then?


Folkert van Heusden

--
To MultiTail einai ena polymorfiko ergaleio gia ta logfiles kai tin
eksodo twn entolwn. Prosferei: filtrarisma, xrwmatismo, sygxwneysi,
diaforetikes provoles. http://www.vanheusden.com/multitail/
----------------------------------------------------------------------
Phone: +31-6-41278122, PGP-key: 1F28D8AE, http://www.vanheusden.com

2007-05-20 21:23:59

by folkert

[permalink] [raw]
Subject: Re: signals logged / [RFC] log out-of-virtual-memory events

> > > + switch(sig) {
> > > + case SIGQUIT:
> > > + case SIGILL:
> > > + case SIGTRAP:
> > > + case SIGABRT:
> > > + case SIGBUS:
> > > + case SIGFPE:
> > > + case SIGSEGV:
> > > + case SIGXCPU:
> > > + case SIGXFSZ:
> > > + case SIGSYS:
> > > + case SIGSTKFLT:
> > Unconditional? That's definitely a very bad idea. If anything only unhandled
> > signals should be printed this way because some programs use them internally.
> Use these signals internally? Afaik these are fatal, stopping the
> process. So using them internally would be a little tricky.
> > But I think your list is far too long anyways.
>
> So, which ones would you like to have removed then?

(and why, of course, to enlighten me: some are educated guesses)


Folkert van Heusden

--
MultiTail ist eine flexible Applikation um Logfiles und Kommando
Eingaben zu ?berpr?fen. Inkl. Filter, Farben, Zusammenf?hren,
Ansichten etc. http://www.vanheusden.com/multitail/
----------------------------------------------------------------------
Phone: +31-6-41278122, PGP-key: 1F28D8AE, http://www.vanheusden.com

2007-05-20 21:27:38

by Andi Kleen

[permalink] [raw]
Subject: Re: signals logged / [RFC] log out-of-virtual-memory events

On Sun, May 20, 2007 at 11:20:36PM +0200, Folkert van Heusden wrote:
> > > + switch(sig) {
> > > + case SIGQUIT:
> > > + case SIGILL:
> > > + case SIGTRAP:
> > > + case SIGABRT:
> > > + case SIGBUS:
> > > + case SIGFPE:
> > > + case SIGSEGV:
> > > + case SIGXCPU:
> > > + case SIGXFSZ:
> > > + case SIGSYS:
> > > + case SIGSTKFLT:
> >
> > Unconditional? That's definitely a very bad idea. If anything only unhandled
> > signals should be printed this way because some programs use them internally.
>
> Use these signals internally? Afaik these are fatal, stopping the
> process. So using them internally would be a little tricky.

All of them are catchable.

>
> > But I think your list is far too long anyways.
>
> So, which ones would you like to have removed then?

SIGFPE at least and the accounting signals are dubious too. SIGQUIT can
be also relatively common.

-Andi

2007-05-20 22:32:24

by Jeff Dike

[permalink] [raw]
Subject: Re: signals logged / [RFC] log out-of-virtual-memory events

On Mon, May 21, 2007 at 12:24:22AM +0200, Andi Kleen wrote:
> > > But I think your list is far too long anyways.
> >
> > So, which ones would you like to have removed then?
>
> SIGFPE at least and the accounting signals are dubious too. SIGQUIT can
> be also relatively common.

And SIGSEGV and SIGBUS - UML catches these internally and handles them.

Jeff

2007-05-21 03:36:12

by Andrew Morton

[permalink] [raw]
Subject: Re: [PATCH 1/2] limit print_fatal_signal() rate (was: [RFC] log out-of-virtual-memory events)

On Sat, 19 May 2007 12:33:04 +0200 (MEST) Andrea Righi <[email protected]> wrote:

> Bernd Eckenfels wrote:
> > In article <[email protected]> you wrote:
> >> printk("%s/%d: potentially unexpected fatal signal %d.\n",
> >> current->comm, current->pid, signr);
> >
> > can we have both KERN_WARNING please?
> >
> > Gruss
> > Bernd
>
> Depends on print_fatal_signals patch.
>
> ---
>
> Limit the rate of print_fatal_signal() to avoid potential denial-of-service
> attacks.
>
> Signed-off-by: Andrea Righi <[email protected]>
>
> diff -urpN linux-2.6.22-rc1-mm1/kernel/signal.c linux-2.6.22-rc1-mm1-vm-log-enomem/kernel/signal.c
> --- linux-2.6.22-rc1-mm1/kernel/signal.c 2007-05-19 11:25:24.000000000 +0200
> +++ linux-2.6.22-rc1-mm1-vm-log-enomem/kernel/signal.c 2007-05-19 11:30:00.000000000 +0200
> @@ -790,7 +790,10 @@ static void print_vmas(void)
>
> static void print_fatal_signal(struct pt_regs *regs, int signr)
> {
> - printk("%s/%d: potentially unexpected fatal signal %d.\n",
> + if (unlikely(!printk_ratelimit()))
> + return;
> +
> + printk(KERN_WARNING "%s/%d: potentially unexpected fatal signal %d.\n",
> current->comm, current->pid, signr);
>
> #ifdef __i386__

Well OK. But vdso-print-fatal-signals.patch is designated not-for-mainline
anyway.

I think the DoS which you identify has been available for a very long time
on ia64, x86_64 and perhaps others.


2007-05-21 03:36:27

by Andrew Morton

[permalink] [raw]
Subject: Re: [PATCH 2/2] log out-of-virtual-memory events (was: [RFC] log out-of-virtual-memory events)

On Sat, 19 May 2007 12:34:01 +0200 (MEST) Andrea Righi <[email protected]> wrote:

> Print informations about userspace processes that fail to allocate new virtual
> memory.

Why is this useful?

2007-05-21 10:44:32

by Andrea Righi

[permalink] [raw]
Subject: Re: [PATCH 1/2] limit print_fatal_signal() rate

Andrew Morton wrote:
> Well OK. But vdso-print-fatal-signals.patch is designated not-for-mainline
> anyway.
>
> I think the DoS which you identify has been available for a very long time
> on ia64, x86_64 and perhaps others.
>

For the mainline a fix could be the following...

---

Limit the rate of the kernel logging for the segfaults of user applications, to
avoid potential message floods or denial-of-service attacks.

Signed-off-by: Andrea Righi <[email protected]>

diff -urpN linux-2.6.22-rc2/arch/avr32/mm/fault.c linux-2.6.22-rc2-limit-segfaults-printk-rate/arch/avr32/mm/fault.c
--- linux-2.6.22-rc2/arch/avr32/mm/fault.c 2007-05-19 13:11:30.000000000 +0200
+++ linux-2.6.22-rc2-limit-segfaults-printk-rate/arch/avr32/mm/fault.c 2007-05-21 11:48:37.000000000 +0200
@@ -158,7 +158,7 @@ bad_area:
up_read(&mm->mmap_sem);

if (user_mode(regs)) {
- if (exception_trace)
+ if (exception_trace && printk_ratelimit())
printk("%s%s[%d]: segfault at %08lx pc %08lx "
"sp %08lx ecr %lu\n",
is_init(tsk) ? KERN_EMERG : KERN_INFO,
diff -urpN linux-2.6.22-rc2/arch/x86_64/mm/fault.c linux-2.6.22-rc2-limit-segfaults-printk-rate/arch/x86_64/mm/fault.c
--- linux-2.6.22-rc2/arch/x86_64/mm/fault.c 2007-05-21 11:42:07.000000000 +0200
+++ linux-2.6.22-rc2-limit-segfaults-printk-rate/arch/x86_64/mm/fault.c 2007-05-21 11:45:55.000000000 +0200
@@ -489,7 +489,8 @@ bad_area_nosemaphore:
(address >> 32))
return;

- if (exception_trace && unhandled_signal(tsk, SIGSEGV)) {
+ if (exception_trace && unhandled_signal(tsk, SIGSEGV) &&
+ printk_ratelimit()) {
printk(
"%s%s[%d]: segfault at %016lx rip %016lx rsp %016lx error %lx\n",
tsk->pid > 1 ? KERN_INFO : KERN_EMERG,

2007-05-21 10:45:21

by Andrea Righi

[permalink] [raw]
Subject: Re: signals logged / [RFC] log out-of-virtual-memory events

Andi Kleen wrote:
>> + switch(sig) {
>> + case SIGQUIT:
>> + case SIGILL:
>> + case SIGTRAP:
>> + case SIGABRT:
>> + case SIGBUS:
>> + case SIGFPE:
>> + case SIGSEGV:
>> + case SIGXCPU:
>> + case SIGXFSZ:
>> + case SIGSYS:
>> + case SIGSTKFLT:
>
> Unconditional? That's definitely a very bad idea. If anything only unhandled
> signals should be printed this way because some programs use them internally.
> But I think your list is far too long anyways.
>
> -Andi
>

Maybe you could use somthing similar to unhandled_signal() in
arch/x86_64/mm/fault.c, but I agree that the list seems a bit too long...

-Andrea

2007-05-21 10:48:40

by Andrea Righi

[permalink] [raw]
Subject: Re: [PATCH 2/2] log out-of-virtual-memory events

Andrew Morton wrote:
> On Sat, 19 May 2007 12:34:01 +0200 (MEST) Andrea Righi <[email protected]> wrote:
>
>> Print informations about userspace processes that fail to allocate new virtual
>> memory.
>
> Why is this useful?
>

Well... in strict overcommit mode (overcommit_memory=2) this is the only way to
track down problems of the (bad-designed) user applications that exit when they
receive a -ENOMEM without logging anything... and, anyway, it could be an
additional aid in figuring out what is going wrong on inside a system. BTW, I
don't think it should be enabled by default, so this is the reason why it should
depend on print_fatal_signals patch.

-Andrea

2007-05-21 11:04:18

by folkert

[permalink] [raw]
Subject: Re: signals logged / [RFC] log out-of-virtual-memory events

> >> + switch(sig) {
> >> + case SIGQUIT:
...
> >> + case SIGSTKFLT:
> >
> > Unconditional? That's definitely a very bad idea. If anything only unhandled
> > signals should be printed this way because some programs use them internally.
> > But I think your list is far too long anyways.
>
> Maybe you could use somthing similar to unhandled_signal() in
> arch/x86_64/mm/fault.c, but I agree that the list seems a bit too long...

What about the following enhancement: I check with sig_fatal if it would
kill the process and only then emit a message. So when an application
takes care itself of handling it nothing is printed.

Signed-off by: Folkert van Heusden <[email protected]>

--- kernel/signal.c.org 2007-05-20 22:47:13.000000000 +0200
+++ kernel/signal.c 2007-05-21 12:59:52.000000000 +0200
@@ -739,6 +739,8 @@
struct sigqueue * q = NULL;
int ret = 0;

+ /* emit some logging for unhandled signals
+ */
+ if (sig_fatal(t, sig))
+ {
+ printk(KERN_WARNING "Sig %d send to %d owned by %d.%d (%s)\n",
+ sig, t -> pid, t -> uid, t -> gid, t -> comm);
+ }
+
/*
* fast-pathed signals for kernel-internal things like SIGSTOP
* or SIGKILL.

of course, this can also be limited to only the interesting signals:

Signed-off by: Folkert van Heusden <[email protected]>

--- kernel/signal.c.org 2007-05-20 22:47:13.000000000 +0200
+++ kernel/signal.c 2007-05-21 12:59:52.000000000 +0200
@@ -739,6 +739,28 @@
struct sigqueue * q = NULL;
int ret = 0;

+ /* emit some logging for nasty signals
+ * especially SIGSEGV and friends aught to be looked at when happening
+ */
+ switch(sig) {
+ case SIGQUIT:
+ case SIGILL:
+ case SIGTRAP:
+ case SIGABRT:
+ case SIGBUS:
+ case SIGFPE:
+ case SIGSEGV:
+ case SIGXCPU:
+ case SIGXFSZ:
+ case SIGSYS:
+ case SIGSTKFLT:
+ if (sig_fatal(t, sig))
+ {
+ printk(KERN_WARNING "Sig %d send to %d owned by %d.%d (%s)\n",
+ sig, t -> pid, t -> uid, t -> gid, t -> comm);
+ }
+ }
+
/*
* fast-pathed signals for kernel-internal things like SIGSTOP
* or SIGKILL.


Folkert van Heusden

--

Multitail - gibkaja utilita po sledovaniju log-fajlov i vyvoda
kommand. Fil'trovanie, raskra?ivanie, slijanie, vizual'noe sravnenie,
i t.d. http://www.vanheusden.com/multitail/
----------------------------------------------------------------------
Phone: +31-6-41278122, PGP-key: 1F28D8AE, http://www.vanheusden.com

2007-05-21 11:46:33

by Gábor Lénárt

[permalink] [raw]
Subject: Re: signals logged / [RFC] log out-of-virtual-memory events

On Sun, May 20, 2007 at 11:20:36PM +0200, Folkert van Heusden wrote:
> > > + switch(sig) {
> > > + case SIGQUIT:
> > > + case SIGILL:
> > > + case SIGTRAP:
> > > + case SIGABRT:
> > > + case SIGBUS:
> > > + case SIGFPE:
> > > + case SIGSEGV:
> > > + case SIGXCPU:
> > > + case SIGXFSZ:
> > > + case SIGSYS:
> > > + case SIGSTKFLT:
> >
> > Unconditional? That's definitely a very bad idea. If anything only unhandled
> > signals should be printed this way because some programs use them internally.
>
> Use these signals internally? Afaik these are fatal, stopping the
> process. So using them internally would be a little tricky.

For example mplayer uses SIGSEGV: it does not check ALL of the possible
error situations, in case of SIGSEGV (a bad stream etc) it quickly
reinit itself from the last position. It's much better in performance (well
sure for a stream without error, or only few of them) than to check all of
the possible problems. Nowdays it may not a problem to check everything but
it WAS when MPlayer born (much slower CPUs). But some signals are also
useful for emulation etc (eg in my own DOS emulator some years ago ....).

--
- G?bor

2007-05-21 12:39:23

by Jan Engelhardt

[permalink] [raw]
Subject: Re: signals logged / [RFC] log out-of-virtual-memory events


On May 21 2007 13:04, Folkert van Heusden wrote:
>
>What about the following enhancement: I check with sig_fatal if it would
>kill the process and only then emit a message. So when an application
>takes care itself of handling it nothing is printed.

>+ /* emit some logging for unhandled signals
>+ */
>+ if (sig_fatal(t, sig))

Not unhandled_signal()?

>+ {

if (sig_fatal(t, sig)) {

>+ printk(KERN_WARNING "Sig %d send to %d owned by %d.%d (%s)\n",

s/send/sent/;

>+ sig, t -> pid, t -> uid, t -> gid, t -> comm);

t->pid, t->uid, t->gid, t->comm);

>+ }
>+
> /*
> * fast-pathed signals for kernel-internal things like SIGSTOP
> * or SIGKILL.
>
>of course, this can also be limited to only the interesting signals:
>
>Signed-off by: Folkert van Heusden <[email protected]>
>
>--- kernel/signal.c.org 2007-05-20 22:47:13.000000000 +0200
>+++ kernel/signal.c 2007-05-21 12:59:52.000000000 +0200
>@@ -739,6 +739,28 @@
> struct sigqueue * q = NULL;
> int ret = 0;
>
>+ /* emit some logging for nasty signals
>+ * especially SIGSEGV and friends aught to be looked at when happening
>+ */
>+ switch(sig) {
>+ case SIGQUIT:
>+ case SIGILL:
>+ case SIGTRAP:
>+ case SIGABRT:
>+ case SIGBUS:
>+ case SIGFPE:
>+ case SIGSEGV:
>+ case SIGXCPU:
>+ case SIGXFSZ:
>+ case SIGSYS:
>+ case SIGSTKFLT:
>+ if (sig_fatal(t, sig))
>+ {
>+ printk(KERN_WARNING "Sig %d send to %d owned by %d.%d (%s)\n",
>+ sig, t -> pid, t -> uid, t -> gid, t -> comm);
>+ }
>+ }
>+
> /*
> * fast-pathed signals for kernel-internal things like SIGSTOP
> * or SIGKILL.
>



Jan
--

2007-05-21 12:47:46

by folkert

[permalink] [raw]
Subject: Re: signals logged / [RFC] log out-of-virtual-memory events

> >What about the following enhancement: I check with sig_fatal if it would
> >kill the process and only then emit a message. So when an application
> >takes care itself of handling it nothing is printed.
> >+ /* emit some logging for unhandled signals
> >+ */
> >+ if (sig_fatal(t, sig))
> Not unhandled_signal()?

Can we already use that one in send_signal? As the signal needs to be
send first I think before we know if it was handled or not? sig_fatal
checks if the handler is set to default - which is it is not taken care
of.

> >+ {
> if (sig_fatal(t, sig)) {
> >+ printk(KERN_WARNING "Sig %d send to %d owned by %d.%d (%s)\n",
> s/send/sent/;
> >+ sig, t -> pid, t -> uid, t -> gid, t -> comm);
> t->pid, t->uid, t->gid, t->comm);


Description:
This patch adds code to the signal-sender making it log a message when
an unhandled fatal signal will be delivered.

Signed-off by: Folkert van Heusden <[email protected]>

--- kernel/signal.c.org 2007-05-20 22:47:13.000000000 +0200
+++ kernel/signal.c 2007-05-21 14:46:05.000000000 +0200
@@ -739,6 +739,12 @@
struct sigqueue * q = NULL;
int ret = 0;

+ /* unhandled fatal signals are logged */
+ if (sig_fatal(t, sig)) {
+ printk(KERN_WARNING "Sig %d sent to %d owned by %d.%d (%s)\n",
+ sig, t->pid, t->uid, t->gid, t->comm);
+ }
+
/*
* fast-pathed signals for kernel-internal things like SIGSTOP
* or SIGKILL.


Folkert van Heusden

--
MultiTail ? uno flexible tool per seguire di logfiles e effettuazione
di commissioni. Feltrare, provedere da colore, merge, 'diff-view',
etc. http://www.vanheusden.com/multitail/
----------------------------------------------------------------------
Phone: +31-6-41278122, PGP-key: 1F28D8AE, http://www.vanheusden.com

2007-05-21 13:58:34

by Andrea Righi

[permalink] [raw]
Subject: Re: signals logged / [RFC] log out-of-virtual-memory events

Folkert van Heusden wrote:
>>> What about the following enhancement: I check with sig_fatal if it would
>>> kill the process and only then emit a message. So when an application
>>> takes care itself of handling it nothing is printed.
>>> + /* emit some logging for unhandled signals
>>> + */
>>> + if (sig_fatal(t, sig))
>> Not unhandled_signal()?
>
> Can we already use that one in send_signal? As the signal needs to be
> send first I think before we know if it was handled or not? sig_fatal
> checks if the handler is set to default - which is it is not taken care
> of.

What about ptrace()'d processes? I don't think we should log signals for them...

-Andrea

2007-05-21 18:59:57

by folkert

[permalink] [raw]
Subject: Re: signals logged / [RFC] log out-of-virtual-memory events

> >>> What about the following enhancement: I check with sig_fatal if it would
> >>> kill the process and only then emit a message. So when an application
> >>> takes care itself of handling it nothing is printed.
> >>> + /* emit some logging for unhandled signals
> >>> + */
> >>> + if (sig_fatal(t, sig))
> >> Not unhandled_signal()?
> > Can we already use that one in send_signal? As the signal needs to be
> > send first I think before we know if it was handled or not? sig_fatal
> > checks if the handler is set to default - which is it is not taken care
> > of.
> What about ptrace()'d processes? I don't think we should log signals for them...

Why not?


Folkert van Heusden

--
MultiTail es una herramienta flexibele para consiguir archivos de log,
y para ejecutar ordenes. Filtrar, a?adir colores, merger y vista de
las differencias. http://www.vanheusden.com/multitail/
----------------------------------------------------------------------
Phone: +31-6-41278122, PGP-key: 1F28D8AE, http://www.vanheusden.com

2007-05-21 22:16:15

by Andrea Righi

[permalink] [raw]
Subject: Re: signals logged / [RFC] log out-of-virtual-memory events

Folkert van Heusden wrote:
>>>>> What about the following enhancement: I check with sig_fatal if it would
>>>>> kill the process and only then emit a message. So when an application
>>>>> takes care itself of handling it nothing is printed.
>>>>> + /* emit some logging for unhandled signals
>>>>> + */
>>>>> + if (sig_fatal(t, sig))
>>>> Not unhandled_signal()?
>>> Can we already use that one in send_signal? As the signal needs to be
>>> send first I think before we know if it was handled or not? sig_fatal
>>> checks if the handler is set to default - which is it is not taken care
>>> of.
>> What about ptrace()'d processes? I don't think we should log signals for them...
>
> Why not?

Maybe sometimes it's useful, maybe not, but I suppose that usually only the
controlling process should care about the critical signals received by the
controlled process. I simply don't think it should be a system issue. For
example I wouldn't like to have a lot of messages in the kernel logs just
because I'm debugging some segfaulting programs with gdb.

-Andrea

2007-05-23 18:01:15

by Satyam Sharma

[permalink] [raw]
Subject: Re: signals logged / [RFC] log out-of-virtual-memory events

On 5/21/07, Folkert van Heusden <[email protected]> wrote:
> > >What about the following enhancement: I check with sig_fatal if it would
> > >kill the process and only then emit a message. So when an application
> > >takes care itself of handling it nothing is printed.
> > >+ /* emit some logging for unhandled signals
> > >+ */
> > >+ if (sig_fatal(t, sig))
> > Not unhandled_signal()?
>
> Can we already use that one in send_signal? As the signal needs to be
> send first I think before we know if it was handled or not? sig_fatal
> checks if the handler is set to default - which is it is not taken care
> of.
>
> > >+ {
> > if (sig_fatal(t, sig)) {
> > >+ printk(KERN_WARNING "Sig %d send to %d owned by %d.%d (%s)\n",
> > s/send/sent/;
> > >+ sig, t -> pid, t -> uid, t -> gid, t -> comm);
> > t->pid, t->uid, t->gid, t->comm);
>
>
> Description:
> This patch adds code to the signal-sender making it log a message when
> an unhandled fatal signal will be delivered.

Gargh ... why does this want to be in the *kernel*'s logs? In any case, can
you please make this KERN_INFO (or lower) instead of KERN_WARNING.

2007-05-23 18:45:46

by folkert

[permalink] [raw]
Subject: Re: signals logged / [RFC] log out-of-virtual-memory events

> >> >+ {
> >> if (sig_fatal(t, sig)) {
> >> >+ printk(KERN_WARNING "Sig %d send to %d owned by %d.%d
> >(%s)\n",
> >> s/send/sent/;
> >> >+ sig, t -> pid, t -> uid, t -> gid, t -> comm);
> >> t->pid, t->uid, t->gid, t->comm);
> >
>
> Gargh ... why does this want to be in the *kernel*'s logs? In any case, can
> you please make this KERN_INFO (or lower) instead of KERN_WARNING.

Description:
This patch adds code to the signal-sender making it log a message when
an unhandled fatal signal will be delivered.

Signed-of by: Folkert van Heusden <[email protected]

--- kernel/signal.c.org 2007-05-20 22:47:13.000000000 +0200
+++ kernel/signal.c 2007-05-21 14:46:05.000000000 +0200
@@ -739,6 +739,12 @@
struct sigqueue * q = NULL;
int ret = 0;

+ /* unhandled fatal signals are logged */
+ if (sig_fatal(t, sig)) {
+ printk(KERN_INFO "Sig %d sent to %d owned by %d.%d (%s)\n",
+ sig, t->pid, t->uid, t->gid, t->comm);
+ }
+
/*
* fast-pathed signals for kernel-internal things like SIGSTOP
* or SIGKILL.


Folkert van Heusden

--
Temperature outside: 21.437500, temperature livingroom:
----------------------------------------------------------------------
Phone: +31-6-41278122, PGP-key: 1F28D8AE, http://www.vanheusden.com

2007-05-24 07:58:54

by Ingo Molnar

[permalink] [raw]
Subject: Re: [PATCH 1/2] limit print_fatal_signal() rate (was: [RFC] log out-of-virtual-memory events)


* Andrew Morton <[email protected]> wrote:

> Well OK. But vdso-print-fatal-signals.patch is designated
> not-for-mainline anyway.

btw., why? It's very, very useful to distro, early-boot-userspace and
glibc development. The only add-on change should be to not print SIGKILL
events. Otherwise it's very much a keeper. Hm?

Ingo

2007-05-24 08:17:12

by Andrew Morton

[permalink] [raw]
Subject: Re: [PATCH 1/2] limit print_fatal_signal() rate (was: [RFC] log out-of-virtual-memory events)

On Thu, 24 May 2007 09:58:35 +0200 Ingo Molnar <[email protected]> wrote:

>
> * Andrew Morton <[email protected]> wrote:
>
> > Well OK. But vdso-print-fatal-signals.patch is designated
> > not-for-mainline anyway.
>
> btw., why?

err, because that's what I decided a year ago. I wonder why ;)

Perhaps because of the DoS thing, but it has a /proc knob and defaults to
off, so it should be OK.

> It's very, very useful to distro, early-boot-userspace and
> glibc development. The only add-on change should be to not print SIGKILL
> events. Otherwise it's very much a keeper. Hm?
>

<promotes it>

2007-05-24 08:51:18

by Andrea Righi

[permalink] [raw]
Subject: Re: [PATCH 1/2] limit print_fatal_signal() rate

Ingo Molnar wrote:
> * Andrew Morton <[email protected]> wrote:
>
>> Well OK. But vdso-print-fatal-signals.patch is designated
>> not-for-mainline anyway.
>
> btw., why? It's very, very useful to distro, early-boot-userspace and
> glibc development. The only add-on change should be to not print SIGKILL
> events. Otherwise it's very much a keeper. Hm?
>

Actually it seems that SIGKILLs are not printed. In get_signal_to_deliver() we have:

[snip]
@@ -1843,6 +1879,8 @@ relock:
* Anything else is fatal, maybe with a core dump.
*/
current->flags |= PF_SIGNALED;
+ if ((signr != SIGKILL) && print_fatal_signals)
+ print_fatal_signal(regs, signr);
if (sig_kernel_coredump(signr)) {
/*
* If it was able to dump core, this kills all
[snip]

-Andrea

2007-05-24 09:55:26

by Ingo Molnar

[permalink] [raw]
Subject: Re: [PATCH 1/2] limit print_fatal_signal() rate (was: [RFC] log out-of-virtual-memory events)


* Andrew Morton <[email protected]> wrote:

> On Thu, 24 May 2007 09:58:35 +0200 Ingo Molnar <[email protected]> wrote:
>
> >
> > * Andrew Morton <[email protected]> wrote:
> >
> > > Well OK. But vdso-print-fatal-signals.patch is designated
> > > not-for-mainline anyway.
> >
> > btw., why?
>
> err, because that's what I decided a year ago. I wonder why ;)
>
> Perhaps because of the DoS thing, but it has a /proc knob and defaults
> to off, so it should be OK.

yeah. There's also a boot option. To address the DoS angle, should i
make it optionally printk_ratelimit() perhaps? (although often the
messages come in streams and skipping a message can be annoying)

> > It's very, very useful to distro, early-boot-userspace and glibc
> > development. The only add-on change should be to not print SIGKILL
> > events. Otherwise it's very much a keeper. Hm?
>
> <promotes it>

thanks :-)

Ingo

2007-05-24 09:58:16

by Ingo Molnar

[permalink] [raw]
Subject: Re: [PATCH 1/2] limit print_fatal_signal() rate (was: [RFC] log out-of-virtual-memory events)


* Ingo Molnar <[email protected]> wrote:

> [...] The only add-on change should be to not print SIGKILL events.

ah, that's already included in the version in -mm.

admittedly, the #ifdef __i386__ is quite lame, but there's no generic
safely-try-to-show-code-at-addr function available at the moment.

Ingo

2007-05-24 09:59:06

by Ingo Molnar

[permalink] [raw]
Subject: Re: [PATCH 1/2] limit print_fatal_signal() rate


* Andrea Righi <[email protected]> wrote:

> Actually it seems that SIGKILLs are not printed. In
> get_signal_to_deliver() we have:
>
> [snip]
> @@ -1843,6 +1879,8 @@ relock:
> * Anything else is fatal, maybe with a core dump.
> */
> current->flags |= PF_SIGNALED;
> + if ((signr != SIGKILL) && print_fatal_signals)
> + print_fatal_signal(regs, signr);

yeah. Either i implemented that and forgot, or someone else implemented
it. :)

Ingo

2007-05-24 16:24:18

by Andrew Morton

[permalink] [raw]
Subject: Re: [PATCH 1/2] limit print_fatal_signal() rate (was: [RFC] log out-of-virtual-memory events)

On Thu, 24 May 2007 11:55:03 +0200 Ingo Molnar <[email protected]> wrote:

>
> * Andrew Morton <[email protected]> wrote:
>
> > On Thu, 24 May 2007 09:58:35 +0200 Ingo Molnar <[email protected]> wrote:
> >
> > >
> > > * Andrew Morton <[email protected]> wrote:
> > >
> > > > Well OK. But vdso-print-fatal-signals.patch is designated
> > > > not-for-mainline anyway.
> > >
> > > btw., why?
> >
> > err, because that's what I decided a year ago. I wonder why ;)
> >
> > Perhaps because of the DoS thing, but it has a /proc knob and defaults
> > to off, so it should be OK.
>
> yeah. There's also a boot option. To address the DoS angle, should i
> make it optionally printk_ratelimit() perhaps? (although often the
> messages come in streams and skipping a message can be annoying)

I don't think so, really. It takes a deliberate act to turn the thing
on, after all.

I we _were_ concerned about the logspam then it might be better to make the
feature turn itself off after 100 messages, rather than ratelimiting it.

2007-06-10 19:53:46

by folkert

[permalink] [raw]
Subject: Re: signals logged / [RFC] log out-of-virtual-memory events

> > >> >+ {
> > >> if (sig_fatal(t, sig)) {
> > >> >+ printk(KERN_WARNING "Sig %d send to %d owned by %d.%d
> > >(%s)\n",
> > >> s/send/sent/;
> > >> >+ sig, t -> pid, t -> uid, t -> gid, t -> comm);
> > >> t->pid, t->uid, t->gid, t->comm);
> > >
> >
> > Gargh ... why does this want to be in the *kernel*'s logs? In any case, can
> > you please make this KERN_INFO (or lower) instead of KERN_WARNING.
> Description:
> This patch adds code to the signal-sender making it log a message when
> an unhandled fatal signal will be delivered.

New version. This one also informs the user about the sende pid/uid of
the signal (when applicable).

Signed-of by: Folkert van Heusden <[email protected]

--- linux/kernel/signal.c.org 2007-05-20 22:47:13.000000000 +0200
+++ linux/kernel/signal.c 2007-06-10 00:21:31.000000000 +0200
@@ -739,6 +739,18 @@
struct sigqueue * q = NULL;
int ret = 0;

+ /* unhandled fatal signals are logged */
+ if (sig_fatal(t, sig)) {
+ if (is_si_special(info))
+ printk(KERN_INFO "Sig %d sent to %d owned by %d.%d (%s)\n",
+ sig, t->pid, t->uid, t->gid, t->comm);
+ else
+ printk(KERN_INFO "Sig %d sent to %d owned by %d.%d (%s), sent by pid %d, uid %d\n",
+ sig, t->pid, t->uid, t->gid, t->comm,
+ info -> _sifields._kill._pid,
+ info -> _sifields._kill._uid);
+ }
+
/*
* fast-pathed signals for kernel-internal things like SIGSTOP
* or SIGKILL.


Folkert van Heusden

--
Feeling generous? -> http://www.vanheusden.com/wishlist.php
----------------------------------------------------------------------
Phone: +31-6-41278122, PGP-key: 1F28D8AE, http://www.vanheusden.com

2007-06-10 20:08:47

by Jiri Kosina

[permalink] [raw]
Subject: Re: signals logged / [RFC] log out-of-virtual-memory events

On Sun, 10 Jun 2007, Folkert van Heusden wrote:

> Signed-of by: Folkert van Heusden <[email protected]

This looks broken BTW.

> + printk(KERN_INFO "Sig %d sent to %d owned by %d.%d (%s), sent by pid %d, uid %d\n",
> + sig, t->pid, t->uid, t->gid, t->comm,
> + info -> _sifields._kill._pid,
> + info -> _sifields._kill._uid);

Am I the only one whose eyes are hurt by these spaces?

--
Jiri Kosina

2007-06-10 20:37:28

by Jan Engelhardt

[permalink] [raw]
Subject: Re: signals logged / [RFC] log out-of-virtual-memory events


On Jun 10 2007 22:06, Jiri Kosina wrote:
>On Sun, 10 Jun 2007, Folkert van Heusden wrote:
>
>> Signed-of by: Folkert van Heusden <[email protected]
>
>This looks broken BTW.
>
>> + printk(KERN_INFO "Sig %d sent to %d owned by %d.%d (%s), sent by pid %d, uid %d\n",
>> + sig, t->pid, t->uid, t->gid, t->comm,
>> + info -> _sifields._kill._pid,
>> + info -> _sifields._kill._uid);
>
>Am I the only one whose eyes are hurt by these spaces?

They were discussed before already. And they were fixed up (t->uid...).
And now new ones got added. Ergh.



Jan
--