We were seeing unexplained segfaults in coreutils processes and other
basic utilities that we tracked down to binfmt_elf failing to load
segments for ld.so. Digging further, the actual problem seems to occur
when a process gets sigkilled while it is still being loaded by the
kernel. In our case when _do_page_fault goes for a retry it will return
early as it first checks for fatal_signal_pending(), so load_elf_interp
also returns with error and as a result search_binary_handler will
force_sigsegv() which is pretty confusing as nothing actually failed
here.
Fixes: 19d860a140be ("handle suicide on late failure exits in execve() in search_binary_handler()")
Reference: https://lkml.org/lkml/2013/2/14/5
Signed-off-by: Ivan Delalande <[email protected]>
---
fs/exec.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/fs/exec.c b/fs/exec.c
index bdd0eacefdf5..6e8007edbb2d 100644
--- a/fs/exec.c
+++ b/fs/exec.c
@@ -1656,7 +1656,8 @@ int search_binary_handler(struct linux_binprm *bprm)
if (retval < 0 && !bprm->mm) {
/* we got to flush_old_exec() and failed after it */
read_unlock(&binfmt_lock);
- force_sigsegv(SIGSEGV, current);
+ if (!fatal_signal_pending(current))
+ force_sigsegv(SIGSEGV, current);
return retval;
}
if (retval != -ENOEXEC || !bprm->file) {
--
2.18.0
Hi Ivan,
2018-07-31 1:56 GMT+01:00 Ivan Delalande <[email protected]>:
> We were seeing unexplained segfaults in coreutils processes and other
> basic utilities that we tracked down to binfmt_elf failing to load
> segments for ld.so. Digging further, the actual problem seems to occur
> when a process gets sigkilled while it is still being loaded by the
> kernel. In our case when _do_page_fault goes for a retry it will return
> early as it first checks for fatal_signal_pending(), so load_elf_interp
> also returns with error and as a result search_binary_handler will
> force_sigsegv() which is pretty confusing as nothing actually failed
> here.
>
> Fixes: 19d860a140be ("handle suicide on late failure exits in execve() in search_binary_handler()")
> Reference: https://lkml.org/lkml/2013/2/14/5
> Signed-off-by: Ivan Delalande <[email protected]>
+Cc: Oleg Nesterov <[email protected]>
+Cc: Andy Lutomirski <[email protected]>
> ---
> fs/exec.c | 3 ++-
> 1 file changed, 2 insertions(+), 1 deletion(-)
>
> diff --git a/fs/exec.c b/fs/exec.c
> index bdd0eacefdf5..6e8007edbb2d 100644
> --- a/fs/exec.c
> +++ b/fs/exec.c
> @@ -1656,7 +1656,8 @@ int search_binary_handler(struct linux_binprm *bprm)
> if (retval < 0 && !bprm->mm) {
> /* we got to flush_old_exec() and failed after it */
> read_unlock(&binfmt_lock);
> - force_sigsegv(SIGSEGV, current);
> + if (!fatal_signal_pending(current))
> + force_sigsegv(SIGSEGV, current);
I would suggest to add something like:
: if (print_fatal_signals)
: pr_info("load_binary() failed: %d\n", retval);
It was interesting to catch that it actually segfaults during loading,
probably will save someone a couple of minutes too ;-)
> return retval;
> }
> if (retval != -ENOEXEC || !bprm->file) {
Thanks,
Dmitry
2018-08-02 20:53 GMT+01:00 Dmitry Safonov <[email protected]>:
> Hi Ivan,
>
> 2018-07-31 1:56 GMT+01:00 Ivan Delalande <[email protected]>:
>> We were seeing unexplained segfaults in coreutils processes and other
>> basic utilities that we tracked down to binfmt_elf failing to load
>> segments for ld.so. Digging further, the actual problem seems to occur
>> when a process gets sigkilled while it is still being loaded by the
>> kernel. In our case when _do_page_fault goes for a retry it will return
>> early as it first checks for fatal_signal_pending(), so load_elf_interp
>> also returns with error and as a result search_binary_handler will
>> force_sigsegv() which is pretty confusing as nothing actually failed
>> here.
>>
>> Fixes: 19d860a140be ("handle suicide on late failure exits in execve() in search_binary_handler()")
>> Reference: https://lkml.org/lkml/2013/2/14/5
>> Signed-off-by: Ivan Delalande <[email protected]>
>
> +Cc: Oleg Nesterov <[email protected]>
> +Cc: Andy Lutomirski <[email protected]>
Also worth to add to commit message an example of previously
user-visible message in dmesg:
[ 1545.889604] potentially unexpected fatal signal 11.
[ 1545.889614] CPU: 2 PID: 7462 Comm: grep Tainted: P O 3.18.28 #1
[ 1545.889617] Hardware name: Celestica D4040/D4040, BIOS 5.6.5 08/18/2016
[ 1545.889621] task: ffff880011282280 ti: ffff880100938000 task.ti:
ffff880100938000
[ 1545.889624] RIP: 0023:[<00000000f760eb70>] [<00000000f760eb70>] 0xf760eb70
[ 1545.889641] RSP: 002b:00000000fffa3454 EFLAGS: 00000296
[ 1545.889644] RAX: fffffffffffffff2 RBX: 00000000f7a5c3e8 RCX: 00000000f7a5c718
[ 1545.889647] RDX: 00000000f7a5b230 RSI: 00000000f7a5c718 RDI: 00000000f757c000
[ 1545.889650] RBP: 00000000f7a5c3e8 R08: 0000000000000000 R09: 0000000000000000
[ 1545.889653] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
[ 1545.889656] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[ 1545.889659] FS: 0000000000000000(0000) GS:ffff88017fb00000(0000)
knlGS:0000000000000000
[ 1545.889662] CS: 0010 DS: 002b ES: 002b CR0: 000000008005003b
[ 1545.889665] CR2: 00000000f77d7838 CR3: 000000010bb90000 CR4: 00000000001007e0
(which now will be suppressed if there was a fatal signal)
>
>> ---
>> fs/exec.c | 3 ++-
>> 1 file changed, 2 insertions(+), 1 deletion(-)
>>
>> diff --git a/fs/exec.c b/fs/exec.c
>> index bdd0eacefdf5..6e8007edbb2d 100644
>> --- a/fs/exec.c
>> +++ b/fs/exec.c
>> @@ -1656,7 +1656,8 @@ int search_binary_handler(struct linux_binprm *bprm)
>> if (retval < 0 && !bprm->mm) {
>> /* we got to flush_old_exec() and failed after it */
>> read_unlock(&binfmt_lock);
>> - force_sigsegv(SIGSEGV, current);
>> + if (!fatal_signal_pending(current))
>> + force_sigsegv(SIGSEGV, current);
>
> I would suggest to add something like:
> : if (print_fatal_signals)
> : pr_info("load_binary() failed: %d\n", retval);
>
> It was interesting to catch that it actually segfaults during loading,
> probably will save someone a couple of minutes too ;-)
Not sure if it's easy to trigger, but it might require a ratelimit too..
>
>> return retval;
>> }
>> if (retval != -ENOEXEC || !bprm->file) {
--
Dmitry
On 08/02, Dmitry Safonov wrote:
>
> Hi Ivan,
>
> 2018-07-31 1:56 GMT+01:00 Ivan Delalande <[email protected]>:
> > We were seeing unexplained segfaults in coreutils processes and other
> > basic utilities that we tracked down to binfmt_elf failing to load
> > segments for ld.so. Digging further, the actual problem seems to occur
> > when a process gets sigkilled while it is still being loaded by the
> > kernel. In our case when _do_page_fault goes for a retry it will return
> > early as it first checks for fatal_signal_pending(), so load_elf_interp
> > also returns with error and as a result search_binary_handler will
> > force_sigsegv() which is pretty confusing as nothing actually failed
> > here.
> >
> > Fixes: 19d860a140be ("handle suicide on late failure exits in execve() in search_binary_handler()")
> > Reference: https://lkml.org/lkml/2013/2/14/5
> > Signed-off-by: Ivan Delalande <[email protected]>
>
> +Cc: Oleg Nesterov <[email protected]>
> +Cc: Andy Lutomirski <[email protected]>
Thanks...
and sorry, I fail to understand the problem and what/how this patch tries to fix.
Hmm. After I read the next email from Dmitry it seems to me that the whole purpose
of this patch is to avoid print_fatal_signal()? If yes, the changelog should clearly
explain this.
> > --- a/fs/exec.c
> > +++ b/fs/exec.c
> > @@ -1656,7 +1656,8 @@ int search_binary_handler(struct linux_binprm *bprm)
> > if (retval < 0 && !bprm->mm) {
> > /* we got to flush_old_exec() and failed after it */
> > read_unlock(&binfmt_lock);
> > - force_sigsegv(SIGSEGV, current);
> > + if (!fatal_signal_pending(current))
> > + force_sigsegv(SIGSEGV, current);
I won't argue, but may be force_sigsegv() should check fatal_signal_pending()
itself. setup_rt_frame() can too fail if fatal_signal_pending() by the same
reason.
Oleg.
Hi,
On Fri, Aug 03, 2018 at 03:39:24PM +0200, Oleg Nesterov wrote:
> On 08/02, Dmitry Safonov wrote:
> > 2018-07-31 1:56 GMT+01:00 Ivan Delalande <[email protected]>:
> > > We were seeing unexplained segfaults in coreutils processes and other
> > > basic utilities that we tracked down to binfmt_elf failing to load
> > > segments for ld.so. Digging further, the actual problem seems to occur
> > > when a process gets sigkilled while it is still being loaded by the
> > > kernel. In our case when _do_page_fault goes for a retry it will return
> > > early as it first checks for fatal_signal_pending(), so load_elf_interp
> > > also returns with error and as a result search_binary_handler will
> > > force_sigsegv() which is pretty confusing as nothing actually failed
> > > here.
> > >
> > > Fixes: 19d860a140be ("handle suicide on late failure exits in execve() in search_binary_handler()")
> > > Reference: https://lkml.org/lkml/2013/2/14/5
> > > Signed-off-by: Ivan Delalande <[email protected]>
> >
> > +Cc: Oleg Nesterov <[email protected]>
> > +Cc: Andy Lutomirski <[email protected]>
>
> Thanks...
>
> and sorry, I fail to understand the problem and what/how this patch tries to fix.
>
> Hmm. After I read the next email from Dmitry it seems to me that the whole purpose
> of this patch is to avoid print_fatal_signal()? If yes, the changelog should clearly
> explain this.
Sorry about that, yes this is purely to avoid printing the segfault
messages for these processes when they were in fact killed.
I'll definitely send a v2 to clarify that, and probably add the helpful
message Dimitry suggested as well.
> > > --- a/fs/exec.c
> > > +++ b/fs/exec.c
> > > @@ -1656,7 +1656,8 @@ int search_binary_handler(struct linux_binprm *bprm)
> > > if (retval < 0 && !bprm->mm) {
> > > /* we got to flush_old_exec() and failed after it */
> > > read_unlock(&binfmt_lock);
> > > - force_sigsegv(SIGSEGV, current);
> > > + if (!fatal_signal_pending(current))
> > > + force_sigsegv(SIGSEGV, current);
>
> I won't argue, but may be force_sigsegv() should check fatal_signal_pending()
> itself. setup_rt_frame() can too fail if fatal_signal_pending() by the same
> reason.
I'm not sure, I think it would feel out of place in force_sigsegv() as
other callers might not expect this check in different contexts. I could
add a similar call to fatal_signal_pending() in signal_setup_done()
though, if you think we can hit the same problem from setup_rt_frame().
Thanks,
--
Ivan Delalande
Arista Networks