2020-05-22 10:07:39

by Sumit Garg

Subject: [PATCH] kdb: Make kdb_printf robust to run in NMI context

While rounding up CPUs via NMIs, its possible that a rounded up CPU
maybe holding a console port lock leading to kgdb master CPU stuck in
a deadlock during invocation of console write operations. So in order
to avoid such a deadlock, invoke bust_spinlocks() prior to invocation
of console handlers.

Also, add a check for console port to be enabled prior to invocation of
corresponding handler.

Suggested-by: Petr Mladek <[email protected]>
Suggested-by: Sergey Senozhatsky <[email protected]>
Signed-off-by: Sumit Garg <[email protected]>
---
kernel/debug/kdb/kdb_io.c | 8 ++++++++
1 file changed, 8 insertions(+)

diff --git a/kernel/debug/kdb/kdb_io.c b/kernel/debug/kdb/kdb_io.c
index 924bc92..e32ece6 100644
--- a/kernel/debug/kdb/kdb_io.c
+++ b/kernel/debug/kdb/kdb_io.c
@@ -699,7 +699,11 @@ int vkdb_printf(enum kdb_msgsrc src, const char *fmt, va_list ap)
}
}
for_each_console(c) {
+ if (!(c->flags & CON_ENABLED))
+ continue;
+ bust_spinlocks(1);
c->write(c, cp, retlen - (cp - kdb_buffer));
+ bust_spinlocks(0);
touch_nmi_watchdog();
}
}
@@ -761,7 +765,11 @@ int vkdb_printf(enum kdb_msgsrc src, const char *fmt, va_list ap)
}
}
for_each_console(c) {
+ if (!(c->flags & CON_ENABLED))
+ continue;
+ bust_spinlocks(1);
c->write(c, moreprompt, strlen(moreprompt));
+ bust_spinlocks(0);
touch_nmi_watchdog();
}

--
2.7.4

2020-05-22 10:40:24

by Daniel Thompson

Subject: Re: [PATCH] kdb: Make kdb_printf robust to run in NMI context

On Fri, May 22, 2020 at 03:32:26PM +0530, Sumit Garg wrote:
> While rounding up CPUs via NMIs, its possible that a rounded up CPU
> maybe holding a console port lock leading to kgdb master CPU stuck in
> a deadlock during invocation of console write operations. So in order
> to avoid such a deadlock, invoke bust_spinlocks() prior to invocation
> of console handlers.
>
> Also, add a check for console port to be enabled prior to invocation of
> corresponding handler.
>
> Suggested-by: Petr Mladek <[email protected]>
> Suggested-by: Sergey Senozhatsky <[email protected]>
> Signed-off-by: Sumit Garg <[email protected]>
> ---
> kernel/debug/kdb/kdb_io.c | 8 ++++++++
> 1 file changed, 8 insertions(+)
>
> diff --git a/kernel/debug/kdb/kdb_io.c b/kernel/debug/kdb/kdb_io.c
> index 924bc92..e32ece6 100644
> --- a/kernel/debug/kdb/kdb_io.c
> +++ b/kernel/debug/kdb/kdb_io.c
> @@ -699,7 +699,11 @@ int vkdb_printf(enum kdb_msgsrc src, const char *fmt, va_list ap)
> }
> }
> for_each_console(c) {
> + if (!(c->flags & CON_ENABLED))
> + continue;
> + bust_spinlocks(1);
> c->write(c, cp, retlen - (cp - kdb_buffer));
> + bust_spinlocks(0);

I think it might actually be better to directly manipulate oops_in_progress
here.

bust_spinlocks(0) will call unblank_screen() and console_unblank() and
I don't think calling these is worth the risk (and whilst kicking klogd
is safe I think its also not required).


Daniel.



> touch_nmi_watchdog();
> }
> }
> @@ -761,7 +765,11 @@ int vkdb_printf(enum kdb_msgsrc src, const char *fmt, va_list ap)
> }
> }
> for_each_console(c) {
> + if (!(c->flags & CON_ENABLED))
> + continue;
> + bust_spinlocks(1);
> c->write(c, moreprompt, strlen(moreprompt));
> + bust_spinlocks(0);
> touch_nmi_watchdog();
> }
>
> --
> 2.7.4
>

2020-05-22 10:57:03

by Sumit Garg

Subject: Re: [PATCH] kdb: Make kdb_printf robust to run in NMI context

On Fri, 22 May 2020 at 16:08, Daniel Thompson
<[email protected]> wrote:
>
> On Fri, May 22, 2020 at 03:32:26PM +0530, Sumit Garg wrote:
> > While rounding up CPUs via NMIs, its possible that a rounded up CPU
> > maybe holding a console port lock leading to kgdb master CPU stuck in
> > a deadlock during invocation of console write operations. So in order
> > to avoid such a deadlock, invoke bust_spinlocks() prior to invocation
> > of console handlers.
> >
> > Also, add a check for console port to be enabled prior to invocation of
> > corresponding handler.
> >
> > Suggested-by: Petr Mladek <[email protected]>
> > Suggested-by: Sergey Senozhatsky <[email protected]>
> > Signed-off-by: Sumit Garg <[email protected]>
> > ---
> > kernel/debug/kdb/kdb_io.c | 8 ++++++++
> > 1 file changed, 8 insertions(+)
> >
> > diff --git a/kernel/debug/kdb/kdb_io.c b/kernel/debug/kdb/kdb_io.c
> > index 924bc92..e32ece6 100644
> > --- a/kernel/debug/kdb/kdb_io.c
> > +++ b/kernel/debug/kdb/kdb_io.c
> > @@ -699,7 +699,11 @@ int vkdb_printf(enum kdb_msgsrc src, const char *fmt, va_list ap)
> > }
> > }
> > for_each_console(c) {
> > + if (!(c->flags & CON_ENABLED))
> > + continue;
> > + bust_spinlocks(1);
> > c->write(c, cp, retlen - (cp - kdb_buffer));
> > + bust_spinlocks(0);
>
> I think it might actually be better to directly manipulate oops_in_progress
> here.
>

Okay will directly manipulate oops_in_progress here.

> bust_spinlocks(0) will call unblank_screen() and console_unblank() and
> I don't think calling these is worth the risk (and whilst kicking klogd
> is safe I think its also not required).

Makes sense.

-Sumit

>
>
> Daniel.
>
>
>
> > touch_nmi_watchdog();
> > }
> > }
> > @@ -761,7 +765,11 @@ int vkdb_printf(enum kdb_msgsrc src, const char *fmt, va_list ap)
> > }
> > }
> > for_each_console(c) {
> > + if (!(c->flags & CON_ENABLED))
> > + continue;
> > + bust_spinlocks(1);
> > c->write(c, moreprompt, strlen(moreprompt));
> > + bust_spinlocks(0);
> > touch_nmi_watchdog();
> > }
> >
> > --
> > 2.7.4
> >