2012-08-21 06:40:43

by David Airlie

[permalink] [raw]
Subject: [PATCH] fbcon: fix race condition between console lock and cursor timer

So we've had a fair few reports of fbcon handover breakage between
efi/vesafb and i915 surface recently, so I dedicated a couple of
days to finding the problem.

Essentially the last thing we saw was the conflicting framebuffer
message and that was all.

So after much tracing with direct netconsole writes (printks
under console_lock not so useful), I think I found the race.

Thread A (driver load) Thread B (timer thread)
unbind_con_driver -> |
bind_con_driver -> |
vc->vc_sw->con_deinit -> |
fbcon_deinit -> |
console_lock() |
| |
| fbcon_flashcursor timer fires
| console_lock() <- blocked for A
|
|
fbcon_del_cursor_timer ->
del_timer_sync
(BOOM)

Of course because all of this is under the console lock,
we never see anything, also since we also just unbound the active
console guess what we never see anything.

Hopefully this fixes the problem for anyone seeing vesafb->kms
driver handoff.

Signed-off-by: David Airlie <[email protected]>
---
drivers/video/console/fbcon.c | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/drivers/video/console/fbcon.c b/drivers/video/console/fbcon.c
index 2e471c2..f8a79fc 100644
--- a/drivers/video/console/fbcon.c
+++ b/drivers/video/console/fbcon.c
@@ -372,8 +372,12 @@ static void fb_flashcursor(struct work_struct *work)
struct vc_data *vc = NULL;
int c;
int mode;
+ int ret;
+
+ ret = console_trylock();
+ if (ret == 0)
+ return;

- console_lock();
if (ops && ops->currcon != -1)
vc = vc_cons[ops->currcon].d;

--
1.7.10.2


2012-08-21 09:10:41

by Alan

[permalink] [raw]
Subject: Re: [PATCH] fbcon: fix race condition between console lock and cursor timer

> So after much tracing with direct netconsole writes (printks
> under console_lock not so useful), I think I found the race.

Direct netconsole write would be a useful patch to have mainline I think
8)

> Hopefully this fixes the problem for anyone seeing vesafb->kms
> driver handoff.

Not really the proper fix but its clear and is probably the best thing to
go in initially with a cc: stable. Can you at least stick a large

+ /* FIXME: we should sort out the unbind locking instead */

on the patch however.

2012-08-21 12:15:04

by Peter Zijlstra

[permalink] [raw]
Subject: Re: [PATCH] fbcon: fix race condition between console lock and cursor timer

On Tue, 2012-08-21 at 16:40 +1000, Dave Airlie wrote:
> So after much tracing with direct netconsole writes (printks
> under console_lock not so useful)

I always use earlyprintk on serial..

2012-08-21 12:16:11

by Peter Zijlstra

[permalink] [raw]
Subject: Re: [PATCH] fbcon: fix race condition between console lock and cursor timer

On Tue, 2012-08-21 at 10:15 +0100, Alan Cox wrote:
> > So after much tracing with direct netconsole writes (printks
> > under console_lock not so useful), I think I found the race.
>
> Direct netconsole write would be a useful patch to have mainline I think
> 8)

could we make that use the earlyprintk infrastructure?

2012-08-21 13:19:08

by Josh Boyer

[permalink] [raw]
Subject: Re: [PATCH] fbcon: fix race condition between console lock and cursor timer

On Tue, Aug 21, 2012 at 2:40 AM, Dave Airlie <[email protected]> wrote:
> So we've had a fair few reports of fbcon handover breakage between
> efi/vesafb and i915 surface recently, so I dedicated a couple of
> days to finding the problem.
>
> Essentially the last thing we saw was the conflicting framebuffer
> message and that was all.
>
> So after much tracing with direct netconsole writes (printks
> under console_lock not so useful), I think I found the race.
>
> Thread A (driver load) Thread B (timer thread)
> unbind_con_driver -> |
> bind_con_driver -> |
> vc->vc_sw->con_deinit -> |
> fbcon_deinit -> |
> console_lock() |
> | |
> | fbcon_flashcursor timer fires
> | console_lock() <- blocked for A
> |
> |
> fbcon_del_cursor_timer ->
> del_timer_sync
> (BOOM)
>
> Of course because all of this is under the console lock,
> we never see anything, also since we also just unbound the active
> console guess what we never see anything.
>
> Hopefully this fixes the problem for anyone seeing vesafb->kms
> driver handoff.
>
> Signed-off-by: David Airlie <[email protected]>
> ---
> drivers/video/console/fbcon.c | 6 +++++-
> 1 file changed, 5 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/video/console/fbcon.c b/drivers/video/console/fbcon.c
> index 2e471c2..f8a79fc 100644
> --- a/drivers/video/console/fbcon.c
> +++ b/drivers/video/console/fbcon.c
> @@ -372,8 +372,12 @@ static void fb_flashcursor(struct work_struct *work)
> struct vc_data *vc = NULL;
> int c;
> int mode;
> + int ret;
> +
> + ret = console_trylock();
> + if (ret == 0)
> + return;
>
> - console_lock();
> if (ops && ops->currcon != -1)
> vc = vc_cons[ops->currcon].d;
>

I have a Dell XPS 8300 machine with a Radeon card in it that started
showing this problem yesterday with 3.6-rc2 kernels. I tested this
patch on top of v3.6-rc2-206-g10c63c9 this morning and the problem
seems to have been cleared up for me. That includes making sure the
grub2 file has the gfxterm set, etc.

I know we've been seeing this quite a bit more on Fedora 17, so we'll
want to have some people test a 3.5 build with it but things are
looking better.

josh

2012-08-22 04:00:58

by Dave Airlie

[permalink] [raw]
Subject: Re: [PATCH] fbcon: fix race condition between console lock and cursor timer

On Tue, Aug 21, 2012 at 7:15 PM, Alan Cox <[email protected]> wrote:
>> So after much tracing with direct netconsole writes (printks
>> under console_lock not so useful), I think I found the race.
>
> Direct netconsole write would be a useful patch to have mainline I think
> 8)

Well I used a one line wrapper around the netconsole write_msg, which
just passed
NULL as the first arg, then sprinkled netconsole_write_msg around the
place, not having
printf stuff could be an annoyance for some people, for this it didn't matter.

Peter I wish I had a serial port to work with :-)

>
> Not really the proper fix but its clear and is probably the best thing to
> go in initially with a cc: stable. Can you at least stick a large
>
> + /* FIXME: we should sort out the unbind locking instead */

Done, and cc stable, I'll send this to Linus via my tree as its fairly
urgent from my pov.

Dave.