2008-02-29 12:49:32

by Simon Huggins

[permalink] [raw]
Subject: Scheduler broken? sdhci issues with scheduling

[ Please Cc me on replies ]

I had a bug with sdhci which Pierre Ossman looked at for me.

In the end essentially the fix was to use HZ=1000 and nothing else.
Pierre seemed to think that this was a bug in the scheduler.

I'm interested if someone who groks the scheduler could take a look at
the mmc/* code and see if it's abusing HZ or if there's anything
obviously wrong there. If it's not then perhaps there really is a bug
in the scheduler?

See
http://list.drzeus.cx/pipermail/sdhci-devel/2008-February/002164.html
http://list.drzeus.cx/pipermail/sdhci-devel/2008-January/002142.html

Here's the log that caused him to suspect the scheduler:
http://list.drzeus.cx/pipermail/sdhci-devel/2007-December/002086.html
and his post saying so:
http://list.drzeus.cx/pipermail/sdhci-devel/2007-December/002087.html

Originally I tried to narrow it down:
http://list.drzeus.cx/pipermail/sdhci-devel/2007-November/002072.html
and shows a commit that seemed to "fix" it for me
http://list.drzeus.cx/pipermail/sdhci-devel/2007-November/002072.html
but apparently the commit I reverted was good
http://list.drzeus.cx/pipermail/sdhci-devel/2007-November/002078.html

I'm happy to provide more information or try different kernels/config
options etc (I really should upgrade the kernel on my laptop anyway).

Thanks,

--
----------( "An excellent suggestion sir, with only two )----------
Simon ----( minor flaws...." - Kryten )---- Nomis
Htag.pl 0.0.22


2008-02-29 13:34:58

by Ingo Molnar

[permalink] [raw]
Subject: Re: Scheduler broken? sdhci issues with scheduling


* Simon Huggins <[email protected]> wrote:

> [ Please Cc me on replies ]
>
> I had a bug with sdhci which Pierre Ossman looked at for me.
>
> In the end essentially the fix was to use HZ=1000 and nothing else.
> Pierre seemed to think that this was a bug in the scheduler.

does the patch below help, even if you keep HZ=100? This doesnt look
like a scheduler issue, it's more of a timer/timing issue. Different HZ
means different msleep() results - and the mmc code does a loop of small
msleep delays.

Ingo

-------------->
---
drivers/mmc/core/core.h | 7 +------
1 file changed, 1 insertion(+), 6 deletions(-)

Index: linux/drivers/mmc/core/core.h
===================================================================
--- linux.orig/drivers/mmc/core/core.h
+++ linux/drivers/mmc/core/core.h
@@ -36,12 +36,7 @@ void mmc_set_timing(struct mmc_host *hos

static inline void mmc_delay(unsigned int ms)
{
- if (ms < 1000 / HZ) {
- cond_resched();
- mdelay(ms);
- } else {
- msleep(ms);
- }
+ mdelay(ms);
}

void mmc_rescan(struct work_struct *work);

2008-02-29 13:52:00

by Ingo Molnar

[permalink] [raw]
Subject: Re: Scheduler broken? sdhci issues with scheduling


* Ingo Molnar <[email protected]> wrote:

> does the patch below help, even if you keep HZ=100? This doesnt look
> like a scheduler issue, it's more of a timer/timing issue. Different
> HZ means different msleep() results - and the mmc code does a loop of
> small msleep delays.

alternatively, instead of applying the first patch, could you apply the
patch below instead? This makes "msleep()" much more precise. Does this
help in the HZ != 1000 case?

Ingo

----------------------->
Subject: hrtimers: use hrtimers so that msleep() sleeps for the requested time
From: Jonathan Corbet <[email protected]>

The problem being addressed here is that the current msleep() will stop
for a minimum of two jiffies, meaning that, on a HZ=100 system,
msleep(1) delays for for about 20ms. In a driver with one such delay
for each of 150 or so register setting operations, the extra time adds
up to a few seconds.

This patch addresses the situation by using hrtimers. On tickless
systems with working timers, msleep(1) now sleeps for 1ms, even with
HZ=100.

Most comments last time were favorable. The one dissenter was Roman,
who worries about the overhead of using hrtimers for this operation; my
understanding is that he would rather see a really_msleep() function for
those who actually want millisecond resolution. I'm not sure how to
characterize what the cost could be, but it can only be buried by the
fact that every call sleeps for some number of milliseconds. On my
system, the several hundred total msleep() calls can't cause any real
overhead, and almost all happen at initialization time.

I still think it would be useful for msleep() to do what it says it does
and not vastly oversleep with small arguments. A quick grep turns up
450 msleep(1) calls in the current mainline. Andrew, if you agree, can
you drop this into -mm? If not, I guess I'll let it go.

Current msleep() snoozes for at least two jiffies, causing msleep(1) to
sleep for at least 20ms on HZ=100 systems. Using hrtimers allows
msleep() to sleep for something much closer to the requested time.

Signed-off-by: Jonathan Corbet <[email protected]>
Signed-off-by: Ingo Molnar <[email protected]>
Signed-off-by: Thomas Gleixner <[email protected]>
---
fs/proc/proc_misc.c | 4 ++--
fs/select.c | 6 ++++--
include/linux/time.h | 8 ++++----
kernel/timer.c | 47 ++++++++++++++++++++++++++++++++++++++---------
4 files changed, 48 insertions(+), 17 deletions(-)

Index: linux/fs/proc/proc_misc.c
===================================================================
--- linux.orig/fs/proc/proc_misc.c
+++ linux/fs/proc/proc_misc.c
@@ -115,9 +115,9 @@ static int uptime_read_proc(char *page,
cputime_to_timespec(idletime, &idle);
len = sprintf(page,"%lu.%02lu %lu.%02lu\n",
(unsigned long) uptime.tv_sec,
- (uptime.tv_nsec / (NSEC_PER_SEC / 100)),
+ (uptime.tv_nsec / (unsigned long)(NSEC_PER_SEC / 100)),
(unsigned long) idle.tv_sec,
- (idle.tv_nsec / (NSEC_PER_SEC / 100)));
+ (idle.tv_nsec / (unsigned long)(NSEC_PER_SEC / 100)));

return proc_calc_metrics(page, start, off, count, eof, len);
}
Index: linux/fs/select.c
===================================================================
--- linux.orig/fs/select.c
+++ linux/fs/select.c
@@ -446,7 +446,8 @@ asmlinkage long sys_pselect7(int n, fd_s
if ((u64)ts.tv_sec >= (u64)MAX_INT64_SECONDS)
timeout = -1; /* infinite */
else {
- timeout = DIV_ROUND_UP(ts.tv_nsec, NSEC_PER_SEC/HZ);
+ timeout = DIV_ROUND_UP(ts.tv_nsec,
+ (unsigned long)NSEC_PER_SEC/HZ);
timeout += ts.tv_sec * HZ;
}
}
@@ -777,7 +778,8 @@ asmlinkage long sys_ppoll(struct pollfd
if ((u64)ts.tv_sec >= (u64)MAX_INT64_SECONDS)
timeout = -1; /* infinite */
else {
- timeout = DIV_ROUND_UP(ts.tv_nsec, NSEC_PER_SEC/HZ);
+ timeout = DIV_ROUND_UP(ts.tv_nsec,
+ (unsigned long)NSEC_PER_SEC/HZ);
timeout += ts.tv_sec * HZ;
}
}
Index: linux/include/linux/time.h
===================================================================
--- linux.orig/include/linux/time.h
+++ linux/include/linux/time.h
@@ -31,11 +31,11 @@ struct timezone {
/* Parameters used to convert the timespec values: */
#define MSEC_PER_SEC 1000L
#define USEC_PER_MSEC 1000L
-#define NSEC_PER_USEC 1000L
-#define NSEC_PER_MSEC 1000000L
+#define NSEC_PER_USEC 1000LL
+#define NSEC_PER_MSEC 1000000LL
#define USEC_PER_SEC 1000000L
-#define NSEC_PER_SEC 1000000000L
-#define FSEC_PER_SEC 1000000000000000L
+#define NSEC_PER_SEC 1000000000LL
+#define FSEC_PER_SEC 1000000000000000LL

static inline int timespec_equal(const struct timespec *a,
const struct timespec *b)
Index: linux/kernel/timer.c
===================================================================
--- linux.orig/kernel/timer.c
+++ linux/kernel/timer.c
@@ -1368,18 +1368,43 @@ void __init init_timers(void)
open_softirq(TIMER_SOFTIRQ, run_timer_softirq, NULL);
}

+
+
+
+static void do_msleep(unsigned int msecs, struct hrtimer_sleeper *sleeper,
+ int sigs)
+{
+ enum hrtimer_mode mode = HRTIMER_MODE_REL;
+ int state = sigs ? TASK_INTERRUPTIBLE : TASK_UNINTERRUPTIBLE;
+
+ /*
+ * This is really just a reworked and simplified version
+ * of do_nanosleep().
+ */
+ hrtimer_init(&sleeper->timer, CLOCK_MONOTONIC, mode);
+ sleeper->timer.expires = ktime_set(0, msecs*NSEC_PER_MSEC);
+ hrtimer_init_sleeper(sleeper, current);
+
+ do {
+ set_current_state(state);
+ hrtimer_start(&sleeper->timer, sleeper->timer.expires, mode);
+ if (sleeper->task)
+ schedule();
+ hrtimer_cancel(&sleeper->timer);
+ mode = HRTIMER_MODE_ABS;
+ } while (sleeper->task && !(sigs && signal_pending(current)));
+}
+
/**
* msleep - sleep safely even with waitqueue interruptions
* @msecs: Time in milliseconds to sleep for
*/
void msleep(unsigned int msecs)
{
- unsigned long timeout = msecs_to_jiffies(msecs) + 1;
+ struct hrtimer_sleeper sleeper;

- while (timeout)
- timeout = schedule_timeout_uninterruptible(timeout);
+ do_msleep(msecs, &sleeper, 0);
}
-
EXPORT_SYMBOL(msleep);

/**
@@ -1388,11 +1413,15 @@ EXPORT_SYMBOL(msleep);
*/
unsigned long msleep_interruptible(unsigned int msecs)
{
- unsigned long timeout = msecs_to_jiffies(msecs) + 1;
+ struct hrtimer_sleeper sleeper;
+ ktime_t left;

- while (timeout && !signal_pending(current))
- timeout = schedule_timeout_interruptible(timeout);
- return jiffies_to_msecs(timeout);
-}
+ do_msleep(msecs, &sleeper, 1);

+ if (!sleeper.task)
+ return 0;
+ left = ktime_sub(sleeper.timer.expires,
+ sleeper.timer.base->get_time());
+ return max(((long) ktime_to_ns(left))/(long)NSEC_PER_MSEC, 1L);
+}
EXPORT_SYMBOL(msleep_interruptible);

2008-02-29 14:00:52

by Ingo Molnar

[permalink] [raw]
Subject: Re: Scheduler broken? sdhci issues with scheduling


* Ingo Molnar <[email protected]> wrote:

> alternatively, instead of applying the first patch, could you apply
> the patch below instead? This makes "msleep()" much more precise. Does
> this help in the HZ != 1000 case?

ok, this had a build failure - use the one below instead.

Ingo

-------------->
Subject: hrtimers: use hrtimers so that msleep() sleeps for the requested time
From: Jonathan Corbet <[email protected]>

The problem being addressed here is that the current msleep() will stop
for a minimum of two jiffies, meaning that, on a HZ=100 system,
msleep(1) delays for for about 20ms. In a driver with one such delay
for each of 150 or so register setting operations, the extra time adds
up to a few seconds.

This patch addresses the situation by using hrtimers. On tickless
systems with working timers, msleep(1) now sleeps for 1ms, even with
HZ=100.

Most comments last time were favorable. The one dissenter was Roman,
who worries about the overhead of using hrtimers for this operation; my
understanding is that he would rather see a really_msleep() function for
those who actually want millisecond resolution. I'm not sure how to
characterize what the cost could be, but it can only be buried by the
fact that every call sleeps for some number of milliseconds. On my
system, the several hundred total msleep() calls can't cause any real
overhead, and almost all happen at initialization time.

I still think it would be useful for msleep() to do what it says it does
and not vastly oversleep with small arguments. A quick grep turns up
450 msleep(1) calls in the current mainline. Andrew, if you agree, can
you drop this into -mm? If not, I guess I'll let it go.

Current msleep() snoozes for at least two jiffies, causing msleep(1) to
sleep for at least 20ms on HZ=100 systems. Using hrtimers allows
msleep() to sleep for something much closer to the requested time.

Signed-off-by: Jonathan Corbet <[email protected]>
Signed-off-by: Ingo Molnar <[email protected]>
Signed-off-by: Thomas Gleixner <[email protected]>
---
fs/proc/proc_misc.c | 4 ++--
fs/select.c | 6 ++++--
kernel/timer.c | 47 ++++++++++++++++++++++++++++++++++++++---------
3 files changed, 44 insertions(+), 13 deletions(-)

Index: linux/fs/proc/proc_misc.c
===================================================================
--- linux.orig/fs/proc/proc_misc.c
+++ linux/fs/proc/proc_misc.c
@@ -115,9 +115,9 @@ static int uptime_read_proc(char *page,
cputime_to_timespec(idletime, &idle);
len = sprintf(page,"%lu.%02lu %lu.%02lu\n",
(unsigned long) uptime.tv_sec,
- (uptime.tv_nsec / (NSEC_PER_SEC / 100)),
+ (uptime.tv_nsec / (unsigned long)(NSEC_PER_SEC / 100)),
(unsigned long) idle.tv_sec,
- (idle.tv_nsec / (NSEC_PER_SEC / 100)));
+ (idle.tv_nsec / (unsigned long)(NSEC_PER_SEC / 100)));

return proc_calc_metrics(page, start, off, count, eof, len);
}
Index: linux/fs/select.c
===================================================================
--- linux.orig/fs/select.c
+++ linux/fs/select.c
@@ -446,7 +446,8 @@ asmlinkage long sys_pselect7(int n, fd_s
if ((u64)ts.tv_sec >= (u64)MAX_INT64_SECONDS)
timeout = -1; /* infinite */
else {
- timeout = DIV_ROUND_UP(ts.tv_nsec, NSEC_PER_SEC/HZ);
+ timeout = DIV_ROUND_UP(ts.tv_nsec,
+ (unsigned long)NSEC_PER_SEC/HZ);
timeout += ts.tv_sec * HZ;
}
}
@@ -777,7 +778,8 @@ asmlinkage long sys_ppoll(struct pollfd
if ((u64)ts.tv_sec >= (u64)MAX_INT64_SECONDS)
timeout = -1; /* infinite */
else {
- timeout = DIV_ROUND_UP(ts.tv_nsec, NSEC_PER_SEC/HZ);
+ timeout = DIV_ROUND_UP(ts.tv_nsec,
+ (unsigned long)NSEC_PER_SEC/HZ);
timeout += ts.tv_sec * HZ;
}
}
Index: linux/kernel/timer.c
===================================================================
--- linux.orig/kernel/timer.c
+++ linux/kernel/timer.c
@@ -1368,18 +1368,43 @@ void __init init_timers(void)
open_softirq(TIMER_SOFTIRQ, run_timer_softirq, NULL);
}

+
+
+
+static void do_msleep(unsigned int msecs, struct hrtimer_sleeper *sleeper,
+ int sigs)
+{
+ enum hrtimer_mode mode = HRTIMER_MODE_REL;
+ int state = sigs ? TASK_INTERRUPTIBLE : TASK_UNINTERRUPTIBLE;
+
+ /*
+ * This is really just a reworked and simplified version
+ * of do_nanosleep().
+ */
+ hrtimer_init(&sleeper->timer, CLOCK_MONOTONIC, mode);
+ sleeper->timer.expires = ktime_set(0, msecs*NSEC_PER_MSEC);
+ hrtimer_init_sleeper(sleeper, current);
+
+ do {
+ set_current_state(state);
+ hrtimer_start(&sleeper->timer, sleeper->timer.expires, mode);
+ if (sleeper->task)
+ schedule();
+ hrtimer_cancel(&sleeper->timer);
+ mode = HRTIMER_MODE_ABS;
+ } while (sleeper->task && !(sigs && signal_pending(current)));
+}
+
/**
* msleep - sleep safely even with waitqueue interruptions
* @msecs: Time in milliseconds to sleep for
*/
void msleep(unsigned int msecs)
{
- unsigned long timeout = msecs_to_jiffies(msecs) + 1;
+ struct hrtimer_sleeper sleeper;

- while (timeout)
- timeout = schedule_timeout_uninterruptible(timeout);
+ do_msleep(msecs, &sleeper, 0);
}
-
EXPORT_SYMBOL(msleep);

/**
@@ -1388,11 +1413,15 @@ EXPORT_SYMBOL(msleep);
*/
unsigned long msleep_interruptible(unsigned int msecs)
{
- unsigned long timeout = msecs_to_jiffies(msecs) + 1;
+ struct hrtimer_sleeper sleeper;
+ ktime_t left;

- while (timeout && !signal_pending(current))
- timeout = schedule_timeout_interruptible(timeout);
- return jiffies_to_msecs(timeout);
-}
+ do_msleep(msecs, &sleeper, 1);

+ if (!sleeper.task)
+ return 0;
+ left = ktime_sub(sleeper.timer.expires,
+ sleeper.timer.base->get_time());
+ return max(((long) ktime_to_ns(left))/(long)NSEC_PER_MSEC, 1L);
+}
EXPORT_SYMBOL(msleep_interruptible);

2008-02-29 19:40:20

by Simon Huggins

[permalink] [raw]
Subject: Re: Scheduler broken? sdhci issues with scheduling

Hi Ingo,

(please cc me again)

On Fri, Feb 29, 2008 at 02:34:33PM +0100, Ingo Molnar wrote:
> * Simon Huggins <[email protected]> wrote:
> > [ Please Cc me on replies ]
> > I had a bug with sdhci which Pierre Ossman looked at for me.
> > In the end essentially the fix was to use HZ=1000 and nothing else.
> > Pierre seemed to think that this was a bug in the scheduler.
> does the patch below help, even if you keep HZ=100? This doesnt look
> like a scheduler issue, it's more of a timer/timing issue. Different HZ
> means different msleep() results - and the mmc code does a loop of small
> msleep delays.

Thanks for looking at it.

I did tests with 2.6.24.3 with HZ=1000 and HZ=100 and as expected the
latter didn't work.

> -------------->
> ---
> drivers/mmc/core/core.h | 7 +------
> 1 file changed, 1 insertion(+), 6 deletions(-)

> Index: linux/drivers/mmc/core/core.h
> ===================================================================
> --- linux.orig/drivers/mmc/core/core.h
> +++ linux/drivers/mmc/core/core.h
> @@ -36,12 +36,7 @@ void mmc_set_timing(struct mmc_host *hos

> static inline void mmc_delay(unsigned int ms)
> {
> - if (ms < 1000 / HZ) {
> - cond_resched();
> - mdelay(ms);
> - } else {
> - msleep(ms);
> - }
> + mdelay(ms);
> }

> void mmc_rescan(struct work_struct *work);

That doesn't work. I did a test with HZ=100 and this patch. I've
attached the log as patch1-log.

On Fri, Feb 29, 2008 at 03:00:24PM +0100, Ingo Molnar wrote:

> * Ingo Molnar <[email protected]> wrote:

> > alternatively, instead of applying the first patch, could you apply
> > the patch below instead? This makes "msleep()" much more precise. Does
> > this help in the HZ != 1000 case?

> ok, this had a build failure - use the one below instead.

That doesn't work either. Again I did a test with HZ=100 and this
patch. I've attached the log as patch2-log.

It had some fuzz but no rejects with 2.6.24.3.

> -------------->
> Subject: hrtimers: use hrtimers so that msleep() sleeps for the requested time
> From: Jonathan Corbet <[email protected]>

> The problem being addressed here is that the current msleep() will stop
> for a minimum of two jiffies, meaning that, on a HZ=100 system,
> msleep(1) delays for for about 20ms. In a driver with one such delay
> for each of 150 or so register setting operations, the extra time adds
> up to a few seconds.

> This patch addresses the situation by using hrtimers. On tickless
> systems with working timers, msleep(1) now sleeps for 1ms, even with
> HZ=100.

> Most comments last time were favorable. The one dissenter was Roman,
> who worries about the overhead of using hrtimers for this operation; my
> understanding is that he would rather see a really_msleep() function for
> those who actually want millisecond resolution. I'm not sure how to
> characterize what the cost could be, but it can only be buried by the
> fact that every call sleeps for some number of milliseconds. On my
> system, the several hundred total msleep() calls can't cause any real
> overhead, and almost all happen at initialization time.

> I still think it would be useful for msleep() to do what it says it does
> and not vastly oversleep with small arguments. A quick grep turns up
> 450 msleep(1) calls in the current mainline. Andrew, if you agree, can
> you drop this into -mm? If not, I guess I'll let it go.

> Current msleep() snoozes for at least two jiffies, causing msleep(1) to
> sleep for at least 20ms on HZ=100 systems. Using hrtimers allows
> msleep() to sleep for something much closer to the requested time.

[snip]


Anything else I can try?

--
----------( "There are times when I think Clint belongs in )----------
----------( my RSS-hole" -- anonymous Debian Planet reader. )----------
Simon ----( )---- Nomis
Htag.pl 0.0.22


Attachments:
(No filename) (3.80 kB)
patch1-log (6.01 kB)
patch2-log (5.74 kB)
Download all attachments

2008-02-29 20:39:40

by Ingo Molnar

[permalink] [raw]
Subject: Re: Scheduler broken? sdhci issues with scheduling


* Simon Huggins <[email protected]> wrote:

> Hi Ingo,
>
> (please cc me again)
>
> On Fri, Feb 29, 2008 at 02:34:33PM +0100, Ingo Molnar wrote:
> > * Simon Huggins <[email protected]> wrote:
> > > [ Please Cc me on replies ]
> > > I had a bug with sdhci which Pierre Ossman looked at for me.
> > > In the end essentially the fix was to use HZ=1000 and nothing else.
> > > Pierre seemed to think that this was a bug in the scheduler.
> > does the patch below help, even if you keep HZ=100? This doesnt look
> > like a scheduler issue, it's more of a timer/timing issue. Different HZ
> > means different msleep() results - and the mmc code does a loop of small
> > msleep delays.
>
> Thanks for looking at it.
>
> I did tests with 2.6.24.3 with HZ=1000 and HZ=100 and as expected the
> latter didn't work.
>
> > -------------->
> > ---
> > drivers/mmc/core/core.h | 7 +------
> > 1 file changed, 1 insertion(+), 6 deletions(-)
>
> > Index: linux/drivers/mmc/core/core.h
> > ===================================================================
> > --- linux.orig/drivers/mmc/core/core.h
> > +++ linux/drivers/mmc/core/core.h
> > @@ -36,12 +36,7 @@ void mmc_set_timing(struct mmc_host *hos
>
> > static inline void mmc_delay(unsigned int ms)
> > {
> > - if (ms < 1000 / HZ) {
> > - cond_resched();
> > - mdelay(ms);
> > - } else {
> > - msleep(ms);
> > - }
> > + mdelay(ms);
> > }
>
> > void mmc_rescan(struct work_struct *work);
>
> That doesn't work. I did a test with HZ=100 and this patch. I've
> attached the log as patch1-log.

> Anything else I can try?

so neither precise, nor imprecise timings help??

Ingo

2008-03-01 12:42:17

by Simon Huggins

[permalink] [raw]
Subject: Re: Scheduler broken? sdhci issues with scheduling

(please cc me)

On Fri, Feb 29, 2008 at 09:39:14PM +0100, Ingo Molnar wrote:
> * Simon Huggins <[email protected]> wrote:
> > On Fri, Feb 29, 2008 at 02:34:33PM +0100, Ingo Molnar wrote:
> > > * Simon Huggins <[email protected]> wrote:
> > > > [ Please Cc me on replies ]
> > > > I had a bug with sdhci which Pierre Ossman looked at for me.
> > > > In the end essentially the fix was to use HZ=1000 and nothing else.
> > > > Pierre seemed to think that this was a bug in the scheduler.
> > > does the patch below help, even if you keep HZ=100? This doesnt look
> > > like a scheduler issue, it's more of a timer/timing issue. Different HZ
> > > means different msleep() results - and the mmc code does a loop of small
> > > msleep delays.
> > Thanks for looking at it.
> > I did tests with 2.6.24.3 with HZ=1000 and HZ=100 and as expected the
> > latter didn't work.
[..]
> > That doesn't work. I did a test with HZ=100 and this patch. I've
> > attached the log as patch1-log.

> > Anything else I can try?
> so neither precise, nor imprecise timings help??

I don't know. I don't really understand the patches I applied I'm
afraid; I'm just reporting the results.

I've cc'd Pierre Ossman so he might have a look from the sdhci point of
view.

Pierre, the thread starts here:
http://lkml.org/lkml/2008/2/29/118

Simon.

--
Just another wannabie | "It's so nice to have a big | Just another fool
----------------------+ strong spud around the house." +-------------------
This message was brought to you by the letter E and the number 13.
htag.pl 0.0.22 -- http://www.earth.li/projectpurple/progs/htag.html

2008-03-01 14:09:21

by Pierre Ossman

[permalink] [raw]
Subject: Re: Scheduler broken? sdhci issues with scheduling

On Sat, 1 Mar 2008 12:42:04 +0000
Simon Huggins <[email protected]> wrote:

> > so neither precise, nor imprecise timings help??
>
> I don't know. I don't really understand the patches I applied I'm
> afraid; I'm just reporting the results.
>
> I've cc'd Pierre Ossman so he might have a look from the sdhci point of
> view.
>
> Pierre, the thread starts here:
> http://lkml.org/lkml/2008/2/29/118
>

I'm afraid I can't add that much...

Ingo, the basic problem is this:

The sdhci controllers are not supposed to signal "card insertion" before they have a stable connection with the card. But most (all?) hardware ignores this and signals immediately. This means we need a delay to handle bouncing connectors. This delay is supposed to be 500 ms, but on Simon's system it is in the order of tens of milliseconds.

The mechanism is not msleep(), nor mdelay(), but a delayed work queue. That's why I told Simon to talk to you as that seems like scheduler territory.

The call chain is:

1. sdhci_irq()
2. sdhci_tasklet_card() <- delay is calculated here
3. mmc_detect_change()
4. mmc_schedule_delayed_work()
5. queue_delayed_work()

msleep() and mdelay() are also affected, but this is the major symptom.

Rgds

--
-- Pierre Ossman

Linux kernel, MMC maintainer http://www.kernel.org
PulseAudio, core developer http://pulseaudio.org
rdesktop, core developer http://www.rdesktop.org

2008-03-02 15:43:05

by Pavel Machek

[permalink] [raw]
Subject: Re: Scheduler broken? sdhci issues with scheduling

On Fri 2008-02-29 11:52:57, Simon Huggins wrote:
> [ Please Cc me on replies ]
>
> I had a bug with sdhci which Pierre Ossman looked at for me.
>
> In the end essentially the fix was to use HZ=1000 and nothing else.
> Pierre seemed to think that this was a bug in the scheduler.
>
> I'm interested if someone who groks the scheduler could take a look at
> the mmc/* code and see if it's abusing HZ or if there's anything
> obviously wrong there. If it's not then perhaps there really is a bug
> in the scheduler?

Is mmc slow or does it just break down? Slow would be expected, but
breaking down should be fixed in MMC, too...

--
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

2008-03-02 18:05:08

by Simon Huggins

[permalink] [raw]
Subject: Re: Scheduler broken? sdhci issues with scheduling

Hiya Pavel,

(please cc me)

On Sun, Mar 02, 2008 at 11:40:31AM +0100, Pavel Machek wrote:
> On Fri 2008-02-29 11:52:57, Simon Huggins wrote:
> > [ Please Cc me on replies ]
> > I had a bug with sdhci which Pierre Ossman looked at for me.
> > In the end essentially the fix was to use HZ=1000 and nothing else.
> > Pierre seemed to think that this was a bug in the scheduler.
> > I'm interested if someone who groks the scheduler could take a look at
> > the mmc/* code and see if it's abusing HZ or if there's anything
> > obviously wrong there. If it's not then perhaps there really is a bug
> > in the scheduler?
> Is mmc slow or does it just break down? Slow would be expected, but
> breaking down should be fixed in MMC, too...

It just doesn't work. It won't detect that a card has been inserted and
doesn't create the device nodes for it though it does log lots of data
where it tries.

Pierre explained further up this thread on why it tries to do what it
does but for some reason he's not seeing the expected behaviour from the
scheduler.

As a user I don't care where the bug is fixed :)

--
Simon Huggins \ "Ah. So you're a waffle man!" - Talkie Toaster
\
http://www.earth.li/~huggie/ htag.pl 0.0.22

2008-03-03 19:44:34

by Pierre Ossman

[permalink] [raw]
Subject: Re: Scheduler broken? sdhci issues with scheduling

On Sun, 2 Mar 2008 11:40:31 +0100
Pavel Machek <[email protected]> wrote:

>
> Is mmc slow or does it just break down? Slow would be expected, but
> breaking down should be fixed in MMC, too...
>

The problem is that the kernel sleeps _less_ than asked for, not more. No hardware can be expected to work if its needed delays aren't respected.

--
-- Pierre Ossman

Linux kernel, MMC maintainer http://www.kernel.org
PulseAudio, core developer http://pulseaudio.org
rdesktop, core developer http://www.rdesktop.org

2008-03-03 20:37:51

by Pavel Machek

[permalink] [raw]
Subject: Re: Scheduler broken? sdhci issues with scheduling

On Mon 2008-03-03 20:43:00, Pierre Ossman wrote:
> On Sun, 2 Mar 2008 11:40:31 +0100
> Pavel Machek <[email protected]> wrote:
>
> >
> > Is mmc slow or does it just break down? Slow would be expected, but
> > breaking down should be fixed in MMC, too...
> >
>
> The problem is that the kernel sleeps _less_ than asked for, not more. No hardware can be expected to work if its needed delays aren't respected.
>

Oops, that's bad. Can you perhaps create small testcase for
scheduler/timer people to play with?

Also... you may want to switch timesources. notsc fixed "wait for much
too long" for me.
Pavel
--
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

2008-03-03 21:07:12

by Pierre Ossman

[permalink] [raw]
Subject: Re: Scheduler broken? sdhci issues with scheduling

On Mon, 3 Mar 2008 21:38:08 +0100
Pavel Machek <[email protected]> wrote:

> On Mon 2008-03-03 20:43:00, Pierre Ossman wrote:
> > The problem is that the kernel sleeps _less_ than asked for, not more. No hardware can be expected to work if its needed delays aren't respected.
> >
>
> Oops, that's bad. Can you perhaps create small testcase for
> scheduler/timer people to play with?
>
> Also... you may want to switch timesources. notsc fixed "wait for much
> too long" for me.

I'm afraid I could never reproduce the problem. So it's Simon that will have to do the tests for now..


--
-- Pierre Ossman

Linux kernel, MMC maintainer http://www.kernel.org
PulseAudio, core developer http://pulseaudio.org
rdesktop, core developer http://www.rdesktop.org

2008-03-13 21:18:05

by Simon Huggins

[permalink] [raw]
Subject: Re: Scheduler broken? sdhci issues with scheduling

'ello Pierre

On Mon, Mar 03, 2008 at 10:05:30PM +0100, Pierre Ossman wrote:
> On Mon, 3 Mar 2008 21:38:08 +0100
> Pavel Machek <[email protected]> wrote:
> > On Mon 2008-03-03 20:43:00, Pierre Ossman wrote:
> > > The problem is that the kernel sleeps _less_ than asked for, not more. No hardware can be expected to work if its needed delays aren't respected.
> > Oops, that's bad. Can you perhaps create small testcase for
> > scheduler/timer people to play with?
> > Also... you may want to switch timesources. notsc fixed "wait for much
> > too long" for me.
> I'm afraid I could never reproduce the problem. So it's Simon that
> will have to do the tests for now..

I'm happy to test bits. What do I need to do to switch time sources for
instance?


Simon

--
... <ICv6> note pour plus tard... jamais paster un /quit dans un client irc
=)

2008-03-15 15:05:22

by Pierre Ossman

[permalink] [raw]
Subject: Re: Scheduler broken? sdhci issues with scheduling

On Thu, 13 Mar 2008 21:17:49 +0000
Simon Huggins <[email protected]> wrote:

>
> I'm happy to test bits. What do I need to do to switch time sources for
> instance?
>

I'm no expert, but you have the kernel parameter "clocksource" documented in Documentation/kernel-parameters. Also check the directory /sys/devices/system/clocksource for assorted runtime changes.

Rgds
--
-- Pierre Ossman

Linux kernel, MMC maintainer http://www.kernel.org
PulseAudio, core developer http://pulseaudio.org
rdesktop, core developer http://www.rdesktop.org