Commit 305e683 introduced a wrap bug that causes task scheduling to fail
after sched_clock() wrap. On a 1000 HZ system with 32bit jiffies, this
occurs after 49.7 days.
Bug was introduced in 2.6.35.12 and is still present in linux-2.6.35.y HEAD.
Symptoms include one task getting all available cpu time while others get
_none_. Setting niceness seems to make things even worse. Running this code
in a new process after wrap completely lock up user space, thus triggering a
watchdog reboot:
{ nice(1); while(1); }
To reproduce bug in reasonable time, one can up HZ. With 16000 HZ, bug occurs
after 3.1 days.
Modifying sched_clock() to wrap when jiffies does triggers bug after 5 mins.
The basic problem seems to be that rq->clock_task get stuck forever with a
really high value when rq->clock starts over from 0.
This fix solves that problem:
diff --git a/kernel/sched.c b/kernel/sched.c
index d40d662..883448f 100644
--- a/kernel/sched.c
+++ b/kernel/sched.c
@@ -657,6 +657,8 @@ inline void update_rq_clock(struct rq *rq)
if (!rq->skip_clock_update)
rq->clock = sched_clock_cpu(cpu_of(rq));
irq_time = irq_time_cpu(cpu);
+ if (rq->clock < rq->clock_task)
+ rq->clock_task = 0;
if (rq->clock - irq_time > rq->clock_task)
rq->clock_task = rq->clock - irq_time;
I can create a proper patch if the above is acceptable.
A more appropriate solution would perhaps be to pull some additional sched
commits into stable branch, like fe44d62 and friends. I don't know enough
about scheduler internals to tell.
All tests were performed on mips32 systems, but all systems with 32bit
jiffies should be affected.
/Thomas
On Sun, Jun 24, 2012 at 07:32:11PM +0200, Thomas Lange wrote:
> Commit 305e683 introduced a wrap bug that causes task scheduling to fail
> after sched_clock() wrap. On a 1000 HZ system with 32bit jiffies, this
> occurs after 49.7 days.
>
> Bug was introduced in 2.6.35.12 and is still present in linux-2.6.35.y HEAD.
Is this an issue in 3.5-rc3?
2.6.35-stable really isn't maintained anymore, I would suggest upgrading
to a more modern kernel version.
thanks,
greg k-h
On Sun, 2012-06-24 at 19:32 +0200, Thomas Lange wrote:
> Bug was introduced in 2.6.35.12 and is still present in linux-2.6.35.y HEAD.
Ok, so nobody cares about that.. what does something recent like 3.5-rc4
do?
If that's fixed, find the patch that fixes it. If not, we'll have a
look.
On Mon, 2012-06-25 at 11:45 +0200, Peter Zijlstra wrote:
> On Sun, 2012-06-24 at 19:32 +0200, Thomas Lange wrote:
> > Bug was introduced in 2.6.35.12 and is still present in linux-2.6.35.y HEAD.
>
> Ok, so nobody cares about that.. what does something recent like 3.5-rc4
> do?
>
> If that's fixed, find the patch that fixes it. If not, we'll have a
> look.
If anything, I think something like the below ought to cure things.
---
kernel/sched/clock.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/kernel/sched/clock.c b/kernel/sched/clock.c
index c685e31..a1a128a 100644
--- a/kernel/sched/clock.c
+++ b/kernel/sched/clock.c
@@ -74,7 +74,7 @@
*/
unsigned long long __attribute__((weak)) sched_clock(void)
{
- return (unsigned long long)(jiffies - INITIAL_JIFFIES)
+ return (unsigned long long)(get_jiffies_64() - INITIAL_JIFFIES)
* (NSEC_PER_SEC / HZ);
}
EXPORT_SYMBOL_GPL(sched_clock);
On 2012-06-25 12:38, Peter Zijlstra wrote:
> On Mon, 2012-06-25 at 11:45 +0200, Peter Zijlstra wrote:
>> On Sun, 2012-06-24 at 19:32 +0200, Thomas Lange wrote:
>>> Bug was introduced in 2.6.35.12 and is still present in linux-2.6.35.y HEAD.
>>
>> Ok, so nobody cares about that.. what does something recent like 3.5-rc4
>> do?
I haven't tested yet. I will try to find the time to do that.
>> If that's fixed, find the patch that fixes it. If not, we'll have a
>> look.
Commit fe44d62 removed the if-clause that caused the problem. Most likely, this
solved the problem.
>
>
> If anything, I think something like the below ought to cure things.
I agree. Either we make sure that sched_clock() never wraps or if it wraps,
that it wraps shortly after boot to catch problems early.
>
> ---
> kernel/sched/clock.c | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/kernel/sched/clock.c b/kernel/sched/clock.c
> index c685e31..a1a128a 100644
> --- a/kernel/sched/clock.c
> +++ b/kernel/sched/clock.c
> @@ -74,7 +74,7 @@
> */
> unsigned long long __attribute__((weak)) sched_clock(void)
> {
> - return (unsigned long long)(jiffies - INITIAL_JIFFIES)
> + return (unsigned long long)(get_jiffies_64() - INITIAL_JIFFIES)
> * (NSEC_PER_SEC / HZ);
> }
> EXPORT_SYMBOL_GPL(sched_clock);
/Thomas
On Mon, 2012-06-25 at 20:49 +0200, Thomas Lange wrote:
> Commit fe44d62 removed the if-clause that caused the problem. Most likely, this
> solved the problem.
Ah, probably. IIRC we also fixed the short wrap of those arm clocks
since.
> > If anything, I think something like the below ought to cure things.
>
> I agree. Either we make sure that sched_clock() never wraps or if it wraps,
> that it wraps shortly after boot to catch problems early.
Right, sched_clock() should only ever wrap on the full u64.