On three of the several paths in entry_64.S that call
do_notify_resume() on the way back to user mode, we fail to properly
check again for newly-arrived work that requires another call to
do_notify_resume() before going to user mode. These paths set the
mask to check only _TIF_NEED_RESCHED, but this is wrong. The other
paths that lead to do_notify_resume() do this correctly already, and
entry_32.S does it correctly in all cases.
All paths back to user mode have to check all the _TIF_WORK_MASK
flags at the last possible stage, with interrupts disabled.
Otherwise, we miss any flags (TIF_SIGPENDING for example) that were
set any time after we entered do_notify_resume(). More work flags
can be set (or left set) synchronously inside do_notify_resume(), as
TIF_SIGPENDING can be, or asynchronously by interrupts or other CPUs
(which then send an asynchronous interrupt).
There are many different scenarios that could hit this bug, most of
them races. The simplest one to demonstrate does not require any
race: when one signal has done handler setup at the check before
returning from a syscall, and there is another signal pending that
should be handled. The second signal's handler should interrupt the
first signal handler before it actually starts (so the interrupted PC
is still at the handler's entry point). Instead, it runs away until
the next kernel entry (next syscall, tick, etc).
This test behaves correctly on 32-bit kernels, and fails on 64-bit
(either 32-bit or 64-bit test binary). With this fix, it works.
#define _GNU_SOURCE
#include <stdio.h>
#include <signal.h>
#include <string.h>
#include <sys/ucontext.h>
#ifndef REG_RIP
#define REG_RIP REG_EIP
#endif
static sig_atomic_t hit1, hit2;
static void
handler (int sig, siginfo_t *info, void *ctx)
{
ucontext_t *uc = ctx;
if ((void *) uc->uc_mcontext.gregs[REG_RIP] == &handler)
{
if (sig == SIGUSR1)
hit1 = 1;
else
hit2 = 1;
}
printf ("%s at %#lx\n", strsignal (sig),
uc->uc_mcontext.gregs[REG_RIP]);
}
int
main (void)
{
struct sigaction sa;
sigset_t set;
sigemptyset (&sa.sa_mask);
sa.sa_flags = SA_SIGINFO;
sa.sa_sigaction = &handler;
if (sigaction (SIGUSR1, &sa, NULL)
|| sigaction (SIGUSR2, &sa, NULL))
return 2;
sigemptyset (&set);
sigaddset (&set, SIGUSR1);
sigaddset (&set, SIGUSR2);
if (sigprocmask (SIG_BLOCK, &set, NULL))
return 3;
printf ("main at %p, handler at %p\n", &main, &handler);
raise (SIGUSR1);
raise (SIGUSR2);
if (sigprocmask (SIG_UNBLOCK, &set, NULL))
return 4;
if (hit1 + hit2 == 1)
{
puts ("PASS");
return 0;
}
puts ("FAIL");
return 1;
}
Signed-off-by: Roland McGrath <[email protected]>
---
arch/x86/kernel/entry_64.S | 7 +++----
1 files changed, 3 insertions(+), 4 deletions(-)
diff --git a/arch/x86/kernel/entry_64.S b/arch/x86/kernel/entry_64.S
index 556a8df..968cf54 100644
--- a/arch/x86/kernel/entry_64.S
+++ b/arch/x86/kernel/entry_64.S
@@ -305,7 +305,7 @@ sysret_signal:
leaq -ARGOFFSET(%rsp),%rdi # &pt_regs -> arg1
xorl %esi,%esi # oldset -> arg2
call ptregscall_common
-1: movl $_TIF_NEED_RESCHED,%edi
+1: movl $_TIF_WORK_MASK,%edi
/* Use IRET because user could have changed frame. This
works because ptregscall_common has called FIXUP_TOP_OF_STACK. */
DISABLE_INTERRUPTS(CLBR_NONE)
@@ -393,7 +393,7 @@ int_signal:
movq %rsp,%rdi # &ptregs -> arg1
xorl %esi,%esi # oldset -> arg2
call do_notify_resume
-1: movl $_TIF_NEED_RESCHED,%edi
+1: movl $_TIF_WORK_MASK,%edi
int_restore_rest:
RESTORE_REST
DISABLE_INTERRUPTS(CLBR_NONE)
@@ -647,9 +647,8 @@ retint_signal:
RESTORE_REST
DISABLE_INTERRUPTS(CLBR_NONE)
TRACE_IRQS_OFF
- movl $_TIF_NEED_RESCHED,%edi
GET_THREAD_INFO(%rcx)
- jmp retint_check
+ jmp retint_with_reschedule
#ifdef CONFIG_PREEMPT
/* Returning to kernel space. Check if we need preemption */
On Thu, 10 Jul 2008, Roland McGrath wrote:
>
> There are many different scenarios that could hit this bug, most of
> them races. The simplest one to demonstrate does not require any
> race: when one signal has done handler setup at the check before
> returning from a syscall, and there is another signal pending that
> should be handled. The second signal's handler should interrupt the
> first signal handler before it actually starts (so the interrupted PC
> is still at the handler's entry point). Instead, it runs away until
> the next kernel entry (next syscall, tick, etc).
I have this dim memory of at least _some_ of this being on purpose.
If you look at old kernels (_really_ old ones - I think it's way before
even the historical git archive, but I didn't take a look), we used to set
up several stack frames at once, so that we'd nest the stack frames
completely.
In other words, the code in do_signal() used to literally be a loop,
something like
while ((signr = get_signal_to_deliver(&info, &ka, regs, NULL)) > 0) {
.. setup signal frame ..
(No, I don't think that's at all accurate of the actual code we used to
have - I just took the current do_signal() code as an example)
And that explicit loop was removed in order for us to have just a single
outstanding signal at a time. I forget the exact details why.
But if you really want that behaviour, then re-introducing the loop would
likely be the better approach (or should be combined), since I think you
effectively just re-introduced it (at a much bigger granularity).
Hmm.
Linus
> But if you really want that behaviour, then re-introducing the loop would
> likely be the better approach (or should be combined), since I think you
> effectively just re-introduced it (at a much bigger granularity).
I don't think so. Firstly, TIF_SIGPENDING is not the only flag in
question. There are other reasons to re-enter do_notify_resume().
If those are set during signal processing et al, they should take
effect before going back to user mode.
Second, there is always a race. Anywhere after the last time the
siglock was held inside do_signal(), there can be an interrupt that
sets TIF_SIGPENDING (or other _TIF_DO_NOTIFY_MASK flags). If you go
on to return to user mode, then it can be a long time before the new
signal is actually delivered (til the next tick).
It really is necessary to check all the _TIF_WORK_MASK flags with
interrupts disabled, last thing. I just don't see how any short
cuts here can be robust. It's simple, it's right, and it's what all
the other paths (and all other arch code I've ever noticed) do.
Since it's necessary to have robust checks in the final part of the
assembly code path anyway, and stacked signals are rare, there is
just no special reason to have a loop in do_signal(). In the common
case it every time retakes the siglock again when unnecessary, with
bad SMP performance effects; optimizing that with a signal_pending()
check just shows why it's simpler not to have a loop. Frankly, I'm
glad we don't have one because it would fix only the scenario that
has a test case that's real easy to write, and leave lying all the
much more hairy ones that will cause someone to spend days and days
some day later tearing his hair out.
Thanks,
Roland
On Thu, 10 Jul 2008, Roland McGrath wrote:
>
> > But if you really want that behaviour, then re-introducing the loop would
> > likely be the better approach (or should be combined), since I think you
> > effectively just re-introduced it (at a much bigger granularity).
>
> I don't think so. Firstly, TIF_SIGPENDING is not the only flag in
> question. There are other reasons to re-enter do_notify_resume().
> If those are set during signal processing et al, they should take
> effect before going back to user mode.
You're ignoring the background question - we expressly _stopped_ doing
this long ago. So the real issue was the ".. if you really .." part.
Do we really? What's the actual downside here?
Linus
On Thu, 10 Jul 2008, Linus Torvalds wrote:
>
> You're ignoring the background question - we expressly _stopped_ doing
> this long ago. So the real issue was the ".. if you really .." part.
Side note: the fact that 32-bit does it seems to imply it's ok, but it
bothers me that I know we changed the signal code at some point. Is the
64-bit code doing the exact _same_ things as 32-bit now?
Linus
> You're ignoring the background question - we expressly _stopped_ doing
> this long ago. So the real issue was the ".. if you really .." part.
>
> Do we really? What's the actual downside here?
I'm not convinced it was real "express". It was never expressed in a
comment or log entry. The change came in (pre-git) with:
[PATCH] x86-64 architecture specific sync for 2.5.8
and commit 10ffdbb8d605be88b148f127ec86452f1364d4f0 "cleaned up slightly"
making the other paths match, with no explanation on the subject.
i386 has never behaved this way, and still doesn't. I would doubt any
other arch ever has. (My fix makes x86_64 and i386 treatment of
_TIF_WORK_MASK and any related signal race issues identical.)
The behavior of the test case I posted is just demonstrably wrong. I
know you're never swayed by the fact that it has always been specified
and documented clearly to behave this way (in the case of multiple
pending signals like the test case). Since it always did on i386, it's
easy to expect that there may be all manner of applications lurking
around that have depended on the correct semantics in subtle (and
probably intermittent) ways their poor users and maintainers may never
figure out.
What really irks me about the thought of leaving this wrong is that we
have spent so much effort lately on establishing a simple rule that when
you set TIF_SIGPENDING it will be acted on. We did this after a lot of
painful time from a lot of people went into tracking down subtle weird
problems and races. So, KISS. Make a rule we can rely on, and then be
damn careful that we don't break the rule. That's been serving us well,
which is to say preventing it going from two people who can keep track
of what's going with signals on any given day, to zero. Now that rule
that kept life barely comprehensible is amended with, unless it's
already inside signals code or some nearby arch code, or it's a race,
or, yeah, I think that's all the cases, but check with--well, noone
really knows, so I don't know who you check with, sorry. You just can't
reason about the code if you don't maintain the invariants.
The "actual" downsides include numerous unknowns, and I always forget
not to be surprised when you aren't scared that we have no idea what-all
the code might actually do. The easy scenarios to think of off hand
have downsides like loss of timely signal delivery, where something can
chew 15ms of CPU after you killed it. If I try all day I can come up
with more specific cases and maybe even some with instantly terrible
outcomes. But I won't think of them all. The worst ones will come up
much later (or are already dogging someone unwitting now), when someone
else sinks lots of time and effort trying to figure out strange
misbehaviors in their systems.
Thanks,
Roland
On Thu, 10 Jul 2008, Roland McGrath wrote:
>
> The "actual" downsides include numerous unknowns, and I always forget
> not to be surprised when you aren't scared that we have no idea what-all
> the code might actually do.
You're just grand-standing here.
The fact is, this has been "broken" for a long long time. A lot of
applications (realistically, probably _all_ current 64-bit apps) have been
tested with the old behaviour.
The thing is, I'm simply not the _least_ nervous about behavior that we've
obviously had for a long time. And you shouldn't be either - nor should
you try to make up some bogus argument to the contrary.
You should be scared about the case that IS NOT tested - which is by
definition the new behaviour (admittedly only for 64-bit apps). Claiming
anything else is just stupid and dishonest.
So stop making up _bad_ arguments. You do have a good one: the fact that
x86-32 apparently works the way it works. But then you just irritate me
with your idiotic other ones about how things are "supposed" to work.
So just be honest: you didn't have any _actual_ downsides that you knew
about. Which was what I wondered about and asked about.
Those things make a big difference when I'm just about ready to cut a new
release. This is _clearly_ not a regression (unless you talk about things
from years and years ago), and apparently you don't have any actual code
that is broken - just your expectations.
In short, it doesn't smell like something I should apply under -rc9 rules.
[ And dammit, I wish more developers would ask themselves that question:
"Does this patch need to go in when we're very late in the -rc
sequence?". Or at least not then try to grand-stand and avoid the
question I asked. ]
Linus
> In short, it doesn't smell like something I should apply under -rc9 rules.
Sorry, never said it was. 2.6.27 is just fine for this.
Since it has indeed been that way so long, I guess I figured
it was obvious this was not a last-minute-urgent sort of thing.
I figured Ingo would put it in one of his trees to be chewed on.
Aside from the test case, which I did indeed contrive after I
found the bug, and the general concern about races, I was only
bitten by this because it fouled up things for changes I'm
working on for .27 or later.
Thanks,
Roland
On Thu, 10 Jul 2008, Roland McGrath wrote:
>
> i386 has never behaved this way, and still doesn't.
Btw, I told you differently, you didn't believe me. So I went back just
because I have an ego easily the size of Manhattan, and when you doubt me,
my ego is hurt.
i386 _has_ very much behaved this way. The system call code literally used
to do
..
cmpl $0,sigpending(%ebx)
jne signal_return
..
and then 'signal_return' case would handle one signal and then return to
user space:
..
call SYMBOL_NAME(do_signal)
#ifdef CONFIG_SMP
GET_CURRENT(%ebx)
#endif
cli
CHECK_SOFTIRQ
je restore_all
..
ie it would call "restore_all" without re-checking signals. I wasn't
hallucinating.
So that's really old code.
The _really_ ancient code (like a decade ago or more) used to actually
loop in do_signal() generating multiple signal stacks in one go. Then, a
long long time ago, that was actually removed, and we expressly only tried
to queue one signal at a time. I do not remember why, but it was before
2.4.0. Probably _long_ before, but it's so long ago my memory is
unreliable.
The "queue multiple signals" code that you claim has always been there is
in fact pretty recent. For x86-32, it was a thing introduced in commit
c3ff8ec31c1249d268cd11390649768a12bec1b9: by _you_, just three years ago,
in 2.6.14, in regular git times.
So the x86-64 behaviour actually matches what the x86-32 behavior was at
the time before things split.
And I'd also like to point out another commit, namely "[PATCH] fix broken
vm86 interrupt/signal handling" (4031ff388138b58e5cd472dccce38828bcb8c706)
that fixed a bug with an endless loop you had introduced in that original
x86-32 code when you fixed this exact same issue back when.
Heh. That's the kind of thing that worries me.
Linus
On Thu, 10 Jul 2008, Linus Torvalds wrote:
>
> So the x86-64 behaviour actually matches what the x86-32 behavior was at
> the time before things split.
>
> And I'd also like to point out another commit, namely "[PATCH] fix broken
> vm86 interrupt/signal handling" (4031ff388138b58e5cd472dccce38828bcb8c706)
> that fixed a bug with an endless loop you had introduced in that original
> x86-32 code when you fixed this exact same issue back when.
>
> Heh. That's the kind of thing that worries me.
That said, seeing the full history of this same thing on the x86-32 side
actually makes me _much_ happier about the patch. Because now I can tell
when we did it, and what problems it seems to have caused (answer:
"apparently none that are possibly relevant on x86-64").
IOW, just seeing the fact that we used to behave the same, and then
changed it on the 32-bit side (with no apparent big issues) and did it
without updating the 64-bit side, makes me just much happier about your
patch. Just because I know know the background to why this was x86-64
only.
So now I'm considering just putting it in before the 2.6.26 release after
all ;)
Linus
On Thu, 10 Jul 2008, Linus Torvalds wrote:
>
> So now I'm considering just putting it in before the 2.6.26 release after
> all ;)
.. and having looked at the code, and thought about it some more, I'm
definitely off the patch again.
The reason is actually exactly the same bug that showed up when you did
this for x86-32 three years ago, and that may in fact still be lurking.
The endless loop of "call do_notify_resume until all the work flags are
zero" is very fragile: it will immediately cause a hard lockup if there is
some circumstance where do_notify_resume will not clear the flag.
And when it comes to signals, there are several cases that can cause
TIF_SIGPENDING to not be cleared:
- confusion about user/kernel mode, where "do_signal()" will return
without doing anything at all if we're in user mode.
This was the bug we hit back in 2005 with a out-of-tree kernel-based
vm86 model (which hopefully has since died a painful death).
- get_signal_to_deliver() returning and not handling the signal.
dequeue_signal() will do this for that collect_signal() case and for
the whole DRI notifier thing. The DRI notifier() case actually clears
TIF_SIGPENDING, but then we do "recalc_sigpending()" in the caller, so
it might get set again.
I do hate that code (I know you do too), and the code _should_ block
the signal that gets ignored (so recalc_sigpending() should keep it
cleared), but it's not entirely obvious. Maybe it gets into an endless
loop of calling the notifier if this case ever triggers?
- recalc_sigpending() expressly does not clear the TIF_SIGPENDING flag if
we hit the "freezing(current)" case. So TIF_SIGPENDING stays set for
freezing() processes. I think (and *hope*) they all get caught by other
means anyway in that whole do_notify_resume() loop, but this is another
of those "the freezer code is insane, I'm not going to try to think it
through" cases.
In short, I think your patch is fine now, but I'm also nervous enough
about it that I'm not going to apply it. Any bugs it could expose look
very unlikely, and if they exist they are probably bugs on 32-bit as we
speak, but call me a worry-wart.
Linus
On Thu, 10 Jul 2008, Linus Torvalds wrote:
>
> - confusion about user/kernel mode, where "do_signal()" will return
> without doing anything at all if we're in user mode.
s/in/not returning to/
Duh.
Linus
* Roland McGrath <[email protected]> wrote:
> On three of the several paths in entry_64.S that call
> do_notify_resume() on the way back to user mode, we fail to properly
> check again for newly-arrived work that requires another call to
> do_notify_resume() before going to user mode. These paths set the
> mask to check only _TIF_NEED_RESCHED, but this is wrong. The other
> paths that lead to do_notify_resume() do this correctly already, and
> entry_32.S does it correctly in all cases.
nice find! Roland, is this related to the thread started by Elias
Oltmanns yesterday:
http://lkml.org/lkml/2008/7/10/57
which is also related to the thread started by Edwin T?r?k:
http://lkml.org/lkml/2008/6/28/50
? The weight of complains seems to be on the 64-bit side, in those
threads 32-bit systems seem to be implicated as well by latencytop.
Perhaps the 64-bit delays are related to the bug you fix and we could
chalk up the 32-bit delays to IO delays?
more people Cc:-ed: if your delays happen on 64-bit x86 systems - does
Roland's patch (also repeated below) fix those delay issues?
Ingo
------------------------->
Subject: x86: fix delayed signals
From: Roland McGrath <[email protected]>
On three of the several paths in entry_64.S that call
do_notify_resume() on the way back to user mode, we fail to properly
check again for newly-arrived work that requires another call to
do_notify_resume() before going to user mode. These paths set the
mask to check only _TIF_NEED_RESCHED, but this is wrong. The other
paths that lead to do_notify_resume() do this correctly already, and
entry_32.S does it correctly in all cases.
All paths back to user mode have to check all the _TIF_WORK_MASK
flags at the last possible stage, with interrupts disabled.
Otherwise, we miss any flags (TIF_SIGPENDING for example) that were
set any time after we entered do_notify_resume(). More work flags
can be set (or left set) synchronously inside do_notify_resume(), as
TIF_SIGPENDING can be, or asynchronously by interrupts or other CPUs
(which then send an asynchronous interrupt).
There are many different scenarios that could hit this bug, most of
them races. The simplest one to demonstrate does not require any
race: when one signal has done handler setup at the check before
returning from a syscall, and there is another signal pending that
should be handled. The second signal's handler should interrupt the
first signal handler before it actually starts (so the interrupted PC
is still at the handler's entry point). Instead, it runs away until
the next kernel entry (next syscall, tick, etc).
This test behaves correctly on 32-bit kernels, and fails on 64-bit
(either 32-bit or 64-bit test binary). With this fix, it works.
#define _GNU_SOURCE
#include <stdio.h>
#include <signal.h>
#include <string.h>
#include <sys/ucontext.h>
#ifndef REG_RIP
#define REG_RIP REG_EIP
#endif
static sig_atomic_t hit1, hit2;
static void
handler (int sig, siginfo_t *info, void *ctx)
{
ucontext_t *uc = ctx;
if ((void *) uc->uc_mcontext.gregs[REG_RIP] == &handler)
{
if (sig == SIGUSR1)
hit1 = 1;
else
hit2 = 1;
}
printf ("%s at %#lx\n", strsignal (sig),
uc->uc_mcontext.gregs[REG_RIP]);
}
int
main (void)
{
struct sigaction sa;
sigset_t set;
sigemptyset (&sa.sa_mask);
sa.sa_flags = SA_SIGINFO;
sa.sa_sigaction = &handler;
if (sigaction (SIGUSR1, &sa, NULL)
|| sigaction (SIGUSR2, &sa, NULL))
return 2;
sigemptyset (&set);
sigaddset (&set, SIGUSR1);
sigaddset (&set, SIGUSR2);
if (sigprocmask (SIG_BLOCK, &set, NULL))
return 3;
printf ("main at %p, handler at %p\n", &main, &handler);
raise (SIGUSR1);
raise (SIGUSR2);
if (sigprocmask (SIG_UNBLOCK, &set, NULL))
return 4;
if (hit1 + hit2 == 1)
{
puts ("PASS");
return 0;
}
puts ("FAIL");
return 1;
}
Signed-off-by: Roland McGrath <[email protected]>
---
arch/x86/kernel/entry_64.S | 7 +++----
1 files changed, 3 insertions(+), 4 deletions(-)
diff --git a/arch/x86/kernel/entry_64.S b/arch/x86/kernel/entry_64.S
index 556a8df..968cf54 100644
--- a/arch/x86/kernel/entry_64.S
+++ b/arch/x86/kernel/entry_64.S
@@ -305,7 +305,7 @@ sysret_signal:
leaq -ARGOFFSET(%rsp),%rdi # &pt_regs -> arg1
xorl %esi,%esi # oldset -> arg2
call ptregscall_common
-1: movl $_TIF_NEED_RESCHED,%edi
+1: movl $_TIF_WORK_MASK,%edi
/* Use IRET because user could have changed frame. This
works because ptregscall_common has called FIXUP_TOP_OF_STACK. */
DISABLE_INTERRUPTS(CLBR_NONE)
@@ -393,7 +393,7 @@ int_signal:
movq %rsp,%rdi # &ptregs -> arg1
xorl %esi,%esi # oldset -> arg2
call do_notify_resume
-1: movl $_TIF_NEED_RESCHED,%edi
+1: movl $_TIF_WORK_MASK,%edi
int_restore_rest:
RESTORE_REST
DISABLE_INTERRUPTS(CLBR_NONE)
@@ -647,9 +647,8 @@ retint_signal:
RESTORE_REST
DISABLE_INTERRUPTS(CLBR_NONE)
TRACE_IRQS_OFF
- movl $_TIF_NEED_RESCHED,%edi
GET_THREAD_INFO(%rcx)
- jmp retint_check
+ jmp retint_with_reschedule
#ifdef CONFIG_PREEMPT
/* Returning to kernel space. Check if we need preemption */
On 2008-07-11 08:46, Ingo Molnar wrote:
> * Roland McGrath <[email protected]> wrote:
>
>
>> On three of the several paths in entry_64.S that call
>> do_notify_resume() on the way back to user mode, we fail to properly
>> check again for newly-arrived work that requires another call to
>> do_notify_resume() before going to user mode. These paths set the
>> mask to check only _TIF_NEED_RESCHED, but this is wrong. The other
>> paths that lead to do_notify_resume() do this correctly already, and
>> entry_32.S does it correctly in all cases.
>>
>
> nice find! Roland, is this related to the thread started by Elias
> Oltmanns yesterday:
>
> http://lkml.org/lkml/2008/7/10/57
>
I will try to get a latency trace using the debug features in
linux-next, but first
I have to bisect a boot failure in linux-next.
> which is also related to the thread started by Edwin T?r?k:
>
> http://lkml.org/lkml/2008/6/28/50
>
> ? The weight of complains seems to be on the 64-bit side, in those
> threads 32-bit systems seem to be implicated as well by latencytop.
> Perhaps the 64-bit delays are related to the bug you fix and we could
> chalk up the 32-bit delays to IO delays?
>
> more people Cc:-ed: if your delays happen on 64-bit x86 systems - does
> Roland's patch (also repeated below) fix those delay issues?
>
>
Thanks for CC-ing me.
I will give the patch a try.
Best regards,
--Edwin
Ingo Molnar <[email protected]> wrote:
> * Roland McGrath <[email protected]> wrote:
>
>> On three of the several paths in entry_64.S that call
>> do_notify_resume() on the way back to user mode, we fail to properly
>> check again for newly-arrived work that requires another call to
>> do_notify_resume() before going to user mode. These paths set the
>> mask to check only _TIF_NEED_RESCHED, but this is wrong. The other
>> paths that lead to do_notify_resume() do this correctly already, and
>> entry_32.S does it correctly in all cases.
>
> nice find! Roland, is this related to the thread started by Elias
> Oltmanns yesterday:
>
> http://lkml.org/lkml/2008/7/10/57
>
> which is also related to the thread started by Edwin T?r?k:
>
> http://lkml.org/lkml/2008/6/28/50
>
> ? The weight of complains seems to be on the 64-bit side, in those
> threads 32-bit systems seem to be implicated as well by latencytop.
> Perhaps the 64-bit delays are related to the bug you fix and we could
> chalk up the 32-bit delays to IO delays?
Since I'm using a 32-bit machine, this patch isn't going to solve my
particular problem. Still, I'm glad to hear that someone is looking at
the assembly bits related to signal handling right now because that's
precisely the part where I gave up to figure out for myself how signal
handling works in the kernel.
As for delayed signal handling on 32-bit platforms due to heavy disk
I/O, I'm still reluctant to accept that signals should be delayed that
much even though everything else that is not related to disk I/O works
perfectly well. Now that I've seen some documentation for ftrace, I'll
give it a go and try produce some helpful data. However, I'm rather busy
right now and may not be able to do so before next week.
Regards,
Elias
On Fri, 11 Jul 2008, Ingo Molnar wrote:
>
> nice find! Roland, is this related to the thread started by Elias
> Oltmanns yesterday:
>
> http://lkml.org/lkml/2008/7/10/57
No.
First off, the delayed sending of signals isn't actually delayed that
much. It's the next kernel entry at the most - certainly *not* across
scheduling points, which the fact that ^Z shows up in the xterm says
happen.
Secondly, it seems to happen on x86-32 too, and it is claimed to be a
regression. Neither of which would be true for this case.
Thirdly, there seems to be no other signals involved (ie it's a single
signal).
I think the IO interactivity report is simply because of IO scheduling
and/or paging out the shell too aggressively. No signals will be delivered
while the process in in 'D' state (the new 'killable' thing being an
exception in _some_ cases, but that's literally just for killing signals,
not backgrounding).
So to look at the original report:
"By sprinkling some printk()s all over the place, I've managed to
establish the following sequence of events taking place in the event of
delayed signal handling as described above:
The first Ctrl+Z event enqueues a SIGTSTP signal which eventually
results in a call to kick_process(). For some reason though, the signal
isn't handled straight away but remains on the queue for some time."
the thing is, kick_process() only helps if the other thread is _running_
on the other CPU. If it's actively waiting for disk, it's a no-op: the
signal handling code expects that the scheduler will either wake it up in
"wake_up_state()" (which won't happen if it is in 'D' state, of course!)
or that the process will just handle it in its own time when it wakes up
when IO completes.
So if things have gotten worse latency, it's most likely simply because
our IO has worse latency - probably because of having more requests
outstanding, or because of unlucky/bad IO scheduler behaviour. The first
thing to do (because it's fairly easy) is to start off testing different
IO schedulers, but also see if there are some cases where we should try to
return early.
In the particular case of Edwin T?r?k, his latency problem seems to b with
"find". That's interesting, because it's one of the cases where we *could*
easily improve on latency, by making "readdir()" return early both for
killable signals, but also for regular signals when we've filled the
buffer partially (because unlike a "read()" system call, we don't promise
to fill the whole buffer _anyway_).
It may be, for example, that the increased latency isn't actually because
the *kernel* has increased latencies at all, but perhaps 'find' uses a
much bigger buffer for it's readdir() code? Anyway, _if_ it's readdir()
that has high latency, the appended patch might make a difference. I think
it's probably a good idea regardless..
But it would be really good to have the thing bisected regardless of what
it is.
Linus
---
This makes 'readdir()' return early if it has a signal pending and it has
filled at least one entry (the -EINTR will never be passed on to user
space: the readdir() system call will return the length of the filled-in
buffer)
fs/readdir.c | 6 ++++++
1 files changed, 6 insertions(+), 0 deletions(-)
diff --git a/fs/readdir.c b/fs/readdir.c
index 4e026e5..ec8c192 100644
--- a/fs/readdir.c
+++ b/fs/readdir.c
@@ -159,6 +159,9 @@ static int filldir(void * __buf, const char * name, int namlen, loff_t offset,
return -EOVERFLOW;
dirent = buf->previous;
if (dirent) {
+ /* Only check signals if we have filled at least one entry! */
+ if (signal_pending(current))
+ return -EINTR;
if (__put_user(offset, &dirent->d_off))
goto efault;
}
@@ -241,6 +244,9 @@ static int filldir64(void * __buf, const char * name, int namlen, loff_t offset,
return -EINVAL;
dirent = buf->previous;
if (dirent) {
+ /* Only check signals if we have filled at least one entry! */
+ if (signal_pending(current))
+ return -EINTR;
if (__put_user(offset, &dirent->d_off))
goto efault;
}
> + /* Only check signals if we have filled at least one entry! */
> + if (signal_pending(current))
> + return -EINTR;
Shouldn't it return the short count?
(And if not, then probably should be -ERESTARTSYS.)
Thanks,
Roland
On Fri, 11 Jul 2008, Linus Torvalds wrote:
>
> But it would be really good to have the thing bisected regardless of what
> it is.
Btw, did any of the impacted people test -rc9? Edwin's report is about
-rc2 and -rc8, and one of the things we fixed since -rc8 is that incorrect
and unintentional nr_zones zeroing that effectively disabled kswapd - and
made everybody do synchronous memory freeing when they wanted to allocate
more memory.. That can play havoc with any interactive stuff.
Linus
On Fri, 11 Jul 2008, Linus Torvalds wrote:
>
> So making the callback just return an error is the right thing to do -
> because the callers will then fix up the file offset in the last directory
> entry and the final return value.
That said, I didn't actually _test_ my patch. That's what users are for!
Linus
On Fri, 11 Jul 2008, Roland McGrath wrote:
>
> > + /* Only check signals if we have filled at least one entry! */
> > + if (signal_pending(current))
> > + return -EINTR;
>
> Shouldn't it return the short count?
> (And if not, then probably should be -ERESTARTSYS.)
readdir (getdents) is a complex system call, and the path is
sys_getdents64() ->
vfs_readdir (with 'filldir64' as a callback) ->
low-level filesystem 'readdir' function ->
filldir64 callback
and what happens is that when that 'filldir64()' callback returns an
error, the low-level filesystem just stops and returns with a success
(because everything worked for _it_ - it was the callback that errored
out).
The 'sys_getdents64()' then does:
..
lastdirent = buf.previous;
if (lastdirent) {
typeof(lastdirent->d_off) d_off = file->f_pos;
error = -EFAULT;
if (__put_user(d_off, &lastdirent->d_off))
goto out_putf;
error = count - buf.count;
}
..
ie we'll do that "return partial buffer size" at that level, not deep in
the callback.
So making the callback just return an error is the right thing to do -
because the callers will then fix up the file offset in the last directory
entry and the final return value.
Linus
On Fri, 11 Jul 2008, Linus Torvalds wrote:
>
> Btw, did any of the impacted people test -rc9? Edwin's report is about
> -rc2 and -rc8, and one of the things we fixed since -rc8 is that incorrect
> and unintentional nr_zones zeroing that effectively disabled kswapd - and
> made everybody do synchronous memory freeing when they wanted to allocate
> more memory.. That can play havoc with any interactive stuff.
Hmm. Edwin's latencytop output includes this (ignoring the _very_ top
entries that are all either CD-ROM media change tests or are interruptible
pipe/select things) at the top:
21 10264428 915514 get_request_wait __make_request generic_make_request
submit_bio xfs_submit_ioend_bio xfs_submit_ioend
xfs_page_state_convert xfs_vm_writepage __writepage
write_cache_pages generic_writepages xfs_vm_writepages
26 3369263 2260529 down xfs_buf_iowait xfs_buf_iostart xfs_buf_read_flags
xfs_trans_read_buf xfs_imap_to_bp xfs_itobp xfs_iread
xfs_iget_core xfs_iget xfs_lookup xfs_vn_lookup 1 17888 17888 down
xfs_buf_iowait xfs_buf_iostart xfs_buf_read_flags
xfs_trans_read_buf xfs_da_do_buf xfs_da_read_buf
xfs_dir2_block_getdents xfs_readdir xfs_file_readdir vfs_readdir
sys_getdents64
..
which says that (a) yes, readdir() is part of the problematic paths, so my
patch may make a difference but also (b) we also have so many writeback
IO's in flight that the write request queue is totally full, and the
writing side is simply waiting for the queue to empty.
I guess (b) isn't a surprise (considering the load), but it does explain
why any IO read will be very much delayed. If the IO scheduler (or the
disk itself - tagged commands etc) doesn't prioritize reads and
effectively always put them ahead of the queue, you can get very very long
latencies just because you have to wait for lots of writes to complete
first.
Linus
On Fri, 11 Jul 2008, Linus Torvalds wrote:
>
> Btw, did any of the impacted people test -rc9? Edwin's report is about
> -rc2 and -rc8, and one of the things we fixed since -rc8 is that incorrect
> and unintentional nr_zones zeroing that effectively disabled kswapd - and
> made everybody do synchronous memory freeing when they wanted to allocate
> more memory.. That can play havoc with any interactive stuff.
No, after testing more, I think Edwin is right.
There _does_ seem to be something wrong in signal handling when the signal
happens during 'D' state (although I guess it could also be an artifact of
other scheduling activity). I just put my machine under heavy read load by
having a background process that does
while : ; do echo 3 > /proc/sys/vm/drop_caches ; sleep 5; done
to flush the caches all the time, and then doing endless loops of "git
grep" and other things, and I can definitely re-create the bad latency.
The best example is doing just
[torvalds@woody ~]$ ls -l /usr/bin
^Ctotal 367720
-rwxr-xr-x 1 root root 43648 2008-07-04 09:35 [
[torvalds@woody ~]$
and there are several seconds between the '^C' and the printout of the two
lines (after which the 'ls' finally dies).
Perhaps more interestingly: when this happens (it doesn't happen all the
time), it will _always_ die at the first two lines of printout for me, on
two different machines: ie I *always* get that exact pattern of those two
lines printed out (except when it reacts immediately, or when I press ^C
so late that it has printed out much more).
And it's not some single system call latency that is long. I checked with
strace, and while the IO load is high and causes the whole 'ls' to take a
long time (up to 10 seconds - /usr/bin is *big*), each individual system
call never takes very long. The longest system call latency I saw was 0.24
seconds, which was for some very unlucky lstat() calls. Most of them were
in the millisecond range, even though they obviously had to do IO.
More tellingly, those two writes are simply not even done with a single
system call. The 'strace' for an 'ls -l' shows
...
write(1, "total 367720\n", 13) = 13 <0.000085>
open("/etc/localtime", O_RDONLY) = 3 <0.000016>
fstat(3, {st_mode=S_IFREG|0644, st_size=2819, ...}) = 0 <0.000004>
fstat(3, {st_mode=S_IFREG|0644, st_size=2819, ...}) = 0 <0.000004>
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f6d996dd000 <0.000006>
read(3, "TZif2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\4\0\0\0\4\0\0\0\0"..., 4096) = 2819 <0.041416>
lseek(3, -1802, SEEK_CUR) = 1017 <0.000005>
read(3, "TZif2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\5\0\0\0\5\0\0\0\0"..., 4096) = 1802 <0.000006>
close(3) = 0 <0.000006>
munmap(0x7f6d996dd000, 4096) = 0 <0.000014>
write(1, "-rwxr-xr-x 1 root root 436"..., 54) = 54 <0.000008>
...
so there is actually *many* system calls in there between the header line
that does the 'total' and the first line of actual real output, but
despite that, it reacts to the ^C consistently after the second line.
Very odd.
And the problem definitely happens on x86-32 too.
It also happens on the text console, so this is not some bad interaction
with X and pty's and xterm or gnome-terminal or something like that. At
first I thought "no way can that actually happen, I think the terminal is
broken and only sends the signal after it has seen output", but no, it
happens for me in text-mode too.
Oleg added to the Cc as the master of signal handling code.
Linus
On Fri, 11 Jul 2008 11:31:26 -0700 (PDT)
Linus Torvalds <[email protected]> wrote:
>
>
> On Fri, 11 Jul 2008, Linus Torvalds wrote:
> >
> > Btw, did any of the impacted people test -rc9? Edwin's report is
> > about -rc2 and -rc8, and one of the things we fixed since -rc8 is
> > that incorrect and unintentional nr_zones zeroing that effectively
> > disabled kswapd - and made everybody do synchronous memory freeing
> > when they wanted to allocate more memory.. That can play havoc with
> > any interactive stuff.
>
> Hmm. Edwin's latencytop output includes this (ignoring the _very_ top
> entries that are all either CD-ROM media change tests or are
> interruptible pipe/select things) at the top:
>
> 21 10264428 915514 get_request_wait __make_request
> generic_make_request submit_bio xfs_submit_ioend_bio xfs_submit_ioend
> xfs_page_state_convert xfs_vm_writepage __writepage
> write_cache_pages generic_writepages xfs_vm_writepages
argh. well I guess this might be useful for this case, but normally
latencytop gives you much more humanly readable data... maybe Edwin
forgot to do "make install" :-(
On Fri, 11 Jul 2008, Linus Torvalds wrote:
>
> No, after testing more, I think Edwin is right.
Naah. It is true that there is a huge delay in doing
ls -l /usr/bin
and then pressing ^C, but after having hit my head against this for a
while, I realized that it has nothing to do with the kernel.
Doing an "strace ls" showed that ls doesn't play any games with signals
etc, which fooled me into looking for a kernel reason.
HOWEVER, it looks like at least fedora does a
alias ls='ls --color=auto'
and it turns out that if you do that --color=auto, then ls will indeed
catch all normal signals and set a "please stop now" flag, instead of
dying immediately. The reason is probably to avoiding leaving the terminal
with some odd color if interrupted in an inconvenient place.
So I was chasing this latency thing totally unnecessarily. It's in user
space (or at least _my_ particular issue was).
Other user cases may obviously be elsewhere.
Linus
On 2008-07-12 02:22, Linus Torvalds wrote:
> On Fri, 11 Jul 2008, Linus Torvalds wrote:
>
>> No, after testing more, I think Edwin is right.
>>
>
> Naah. It is true that there is a huge delay in doing
>
> ls -l /usr/bin
>
> and then pressing ^C, but after having hit my head against this for a
> while, I realized that it has nothing to do with the kernel.
>
> Doing an "strace ls" showed that ls doesn't play any games with signals
> etc, which fooled me into looking for a kernel reason.
>
> HOWEVER, it looks like at least fedora does a
>
> alias ls='ls --color=auto'
>
> and it turns out that if you do that --color=auto, then ls will indeed
> catch all normal signals and set a "please stop now" flag, instead of
> dying immediately. The reason is probably to avoiding leaving the terminal
> with some odd color if interrupted in an inconvenient place.
>
> So I was chasing this latency thing totally unnecessarily. It's in user
> space (or at least _my_ particular issue was).
>
> Other user cases may obviously be elsewhere.
Thanks for looking into the problem.
I did some more testing, and here are my results so far (with my
testcase of running dd and find):
On my 32-bit box (slow disks, SMP, XFS filesystem) 2.6.26-rc9 behaves
the same as 2.6.26-rc8, I can reliably reproduce a 2-3 second latency
[1] between pressing ^C the first time, and the shell returning (on the
text console too).
Using ftrace available from tip/master, I see up to 3 seconds of delay
between kill_pgrp and detach_pid (and during that time I can press ^C
again, leading to 2-3 kill_pgrp calls)
On my 64-bit box (2 disks in raid-0, UP, reiserfs filesystem) 2.6.25 and
2.6.26-rc9 behave the same, and most of the time (10-20 times in a row)
find responds to ^C instantly.
However in _some_ cases find doesn't respond to ^C for a very long time
(~30 seconds), and when this happens I can't do anything else but switch
consoles,
starting another process (latencytop -d) hangs, and so does any other
external command.
I have tried to get a latencytop output from the 64-bit box, but I
couldn't reproduce the long latency when latencytop was running.
I have reproduced the latency after launching/killing find 20+ times,
but after I started latencytop I couldn't reproduce.
But I can no longer reproduce it now, even if I stop latencytop (and
latencytop sysctl _is_ 0).
I haven't yet tried ftrace on this box, and neither did I try Roland's
patch yet. I will try that now, and hopefuly come back with some numbers
shortly.
[1]
I switch to text console, then I run this script in the background:
cd /debug/tracing
echo kill_pgrp detach_pid\
>set_ftrace_filter
echo ftrace >current_tracer
echo 1 >tracing_enabled
sleep 60
echo 0 >tracing_enabled
cat trace >/tmp/trace.out
cat latency_trace >/tmp/latencytrace.out
Then I run this script in the background, and wait until in shows
'Waiting for dd to finish':
dd if=/dev/zero of=xt bs=100M count=4&
dd if=/dev/zero of=yt bs=100M count=4&
rm xt
rm yt
echo "Waiting for dd to finish..."
wait %1 %2
echo "Done"
Then I start 'find / >/dev/null 2>/dev/null, and type ^C after about a
second'.
trace and latencytrace outputs below (for the 32-bit box):
# tracer: ftrace
#
# TASK-PID CPU# TIMESTAMP FUNCTION
# | | | | |
events/0-9 [00] 450.146629: kill_pgrp <-n_tty_receive_buf
bash-5821 [00] 450.386653: detach_pid <-release_task
events/0-9 [00] 451.631327: kill_pgrp <-n_tty_receive_buf
events/0-9 [00] 452.394820: kill_pgrp <-n_tty_receive_buf
events/0-9 [00] 453.127932: kill_pgrp <-n_tty_receive_buf
bash-5821 [00] 453.143686: detach_pid <-release_task
bash-5821 [00] 453.143686: detach_pid <-release_task
bash-5821 [00] 453.143686: detach_pid <-release_task
^^^^^^^^^^^^ I had to press ^C 3 times here, and there is a ~2 sec delay
events/0-9 [00] 454.164452: kill_pgrp <-n_tty_receive_buf
events/0-9 [00] 455.027774: kill_pgrp <-n_tty_receive_buf
events/0-9 [00] 455.762679: kill_pgrp <-n_tty_receive_buf
bash-5821 [00] 455.770332: detach_pid <-release_task
bash-5821 [00] 455.770332: detach_pid <-release_task
bash-5821 [00] 455.770332: detach_pid <-release_task
^^^^^^^ again, a 1 second delay
events/0-9 [00] 456.718620: kill_pgrp <-n_tty_receive_buf
bash-5821 [00] 456.862658: detach_pid <-release_task
bash-5821 [00] 456.862658: detach_pid <-release_task
bash-5821 [00] 456.862658: detach_pid <-release_task
events/0-9 [00] 457.880204: kill_pgrp <-n_tty_receive_buf
bash-5821 [00] 457.908467: detach_pid <-release_task
bash-5821 [00] 457.908467: detach_pid <-release_task
bash-5821 [00] 457.908467: detach_pid <-release_task
events/0-9 [00] 459.238903: kill_pgrp <-n_tty_receive_buf
bash-5821 [00] 460.081542: detach_pid <-release_task
bash-5821 [00] 460.081542: detach_pid <-release_task
bash-5821 [00] 460.081542: detach_pid <-release_task
events/0-9 [00] 460.083701: kill_pgrp <-n_tty_receive_buf
events/0-9 [00] 462.246407: kill_pgrp <-n_tty_receive_buf
events/0-9 [00] 462.975581: kill_pgrp <-n_tty_receive_buf
bash-5821 [00] 463.583139: detach_pid <-release_task
^^^^^^^^^^^^^^^^^^^ 3.5 second delay here
bash-5821 [00] 463.583139: detach_pid <-release_task
bash-5821 [00] 463.583139: detach_pid <-release_task
events/0-9 [00] 463.699181: kill_pgrp <-n_tty_receive_buf
events/0-9 [00] 464.735022: kill_pgrp <-n_tty_receive_buf
bash-5821 [00] 465.094654: detach_pid <-release_task
bash-5821 [00] 465.094654: detach_pid <-release_task
bash-5821 [00] 465.094654: detach_pid <-release_task
events/0-9 [00] 466.537663: kill_pgrp <-n_tty_receive_buf
bash-5821 [00] 466.945909: detach_pid <-release_task
# tracer: ftrace
#
ftrace latency trace v1.1.5 on 2.6.26-rc9-tip-01669-g639a570
--------------------------------------------------------------------
latency: 0 us, #210/210, CPU#0 | (M:unknown VP:0, KP:0, SP:0 HP:0 #P:2)
-----------------
| task: -0 (uid:0 nice:0 policy:0 rt_prio:0)
-----------------
# _------=> CPU#
# / _-----=> irqs-off
# | / _----=> need-resched
# || / _---=> hardirq/softirq
# ||| / _--=> preempt-depth
# |||| /
# ||||| delay
# cmd pid ||||| time | caller
# \ / ||||| \ | /
kstopmac-6225 0d..3 10us : detach_pid (release_task)
kstopmac-6225 0d..3 10us : detach_pid (release_task)
kstopmac-6225 0d..3 10us : detach_pid (release_task)
init-1 0d..3 10us : detach_pid (release_task)
init-1 0d..3 10us : detach_pid (release_task)
init-1 0d..3 10us!: detach_pid (release_task)
sh-6228 1d..3 3944272us+: detach_pid (release_task)
sh-6228 1d..3 3944273us : detach_pid (release_task)
sh-6228 1d..3 3944274us!: detach_pid (release_task)
sh-6228 1d..3 3945744us+: detach_pid (release_task)
sh-6228 1d..3 3945745us : detach_pid (release_task)
sh-6228 1d..3 3945746us!: detach_pid (release_task)
hal-syst-6233 0d..3 4156674us : detach_pid (release_task)
hal-syst-6233 0d..3 4156674us : detach_pid (release_task)
hal-syst-6233 0d..3 4156674us!: detach_pid (release_task)
hald-run-5416 0d..3 4160008us : detach_pid (release_task)
hald-run-5416 0d..3 4160008us : detach_pid (release_task)
hald-run-5416 0d..3 4160008us!: detach_pid (release_task)
events/0-9 0.... 9523335us!: kill_pgrp (n_tty_receive_buf)
bash-5821 0d..3 9763359us : detach_pid (release_task)
bash-5821 0d..3 9763359us : detach_pid (release_task)
bash-5821 0d..3 9763359us!: detach_pid (release_task)
hal-syst-6236 0d..3 10167777us : detach_pid (release_task)
hal-syst-6236 0d..3 10167779us : detach_pid (release_task)
hal-syst-6236 0d..3 10167779us!: detach_pid (release_task)
hald-run-5416 0d..3 10172764us : detach_pid (release_task)
hald-run-5416 0d..3 10172765us : detach_pid (release_task)
hald-run-5416 0d..3 10172765us!: detach_pid (release_task)
events/0-9 0.... 11008033us!: kill_pgrp (n_tty_receive_buf)
events/0-9 0.... 11771525us!: kill_pgrp (n_tty_receive_buf)
events/0-9 0.... 12504637us!: kill_pgrp (n_tty_receive_buf)
bash-5821 0d..3 12520392us : detach_pid (release_task)
bash-5821 0d..3 12520392us : detach_pid (release_task)
bash-5821 0d..3 12520392us!: detach_pid (release_task)
events/0-9 0.... 13541157us!: kill_pgrp (n_tty_receive_buf)
events/0-9 0.... 14404479us!: kill_pgrp (n_tty_receive_buf)
events/0-9 0.... 15139385us!: kill_pgrp (n_tty_receive_buf)
bash-5821 0d..3 15147037us : detach_pid (release_task)
bash-5821 0d..3 15147037us : detach_pid (release_task)
bash-5821 0d..3 15147037us!: detach_pid (release_task)
events/0-9 0.... 16095325us!: kill_pgrp (n_tty_receive_buf)
hal-syst-6242 0d..3 16180193us : detach_pid (release_task)
hal-syst-6242 0d..3 16180193us : detach_pid (release_task)
hal-syst-6242 0d..3 16180193us!: detach_pid (release_task)
hald-run-5416 0d..3 16183420us : detach_pid (release_task)
hald-run-5416 0d..3 16183420us : detach_pid (release_task)
hald-run-5416 0d..3 16183420us!: detach_pid (release_task)
bash-5821 0d..3 16239363us : detach_pid (release_task)
bash-5821 0d..3 16239363us : detach_pid (release_task)
bash-5821 0d..3 16239363us!: detach_pid (release_task)
events/0-9 0.... 17256910us!: kill_pgrp (n_tty_receive_buf)
bash-5821 0d..3 17285172us : detach_pid (release_task)
bash-5821 0d..3 17285172us : detach_pid (release_task)
bash-5821 0d..3 17285172us!: detach_pid (release_task)
events/0-9 0.... 18615608us!: kill_pgrp (n_tty_receive_buf)
bash-5821 0d..3 19458248us : detach_pid (release_task)
bash-5821 0d..3 19458248us : detach_pid (release_task)
bash-5821 0d..3 19458248us!: detach_pid (release_task)
events/0-9 0.... 19460406us!: kill_pgrp (n_tty_receive_buf)
events/0-9 0.... 21623112us!: kill_pgrp (n_tty_receive_buf)
hal-syst-6247 0d..3 22194451us : detach_pid (release_task)
hal-syst-6247 0d..3 22194453us : detach_pid (release_task)
hal-syst-6247 0d..3 22194453us!: detach_pid (release_task)
hald-run-5416 0d..3 22199518us : detach_pid (release_task)
hald-run-5416 0d..3 22199520us : detach_pid (release_task)
hald-run-5416 0d..3 22199520us!: detach_pid (release_task)
events/0-9 0.... 22352287us!: kill_pgrp (n_tty_receive_buf)
bash-5821 0d..3 22959844us : detach_pid (release_task)
bash-5821 0d..3 22959844us : detach_pid (release_task)
bash-5821 0d..3 22959844us!: detach_pid (release_task)
events/0-9 0.... 23075886us!: kill_pgrp (n_tty_receive_buf)
events/0-9 0.... 24111727us!: kill_pgrp (n_tty_receive_buf)
bash-5821 0d..3 24471359us : detach_pid (release_task)
bash-5821 0d..3 24471359us : detach_pid (release_task)
bash-5821 0d..3 24471359us!: detach_pid (release_task)
events/0-9 0.... 25914368us!: kill_pgrp (n_tty_receive_buf)
bash-5821 0d..3 26322615us : detach_pid (release_task)
bash-5821 0d..3 26322615us : detach_pid (release_task)
bash-5821 0d..3 26322615us!: detach_pid (release_task)
hal-syst-6252 0d..3 28207618us : detach_pid (release_task)
hal-syst-6252 0d..3 28207619us : detach_pid (release_task)
hal-syst-6252 0d..3 28207620us!: detach_pid (release_task)
hald-run-5416 0d..3 28212746us : detach_pid (release_task)
hald-run-5416 0d..3 28212747us : detach_pid (release_task)
hald-run-5416 0d..3 28212748us!: detach_pid (release_task)
events/0-9 0.... 28473354us!: kill_pgrp (n_tty_receive_buf)
events/0-9 0.... 29208378us!: kill_pgrp (n_tty_receive_buf)
pdflush-236 1d..3 29463682us+: detach_pid (release_task)
pdflush-236 1d..3 29463684us : detach_pid (release_task)
pdflush-236 1d..3 29463685us!: detach_pid (release_task)
bash-5821 0d..3 29633203us : detach_pid (release_task)
bash-5821 0d..3 29633203us : detach_pid (release_task)
bash-5821 0d..3 29633203us!: detach_pid (release_task)
events/0-9 0.... 31177193us!: kill_pgrp (n_tty_receive_buf)
events/0-9 0.... 32081149us!: kill_pgrp (n_tty_receive_buf)
bash-5821 0d..3 32096726us : detach_pid (release_task)
bash-5821 0d..3 32096726us : detach_pid (release_task)
bash-5821 0d..3 32096726us!: detach_pid (release_task)
hal-syst-6255 0d..3 34221092us : detach_pid (release_task)
hal-syst-6255 0d..3 34221094us : detach_pid (release_task)
hal-syst-6255 0d..3 34221095us!: detach_pid (release_task)
hald-run-5416 0d..3 34226008us+: detach_pid (release_task)
hald-run-5416 0d..3 34226009us : detach_pid (release_task)
hald-run-5416 0d..3 34226010us!: detach_pid (release_task)
events/0-9 0.... 34943377us!: kill_pgrp (n_tty_receive_buf)
bash-5821 0d..3 34953441us : detach_pid (release_task)
bash-5821 0d..3 34953441us : detach_pid (release_task)
bash-5821 0d..3 34953441us!: detach_pid (release_task)
sh-6228 1d..3 38434166us : detach_pid (release_task)
sh-6228 1d..3 38434166us : detach_pid (release_task)
sh-6228 1d..3 38434166us!: detach_pid (release_task)
sh-6228 1d..3 38740305us+: detach_pid (release_task)
sh-6228 1d..3 38740307us : detach_pid (release_task)
sh-6228 1d..3 38740308us!: detach_pid (release_task)
bash-5821 0d..3 38744188us : detach_pid (release_task)
bash-5821 0d..3 38744188us : detach_pid (release_task)
bash-5821 0d..3 38744188us!: detach_pid (release_task)
bash-5821 0d..3 40161326us : detach_pid (release_task)
bash-5821 0d..3 40161327us : detach_pid (release_task)
bash-5821 0d..3 40161328us!: detach_pid (release_task)
hal-syst-6259 0d..3 40233791us : detach_pid (release_task)
hal-syst-6259 0d..3 40233792us : detach_pid (release_task)
hal-syst-6259 0d..3 40233793us!: detach_pid (release_task)
hald-run-5416 0d..3 40238677us : detach_pid (release_task)
hald-run-5416 0d..3 40238678us+: detach_pid (release_task)
hald-run-5416 0d..3 40238680us!: detach_pid (release_task)
bash-5821 1d..3 43913077us : detach_pid (release_task)
bash-5821 1d..3 43913077us : detach_pid (release_task)
bash-5821 1d..3 43913077us!: detach_pid (release_task)
bash-5821 0d..3 44968815us : detach_pid (release_task)
bash-5821 0d..3 44968815us : detach_pid (release_task)
bash-5821 0d..3 44968815us!: detach_pid (release_task)
hal-syst-6263 0d..3 46247842us : detach_pid (release_task)
hal-syst-6263 0d..3 46247844us : detach_pid (release_task)
hal-syst-6263 0d..3 46247845us!: detach_pid (release_task)
hald-run-5416 0d..3 46252860us : detach_pid (release_task)
hald-run-5416 0d..3 46252861us : detach_pid (release_task)
hald-run-5416 0d..3 46252862us!: detach_pid (release_task)
nullmail-5260 1d..3 46591091us : detach_pid (release_task)
nullmail-5260 1d..3 46591092us : detach_pid (release_task)
nullmail-5260 1d..3 46591093us!: detach_pid (release_task)
nullmail-5260 1d..3 46594439us : detach_pid (release_task)
nullmail-5260 1d..3 46594439us : detach_pid (release_task)
nullmail-5260 1d..3 46594439us!: detach_pid (release_task)
nullmail-5260 1d..3 46596485us : detach_pid (release_task)
nullmail-5260 1d..3 46596486us : detach_pid (release_task)
nullmail-5260 1d..3 46596487us!: detach_pid (release_task)
nullmail-5260 1d..3 46599932us : detach_pid (release_task)
nullmail-5260 1d..3 46599933us : detach_pid (release_task)
nullmail-5260 1d..3 46599934us!: detach_pid (release_task)
nullmail-5260 0d..3 46627792us : detach_pid (release_task)
nullmail-5260 0d..3 46627792us : detach_pid (release_task)
nullmail-5260 0d..3 46627792us!: detach_pid (release_task)
nullmail-5260 0d..3 46636851us : detach_pid (release_task)
nullmail-5260 0d..3 46636851us : detach_pid (release_task)
nullmail-5260 0d..3 46636851us!: detach_pid (release_task)
nullmail-5260 0d..3 46647738us : detach_pid (release_task)
nullmail-5260 0d..3 46647740us : detach_pid (release_task)
nullmail-5260 0d..3 46647740us!: detach_pid (release_task)
nullmail-5260 0d..3 46656857us : detach_pid (release_task)
nullmail-5260 0d..3 46656857us : detach_pid (release_task)
nullmail-5260 0d..3 46656857us!: detach_pid (release_task)
nullmail-5260 0d..3 46666791us : detach_pid (release_task)
nullmail-5260 0d..3 46666791us : detach_pid (release_task)
nullmail-5260 0d..3 46666791us!: detach_pid (release_task)
nullmail-5260 0d..3 46680090us : detach_pid (release_task)
nullmail-5260 0d..3 46680090us : detach_pid (release_task)
nullmail-5260 0d..3 46680090us!: detach_pid (release_task)
nullmail-5260 0d..3 46683330us : detach_pid (release_task)
nullmail-5260 0d..3 46683330us : detach_pid (release_task)
nullmail-5260 0d..3 46683330us!: detach_pid (release_task)
nullmail-5260 1d..3 46684182us : detach_pid (release_task)
nullmail-5260 1d..3 46684182us : detach_pid (release_task)
nullmail-5260 1d..3 46684182us!: detach_pid (release_task)
nullmail-5260 1d..3 46687500us : detach_pid (release_task)
nullmail-5260 1d..3 46687500us : detach_pid (release_task)
nullmail-5260 1d..3 46687500us!: detach_pid (release_task)
nullmail-5260 1d..3 46692670us : detach_pid (release_task)
nullmail-5260 1d..3 46692670us : detach_pid (release_task)
nullmail-5260 1d..3 46692670us!: detach_pid (release_task)
bash-5821 1d..3 47327646us : detach_pid (release_task)
bash-5821 1d..3 47327647us : detach_pid (release_task)
bash-5821 1d..3 47327648us!: detach_pid (release_task)
bash-6280 1d..3 47331181us : detach_pid (release_task)
bash-6280 1d..3 47331181us : detach_pid (release_task)
bash-6280 1d..3 47331181us!: detach_pid (release_task)
bash-5821 1d..3 47331699us : detach_pid (release_task)
bash-5821 1d..3 47331700us : detach_pid (release_task)
bash-5821 1d..3 47331701us!: detach_pid (release_task)
bash-6282 1d..3 47334166us : detach_pid (release_task)
bash-6282 1d..3 47334166us : detach_pid (release_task)
bash-6282 1d..3 47334166us : detach_pid (release_task)
bash-5821 1d..3 47334166us : detach_pid (release_task)
bash-5821 1d..3 47334166us : detach_pid (release_task)
bash-5821 1d..3 47334166us!: detach_pid (release_task)
hal-syst-6284 0d..3 52261078us : detach_pid (release_task)
hal-syst-6284 0d..3 52261079us : detach_pid (release_task)
hal-syst-6284 0d..3 52261080us!: detach_pid (release_task)
hald-run-5416 0d..3 52266033us+: detach_pid (release_task)
hald-run-5416 0d..3 52266034us : detach_pid (release_task)
hald-run-5416 0d..3 52266035us!: detach_pid (release_task)
hal-syst-6286 0d..3 58273784us : detach_pid (release_task)
hal-syst-6286 0d..3 58273786us : detach_pid (release_task)
hal-syst-6286 0d..3 58273786us!: detach_pid (release_task)
hald-run-5416 0d..3 58278681us : detach_pid (release_task)
hald-run-5416 0d..3 58278682us : detach_pid (release_task)
hald-run-5416 0d..3 58278683us!: detach_pid (release_task)
sh-6224 0d..3 60000916us+: detach_pid (release_task)
sh-6224 0d..3 60000918us : detach_pid (release_task)
sh-6224 0d..3 60000919us : detach_pid (release_task)
vim:ft=help
Best regards,
--Edwin
On 2008-07-12 01:53, Arjan van de Ven wrote:
> On Fri, 11 Jul 2008 11:31:26 -0700 (PDT)
> Linus Torvalds <[email protected]> wrote:
>
>
>> On Fri, 11 Jul 2008, Linus Torvalds wrote:
>>
>>> Btw, did any of the impacted people test -rc9? Edwin's report is
>>> about -rc2 and -rc8, and one of the things we fixed since -rc8 is
>>> that incorrect and unintentional nr_zones zeroing that effectively
>>> disabled kswapd - and made everybody do synchronous memory freeing
>>> when they wanted to allocate more memory.. That can play havoc with
>>> any interactive stuff.
>>>
>> Hmm. Edwin's latencytop output includes this (ignoring the _very_ top
>> entries that are all either CD-ROM media change tests or are
>> interruptible pipe/select things) at the top:
>>
>> 21 10264428 915514 get_request_wait __make_request
>> generic_make_request submit_bio xfs_submit_ioend_bio xfs_submit_ioend
>> xfs_page_state_convert xfs_vm_writepage __writepage
>> write_cache_pages generic_writepages xfs_vm_writepages
>>
>
>
> argh. well I guess this might be useful for this case, but normally
> latencytop gives you much more humanly readable data... maybe Edwin
> forgot to do "make install" :-(
I forgot to use 'latencytop -d', I was running latencytop and when I've
seen something interesting I did a 'cat /proc/latency_stats' instead of
'latencytop -d'.
I'll try to use latencytop -d next time.
Best regards,
--Edwin
Linus Torvalds <[email protected]> writes:
>
> That said, seeing the full history of this same thing on the x86-32 side
> actually makes me _much_ happier about the patch. Because now I can tell
> when we did it, and what problems it seems to have caused (answer:
> "apparently none that are possibly relevant on x86-64").
FWIW I agree with Roland that the 64bit behaviour has a race window
that can lead to delayed signals. I had actually fixed this some
months ago myself in my tree because this was noticed by David (who
was comparing to sparc I thin) during code review, but never pushed it
out because it didn't get enough testing (it only ran on my
workstation for some time)
I think his patch is actually doing way too much work though, especially
it is not needed to repeat all checks for the full work mask. I'm
appending the patch I was using, which only rechecks signals.
Obviously it's still not something that should be used without a lot
of testing.
But in theory it's ok.
> So now I'm considering just putting it in before the 2.6.26 release after
> all ;)
That would seem rather hazardous.
-Andi
---
Fix lost signal race on x86-64
Originally noticed by Dave Miller during code review
Signed-off-by: Andi Kleen <[email protected]>
---
arch/x86/kernel/entry_64.S | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
Index: linux/arch/x86/kernel/entry_64.S
===================================================================
--- linux.orig/arch/x86/kernel/entry_64.S
+++ linux/arch/x86/kernel/entry_64.S
@@ -305,7 +305,7 @@ sysret_signal:
leaq -ARGOFFSET(%rsp),%rdi # &pt_regs -> arg1
xorl %esi,%esi # oldset -> arg2
call ptregscall_common
-1: movl $_TIF_NEED_RESCHED,%edi
+1: movl $_TIF_NEED_RESCHED|_TIF_SIGPENDING,%edi
/* Use IRET because user could have changed frame. This
works because ptregscall_common has called FIXUP_TOP_OF_STACK. */
DISABLE_INTERRUPTS(CLBR_NONE)
@@ -393,7 +393,7 @@ int_signal:
movq %rsp,%rdi # &ptregs -> arg1
xorl %esi,%esi # oldset -> arg2
call do_notify_resume
-1: movl $_TIF_NEED_RESCHED,%edi
+1: movl $_TIF_NEED_RESCHED|_TIF_SIGPENDING,%edi
int_restore_rest:
RESTORE_REST
DISABLE_INTERRUPTS(CLBR_NONE)
@@ -647,7 +647,7 @@ retint_signal:
RESTORE_REST
DISABLE_INTERRUPTS(CLBR_NONE)
TRACE_IRQS_OFF
- movl $_TIF_NEED_RESCHED,%edi
+ movl $_TIF_NEED_RESCHED|_TIF_SIGPENDING,%edi
GET_THREAD_INFO(%rcx)
jmp retint_check
Linus Torvalds <[email protected]> writes:
Linus, the fact that is not explained by your theory is why
Ctrl-Z+kill works but Ctrl-C doesn't. On the signal side these
both should be implemented in the same way -- if there was
really a missing signal_pending() check somewhere then Ctrl-Z
(and kill) should be delayed too.
Now doing faster signal processing is always a good thing in general,
but I'm a little doubtful your patch will help with that
particular problem.
-Andi
On 2008-07-12 13:32, T?r?k Edwin wrote:
> I haven't yet tried ftrace on this box, and neither did I try Roland's
> patch yet. I will try that now, and hopefuly come back with some numbers
> shortly.
>
tip/master of today already includes Roland's patch, so I have tested it
without even knowing on the 64-bit box.
[After gathering all the traces, I wanted to repeat everything with the
patch applied, but there was no need since its already there].
I can say that I didn't notice any difference in the delays (I still get
24 - 30 second delays).
I also tried Linus's readdir patch, the latency is around 27 seconds
(was 30 before). I am not sure if it is an improvement, or measurement
noise.
Also I wasn't able to reproduce the 'Ctrl+Z works faster than Ctrl+C'
symptom, perhaps it is just a coincidence: by the time I press Ctrl+Z to
interrupt, enough time has passed (30+ secs), that the same effect would
have been obtained by another Ctrl+C, or by just waiting.
On 32-bit, I think the 2-3 seconds latency can be attributed to I/O
delays, and I think its acceptable given the slow disk (5k4 rpm sata).
On 64-bit, which has faster disks, the latency is usually much smaller
(responds to Ctrl+C instantly), but ocasionally the maximum latency is
30 seconds.
That is too much, I am sure the disks would be able to seek and give the
data required much sooner (a 7k2 rpm, and a 10k rpm disk in raid0).
I think there is something more going on here than just possibly
delaying a signal, I cannot even start the dump_all.sh script when the
30 second latency occurs,
and its just a short script (that *was* launched before, should be in
cache), and it mostly reads /proc stuff.
What should I do next? Is there anything else I could trace to help you
figure out where is the delay?
I am using 2.6.26-rc9-tip for tracing, latest commit is:
commit d47d504b9a39555b9dce1dc2d1974795a87534ba
Author: Ingo Molnar <[email protected]>
Date: Sat Jul 12 09:37:25 2008 +0200
.config is appended to this mail[6].
To reproduce the latency I do the following on a text console (scripts
below [1][2]):
# ./tracing.sh &
# ./test.sh &
When I see the latency [3]:
# . ./dump_all.sh
As another user on a different tty:
$ find / >/dev/null 2>/dev/null
^C
Repeat until I see that find won't respond for more than a few seconds
(10 sec+):
$ find / >/dev/null 2>/dev/null
^C^C ^C^C ^C^C ^C^C ^C^C ^C^C ^C^C ^C^C ^C^C ^C^C ^C^C
Switch to the root tty, execute the dump_all command, unfortunately the
command only launches *after* find is killed (so after the huge latency
finished), but
I think it still gathers some useful info [CFS debugging is turned on,
and so is kernel.latencytop].
Here are some snippets from the same latency trace, 2.6.26-rc9-tip +
Linus's readdir patch (see the full output at the end [4], [5], [6]):
latencytop -d
[max 27565.9msec] Walking directory tree
- 344.92 msec (18.7%)
[max 23876.6msec] stat() operation
- 548.89 msec (6.2%)
[max 23778.5msec] Creating block layer request
- 1119.56 msec (11.5%)
[max 23669.0msec] Reading directory content
- 985.11 msec (4.0%)
[max 23575.4msec] Page fault
- 211.78 msec (4.5%)
[max 20232.7msec] Pagecache sync readahead
- 4058.77 msec (3.1%)
[max 3020.4msec] fsync() on a file
- 572.14 msec (1.1%)
/debug/tracing/trace
events/0-5 [00] 163.271126: kill_pgrp <-n_tty_receive_buf
events/0-5 [00] 164.148165: kill_pgrp <-n_tty_receive_buf
events/0-5 [00] 164.402391: kill_pgrp <-n_tty_receive_buf
[... more of these ...]
events/0-5 [00] 175.516874: kill_pgrp <-n_tty_receive_buf
gzip-4600 [00] 175.710875: detach_pid <-release_task
gzip-4600 [00] 175.710876: detach_pid <-release_task
gzip-4600 [00] 175.710877: detach_pid <-release_task
events/0-5 [00] 176.068271: kill_pgrp <-n_tty_receive_buf
gzip-4600 [00] 176.104521: detach_pid <-release_task
gzip-4600 [00] 176.104523: detach_pid <-release_task
gzip-4600 [00] 176.104523: detach_pid <-release_task
events/0-5 [00] 176.633643: kill_pgrp <-n_tty_receive_buf
events/0-5 [00] 177.229960: kill_pgrp <-n_tty_receive_buf
events/0-5 [00] 177.837424: kill_pgrp <-n_tty_receive_buf
events/0-5 [00] 178.430948: kill_pgrp <-n_tty_receive_buf
events/0-5 [00] 178.982075: kill_pgrp <-n_tty_receive_buf
events/0-5 [00] 179.415018: kill_pgrp <-n_tty_receive_buf
tracing.sh-4779 [00] 180.920587: detach_pid <-release_task
tracing.sh-4779 [00] 180.920588: detach_pid <-release_task
tracing.sh-4779 [00] 180.920588: detach_pid <-release_task
bash-4745 [00] 188.162442: detach_pid <-release_tas
This is a 25 second latency.
I am not sure what the time units in the scheduler debug info are, but
assuming they are miliseconds, they show a 24s se.block_max:
bash (4677, #threads: 1)
---------------------------------------------------------
se.exec_start : 739851.985209
se.vruntime : 242558.101836
se.sum_exec_runtime : 844.881816
se.avg_overlap : 0.021806
se.wait_start : 0.000000
se.sleep_start : 739851.985209
se.block_start : 0.000000
se.sleep_max : 93374.991561
se.block_max : 24867.990819
[1]: tracing.sh
#!/bin/sh
cd /debug/tracing
sysctl kernel.latencytop=1
echo kill_pgrp detach_pid\
>set_ftrace_filter
echo ftrace >current_tracer
echo 1 >tracing_enabled
sleep 60
echo 0 >tracing_enabled
cat trace >/tmp/trace.out
cat latency_trace >/tmp/latencytrace.out
echo "DONE"
[2]: test.sh
dd if=/dev/zero of=x1 bs=100M count=20&
dd if=/dev/zero of=x2 bs=100M count=20&
dd if=/dev/zero of=x3 bs=100M count=20&
dd if=/dev/zero of=x4 bs=100M count=20&
dd if=/dev/zero of=x5 bs=100M count=20&
wait %1 %2 %3 %4 $5
rm x1 x2 x3 x4 x5
[3] dump_all.sh
#!/bin/sh
#try to launch everything asap
./cfs-debug-info.sh&
cat /proc/latency_stats >latency.raw&
latencytop -d >latency.dump&
[4] latency.dump
[max 27565.9msec] Walking directory tree
- 344.92 msec (18.7%)
[max 23876.6msec] stat() operation
- 548.89 msec (6.2%)
[max 23778.5msec] Creating block layer request
- 1119.56 msec (11.5%)
[max 23669.0msec] Reading directory content
- 985.11 msec (4.0%)
[max 23575.4msec] Page fault
- 211.78 msec (4.5%)
[max 20232.7msec] Pagecache sync readahead
- 4058.77 msec (3.1%)
[max 3020.4msec] fsync() on a file
- 572.14 msec (1.1%)
[max 1326.8msec] Syncing inodes
- 315.55 msec (44.3%)
[max 1181.5msec] Writing back inodes
- 29.98 msec (1.6%)
[max 1151.1msec] Deleting an inode
- 36.61 msec (0.3%)
[max 973.9msec] Writing a page to disk
- 43.24 msec (0.3%)
[max 701.8msec] Writing buffer to disk (synchronous)
- 41.91 msec (0.7%)
[max 123.0msec] Executing a program
- 29.36 msec (0.2%)
[max 97.8msec] synchronous write
- 8.32 msec (3.3%)
[max 41.5msec] queue_log_writer do_journal_end do_journal_begin_r
journal_begin reiserfs_persistent_transaction reiserfs_get_block
__block_prepare_write block_write_begin reiserfs_write_begin
generic_file_buffered_write __generic_file_aio_write_nolock
generic_file_aio_write - 9.12 msec (0.0%)
[max 18.5msec] congestion_wait __alloc_pages_internal __alloc_pages
__do_page_cache_readahead force_page_cache_readahead sys_readahead
system_call_after_swapgs - 18.41 msec (0.0%)
[max 17.1msec] congestion_wait __alloc_pages_internal __alloc_pages
find_or_create_page __getblk do_journal_end journal_end
reiserfs_end_persistent_transaction reiserfs_write_end
generic_file_buffered_write __generic_file_aio_write_nolock
generic_file_aio_write - 11.38 msec (0.0%)
[max 16.7msec] Fork() system call
- 8.98 msec (0.0%)
[max 13.0msec] Writing to file
- 10.03 msec (0.0%)
[max 7.4msec] Executing raw SCSI command
- 0.51 msec (0.0%)
[max 4.7msec] Waiting for a process to die
- 0.07 msec (0.0%)
[max 4.5msec] Reading from a pipe
- 0.64 msec (0.0%)
[max 4.4msec] Waiting for event (poll)
- 4.31 msec (0.0%)
[max 3.9msec] opening cdrom device
- 0.61 msec (0.0%)
[max 3.2msec] Waiting for event (select)
- 0.80 msec (0.0%)
[max 1.4msec] congestion_wait __alloc_pages_internal __alloc_pages
find_or_create_page __getblk get_empty_nodes fix_nodes
reiserfs_paste_into_item reiserfs_get_block __block_prepare_write
block_write_begin reiserfs_write_begin - 1.36 msec (0.0%)
[max 0.5msec] SCSI cdrom ioctl
- 0.25 msec (0.0%)
[max 0.0msec] Waiting for event (epoll)
- 0.01 msec (0.0%)
[max 0.0msec] Waiting for TTY input
- 0.01 msec (0.0%)
[max 0.0msec] Writing data to TTY
- 0.00 msec (0.0%)
[5] Everything else is available as a .tar.gz from here (it is 3.1M
unpacked, and didn't want to flood your inboxes):
http://edwintorok.googlepages.com/traces.tar.gz
[6]
#
# Automatically generated make config: don't edit
# Linux kernel version: 2.6.26-rc9
# Sat Jul 12 13:39:35 2008
#
CONFIG_64BIT=y
# CONFIG_X86_32 is not set
CONFIG_X86_64=y
CONFIG_X86=y
CONFIG_ARCH_DEFCONFIG="arch/x86/configs/x86_64_defconfig"
# CONFIG_GENERIC_LOCKBREAK is not set
CONFIG_GENERIC_TIME=y
CONFIG_GENERIC_CMOS_UPDATE=y
CONFIG_CLOCKSOURCE_WATCHDOG=y
CONFIG_GENERIC_CLOCKEVENTS=y
CONFIG_GENERIC_CLOCKEVENTS_BROADCAST=y
CONFIG_LOCKDEP_SUPPORT=y
CONFIG_STACKTRACE_SUPPORT=y
CONFIG_HAVE_LATENCYTOP_SUPPORT=y
CONFIG_FAST_CMPXCHG_LOCAL=y
CONFIG_MMU=y
CONFIG_ZONE_DMA=y
CONFIG_GENERIC_ISA_DMA=y
CONFIG_GENERIC_IOMAP=y
CONFIG_GENERIC_BUG=y
CONFIG_GENERIC_HWEIGHT=y
# CONFIG_GENERIC_GPIO is not set
CONFIG_ARCH_MAY_HAVE_PC_FDC=y
CONFIG_RWSEM_GENERIC_SPINLOCK=y
# CONFIG_RWSEM_XCHGADD_ALGORITHM is not set
# CONFIG_ARCH_HAS_ILOG2_U32 is not set
# CONFIG_ARCH_HAS_ILOG2_U64 is not set
CONFIG_ARCH_HAS_CPU_IDLE_WAIT=y
CONFIG_GENERIC_CALIBRATE_DELAY=y
CONFIG_GENERIC_TIME_VSYSCALL=y
CONFIG_ARCH_HAS_CPU_RELAX=y
CONFIG_ARCH_HAS_CACHE_LINE_SIZE=y
# CONFIG_HAVE_SETUP_PER_CPU_AREA is not set
# CONFIG_HAVE_CPUMASK_OF_CPU_MAP is not set
CONFIG_ARCH_HIBERNATION_POSSIBLE=y
CONFIG_ARCH_SUSPEND_POSSIBLE=y
CONFIG_ZONE_DMA32=y
CONFIG_ARCH_POPULATES_NODE_MAP=y
CONFIG_AUDIT_ARCH=y
CONFIG_ARCH_SUPPORTS_AOUT=y
CONFIG_ARCH_SUPPORTS_OPTIMIZED_INLINING=y
CONFIG_GENERIC_HARDIRQS=y
CONFIG_GENERIC_IRQ_PROBE=y
CONFIG_X86_BIOS_REBOOT=y
CONFIG_X86_TRAMPOLINE=y
# CONFIG_KTIME_SCALAR is not set
CONFIG_DEFCONFIG_LIST="/lib/modules/$UNAME_RELEASE/.config"
#
# General setup
#
CONFIG_EXPERIMENTAL=y
CONFIG_BROKEN_ON_SMP=y
CONFIG_INIT_ENV_ARG_LIMIT=32
CONFIG_LOCALVERSION=""
# CONFIG_LOCALVERSION_AUTO is not set
CONFIG_SWAP=y
CONFIG_SYSVIPC=y
CONFIG_SYSVIPC_SYSCTL=y
CONFIG_POSIX_MQUEUE=y
CONFIG_BSD_PROCESS_ACCT=y
CONFIG_BSD_PROCESS_ACCT_V3=y
# CONFIG_TASKSTATS is not set
CONFIG_AUDIT=y
# CONFIG_AUDITSYSCALL is not set
CONFIG_IKCONFIG=y
CONFIG_IKCONFIG_PROC=y
CONFIG_LOG_BUF_SHIFT=16
# CONFIG_CGROUPS is not set
CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y
CONFIG_GROUP_SCHED=y
CONFIG_FAIR_GROUP_SCHED=y
# CONFIG_RT_GROUP_SCHED is not set
CONFIG_USER_SCHED=y
# CONFIG_CGROUP_SCHED is not set
# CONFIG_SYSFS_DEPRECATED_V2 is not set
CONFIG_RELAY=y
CONFIG_NAMESPACES=y
# CONFIG_UTS_NS is not set
# CONFIG_IPC_NS is not set
# CONFIG_USER_NS is not set
# CONFIG_PID_NS is not set
CONFIG_BLK_DEV_INITRD=y
CONFIG_INITRAMFS_SOURCE=""
# CONFIG_CC_OPTIMIZE_FOR_SIZE is not set
CONFIG_SYSCTL=y
# CONFIG_EMBEDDED is not set
CONFIG_UID16=y
CONFIG_SYSCTL_SYSCALL=y
CONFIG_SYSCTL_SYSCALL_CHECK=y
CONFIG_KALLSYMS=y
CONFIG_KALLSYMS_ALL=y
# CONFIG_KALLSYMS_EXTRA_PASS is not set
CONFIG_HOTPLUG=y
CONFIG_PRINTK=y
CONFIG_BUG=y
CONFIG_ELF_CORE=y
CONFIG_PCSPKR_PLATFORM=y
# CONFIG_COMPAT_BRK is not set
CONFIG_BASE_FULL=y
CONFIG_FUTEX=y
CONFIG_ANON_INODES=y
CONFIG_EPOLL=y
CONFIG_SIGNALFD=y
CONFIG_TIMERFD=y
CONFIG_EVENTFD=y
CONFIG_SHMEM=y
CONFIG_VM_EVENT_COUNTERS=y
CONFIG_SLUB_DEBUG=y
# CONFIG_SLAB is not set
CONFIG_SLUB=y
# CONFIG_SLOB is not set
CONFIG_PROFILING=y
CONFIG_MARKERS=y
CONFIG_OPROFILE=m
CONFIG_HAVE_OPROFILE=y
CONFIG_KPROBES=y
CONFIG_KRETPROBES=y
CONFIG_HAVE_KPROBES=y
CONFIG_HAVE_KRETPROBES=y
# CONFIG_HAVE_DMA_ATTRS is not set
# CONFIG_USE_GENERIC_SMP_HELPERS is not set
CONFIG_PROC_PAGE_MONITOR=y
# CONFIG_HAVE_GENERIC_DMA_COHERENT is not set
CONFIG_SLABINFO=y
CONFIG_RT_MUTEXES=y
# CONFIG_TINY_SHMEM is not set
CONFIG_BASE_SMALL=0
CONFIG_MODULES=y
# CONFIG_MODULE_FORCE_LOAD is not set
CONFIG_MODULE_UNLOAD=y
CONFIG_MODULE_FORCE_UNLOAD=y
CONFIG_MODVERSIONS=y
# CONFIG_MODULE_SRCVERSION_ALL is not set
CONFIG_KMOD=y
CONFIG_BLOCK=y
CONFIG_BLK_DEV_IO_TRACE=y
# CONFIG_BLK_DEV_BSG is not set
CONFIG_BLOCK_COMPAT=y
#
# IO Schedulers
#
CONFIG_IOSCHED_NOOP=y
CONFIG_IOSCHED_AS=y
CONFIG_IOSCHED_DEADLINE=y
CONFIG_IOSCHED_CFQ=y
# CONFIG_DEFAULT_AS is not set
# CONFIG_DEFAULT_DEADLINE is not set
CONFIG_DEFAULT_CFQ=y
# CONFIG_DEFAULT_NOOP is not set
CONFIG_DEFAULT_IOSCHED="cfq"
CONFIG_CLASSIC_RCU=y
#
# Processor type and features
#
CONFIG_TICK_ONESHOT=y
CONFIG_NO_HZ=y
CONFIG_HIGH_RES_TIMERS=y
CONFIG_GENERIC_CLOCKEVENTS_BUILD=y
# CONFIG_SMP is not set
CONFIG_X86_FIND_SMP_CONFIG=y
CONFIG_X86_MPPARSE=y
CONFIG_X86_PC=y
# CONFIG_X86_ELAN is not set
# CONFIG_X86_VOYAGER is not set
# CONFIG_X86_GENERICARCH is not set
# CONFIG_X86_RDC321X is not set
# CONFIG_X86_VSMP is not set
# CONFIG_PARAVIRT_GUEST is not set
# CONFIG_MEMTEST is not set
# CONFIG_M386 is not set
# CONFIG_M486 is not set
# CONFIG_M586 is not set
# CONFIG_M586TSC is not set
# CONFIG_M586MMX is not set
# CONFIG_M686 is not set
# CONFIG_MPENTIUMII is not set
# CONFIG_MPENTIUMIII is not set
# CONFIG_MPENTIUMM is not set
# CONFIG_MPENTIUM4 is not set
# CONFIG_MK6 is not set
# CONFIG_MK7 is not set
CONFIG_MK8=y
# CONFIG_MCRUSOE is not set
# CONFIG_MEFFICEON is not set
# CONFIG_MWINCHIPC6 is not set
# CONFIG_MWINCHIP2 is not set
# CONFIG_MWINCHIP3D is not set
# CONFIG_MGEODEGX1 is not set
# CONFIG_MGEODE_LX is not set
# CONFIG_MCYRIXIII is not set
# CONFIG_MVIAC3_2 is not set
# CONFIG_MVIAC7 is not set
# CONFIG_MPSC is not set
# CONFIG_MCORE2 is not set
# CONFIG_GENERIC_CPU is not set
CONFIG_X86_CPU=y
CONFIG_X86_L1_CACHE_BYTES=64
CONFIG_X86_INTERNODE_CACHE_BYTES=64
CONFIG_X86_CMPXCHG=y
CONFIG_X86_L1_CACHE_SHIFT=6
CONFIG_X86_WP_WORKS_OK=y
CONFIG_X86_GOOD_APIC=y
CONFIG_X86_INTEL_USERCOPY=y
CONFIG_X86_USE_PPRO_CHECKSUM=y
CONFIG_X86_TSC=y
CONFIG_X86_CMPXCHG64=y
CONFIG_X86_CMOV=y
CONFIG_X86_MINIMUM_CPU_FAMILY=64
CONFIG_X86_DEBUGCTLMSR=y
CONFIG_X86_DS=y
CONFIG_X86_PTRACE_BTS=y
CONFIG_HPET_TIMER=y
CONFIG_HPET_EMULATE_RTC=y
CONFIG_DMI=y
CONFIG_GART_IOMMU=y
# CONFIG_CALGARY_IOMMU is not set
# CONFIG_AMD_IOMMU is not set
CONFIG_SWIOTLB=y
CONFIG_IOMMU_HELPER=y
# CONFIG_PREEMPT_NONE is not set
CONFIG_PREEMPT_VOLUNTARY=y
# CONFIG_PREEMPT is not set
CONFIG_X86_LOCAL_APIC=y
CONFIG_X86_IO_APIC=y
CONFIG_X86_MCE=y
CONFIG_X86_MCE_INTEL=y
CONFIG_X86_MCE_AMD=y
# CONFIG_I8K is not set
CONFIG_MICROCODE=m
CONFIG_MICROCODE_OLD_INTERFACE=y
CONFIG_X86_MSR=m
CONFIG_X86_CPUID=m
CONFIG_ARCH_SPARSEMEM_DEFAULT=y
CONFIG_ARCH_SPARSEMEM_ENABLE=y
CONFIG_ARCH_SELECT_MEMORY_MODEL=y
CONFIG_ILLEGAL_POINTER_VALUE=0xffffc10000000000
CONFIG_SELECT_MEMORY_MODEL=y
# CONFIG_FLATMEM_MANUAL is not set
# CONFIG_DISCONTIGMEM_MANUAL is not set
CONFIG_SPARSEMEM_MANUAL=y
CONFIG_SPARSEMEM=y
CONFIG_HAVE_MEMORY_PRESENT=y
# CONFIG_SPARSEMEM_STATIC is not set
CONFIG_SPARSEMEM_EXTREME=y
CONFIG_SPARSEMEM_VMEMMAP_ENABLE=y
CONFIG_SPARSEMEM_VMEMMAP=y
#
# Memory hotplug is currently incompatible with Software Suspend
#
CONFIG_PAGEFLAGS_EXTENDED=y
CONFIG_SPLIT_PTLOCK_CPUS=4
CONFIG_RESOURCES_64BIT=y
CONFIG_ZONE_DMA_FLAG=1
CONFIG_BOUNCE=y
CONFIG_VIRT_TO_BUS=y
CONFIG_MTRR=y
CONFIG_MTRR_SANITIZER=y
CONFIG_MTRR_SANITIZER_ENABLE_DEFAULT=0
CONFIG_MTRR_SANITIZER_SPARE_REG_NR_DEFAULT=1
# CONFIG_X86_PAT is not set
# CONFIG_EFI is not set
CONFIG_SECCOMP=y
# CONFIG_CC_STACKPROTECTOR is not set
# CONFIG_HZ_100 is not set
# CONFIG_HZ_250 is not set
CONFIG_HZ_300=y
# CONFIG_HZ_1000 is not set
CONFIG_HZ=300
CONFIG_SCHED_HRTICK=y
CONFIG_KEXEC=y
# CONFIG_CRASH_DUMP is not set
CONFIG_PHYSICAL_START=0x200000
# CONFIG_RELOCATABLE is not set
CONFIG_PHYSICAL_ALIGN=0x200000
# CONFIG_COMPAT_VDSO is not set
CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y
#
# Power management options
#
CONFIG_ARCH_HIBERNATION_HEADER=y
CONFIG_PM=y
# CONFIG_PM_DEBUG is not set
CONFIG_PM_SLEEP=y
CONFIG_SUSPEND=y
CONFIG_SUSPEND_FREEZER=y
CONFIG_HIBERNATION=y
CONFIG_PM_STD_PARTITION=""
CONFIG_ACPI=y
CONFIG_ACPI_SLEEP=y
CONFIG_ACPI_PROCFS=y
# CONFIG_ACPI_PROCFS_POWER is not set
CONFIG_ACPI_SYSFS_POWER=y
CONFIG_ACPI_PROC_EVENT=y
CONFIG_ACPI_AC=m
CONFIG_ACPI_BATTERY=m
CONFIG_ACPI_BUTTON=m
CONFIG_ACPI_VIDEO=m
CONFIG_ACPI_FAN=m
CONFIG_ACPI_DOCK=m
# CONFIG_ACPI_BAY is not set
CONFIG_ACPI_PROCESSOR=m
CONFIG_ACPI_THERMAL=m
CONFIG_ACPI_WMI=y
CONFIG_ACPI_ASUS=m
CONFIG_ACPI_TOSHIBA=m
# CONFIG_ACPI_CUSTOM_DSDT is not set
CONFIG_ACPI_BLACKLIST_YEAR=0
# CONFIG_ACPI_DEBUG is not set
CONFIG_ACPI_EC=y
CONFIG_ACPI_POWER=y
CONFIG_ACPI_SYSTEM=y
CONFIG_X86_PM_TIMER=y
CONFIG_ACPI_CONTAINER=m
CONFIG_ACPI_SBS=m
#
# CPU Frequency scaling
#
CONFIG_CPU_FREQ=y
CONFIG_CPU_FREQ_TABLE=m
# CONFIG_CPU_FREQ_DEBUG is not set
CONFIG_CPU_FREQ_STAT=m
# CONFIG_CPU_FREQ_STAT_DETAILS is not set
CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE=y
# CONFIG_CPU_FREQ_DEFAULT_GOV_POWERSAVE is not set
# CONFIG_CPU_FREQ_DEFAULT_GOV_USERSPACE is not set
# CONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND is not set
# CONFIG_CPU_FREQ_DEFAULT_GOV_CONSERVATIVE is not set
CONFIG_CPU_FREQ_GOV_PERFORMANCE=y
CONFIG_CPU_FREQ_GOV_POWERSAVE=m
CONFIG_CPU_FREQ_GOV_USERSPACE=y
# CONFIG_CPU_FREQ_GOV_ONDEMAND is not set
CONFIG_CPU_FREQ_GOV_CONSERVATIVE=m
#
# CPUFreq processor drivers
#
CONFIG_X86_ACPI_CPUFREQ=m
CONFIG_X86_POWERNOW_K8=m
CONFIG_X86_POWERNOW_K8_ACPI=y
# CONFIG_X86_SPEEDSTEP_CENTRINO is not set
# CONFIG_X86_P4_CLOCKMOD is not set
#
# shared options
#
# CONFIG_X86_ACPI_CPUFREQ_PROC_INTF is not set
# CONFIG_X86_SPEEDSTEP_LIB is not set
CONFIG_CPU_IDLE=y
CONFIG_CPU_IDLE_GOV_LADDER=y
CONFIG_CPU_IDLE_GOV_MENU=y
#
# Bus options (PCI etc.)
#
CONFIG_PCI=y
CONFIG_PCI_DIRECT=y
CONFIG_PCI_MMCONFIG=y
CONFIG_PCI_DOMAINS=y
# CONFIG_DMAR is not set
CONFIG_PCIEPORTBUS=y
CONFIG_HOTPLUG_PCI_PCIE=m
CONFIG_PCIEAER=y
# CONFIG_PCIEASPM is not set
CONFIG_ARCH_SUPPORTS_MSI=y
CONFIG_PCI_MSI=y
CONFIG_PCI_LEGACY=y
# CONFIG_PCI_DEBUG is not set
CONFIG_HT_IRQ=y
CONFIG_ISA_DMA_API=y
CONFIG_K8_NB=y
# CONFIG_PCCARD is not set
CONFIG_HOTPLUG_PCI=m
CONFIG_HOTPLUG_PCI_FAKE=m
CONFIG_HOTPLUG_PCI_ACPI=m
CONFIG_HOTPLUG_PCI_ACPI_IBM=m
CONFIG_HOTPLUG_PCI_CPCI=y
CONFIG_HOTPLUG_PCI_CPCI_ZT5550=m
CONFIG_HOTPLUG_PCI_CPCI_GENERIC=m
CONFIG_HOTPLUG_PCI_SHPC=m
#
# Executable file formats / Emulations
#
CONFIG_BINFMT_ELF=y
CONFIG_COMPAT_BINFMT_ELF=y
CONFIG_BINFMT_MISC=m
CONFIG_IA32_EMULATION=y
CONFIG_IA32_AOUT=y
CONFIG_COMPAT=y
CONFIG_COMPAT_FOR_U64_ALIGNMENT=y
CONFIG_SYSVIPC_COMPAT=y
#
# Networking
#
CONFIG_NET=y
#
# Networking options
#
CONFIG_PACKET=y
CONFIG_PACKET_MMAP=y
CONFIG_UNIX=y
CONFIG_XFRM=y
CONFIG_XFRM_USER=m
# CONFIG_XFRM_SUB_POLICY is not set
# CONFIG_XFRM_MIGRATE is not set
# CONFIG_XFRM_STATISTICS is not set
CONFIG_NET_KEY=m
# CONFIG_NET_KEY_MIGRATE is not set
CONFIG_INET=y
CONFIG_IP_MULTICAST=y
CONFIG_IP_ADVANCED_ROUTER=y
CONFIG_ASK_IP_FIB_HASH=y
# CONFIG_IP_FIB_TRIE is not set
CONFIG_IP_FIB_HASH=y
CONFIG_IP_MULTIPLE_TABLES=y
CONFIG_IP_ROUTE_MULTIPATH=y
CONFIG_IP_ROUTE_VERBOSE=y
# CONFIG_IP_PNP is not set
CONFIG_NET_IPIP=m
CONFIG_NET_IPGRE=m
CONFIG_NET_IPGRE_BROADCAST=y
CONFIG_IP_MROUTE=y
CONFIG_IP_PIMSM_V1=y
CONFIG_IP_PIMSM_V2=y
# CONFIG_ARPD is not set
CONFIG_SYN_COOKIES=y
CONFIG_INET_AH=m
CONFIG_INET_ESP=m
CONFIG_INET_IPCOMP=m
CONFIG_INET_XFRM_TUNNEL=m
CONFIG_INET_TUNNEL=m
CONFIG_INET_XFRM_MODE_TRANSPORT=m
CONFIG_INET_XFRM_MODE_TUNNEL=m
CONFIG_INET_XFRM_MODE_BEET=m
CONFIG_INET_LRO=m
CONFIG_INET_DIAG=m
CONFIG_INET_TCP_DIAG=m
CONFIG_TCP_CONG_ADVANCED=y
CONFIG_TCP_CONG_BIC=y
CONFIG_TCP_CONG_CUBIC=m
CONFIG_TCP_CONG_WESTWOOD=m
CONFIG_TCP_CONG_HTCP=m
CONFIG_TCP_CONG_HSTCP=m
CONFIG_TCP_CONG_HYBLA=m
CONFIG_TCP_CONG_VEGAS=m
CONFIG_TCP_CONG_SCALABLE=m
CONFIG_TCP_CONG_LP=m
CONFIG_TCP_CONG_VENO=m
# CONFIG_TCP_CONG_YEAH is not set
# CONFIG_TCP_CONG_ILLINOIS is not set
CONFIG_DEFAULT_BIC=y
# CONFIG_DEFAULT_CUBIC is not set
# CONFIG_DEFAULT_HTCP is not set
# CONFIG_DEFAULT_VEGAS is not set
# CONFIG_DEFAULT_WESTWOOD is not set
# CONFIG_DEFAULT_RENO is not set
CONFIG_DEFAULT_TCP_CONG="bic"
# CONFIG_TCP_MD5SIG is not set
CONFIG_IP_VS=m
# CONFIG_IP_VS_DEBUG is not set
CONFIG_IP_VS_TAB_BITS=12
#
# IPVS transport protocol load balancing support
#
CONFIG_IP_VS_PROTO_TCP=y
CONFIG_IP_VS_PROTO_UDP=y
CONFIG_IP_VS_PROTO_ESP=y
CONFIG_IP_VS_PROTO_AH=y
#
# IPVS scheduler
#
CONFIG_IP_VS_RR=m
CONFIG_IP_VS_WRR=m
CONFIG_IP_VS_LC=m
CONFIG_IP_VS_WLC=m
CONFIG_IP_VS_LBLC=m
CONFIG_IP_VS_LBLCR=m
CONFIG_IP_VS_DH=m
CONFIG_IP_VS_SH=m
CONFIG_IP_VS_SED=m
CONFIG_IP_VS_NQ=m
#
# IPVS application helper
#
CONFIG_IP_VS_FTP=m
CONFIG_IPV6=m
CONFIG_IPV6_PRIVACY=y
# CONFIG_IPV6_ROUTER_PREF is not set
# CONFIG_IPV6_OPTIMISTIC_DAD is not set
CONFIG_INET6_AH=m
CONFIG_INET6_ESP=m
CONFIG_INET6_IPCOMP=m
# CONFIG_IPV6_MIP6 is not set
CONFIG_INET6_XFRM_TUNNEL=m
CONFIG_INET6_TUNNEL=m
CONFIG_INET6_XFRM_MODE_TRANSPORT=m
CONFIG_INET6_XFRM_MODE_TUNNEL=m
CONFIG_INET6_XFRM_MODE_BEET=m
CONFIG_INET6_XFRM_MODE_ROUTEOPTIMIZATION=m
CONFIG_IPV6_SIT=m
CONFIG_IPV6_NDISC_NODETYPE=y
CONFIG_IPV6_TUNNEL=m
# CONFIG_IPV6_MULTIPLE_TABLES is not set
# CONFIG_IPV6_MROUTE is not set
# CONFIG_NETLABEL is not set
CONFIG_NETWORK_SECMARK=y
CONFIG_NETFILTER=y
# CONFIG_NETFILTER_DEBUG is not set
# CONFIG_NETFILTER_ADVANCED is not set
#
# Core Netfilter Configuration
#
CONFIG_NETFILTER_NETLINK=m
CONFIG_NETFILTER_NETLINK_LOG=m
CONFIG_NF_CONNTRACK=m
CONFIG_NF_CONNTRACK_SECMARK=y
CONFIG_NF_CONNTRACK_FTP=m
CONFIG_NF_CONNTRACK_IRC=m
CONFIG_NF_CONNTRACK_SIP=m
CONFIG_NF_CT_NETLINK=m
CONFIG_NETFILTER_XTABLES=m
CONFIG_NETFILTER_XT_TARGET_MARK=m
CONFIG_NETFILTER_XT_TARGET_NFLOG=m
CONFIG_NETFILTER_XT_TARGET_SECMARK=m
CONFIG_NETFILTER_XT_TARGET_CONNSECMARK=m
CONFIG_NETFILTER_XT_TARGET_TCPMSS=m
CONFIG_NETFILTER_XT_MATCH_CONNTRACK=m
CONFIG_NETFILTER_XT_MATCH_MARK=m
CONFIG_NETFILTER_XT_MATCH_POLICY=m
CONFIG_NETFILTER_XT_MATCH_STATE=m
#
# IP: Netfilter Configuration
#
CONFIG_NF_CONNTRACK_IPV4=m
CONFIG_NF_CONNTRACK_PROC_COMPAT=y
CONFIG_IP_NF_IPTABLES=m
CONFIG_IP_NF_FILTER=m
CONFIG_IP_NF_TARGET_REJECT=m
CONFIG_IP_NF_TARGET_LOG=m
CONFIG_IP_NF_TARGET_ULOG=m
CONFIG_NF_NAT=m
CONFIG_NF_NAT_NEEDED=y
CONFIG_IP_NF_TARGET_MASQUERADE=m
CONFIG_NF_NAT_FTP=m
CONFIG_NF_NAT_IRC=m
# CONFIG_NF_NAT_TFTP is not set
# CONFIG_NF_NAT_AMANDA is not set
# CONFIG_NF_NAT_PPTP is not set
# CONFIG_NF_NAT_H323 is not set
CONFIG_NF_NAT_SIP=m
CONFIG_IP_NF_MANGLE=m
#
# IPv6: Netfilter Configuration
#
CONFIG_NF_CONNTRACK_IPV6=m
CONFIG_IP6_NF_IPTABLES=m
CONFIG_IP6_NF_MATCH_IPV6HEADER=m
CONFIG_IP6_NF_FILTER=m
CONFIG_IP6_NF_TARGET_LOG=m
CONFIG_IP6_NF_TARGET_REJECT=m
CONFIG_IP6_NF_MANGLE=m
CONFIG_IP_DCCP=m
CONFIG_INET_DCCP_DIAG=m
CONFIG_IP_DCCP_ACKVEC=y
#
# DCCP CCIDs Configuration (EXPERIMENTAL)
#
CONFIG_IP_DCCP_CCID2=m
# CONFIG_IP_DCCP_CCID2_DEBUG is not set
CONFIG_IP_DCCP_CCID3=m
# CONFIG_IP_DCCP_CCID3_DEBUG is not set
CONFIG_IP_DCCP_CCID3_RTO=100
CONFIG_IP_DCCP_TFRC_LIB=m
#
# DCCP Kernel Hacking
#
# CONFIG_IP_DCCP_DEBUG is not set
# CONFIG_NET_DCCPPROBE is not set
CONFIG_IP_SCTP=m
# CONFIG_SCTP_DBG_MSG is not set
# CONFIG_SCTP_DBG_OBJCNT is not set
# CONFIG_SCTP_HMAC_NONE is not set
# CONFIG_SCTP_HMAC_SHA1 is not set
CONFIG_SCTP_HMAC_MD5=y
CONFIG_TIPC=m
CONFIG_TIPC_ADVANCED=y
CONFIG_TIPC_ZONES=3
CONFIG_TIPC_CLUSTERS=1
CONFIG_TIPC_NODES=255
CONFIG_TIPC_SLAVE_NODES=0
CONFIG_TIPC_PORTS=8191
CONFIG_TIPC_LOG=0
# CONFIG_TIPC_DEBUG is not set
CONFIG_ATM=y
CONFIG_ATM_CLIP=y
# CONFIG_ATM_CLIP_NO_ICMP is not set
CONFIG_ATM_LANE=m
CONFIG_ATM_MPOA=m
CONFIG_ATM_BR2684=m
# CONFIG_ATM_BR2684_IPFILTER is not set
CONFIG_BRIDGE=m
CONFIG_VLAN_8021Q=m
CONFIG_DECNET=m
# CONFIG_DECNET_ROUTER is not set
CONFIG_LLC=m
CONFIG_LLC2=m
CONFIG_IPX=m
# CONFIG_IPX_INTERN is not set
CONFIG_ATALK=m
CONFIG_DEV_APPLETALK=m
CONFIG_IPDDP=m
CONFIG_IPDDP_ENCAP=y
CONFIG_IPDDP_DECAP=y
CONFIG_X25=m
CONFIG_LAPB=m
CONFIG_ECONET=m
CONFIG_ECONET_AUNUDP=y
CONFIG_ECONET_NATIVE=y
CONFIG_WAN_ROUTER=m
CONFIG_NET_SCHED=y
#
# Queueing/Scheduling
#
CONFIG_NET_SCH_CBQ=m
CONFIG_NET_SCH_HTB=m
CONFIG_NET_SCH_HFSC=m
CONFIG_NET_SCH_ATM=m
CONFIG_NET_SCH_PRIO=m
CONFIG_NET_SCH_RED=m
CONFIG_NET_SCH_SFQ=m
CONFIG_NET_SCH_TEQL=m
CONFIG_NET_SCH_TBF=m
CONFIG_NET_SCH_GRED=m
CONFIG_NET_SCH_DSMARK=m
CONFIG_NET_SCH_NETEM=m
CONFIG_NET_SCH_INGRESS=m
#
# Classification
#
CONFIG_NET_CLS=y
CONFIG_NET_CLS_BASIC=m
CONFIG_NET_CLS_TCINDEX=m
CONFIG_NET_CLS_ROUTE4=m
CONFIG_NET_CLS_ROUTE=y
CONFIG_NET_CLS_FW=m
CONFIG_NET_CLS_U32=m
CONFIG_CLS_U32_PERF=y
CONFIG_CLS_U32_MARK=y
CONFIG_NET_CLS_RSVP=m
CONFIG_NET_CLS_RSVP6=m
# CONFIG_NET_CLS_FLOW is not set
CONFIG_NET_EMATCH=y
CONFIG_NET_EMATCH_STACK=32
CONFIG_NET_EMATCH_CMP=m
CONFIG_NET_EMATCH_NBYTE=m
CONFIG_NET_EMATCH_U32=m
CONFIG_NET_EMATCH_META=m
CONFIG_NET_EMATCH_TEXT=m
CONFIG_NET_CLS_ACT=y
CONFIG_NET_ACT_POLICE=m
CONFIG_NET_ACT_GACT=m
CONFIG_GACT_PROB=y
CONFIG_NET_ACT_MIRRED=m
CONFIG_NET_ACT_IPT=m
CONFIG_NET_ACT_NAT=m
CONFIG_NET_ACT_PEDIT=m
CONFIG_NET_ACT_SIMP=m
CONFIG_NET_CLS_IND=y
CONFIG_NET_SCH_FIFO=y
#
# Network testing
#
CONFIG_NET_PKTGEN=m
# CONFIG_NET_TCPPROBE is not set
# CONFIG_HAMRADIO is not set
# CONFIG_CAN is not set
# CONFIG_IRDA is not set
# CONFIG_BT is not set
CONFIG_AF_RXRPC=m
# CONFIG_AF_RXRPC_DEBUG is not set
# CONFIG_RXKAD is not set
CONFIG_FIB_RULES=y
#
# Wireless
#
# CONFIG_CFG80211 is not set
# CONFIG_WIRELESS_EXT is not set
# CONFIG_MAC80211 is not set
# CONFIG_IEEE80211 is not set
# CONFIG_RFKILL is not set
# CONFIG_NET_9P is not set
#
# Device Drivers
#
#
# Generic Driver Options
#
CONFIG_UEVENT_HELPER_PATH="/sbin/hotplug"
CONFIG_STANDALONE=y
CONFIG_PREVENT_FIRMWARE_BUILD=y
CONFIG_FW_LOADER=m
# CONFIG_DEBUG_DRIVER is not set
# CONFIG_DEBUG_DEVRES is not set
# CONFIG_SYS_HYPERVISOR is not set
CONFIG_CONNECTOR=m
CONFIG_MTD=m
# CONFIG_MTD_DEBUG is not set
# CONFIG_MTD_CONCAT is not set
CONFIG_MTD_PARTITIONS=y
# CONFIG_MTD_REDBOOT_PARTS is not set
# CONFIG_MTD_AR7_PARTS is not set
#
# User Modules And Translation Layers
#
# CONFIG_MTD_CHAR is not set
# CONFIG_MTD_BLKDEVS is not set
# CONFIG_MTD_BLOCK is not set
# CONFIG_MTD_BLOCK_RO is not set
# CONFIG_FTL is not set
# CONFIG_NFTL is not set
# CONFIG_INFTL is not set
# CONFIG_RFD_FTL is not set
# CONFIG_SSFDC is not set
# CONFIG_MTD_OOPS is not set
#
# RAM/ROM/Flash chip drivers
#
# CONFIG_MTD_CFI is not set
CONFIG_MTD_JEDECPROBE=m
CONFIG_MTD_GEN_PROBE=m
# CONFIG_MTD_CFI_ADV_OPTIONS is not set
CONFIG_MTD_MAP_BANK_WIDTH_1=y
CONFIG_MTD_MAP_BANK_WIDTH_2=y
CONFIG_MTD_MAP_BANK_WIDTH_4=y
# CONFIG_MTD_MAP_BANK_WIDTH_8 is not set
# CONFIG_MTD_MAP_BANK_WIDTH_16 is not set
# CONFIG_MTD_MAP_BANK_WIDTH_32 is not set
CONFIG_MTD_CFI_I1=y
CONFIG_MTD_CFI_I2=y
# CONFIG_MTD_CFI_I4 is not set
# CONFIG_MTD_CFI_I8 is not set
# CONFIG_MTD_CFI_INTELEXT is not set
CONFIG_MTD_CFI_AMDSTD=m
# CONFIG_MTD_CFI_STAA is not set
CONFIG_MTD_CFI_UTIL=m
CONFIG_MTD_RAM=m
# CONFIG_MTD_ROM is not set
# CONFIG_MTD_ABSENT is not set
#
# Mapping drivers for chip access
#
# CONFIG_MTD_COMPLEX_MAPPINGS is not set
# CONFIG_MTD_PHYSMAP is not set
# CONFIG_MTD_TS5500 is not set
# CONFIG_MTD_AMD76XROM is not set
# CONFIG_MTD_ICHXROM is not set
# CONFIG_MTD_ESB2ROM is not set
# CONFIG_MTD_CK804XROM is not set
# CONFIG_MTD_SCB2_FLASH is not set
# CONFIG_MTD_NETtel is not set
# CONFIG_MTD_L440GX is not set
# CONFIG_MTD_INTEL_VR_NOR is not set
# CONFIG_MTD_PLATRAM is not set
#
# Self-contained MTD device drivers
#
# CONFIG_MTD_PMC551 is not set
# CONFIG_MTD_SLRAM is not set
# CONFIG_MTD_PHRAM is not set
# CONFIG_MTD_MTDRAM is not set
# CONFIG_MTD_BLOCK2MTD is not set
#
# Disk-On-Chip Device Drivers
#
# CONFIG_MTD_DOC2000 is not set
# CONFIG_MTD_DOC2001 is not set
# CONFIG_MTD_DOC2001PLUS is not set
# CONFIG_MTD_NAND is not set
# CONFIG_MTD_ONENAND is not set
#
# UBI - Unsorted block images
#
# CONFIG_MTD_UBI is not set
CONFIG_PARPORT=m
CONFIG_PARPORT_PC=m
CONFIG_PARPORT_SERIAL=m
CONFIG_PARPORT_PC_FIFO=y
CONFIG_PARPORT_PC_SUPERIO=y
# CONFIG_PARPORT_GSC is not set
CONFIG_PARPORT_AX88796=m
CONFIG_PARPORT_1284=y
CONFIG_PARPORT_NOT_PC=y
CONFIG_PNP=y
# CONFIG_PNP_DEBUG is not set
#
# Protocols
#
CONFIG_PNPACPI=y
CONFIG_BLK_DEV=y
CONFIG_BLK_DEV_FD=m
# CONFIG_PARIDE is not set
# CONFIG_BLK_CPQ_DA is not set
# CONFIG_BLK_CPQ_CISS_DA is not set
# CONFIG_BLK_DEV_DAC960 is not set
# CONFIG_BLK_DEV_UMEM is not set
# CONFIG_BLK_DEV_COW_COMMON is not set
CONFIG_BLK_DEV_LOOP=m
CONFIG_BLK_DEV_CRYPTOLOOP=m
CONFIG_BLK_DEV_NBD=m
# CONFIG_BLK_DEV_SX8 is not set
# CONFIG_BLK_DEV_UB is not set
CONFIG_BLK_DEV_RAM=y
CONFIG_BLK_DEV_RAM_COUNT=16
CONFIG_BLK_DEV_RAM_SIZE=65536
# CONFIG_BLK_DEV_XIP is not set
CONFIG_CDROM_PKTCDVD=m
CONFIG_CDROM_PKTCDVD_BUFFERS=8
# CONFIG_CDROM_PKTCDVD_WCACHE is not set
# CONFIG_ATA_OVER_ETH is not set
CONFIG_MISC_DEVICES=y
# CONFIG_IBM_ASM is not set
# CONFIG_PHANTOM is not set
# CONFIG_EEPROM_93CX6 is not set
# CONFIG_SGI_IOC4 is not set
CONFIG_TIFM_CORE=m
# CONFIG_TIFM_7XX1 is not set
# CONFIG_FUJITSU_LAPTOP is not set
# CONFIG_MSI_LAPTOP is not set
# CONFIG_SONY_LAPTOP is not set
# CONFIG_THINKPAD_ACPI is not set
# CONFIG_INTEL_MENLOW is not set
# CONFIG_EEEPC_LAPTOP is not set
# CONFIG_ENCLOSURE_SERVICES is not set
CONFIG_HAVE_IDE=y
CONFIG_IDE=m
CONFIG_BLK_DEV_IDE=m
#
# Please see Documentation/ide/ide.txt for help/info on IDE drives
#
# CONFIG_BLK_DEV_IDE_SATA is not set
# CONFIG_BLK_DEV_IDEDISK is not set
# CONFIG_IDEDISK_MULTI_MODE is not set
CONFIG_BLK_DEV_IDECD=m
CONFIG_BLK_DEV_IDECD_VERBOSE_ERRORS=y
# CONFIG_BLK_DEV_IDETAPE is not set
CONFIG_BLK_DEV_IDEFLOPPY=m
# CONFIG_BLK_DEV_IDESCSI is not set
CONFIG_BLK_DEV_IDEACPI=y
# CONFIG_IDE_TASK_IOCTL is not set
CONFIG_IDE_PROC_FS=y
#
# IDE chipset support/bugfixes
#
# CONFIG_IDE_GENERIC is not set
# CONFIG_BLK_DEV_PLATFORM is not set
# CONFIG_BLK_DEV_CMD640 is not set
# CONFIG_BLK_DEV_IDEPNP is not set
CONFIG_BLK_DEV_IDEDMA_SFF=y
#
# PCI IDE chipsets support
#
CONFIG_BLK_DEV_IDEPCI=y
# CONFIG_BLK_DEV_OFFBOARD is not set
CONFIG_BLK_DEV_GENERIC=m
# CONFIG_BLK_DEV_OPTI621 is not set
# CONFIG_BLK_DEV_RZ1000 is not set
CONFIG_BLK_DEV_IDEDMA_PCI=y
# CONFIG_BLK_DEV_AEC62XX is not set
# CONFIG_BLK_DEV_ALI15X3 is not set
CONFIG_BLK_DEV_AMD74XX=m
# CONFIG_BLK_DEV_ATIIXP is not set
# CONFIG_BLK_DEV_CMD64X is not set
# CONFIG_BLK_DEV_TRIFLEX is not set
# CONFIG_BLK_DEV_CY82C693 is not set
# CONFIG_BLK_DEV_CS5520 is not set
# CONFIG_BLK_DEV_CS5530 is not set
# CONFIG_BLK_DEV_HPT34X is not set
# CONFIG_BLK_DEV_HPT366 is not set
# CONFIG_BLK_DEV_JMICRON is not set
# CONFIG_BLK_DEV_SC1200 is not set
# CONFIG_BLK_DEV_PIIX is not set
# CONFIG_BLK_DEV_IT8213 is not set
# CONFIG_BLK_DEV_IT821X is not set
# CONFIG_BLK_DEV_NS87415 is not set
# CONFIG_BLK_DEV_PDC202XX_OLD is not set
# CONFIG_BLK_DEV_PDC202XX_NEW is not set
# CONFIG_BLK_DEV_SVWKS is not set
# CONFIG_BLK_DEV_SIIMAGE is not set
# CONFIG_BLK_DEV_SIS5513 is not set
# CONFIG_BLK_DEV_SLC90E66 is not set
# CONFIG_BLK_DEV_TRM290 is not set
# CONFIG_BLK_DEV_VIA82CXXX is not set
# CONFIG_BLK_DEV_TC86C001 is not set
CONFIG_BLK_DEV_IDEDMA=y
# CONFIG_BLK_DEV_HD_ONLY is not set
# CONFIG_BLK_DEV_HD is not set
#
# SCSI device support
#
CONFIG_RAID_ATTRS=m
CONFIG_SCSI=y
CONFIG_SCSI_DMA=y
CONFIG_SCSI_TGT=m
CONFIG_SCSI_NETLINK=y
CONFIG_SCSI_PROC_FS=y
#
# SCSI support type (disk, tape, CD-ROM)
#
CONFIG_BLK_DEV_SD=m
# CONFIG_CHR_DEV_ST is not set
# CONFIG_CHR_DEV_OSST is not set
CONFIG_BLK_DEV_SR=m
CONFIG_BLK_DEV_SR_VENDOR=y
CONFIG_CHR_DEV_SG=m
CONFIG_CHR_DEV_SCH=m
#
# Some SCSI devices (e.g. CD jukebox) support multiple LUNs
#
CONFIG_SCSI_MULTI_LUN=y
CONFIG_SCSI_CONSTANTS=y
CONFIG_SCSI_LOGGING=y
CONFIG_SCSI_SCAN_ASYNC=y
CONFIG_SCSI_WAIT_SCAN=m
#
# SCSI Transports
#
CONFIG_SCSI_SPI_ATTRS=m
CONFIG_SCSI_FC_ATTRS=m
# CONFIG_SCSI_FC_TGT_ATTRS is not set
CONFIG_SCSI_ISCSI_ATTRS=m
CONFIG_SCSI_SAS_ATTRS=m
CONFIG_SCSI_SAS_LIBSAS=m
# CONFIG_SCSI_SAS_ATA is not set
# CONFIG_SCSI_SAS_HOST_SMP is not set
# CONFIG_SCSI_SAS_LIBSAS_DEBUG is not set
CONFIG_SCSI_SRP_ATTRS=m
CONFIG_SCSI_SRP_TGT_ATTRS=y
CONFIG_SCSI_LOWLEVEL=y
# CONFIG_ISCSI_TCP is not set
# CONFIG_BLK_DEV_3W_XXXX_RAID is not set
# CONFIG_SCSI_3W_9XXX is not set
# CONFIG_SCSI_ACARD is not set
# CONFIG_SCSI_AACRAID is not set
# CONFIG_SCSI_AIC7XXX is not set
# CONFIG_SCSI_AIC7XXX_OLD is not set
# CONFIG_SCSI_AIC79XX is not set
# CONFIG_SCSI_AIC94XX is not set
# CONFIG_SCSI_DPT_I2O is not set
# CONFIG_SCSI_ADVANSYS is not set
# CONFIG_SCSI_ARCMSR is not set
# CONFIG_MEGARAID_NEWGEN is not set
# CONFIG_MEGARAID_LEGACY is not set
# CONFIG_MEGARAID_SAS is not set
# CONFIG_SCSI_HPTIOP is not set
# CONFIG_SCSI_BUSLOGIC is not set
# CONFIG_SCSI_DMX3191D is not set
# CONFIG_SCSI_EATA is not set
# CONFIG_SCSI_FUTURE_DOMAIN is not set
# CONFIG_SCSI_GDTH is not set
# CONFIG_SCSI_IPS is not set
# CONFIG_SCSI_INITIO is not set
# CONFIG_SCSI_INIA100 is not set
# CONFIG_SCSI_PPA is not set
# CONFIG_SCSI_IMM is not set
# CONFIG_SCSI_MVSAS is not set
# CONFIG_SCSI_STEX is not set
# CONFIG_SCSI_SYM53C8XX_2 is not set
# CONFIG_SCSI_IPR is not set
# CONFIG_SCSI_QLOGIC_1280 is not set
# CONFIG_SCSI_QLA_FC is not set
# CONFIG_SCSI_QLA_ISCSI is not set
# CONFIG_SCSI_LPFC is not set
# CONFIG_SCSI_DC395x is not set
# CONFIG_SCSI_DC390T is not set
# CONFIG_SCSI_DEBUG is not set
# CONFIG_SCSI_SRP is not set
CONFIG_ATA=m
# CONFIG_ATA_NONSTANDARD is not set
CONFIG_ATA_ACPI=y
CONFIG_SATA_PMP=y
CONFIG_SATA_AHCI=m
CONFIG_SATA_SIL24=m
CONFIG_ATA_SFF=y
# CONFIG_SATA_SVW is not set
# CONFIG_ATA_PIIX is not set
# CONFIG_SATA_MV is not set
CONFIG_SATA_NV=m
# CONFIG_PDC_ADMA is not set
# CONFIG_SATA_QSTOR is not set
# CONFIG_SATA_PROMISE is not set
# CONFIG_SATA_SX4 is not set
CONFIG_SATA_SIL=m
# CONFIG_SATA_SIS is not set
# CONFIG_SATA_ULI is not set
# CONFIG_SATA_VIA is not set
# CONFIG_SATA_VITESSE is not set
# CONFIG_SATA_INIC162X is not set
# CONFIG_PATA_ACPI is not set
# CONFIG_PATA_ALI is not set
CONFIG_PATA_AMD=m
# CONFIG_PATA_ARTOP is not set
# CONFIG_PATA_ATIIXP is not set
# CONFIG_PATA_CMD640_PCI is not set
# CONFIG_PATA_CMD64X is not set
# CONFIG_PATA_CS5520 is not set
# CONFIG_PATA_CS5530 is not set
# CONFIG_PATA_CYPRESS is not set
# CONFIG_PATA_EFAR is not set
# CONFIG_ATA_GENERIC is not set
# CONFIG_PATA_HPT366 is not set
# CONFIG_PATA_HPT37X is not set
# CONFIG_PATA_HPT3X2N is not set
# CONFIG_PATA_HPT3X3 is not set
# CONFIG_PATA_IT821X is not set
# CONFIG_PATA_IT8213 is not set
# CONFIG_PATA_JMICRON is not set
# CONFIG_PATA_TRIFLEX is not set
# CONFIG_PATA_MARVELL is not set
# CONFIG_PATA_MPIIX is not set
# CONFIG_PATA_OLDPIIX is not set
# CONFIG_PATA_NETCELL is not set
# CONFIG_PATA_NINJA32 is not set
# CONFIG_PATA_NS87410 is not set
# CONFIG_PATA_NS87415 is not set
# CONFIG_PATA_OPTI is not set
# CONFIG_PATA_OPTIDMA is not set
# CONFIG_PATA_PDC_OLD is not set
# CONFIG_PATA_RADISYS is not set
# CONFIG_PATA_RZ1000 is not set
# CONFIG_PATA_SC1200 is not set
# CONFIG_PATA_SERVERWORKS is not set
# CONFIG_PATA_PDC2027X is not set
# CONFIG_PATA_SIL680 is not set
# CONFIG_PATA_SIS is not set
# CONFIG_PATA_VIA is not set
# CONFIG_PATA_WINBOND is not set
# CONFIG_PATA_SCH is not set
CONFIG_MD=y
CONFIG_BLK_DEV_MD=y
CONFIG_MD_LINEAR=m
CONFIG_MD_RAID0=y
CONFIG_MD_RAID1=y
CONFIG_MD_RAID10=m
CONFIG_MD_RAID456=m
CONFIG_MD_RAID5_RESHAPE=y
CONFIG_MD_MULTIPATH=m
CONFIG_MD_FAULTY=m
CONFIG_BLK_DEV_DM=m
# CONFIG_DM_DEBUG is not set
CONFIG_DM_CRYPT=m
CONFIG_DM_SNAPSHOT=m
CONFIG_DM_MIRROR=m
CONFIG_DM_ZERO=m
CONFIG_DM_MULTIPATH=m
CONFIG_DM_MULTIPATH_EMC=m
# CONFIG_DM_MULTIPATH_RDAC is not set
# CONFIG_DM_MULTIPATH_HP is not set
# CONFIG_DM_DELAY is not set
# CONFIG_DM_UEVENT is not set
# CONFIG_FUSION is not set
#
# IEEE 1394 (FireWire) support
#
#
# Enable only one of the two stacks, unless you know what you are doing
#
# CONFIG_FIREWIRE is not set
# CONFIG_IEEE1394 is not set
CONFIG_I2O=m
CONFIG_I2O_LCT_NOTIFY_ON_CHANGES=y
CONFIG_I2O_EXT_ADAPTEC=y
CONFIG_I2O_EXT_ADAPTEC_DMA64=y
CONFIG_I2O_CONFIG=m
CONFIG_I2O_CONFIG_OLD_IOCTL=y
CONFIG_I2O_BUS=m
CONFIG_I2O_BLOCK=m
CONFIG_I2O_SCSI=m
CONFIG_I2O_PROC=m
# CONFIG_MACINTOSH_DRIVERS is not set
CONFIG_NETDEVICES=y
# CONFIG_NETDEVICES_MULTIQUEUE is not set
CONFIG_IFB=m
CONFIG_DUMMY=m
CONFIG_BONDING=m
CONFIG_MACVLAN=m
CONFIG_EQUALIZER=m
CONFIG_TUN=m
CONFIG_VETH=m
# CONFIG_NET_SB1000 is not set
# CONFIG_ARCNET is not set
CONFIG_PHYLIB=m
#
# MII PHY device drivers
#
CONFIG_MARVELL_PHY=m
CONFIG_DAVICOM_PHY=m
CONFIG_QSEMI_PHY=m
CONFIG_LXT_PHY=m
CONFIG_CICADA_PHY=m
CONFIG_VITESSE_PHY=m
CONFIG_SMSC_PHY=m
CONFIG_BROADCOM_PHY=m
CONFIG_ICPLUS_PHY=m
CONFIG_REALTEK_PHY=m
# CONFIG_MDIO_BITBANG is not set
CONFIG_NET_ETHERNET=y
CONFIG_MII=m
# CONFIG_HAPPYMEAL is not set
# CONFIG_SUNGEM is not set
# CONFIG_CASSINI is not set
# CONFIG_NET_VENDOR_3COM is not set
# CONFIG_NET_TULIP is not set
# CONFIG_HP100 is not set
# CONFIG_IBM_NEW_EMAC_ZMII is not set
# CONFIG_IBM_NEW_EMAC_RGMII is not set
# CONFIG_IBM_NEW_EMAC_TAH is not set
# CONFIG_IBM_NEW_EMAC_EMAC4 is not set
CONFIG_NET_PCI=y
# CONFIG_PCNET32 is not set
# CONFIG_AMD8111_ETH is not set
# CONFIG_ADAPTEC_STARFIRE is not set
# CONFIG_B44 is not set
CONFIG_FORCEDETH=m
# CONFIG_FORCEDETH_NAPI is not set
# CONFIG_EEPRO100 is not set
# CONFIG_E100 is not set
# CONFIG_FEALNX is not set
# CONFIG_NATSEMI is not set
# CONFIG_NE2K_PCI is not set
# CONFIG_8139CP is not set
# CONFIG_8139TOO is not set
# CONFIG_R6040 is not set
# CONFIG_SIS900 is not set
# CONFIG_EPIC100 is not set
# CONFIG_SUNDANCE is not set
# CONFIG_VIA_RHINE is not set
# CONFIG_SC92031 is not set
# CONFIG_NET_POCKET is not set
# CONFIG_NETDEV_1000 is not set
# CONFIG_NETDEV_10000 is not set
# CONFIG_TR is not set
#
# Wireless LAN
#
# CONFIG_WLAN_PRE80211 is not set
# CONFIG_WLAN_80211 is not set
# CONFIG_IWLWIFI_LEDS is not set
#
# USB Network Adapters
#
CONFIG_USB_CATC=m
CONFIG_USB_KAWETH=m
CONFIG_USB_PEGASUS=m
CONFIG_USB_RTL8150=m
CONFIG_USB_USBNET=m
CONFIG_USB_NET_AX8817X=m
CONFIG_USB_NET_CDCETHER=m
CONFIG_USB_NET_DM9601=m
CONFIG_USB_NET_GL620A=m
CONFIG_USB_NET_NET1080=m
CONFIG_USB_NET_PLUSB=m
CONFIG_USB_NET_MCS7830=m
CONFIG_USB_NET_RNDIS_HOST=m
CONFIG_USB_NET_CDC_SUBSET=m
CONFIG_USB_ALI_M5632=y
CONFIG_USB_AN2720=y
CONFIG_USB_BELKIN=y
CONFIG_USB_ARMLINUX=y
CONFIG_USB_EPSON2888=y
# CONFIG_USB_KC2190 is not set
CONFIG_USB_NET_ZAURUS=m
# CONFIG_WAN is not set
# CONFIG_ATM_DRIVERS is not set
# CONFIG_FDDI is not set
# CONFIG_HIPPI is not set
# CONFIG_PLIP is not set
CONFIG_PPP=m
CONFIG_PPP_MULTILINK=y
CONFIG_PPP_FILTER=y
CONFIG_PPP_ASYNC=m
CONFIG_PPP_SYNC_TTY=m
CONFIG_PPP_DEFLATE=m
CONFIG_PPP_BSDCOMP=m
CONFIG_PPP_MPPE=m
CONFIG_PPPOE=m
CONFIG_PPPOATM=m
# CONFIG_PPPOL2TP is not set
# CONFIG_SLIP is not set
CONFIG_SLHC=m
# CONFIG_NET_FC is not set
CONFIG_NETCONSOLE=m
# CONFIG_NETCONSOLE_DYNAMIC is not set
CONFIG_NETPOLL=y
# CONFIG_NETPOLL_TRAP is not set
CONFIG_NET_POLL_CONTROLLER=y
# CONFIG_ISDN is not set
# CONFIG_PHONE is not set
#
# Input device support
#
CONFIG_INPUT=y
CONFIG_INPUT_FF_MEMLESS=m
CONFIG_INPUT_POLLDEV=m
#
# Userland interfaces
#
CONFIG_INPUT_MOUSEDEV=y
CONFIG_INPUT_MOUSEDEV_PSAUX=y
CONFIG_INPUT_MOUSEDEV_SCREEN_X=1024
CONFIG_INPUT_MOUSEDEV_SCREEN_Y=768
CONFIG_INPUT_JOYDEV=m
CONFIG_INPUT_EVDEV=m
# CONFIG_INPUT_EVBUG is not set
#
# Input Device Drivers
#
CONFIG_INPUT_KEYBOARD=y
CONFIG_KEYBOARD_ATKBD=y
CONFIG_KEYBOARD_SUNKBD=m
CONFIG_KEYBOARD_LKKBD=m
CONFIG_KEYBOARD_XTKBD=m
CONFIG_KEYBOARD_NEWTON=m
CONFIG_KEYBOARD_STOWAWAY=m
CONFIG_INPUT_MOUSE=y
CONFIG_MOUSE_PS2=m
CONFIG_MOUSE_PS2_ALPS=y
CONFIG_MOUSE_PS2_LOGIPS2PP=y
CONFIG_MOUSE_PS2_SYNAPTICS=y
CONFIG_MOUSE_PS2_LIFEBOOK=y
CONFIG_MOUSE_PS2_TRACKPOINT=y
# CONFIG_MOUSE_PS2_TOUCHKIT is not set
CONFIG_MOUSE_SERIAL=m
# CONFIG_MOUSE_APPLETOUCH is not set
CONFIG_MOUSE_VSXXXAA=m
CONFIG_INPUT_JOYSTICK=y
CONFIG_JOYSTICK_ANALOG=m
CONFIG_JOYSTICK_A3D=m
CONFIG_JOYSTICK_ADI=m
CONFIG_JOYSTICK_COBRA=m
CONFIG_JOYSTICK_GF2K=m
CONFIG_JOYSTICK_GRIP=m
CONFIG_JOYSTICK_GRIP_MP=m
CONFIG_JOYSTICK_GUILLEMOT=m
CONFIG_JOYSTICK_INTERACT=m
CONFIG_JOYSTICK_SIDEWINDER=m
CONFIG_JOYSTICK_TMDC=m
CONFIG_JOYSTICK_IFORCE=m
CONFIG_JOYSTICK_IFORCE_USB=y
CONFIG_JOYSTICK_IFORCE_232=y
CONFIG_JOYSTICK_WARRIOR=m
CONFIG_JOYSTICK_MAGELLAN=m
CONFIG_JOYSTICK_SPACEORB=m
CONFIG_JOYSTICK_SPACEBALL=m
CONFIG_JOYSTICK_STINGER=m
CONFIG_JOYSTICK_TWIDJOY=m
# CONFIG_JOYSTICK_ZHENHUA is not set
CONFIG_JOYSTICK_DB9=m
CONFIG_JOYSTICK_GAMECON=m
CONFIG_JOYSTICK_TURBOGRAFX=m
CONFIG_JOYSTICK_JOYDUMP=m
# CONFIG_JOYSTICK_XPAD is not set
# CONFIG_INPUT_TABLET is not set
# CONFIG_INPUT_TOUCHSCREEN is not set
CONFIG_INPUT_MISC=y
CONFIG_INPUT_PCSPKR=m
# CONFIG_INPUT_ATLAS_BTNS is not set
# CONFIG_INPUT_ATI_REMOTE is not set
# CONFIG_INPUT_ATI_REMOTE2 is not set
# CONFIG_INPUT_KEYSPAN_REMOTE is not set
# CONFIG_INPUT_POWERMATE is not set
# CONFIG_INPUT_YEALINK is not set
CONFIG_INPUT_UINPUT=m
#
# Hardware I/O ports
#
CONFIG_SERIO=y
CONFIG_SERIO_I8042=y
CONFIG_SERIO_SERPORT=m
CONFIG_SERIO_CT82C710=m
CONFIG_SERIO_PARKBD=m
CONFIG_SERIO_PCIPS2=m
CONFIG_SERIO_LIBPS2=y
CONFIG_SERIO_RAW=m
CONFIG_GAMEPORT=m
CONFIG_GAMEPORT_NS558=m
CONFIG_GAMEPORT_L4=m
CONFIG_GAMEPORT_EMU10K1=m
CONFIG_GAMEPORT_FM801=m
#
# Character devices
#
CONFIG_VT=y
CONFIG_VT_CONSOLE=y
CONFIG_HW_CONSOLE=y
# CONFIG_VT_HW_CONSOLE_BINDING is not set
CONFIG_DEVKMEM=y
# CONFIG_SERIAL_NONSTANDARD is not set
# CONFIG_NOZOMI is not set
#
# Serial drivers
#
CONFIG_SERIAL_8250=y
CONFIG_SERIAL_8250_CONSOLE=y
CONFIG_FIX_EARLYCON_MEM=y
CONFIG_SERIAL_8250_PCI=y
CONFIG_SERIAL_8250_PNP=y
CONFIG_SERIAL_8250_NR_UARTS=16
CONFIG_SERIAL_8250_RUNTIME_UARTS=4
CONFIG_SERIAL_8250_EXTENDED=y
CONFIG_SERIAL_8250_MANY_PORTS=y
CONFIG_SERIAL_8250_SHARE_IRQ=y
# CONFIG_SERIAL_8250_DETECT_IRQ is not set
CONFIG_SERIAL_8250_RSA=y
#
# Non-8250 serial port support
#
CONFIG_SERIAL_CORE=y
CONFIG_SERIAL_CORE_CONSOLE=y
CONFIG_SERIAL_JSM=m
CONFIG_UNIX98_PTYS=y
# CONFIG_LEGACY_PTYS is not set
CONFIG_PRINTER=m
# CONFIG_LP_CONSOLE is not set
CONFIG_PPDEV=m
CONFIG_IPMI_HANDLER=m
# CONFIG_IPMI_PANIC_EVENT is not set
CONFIG_IPMI_DEVICE_INTERFACE=m
CONFIG_IPMI_SI=m
CONFIG_IPMI_WATCHDOG=m
CONFIG_IPMI_POWEROFF=m
CONFIG_HW_RANDOM=y
CONFIG_HW_RANDOM_INTEL=m
CONFIG_HW_RANDOM_AMD=m
CONFIG_NVRAM=m
# CONFIG_R3964 is not set
# CONFIG_APPLICOM is not set
# CONFIG_MWAVE is not set
# CONFIG_PC8736x_GPIO is not set
# CONFIG_RAW_DRIVER is not set
CONFIG_HPET=y
CONFIG_HPET_MMAP=y
CONFIG_HANGCHECK_TIMER=m
# CONFIG_TCG_TPM is not set
# CONFIG_TELCLOCK is not set
CONFIG_DEVPORT=y
CONFIG_I2C=m
CONFIG_I2C_BOARDINFO=y
CONFIG_I2C_CHARDEV=m
CONFIG_I2C_ALGOBIT=m
#
# I2C Hardware Bus support
#
CONFIG_I2C_ALI1535=m
CONFIG_I2C_ALI1563=m
CONFIG_I2C_ALI15X3=m
CONFIG_I2C_AMD756=m
CONFIG_I2C_AMD756_S4882=m
CONFIG_I2C_AMD8111=m
CONFIG_I2C_I801=m
CONFIG_I2C_I810=m
CONFIG_I2C_PIIX4=m
CONFIG_I2C_NFORCE2=m
CONFIG_I2C_OCORES=m
CONFIG_I2C_PARPORT=m
CONFIG_I2C_PARPORT_LIGHT=m
CONFIG_I2C_PROSAVAGE=m
CONFIG_I2C_SAVAGE4=m
# CONFIG_I2C_SIMTEC is not set
CONFIG_I2C_SIS5595=m
CONFIG_I2C_SIS630=m
CONFIG_I2C_SIS96X=m
# CONFIG_I2C_TAOS_EVM is not set
CONFIG_I2C_STUB=m
# CONFIG_I2C_TINY_USB is not set
CONFIG_I2C_VIA=m
CONFIG_I2C_VIAPRO=m
CONFIG_I2C_VOODOO3=m
# CONFIG_I2C_PCA_PLATFORM is not set
#
# Miscellaneous I2C Chip support
#
CONFIG_DS1682=m
CONFIG_SENSORS_EEPROM=m
CONFIG_SENSORS_PCF8574=m
CONFIG_PCF8575=m
CONFIG_SENSORS_PCF8591=m
CONFIG_SENSORS_MAX6875=m
# CONFIG_SENSORS_TSL2550 is not set
# CONFIG_I2C_DEBUG_CORE is not set
# CONFIG_I2C_DEBUG_ALGO is not set
# CONFIG_I2C_DEBUG_BUS is not set
# CONFIG_I2C_DEBUG_CHIP is not set
# CONFIG_SPI is not set
# CONFIG_W1 is not set
CONFIG_POWER_SUPPLY=y
# CONFIG_POWER_SUPPLY_DEBUG is not set
# CONFIG_PDA_POWER is not set
# CONFIG_BATTERY_DS2760 is not set
CONFIG_HWMON=y
CONFIG_HWMON_VID=m
CONFIG_SENSORS_ABITUGURU=m
# CONFIG_SENSORS_ABITUGURU3 is not set
# CONFIG_SENSORS_AD7418 is not set
CONFIG_SENSORS_ADM1021=m
CONFIG_SENSORS_ADM1025=m
CONFIG_SENSORS_ADM1026=m
# CONFIG_SENSORS_ADM1029 is not set
CONFIG_SENSORS_ADM1031=m
CONFIG_SENSORS_ADM9240=m
CONFIG_SENSORS_ADT7470=m
CONFIG_SENSORS_ADT7473=m
CONFIG_SENSORS_K8TEMP=m
CONFIG_SENSORS_ASB100=m
CONFIG_SENSORS_ATXP1=m
CONFIG_SENSORS_DS1621=m
CONFIG_SENSORS_I5K_AMB=m
CONFIG_SENSORS_F71805F=m
CONFIG_SENSORS_F71882FG=m
CONFIG_SENSORS_F75375S=m
CONFIG_SENSORS_FSCHER=m
CONFIG_SENSORS_FSCPOS=m
CONFIG_SENSORS_FSCHMD=m
CONFIG_SENSORS_GL518SM=m
CONFIG_SENSORS_GL520SM=m
# CONFIG_SENSORS_CORETEMP is not set
# CONFIG_SENSORS_IBMAEM is not set
CONFIG_SENSORS_IBMPEX=m
CONFIG_SENSORS_IT87=m
CONFIG_SENSORS_LM63=m
CONFIG_SENSORS_LM75=m
CONFIG_SENSORS_LM77=m
CONFIG_SENSORS_LM78=m
CONFIG_SENSORS_LM80=m
CONFIG_SENSORS_LM83=m
CONFIG_SENSORS_LM85=m
CONFIG_SENSORS_LM87=m
CONFIG_SENSORS_LM90=m
CONFIG_SENSORS_LM92=m
CONFIG_SENSORS_LM93=m
CONFIG_SENSORS_MAX1619=m
# CONFIG_SENSORS_MAX6650 is not set
CONFIG_SENSORS_PC87360=m
CONFIG_SENSORS_PC87427=m
CONFIG_SENSORS_SIS5595=m
CONFIG_SENSORS_DME1737=m
CONFIG_SENSORS_SMSC47M1=m
CONFIG_SENSORS_SMSC47M192=m
CONFIG_SENSORS_SMSC47B397=m
CONFIG_SENSORS_ADS7828=m
CONFIG_SENSORS_THMC50=m
CONFIG_SENSORS_VIA686A=m
CONFIG_SENSORS_VT1211=m
CONFIG_SENSORS_VT8231=m
CONFIG_SENSORS_W83781D=m
CONFIG_SENSORS_W83791D=m
CONFIG_SENSORS_W83792D=m
CONFIG_SENSORS_W83793=m
CONFIG_SENSORS_W83L785TS=m
CONFIG_SENSORS_W83L786NG=m
CONFIG_SENSORS_W83627HF=m
CONFIG_SENSORS_W83627EHF=m
CONFIG_SENSORS_HDAPS=m
# CONFIG_SENSORS_APPLESMC is not set
# CONFIG_HWMON_DEBUG_CHIP is not set
CONFIG_THERMAL=y
# CONFIG_THERMAL_HWMON is not set
CONFIG_WATCHDOG=y
# CONFIG_WATCHDOG_NOWAYOUT is not set
#
# Watchdog Device Drivers
#
CONFIG_SOFT_WATCHDOG=m
CONFIG_ACQUIRE_WDT=m
CONFIG_ADVANTECH_WDT=m
CONFIG_ALIM1535_WDT=m
CONFIG_ALIM7101_WDT=m
CONFIG_SC520_WDT=m
CONFIG_EUROTECH_WDT=m
CONFIG_IB700_WDT=m
CONFIG_IBMASR=m
CONFIG_WAFER_WDT=m
CONFIG_I6300ESB_WDT=m
CONFIG_ITCO_WDT=m
# CONFIG_ITCO_VENDOR_SUPPORT is not set
CONFIG_IT8712F_WDT=m
# CONFIG_HP_WATCHDOG is not set
CONFIG_SC1200_WDT=m
CONFIG_PC87413_WDT=m
CONFIG_60XX_WDT=m
CONFIG_SBC8360_WDT=m
CONFIG_CPU5_WDT=m
CONFIG_SMSC37B787_WDT=m
CONFIG_W83627HF_WDT=m
CONFIG_W83697HF_WDT=m
CONFIG_W83877F_WDT=m
CONFIG_W83977F_WDT=m
CONFIG_MACHZ_WDT=m
CONFIG_SBC_EPX_C3_WATCHDOG=m
#
# PCI-based Watchdog Cards
#
CONFIG_PCIPCWATCHDOG=m
CONFIG_WDTPCI=m
CONFIG_WDT_501_PCI=y
#
# USB-based Watchdog Cards
#
CONFIG_USBPCWATCHDOG=m
#
# Sonics Silicon Backplane
#
CONFIG_SSB_POSSIBLE=y
# CONFIG_SSB is not set
#
# Multifunction device drivers
#
# CONFIG_MFD_SM501 is not set
# CONFIG_HTC_PASIC3 is not set
#
# Multimedia devices
#
#
# Multimedia core support
#
CONFIG_VIDEO_DEV=m
CONFIG_VIDEO_V4L2_COMMON=m
CONFIG_VIDEO_ALLOW_V4L1=y
CONFIG_VIDEO_V4L1_COMPAT=y
# CONFIG_DVB_CORE is not set
CONFIG_VIDEO_MEDIA=m
#
# Multimedia drivers
#
# CONFIG_MEDIA_ATTACH is not set
CONFIG_MEDIA_TUNER=m
# CONFIG_MEDIA_TUNER_CUSTOMIZE is not set
CONFIG_MEDIA_TUNER_SIMPLE=m
CONFIG_MEDIA_TUNER_TDA8290=m
CONFIG_MEDIA_TUNER_TDA9887=m
CONFIG_MEDIA_TUNER_TEA5761=m
CONFIG_MEDIA_TUNER_TEA5767=m
CONFIG_MEDIA_TUNER_MT20XX=m
CONFIG_MEDIA_TUNER_XC2028=m
CONFIG_MEDIA_TUNER_XC5000=m
CONFIG_VIDEO_V4L2=m
CONFIG_VIDEO_V4L1=m
# CONFIG_VIDEO_CAPTURE_DRIVERS is not set
# CONFIG_RADIO_ADAPTERS is not set
# CONFIG_DAB is not set
#
# Graphics support
#
CONFIG_AGP=y
CONFIG_AGP_AMD64=y
# CONFIG_AGP_INTEL is not set
# CONFIG_AGP_SIS is not set
# CONFIG_AGP_VIA is not set
# CONFIG_DRM is not set
CONFIG_VGASTATE=m
CONFIG_VIDEO_OUTPUT_CONTROL=m
CONFIG_FB=y
CONFIG_FIRMWARE_EDID=y
# CONFIG_FB_DDC is not set
CONFIG_FB_CFB_FILLRECT=y
CONFIG_FB_CFB_COPYAREA=y
CONFIG_FB_CFB_IMAGEBLIT=y
# CONFIG_FB_CFB_REV_PIXELS_IN_BYTE is not set
# CONFIG_FB_SYS_FILLRECT is not set
# CONFIG_FB_SYS_COPYAREA is not set
# CONFIG_FB_SYS_IMAGEBLIT is not set
# CONFIG_FB_FOREIGN_ENDIAN is not set
# CONFIG_FB_SYS_FOPS is not set
# CONFIG_FB_SVGALIB is not set
# CONFIG_FB_MACMODES is not set
# CONFIG_FB_BACKLIGHT is not set
CONFIG_FB_MODE_HELPERS=y
CONFIG_FB_TILEBLITTING=y
#
# Frame buffer hardware drivers
#
# CONFIG_FB_CIRRUS is not set
# CONFIG_FB_PM2 is not set
# CONFIG_FB_CYBER2000 is not set
# CONFIG_FB_ARC is not set
# CONFIG_FB_ASILIANT is not set
# CONFIG_FB_IMSTT is not set
CONFIG_FB_VGA16=m
# CONFIG_FB_UVESA is not set
CONFIG_FB_VESA=y
# CONFIG_FB_EFI is not set
# CONFIG_FB_N411 is not set
# CONFIG_FB_HGA is not set
# CONFIG_FB_S1D13XXX is not set
# CONFIG_FB_NVIDIA is not set
# CONFIG_FB_RIVA is not set
# CONFIG_FB_LE80578 is not set
# CONFIG_FB_INTEL is not set
# CONFIG_FB_MATROX is not set
# CONFIG_FB_RADEON is not set
# CONFIG_FB_ATY128 is not set
# CONFIG_FB_ATY is not set
# CONFIG_FB_S3 is not set
# CONFIG_FB_SAVAGE is not set
# CONFIG_FB_SIS is not set
# CONFIG_FB_NEOMAGIC is not set
# CONFIG_FB_KYRO is not set
# CONFIG_FB_3DFX is not set
# CONFIG_FB_VOODOO1 is not set
# CONFIG_FB_VT8623 is not set
# CONFIG_FB_TRIDENT is not set
# CONFIG_FB_ARK is not set
# CONFIG_FB_PM3 is not set
# CONFIG_FB_GEODE is not set
# CONFIG_FB_VIRTUAL is not set
CONFIG_BACKLIGHT_LCD_SUPPORT=y
# CONFIG_LCD_CLASS_DEVICE is not set
CONFIG_BACKLIGHT_CLASS_DEVICE=y
# CONFIG_BACKLIGHT_CORGI is not set
# CONFIG_BACKLIGHT_PROGEAR is not set
#
# Display device support
#
# CONFIG_DISPLAY_SUPPORT is not set
#
# Console display driver support
#
CONFIG_VGA_CONSOLE=y
# CONFIG_VGACON_SOFT_SCROLLBACK is not set
CONFIG_VIDEO_SELECT=y
CONFIG_DUMMY_CONSOLE=y
CONFIG_FRAMEBUFFER_CONSOLE=y
# CONFIG_FRAMEBUFFER_CONSOLE_DETECT_PRIMARY is not set
CONFIG_FRAMEBUFFER_CONSOLE_ROTATION=y
# CONFIG_FONTS is not set
CONFIG_FONT_8x8=y
CONFIG_FONT_8x16=y
# CONFIG_LOGO is not set
#
# Sound
#
CONFIG_SOUND=m
#
# Advanced Linux Sound Architecture
#
CONFIG_SND=m
CONFIG_SND_TIMER=m
CONFIG_SND_PCM=m
CONFIG_SND_HWDEP=m
CONFIG_SND_RAWMIDI=m
CONFIG_SND_SEQUENCER=m
CONFIG_SND_SEQ_DUMMY=m
CONFIG_SND_OSSEMUL=y
CONFIG_SND_MIXER_OSS=m
CONFIG_SND_PCM_OSS=m
CONFIG_SND_PCM_OSS_PLUGINS=y
CONFIG_SND_SEQUENCER_OSS=y
# CONFIG_SND_DYNAMIC_MINORS is not set
CONFIG_SND_SUPPORT_OLD_API=y
CONFIG_SND_VERBOSE_PROCFS=y
# CONFIG_SND_VERBOSE_PRINTK is not set
# CONFIG_SND_DEBUG is not set
CONFIG_SND_VMASTER=y
#
# Generic devices
#
# CONFIG_SND_PCSP is not set
CONFIG_SND_MPU401_UART=m
CONFIG_SND_OPL3_LIB=m
CONFIG_SND_VX_LIB=m
CONFIG_SND_AC97_CODEC=m
CONFIG_SND_DUMMY=m
CONFIG_SND_VIRMIDI=m
# CONFIG_SND_MTPAV is not set
# CONFIG_SND_MTS64 is not set
CONFIG_SND_SERIAL_U16550=m
CONFIG_SND_MPU401=m
# CONFIG_SND_PORTMAN2X4 is not set
CONFIG_SND_SB_COMMON=m
#
# PCI devices
#
CONFIG_SND_AD1889=m
CONFIG_SND_ALS300=m
CONFIG_SND_ALS4000=m
CONFIG_SND_ALI5451=m
CONFIG_SND_ATIIXP=m
CONFIG_SND_ATIIXP_MODEM=m
CONFIG_SND_AU8810=m
CONFIG_SND_AU8820=m
CONFIG_SND_AU8830=m
# CONFIG_SND_AW2 is not set
CONFIG_SND_AZT3328=m
CONFIG_SND_BT87X=m
# CONFIG_SND_BT87X_OVERCLOCK is not set
CONFIG_SND_CA0106=m
CONFIG_SND_CMIPCI=m
# CONFIG_SND_OXYGEN is not set
CONFIG_SND_CS4281=m
CONFIG_SND_CS46XX=m
CONFIG_SND_CS46XX_NEW_DSP=y
# CONFIG_SND_CS5530 is not set
CONFIG_SND_DARLA20=m
CONFIG_SND_GINA20=m
CONFIG_SND_LAYLA20=m
CONFIG_SND_DARLA24=m
CONFIG_SND_GINA24=m
CONFIG_SND_LAYLA24=m
CONFIG_SND_MONA=m
CONFIG_SND_MIA=m
CONFIG_SND_ECHO3G=m
CONFIG_SND_INDIGO=m
CONFIG_SND_INDIGOIO=m
CONFIG_SND_INDIGODJ=m
CONFIG_SND_EMU10K1=m
CONFIG_SND_EMU10K1X=m
CONFIG_SND_ENS1370=m
CONFIG_SND_ENS1371=m
CONFIG_SND_ES1938=m
CONFIG_SND_ES1968=m
CONFIG_SND_FM801=m
CONFIG_SND_FM801_TEA575X_BOOL=y
CONFIG_SND_FM801_TEA575X=m
CONFIG_SND_HDA_INTEL=m
CONFIG_SND_HDA_HWDEP=y
CONFIG_SND_HDA_CODEC_REALTEK=y
CONFIG_SND_HDA_CODEC_ANALOG=y
CONFIG_SND_HDA_CODEC_SIGMATEL=y
CONFIG_SND_HDA_CODEC_VIA=y
CONFIG_SND_HDA_CODEC_ATIHDMI=y
CONFIG_SND_HDA_CODEC_CONEXANT=y
CONFIG_SND_HDA_CODEC_CMEDIA=y
CONFIG_SND_HDA_CODEC_SI3054=y
CONFIG_SND_HDA_GENERIC=y
CONFIG_SND_HDA_POWER_SAVE=y
CONFIG_SND_HDA_POWER_SAVE_DEFAULT=200
CONFIG_SND_HDSP=m
CONFIG_SND_HDSPM=m
# CONFIG_SND_HIFIER is not set
CONFIG_SND_ICE1712=m
CONFIG_SND_ICE1724=m
CONFIG_SND_INTEL8X0=m
CONFIG_SND_INTEL8X0M=m
CONFIG_SND_KORG1212=m
CONFIG_SND_KORG1212_FIRMWARE_IN_KERNEL=y
CONFIG_SND_MAESTRO3=m
CONFIG_SND_MAESTRO3_FIRMWARE_IN_KERNEL=y
CONFIG_SND_MIXART=m
CONFIG_SND_NM256=m
CONFIG_SND_PCXHR=m
CONFIG_SND_RIPTIDE=m
CONFIG_SND_RME32=m
CONFIG_SND_RME96=m
CONFIG_SND_RME9652=m
CONFIG_SND_SONICVIBES=m
CONFIG_SND_TRIDENT=m
CONFIG_SND_VIA82XX=m
CONFIG_SND_VIA82XX_MODEM=m
# CONFIG_SND_VIRTUOSO is not set
CONFIG_SND_VX222=m
CONFIG_SND_YMFPCI=m
CONFIG_SND_YMFPCI_FIRMWARE_IN_KERNEL=y
CONFIG_SND_AC97_POWER_SAVE=y
CONFIG_SND_AC97_POWER_SAVE_DEFAULT=200
#
# USB devices
#
CONFIG_SND_USB_AUDIO=m
CONFIG_SND_USB_USX2Y=m
# CONFIG_SND_USB_CAIAQ is not set
#
# System on Chip audio support
#
# CONFIG_SND_SOC is not set
#
# ALSA SoC audio for Freescale SOCs
#
#
# SoC Audio for the Texas Instruments OMAP
#
#
# Open Sound System
#
# CONFIG_SOUND_PRIME is not set
CONFIG_AC97_BUS=m
CONFIG_HID_SUPPORT=y
CONFIG_HID=m
# CONFIG_HID_DEBUG is not set
CONFIG_HIDRAW=y
#
# USB Input Devices
#
CONFIG_USB_HID=m
CONFIG_USB_HIDINPUT_POWERBOOK=y
# CONFIG_HID_FF is not set
CONFIG_USB_HIDDEV=y
#
# USB HID Boot Protocol drivers
#
CONFIG_USB_KBD=m
CONFIG_USB_MOUSE=m
CONFIG_USB_SUPPORT=y
CONFIG_USB_ARCH_HAS_HCD=y
CONFIG_USB_ARCH_HAS_OHCI=y
CONFIG_USB_ARCH_HAS_EHCI=y
CONFIG_USB=y
# CONFIG_USB_DEBUG is not set
CONFIG_USB_ANNOUNCE_NEW_DEVICES=y
#
# Miscellaneous USB options
#
CONFIG_USB_DEVICEFS=y
CONFIG_USB_DEVICE_CLASS=y
# CONFIG_USB_DYNAMIC_MINORS is not set
# CONFIG_USB_SUSPEND is not set
# CONFIG_USB_OTG is not set
#
# USB Host Controller Drivers
#
# CONFIG_USB_C67X00_HCD is not set
CONFIG_USB_EHCI_HCD=m
CONFIG_USB_EHCI_ROOT_HUB_TT=y
# CONFIG_USB_EHCI_TT_NEWSCHED is not set
CONFIG_USB_ISP116X_HCD=m
# CONFIG_USB_ISP1760_HCD is not set
CONFIG_USB_OHCI_HCD=m
# CONFIG_USB_OHCI_BIG_ENDIAN_DESC is not set
# CONFIG_USB_OHCI_BIG_ENDIAN_MMIO is not set
CONFIG_USB_OHCI_LITTLE_ENDIAN=y
CONFIG_USB_UHCI_HCD=m
CONFIG_USB_U132_HCD=m
CONFIG_USB_SL811_HCD=m
# CONFIG_USB_R8A66597_HCD is not set
#
# USB Device Class drivers
#
CONFIG_USB_ACM=m
CONFIG_USB_PRINTER=m
# CONFIG_USB_WDM is not set
#
# NOTE: USB_STORAGE enables SCSI, and 'SCSI disk support'
#
#
# may also be needed; see USB_STORAGE Help for more information
#
CONFIG_USB_STORAGE=m
# CONFIG_USB_STORAGE_DEBUG is not set
CONFIG_USB_STORAGE_DATAFAB=y
CONFIG_USB_STORAGE_FREECOM=y
CONFIG_USB_STORAGE_ISD200=y
CONFIG_USB_STORAGE_DPCM=y
CONFIG_USB_STORAGE_USBAT=y
CONFIG_USB_STORAGE_SDDR09=y
CONFIG_USB_STORAGE_SDDR55=y
CONFIG_USB_STORAGE_JUMPSHOT=y
CONFIG_USB_STORAGE_ALAUDA=y
# CONFIG_USB_STORAGE_ONETOUCH is not set
CONFIG_USB_STORAGE_KARMA=y
# CONFIG_USB_STORAGE_CYPRESS_ATACB is not set
# CONFIG_USB_LIBUSUAL is not set
#
# USB Imaging devices
#
CONFIG_USB_MDC800=m
CONFIG_USB_MICROTEK=m
CONFIG_USB_MON=y
#
# USB port drivers
#
CONFIG_USB_USS720=m
CONFIG_USB_SERIAL=m
CONFIG_USB_EZUSB=y
CONFIG_USB_SERIAL_GENERIC=y
CONFIG_USB_SERIAL_AIRCABLE=m
CONFIG_USB_SERIAL_AIRPRIME=m
CONFIG_USB_SERIAL_ARK3116=m
CONFIG_USB_SERIAL_BELKIN=m
# CONFIG_USB_SERIAL_CH341 is not set
CONFIG_USB_SERIAL_WHITEHEAT=m
CONFIG_USB_SERIAL_DIGI_ACCELEPORT=m
CONFIG_USB_SERIAL_CP2101=m
CONFIG_USB_SERIAL_CYPRESS_M8=m
CONFIG_USB_SERIAL_EMPEG=m
CONFIG_USB_SERIAL_FTDI_SIO=m
CONFIG_USB_SERIAL_FUNSOFT=m
CONFIG_USB_SERIAL_VISOR=m
CONFIG_USB_SERIAL_IPAQ=m
CONFIG_USB_SERIAL_IR=m
CONFIG_USB_SERIAL_EDGEPORT=m
CONFIG_USB_SERIAL_EDGEPORT_TI=m
CONFIG_USB_SERIAL_GARMIN=m
CONFIG_USB_SERIAL_IPW=m
# CONFIG_USB_SERIAL_IUU is not set
# CONFIG_USB_SERIAL_KEYSPAN_PDA is not set
# CONFIG_USB_SERIAL_KEYSPAN is not set
CONFIG_USB_SERIAL_KLSI=m
CONFIG_USB_SERIAL_KOBIL_SCT=m
CONFIG_USB_SERIAL_MCT_U232=m
CONFIG_USB_SERIAL_MOS7720=m
CONFIG_USB_SERIAL_MOS7840=m
# CONFIG_USB_SERIAL_MOTOROLA is not set
CONFIG_USB_SERIAL_NAVMAN=m
CONFIG_USB_SERIAL_PL2303=m
# CONFIG_USB_SERIAL_OTI6858 is not set
# CONFIG_USB_SERIAL_SPCP8X5 is not set
CONFIG_USB_SERIAL_HP4X=m
CONFIG_USB_SERIAL_SAFE=m
# CONFIG_USB_SERIAL_SAFE_PADDED is not set
CONFIG_USB_SERIAL_SIERRAWIRELESS=m
CONFIG_USB_SERIAL_TI=m
CONFIG_USB_SERIAL_CYBERJACK=m
CONFIG_USB_SERIAL_XIRCOM=m
CONFIG_USB_SERIAL_OPTION=m
CONFIG_USB_SERIAL_OMNINET=m
CONFIG_USB_SERIAL_DEBUG=m
#
# USB Miscellaneous drivers
#
# CONFIG_USB_EMI62 is not set
# CONFIG_USB_EMI26 is not set
CONFIG_USB_ADUTUX=m
CONFIG_USB_AUERSWALD=m
CONFIG_USB_RIO500=m
CONFIG_USB_LEGOTOWER=m
CONFIG_USB_LCD=m
# CONFIG_USB_BERRY_CHARGE is not set
CONFIG_USB_LED=m
CONFIG_USB_CYPRESS_CY7C63=m
CONFIG_USB_CYTHERM=m
CONFIG_USB_PHIDGET=m
CONFIG_USB_PHIDGETKIT=m
CONFIG_USB_PHIDGETMOTORCONTROL=m
CONFIG_USB_PHIDGETSERVO=m
CONFIG_USB_IDMOUSE=m
CONFIG_USB_FTDI_ELAN=m
CONFIG_USB_APPLEDISPLAY=m
CONFIG_USB_SISUSBVGA=m
CONFIG_USB_SISUSBVGA_CON=y
CONFIG_USB_LD=m
CONFIG_USB_TRANCEVIBRATOR=m
CONFIG_USB_IOWARRIOR=m
CONFIG_USB_TEST=m
# CONFIG_USB_ISIGHTFW is not set
CONFIG_USB_ATM=m
CONFIG_USB_SPEEDTOUCH=m
CONFIG_USB_CXACRU=m
CONFIG_USB_UEAGLEATM=m
CONFIG_USB_XUSBATM=m
CONFIG_USB_GADGET=m
# CONFIG_USB_GADGET_DEBUG is not set
# CONFIG_USB_GADGET_DEBUG_FILES is not set
# CONFIG_USB_GADGET_DEBUG_FS is not set
CONFIG_USB_GADGET_SELECTED=y
# CONFIG_USB_GADGET_AMD5536UDC is not set
# CONFIG_USB_GADGET_ATMEL_USBA is not set
# CONFIG_USB_GADGET_FSL_USB2 is not set
CONFIG_USB_GADGET_NET2280=y
CONFIG_USB_NET2280=m
# CONFIG_USB_GADGET_PXA2XX is not set
# CONFIG_USB_GADGET_M66592 is not set
# CONFIG_USB_GADGET_PXA27X is not set
# CONFIG_USB_GADGET_GOKU is not set
# CONFIG_USB_GADGET_LH7A40X is not set
# CONFIG_USB_GADGET_OMAP is not set
# CONFIG_USB_GADGET_S3C2410 is not set
# CONFIG_USB_GADGET_AT91 is not set
# CONFIG_USB_GADGET_DUMMY_HCD is not set
CONFIG_USB_GADGET_DUALSPEED=y
CONFIG_USB_ZERO=m
CONFIG_USB_ETH=m
CONFIG_USB_ETH_RNDIS=y
CONFIG_USB_GADGETFS=m
CONFIG_USB_FILE_STORAGE=m
# CONFIG_USB_FILE_STORAGE_TEST is not set
CONFIG_USB_G_SERIAL=m
CONFIG_USB_MIDI_GADGET=m
# CONFIG_USB_G_PRINTER is not set
# CONFIG_MMC is not set
# CONFIG_MEMSTICK is not set
# CONFIG_NEW_LEDS is not set
# CONFIG_ACCESSIBILITY is not set
# CONFIG_INFINIBAND is not set
# CONFIG_EDAC is not set
CONFIG_RTC_LIB=m
CONFIG_RTC_CLASS=m
#
# RTC interfaces
#
CONFIG_RTC_INTF_SYSFS=y
CONFIG_RTC_INTF_PROC=y
CONFIG_RTC_INTF_DEV=y
# CONFIG_RTC_INTF_DEV_UIE_EMUL is not set
# CONFIG_RTC_DRV_TEST is not set
#
# I2C RTC drivers
#
CONFIG_RTC_DRV_DS1307=m
CONFIG_RTC_DRV_DS1374=m
CONFIG_RTC_DRV_DS1672=m
# CONFIG_RTC_DRV_MAX6900 is not set
CONFIG_RTC_DRV_RS5C372=m
CONFIG_RTC_DRV_ISL1208=m
CONFIG_RTC_DRV_X1205=m
CONFIG_RTC_DRV_PCF8563=m
# CONFIG_RTC_DRV_PCF8583 is not set
# CONFIG_RTC_DRV_M41T80 is not set
# CONFIG_RTC_DRV_S35390A is not set
# CONFIG_RTC_DRV_FM3130 is not set
#
# SPI RTC drivers
#
#
# Platform RTC drivers
#
CONFIG_RTC_DRV_CMOS=m
CONFIG_RTC_DRV_DS1511=m
CONFIG_RTC_DRV_DS1553=m
CONFIG_RTC_DRV_DS1742=m
CONFIG_RTC_DRV_STK17TA8=m
CONFIG_RTC_DRV_M48T86=m
CONFIG_RTC_DRV_M48T59=m
CONFIG_RTC_DRV_V3020=m
#
# on-CPU RTC drivers
#
CONFIG_DMADEVICES=y
#
# DMA Devices
#
CONFIG_INTEL_IOATDMA=m
CONFIG_DMA_ENGINE=y
#
# DMA Clients
#
CONFIG_NET_DMA=y
CONFIG_DCA=m
# CONFIG_AUXDISPLAY is not set
CONFIG_UIO=m
# CONFIG_UIO_CIF is not set
# CONFIG_UIO_SMX is not set
#
# Firmware Drivers
#
CONFIG_EDD=m
# CONFIG_EDD_OFF is not set
CONFIG_FIRMWARE_MEMMAP=y
CONFIG_DELL_RBU=m
CONFIG_DCDBAS=m
CONFIG_DMIID=y
# CONFIG_ISCSI_IBFT_FIND is not set
#
# File systems
#
CONFIG_EXT2_FS=y
CONFIG_EXT2_FS_XATTR=y
CONFIG_EXT2_FS_POSIX_ACL=y
CONFIG_EXT2_FS_SECURITY=y
# CONFIG_EXT2_FS_XIP is not set
CONFIG_EXT3_FS=m
CONFIG_EXT3_FS_XATTR=y
CONFIG_EXT3_FS_POSIX_ACL=y
CONFIG_EXT3_FS_SECURITY=y
CONFIG_EXT4DEV_FS=m
CONFIG_EXT4DEV_FS_XATTR=y
CONFIG_EXT4DEV_FS_POSIX_ACL=y
CONFIG_EXT4DEV_FS_SECURITY=y
CONFIG_JBD=m
# CONFIG_JBD_DEBUG is not set
CONFIG_JBD2=m
# CONFIG_JBD2_DEBUG is not set
CONFIG_FS_MBCACHE=y
CONFIG_REISERFS_FS=m
# CONFIG_REISERFS_CHECK is not set
# CONFIG_REISERFS_PROC_INFO is not set
CONFIG_REISERFS_FS_XATTR=y
CONFIG_REISERFS_FS_POSIX_ACL=y
CONFIG_REISERFS_FS_SECURITY=y
CONFIG_JFS_FS=m
CONFIG_JFS_POSIX_ACL=y
CONFIG_JFS_SECURITY=y
# CONFIG_JFS_DEBUG is not set
# CONFIG_JFS_STATISTICS is not set
CONFIG_FS_POSIX_ACL=y
CONFIG_XFS_FS=m
CONFIG_XFS_QUOTA=y
CONFIG_XFS_POSIX_ACL=y
CONFIG_XFS_RT=y
# CONFIG_XFS_DEBUG is not set
CONFIG_GFS2_FS=m
CONFIG_GFS2_FS_LOCKING_NOLOCK=m
CONFIG_GFS2_FS_LOCKING_DLM=m
# CONFIG_OCFS2_FS is not set
CONFIG_DNOTIFY=y
CONFIG_INOTIFY=y
CONFIG_INOTIFY_USER=y
CONFIG_QUOTA=y
CONFIG_QUOTA_NETLINK_INTERFACE=y
# CONFIG_PRINT_QUOTA_WARNING is not set
CONFIG_QFMT_V1=m
CONFIG_QFMT_V2=m
CONFIG_QUOTACTL=y
CONFIG_AUTOFS_FS=m
CONFIG_AUTOFS4_FS=m
CONFIG_FUSE_FS=m
CONFIG_GENERIC_ACL=y
#
# CD-ROM/DVD Filesystems
#
CONFIG_ISO9660_FS=m
CONFIG_JOLIET=y
CONFIG_ZISOFS=y
CONFIG_UDF_FS=m
CONFIG_UDF_NLS=y
#
# DOS/FAT/NT Filesystems
#
CONFIG_FAT_FS=m
CONFIG_MSDOS_FS=m
CONFIG_VFAT_FS=m
CONFIG_FAT_DEFAULT_CODEPAGE=437
CONFIG_FAT_DEFAULT_IOCHARSET="iso8859-1"
CONFIG_NTFS_FS=m
# CONFIG_NTFS_DEBUG is not set
CONFIG_NTFS_RW=y
#
# Pseudo filesystems
#
CONFIG_PROC_FS=y
CONFIG_PROC_KCORE=y
CONFIG_PROC_SYSCTL=y
CONFIG_SYSFS=y
CONFIG_TMPFS=y
CONFIG_TMPFS_POSIX_ACL=y
# CONFIG_HUGETLBFS is not set
# CONFIG_HUGETLB_PAGE is not set
CONFIG_CONFIGFS_FS=m
#
# Miscellaneous filesystems
#
CONFIG_ADFS_FS=m
# CONFIG_ADFS_FS_RW is not set
CONFIG_AFFS_FS=m
CONFIG_ECRYPT_FS=m
CONFIG_HFS_FS=m
CONFIG_HFSPLUS_FS=m
CONFIG_BEFS_FS=m
# CONFIG_BEFS_DEBUG is not set
CONFIG_BFS_FS=m
CONFIG_EFS_FS=m
# CONFIG_JFFS2_FS is not set
CONFIG_CRAMFS=y
CONFIG_VXFS_FS=m
CONFIG_MINIX_FS=m
CONFIG_HPFS_FS=m
CONFIG_QNX4FS_FS=m
CONFIG_ROMFS_FS=m
CONFIG_SYSV_FS=m
CONFIG_UFS_FS=m
# CONFIG_UFS_FS_WRITE is not set
# CONFIG_UFS_DEBUG is not set
CONFIG_NETWORK_FILESYSTEMS=y
# CONFIG_NFS_FS is not set
CONFIG_NFSD=m
CONFIG_NFSD_V3=y
# CONFIG_NFSD_V3_ACL is not set
# CONFIG_NFSD_V4 is not set
CONFIG_LOCKD=m
CONFIG_LOCKD_V4=y
CONFIG_EXPORTFS=m
CONFIG_NFS_COMMON=y
CONFIG_SUNRPC=m
# CONFIG_SUNRPC_BIND34 is not set
# CONFIG_RPCSEC_GSS_KRB5 is not set
# CONFIG_RPCSEC_GSS_SPKM3 is not set
# CONFIG_SMB_FS is not set
# CONFIG_CIFS is not set
# CONFIG_NCP_FS is not set
# CONFIG_CODA_FS is not set
# CONFIG_AFS_FS is not set
#
# Partition Types
#
CONFIG_PARTITION_ADVANCED=y
CONFIG_ACORN_PARTITION=y
# CONFIG_ACORN_PARTITION_CUMANA is not set
# CONFIG_ACORN_PARTITION_EESOX is not set
CONFIG_ACORN_PARTITION_ICS=y
# CONFIG_ACORN_PARTITION_ADFS is not set
# CONFIG_ACORN_PARTITION_POWERTEC is not set
CONFIG_ACORN_PARTITION_RISCIX=y
CONFIG_OSF_PARTITION=y
CONFIG_AMIGA_PARTITION=y
CONFIG_ATARI_PARTITION=y
CONFIG_MAC_PARTITION=y
CONFIG_MSDOS_PARTITION=y
CONFIG_BSD_DISKLABEL=y
CONFIG_MINIX_SUBPARTITION=y
CONFIG_SOLARIS_X86_PARTITION=y
CONFIG_UNIXWARE_DISKLABEL=y
CONFIG_LDM_PARTITION=y
# CONFIG_LDM_DEBUG is not set
CONFIG_SGI_PARTITION=y
CONFIG_ULTRIX_PARTITION=y
CONFIG_SUN_PARTITION=y
CONFIG_KARMA_PARTITION=y
CONFIG_EFI_PARTITION=y
# CONFIG_SYSV68_PARTITION is not set
CONFIG_NLS=y
CONFIG_NLS_DEFAULT="iso8859-1"
CONFIG_NLS_CODEPAGE_437=m
CONFIG_NLS_CODEPAGE_737=m
CONFIG_NLS_CODEPAGE_775=m
CONFIG_NLS_CODEPAGE_850=m
CONFIG_NLS_CODEPAGE_852=m
CONFIG_NLS_CODEPAGE_855=m
CONFIG_NLS_CODEPAGE_857=m
CONFIG_NLS_CODEPAGE_860=m
CONFIG_NLS_CODEPAGE_861=m
CONFIG_NLS_CODEPAGE_862=m
CONFIG_NLS_CODEPAGE_863=m
CONFIG_NLS_CODEPAGE_864=m
CONFIG_NLS_CODEPAGE_865=m
CONFIG_NLS_CODEPAGE_866=m
CONFIG_NLS_CODEPAGE_869=m
CONFIG_NLS_CODEPAGE_936=m
CONFIG_NLS_CODEPAGE_950=m
CONFIG_NLS_CODEPAGE_932=m
CONFIG_NLS_CODEPAGE_949=m
CONFIG_NLS_CODEPAGE_874=m
CONFIG_NLS_ISO8859_8=m
CONFIG_NLS_CODEPAGE_1250=m
CONFIG_NLS_CODEPAGE_1251=m
CONFIG_NLS_ASCII=m
CONFIG_NLS_ISO8859_1=m
CONFIG_NLS_ISO8859_2=m
CONFIG_NLS_ISO8859_3=m
CONFIG_NLS_ISO8859_4=m
CONFIG_NLS_ISO8859_5=m
CONFIG_NLS_ISO8859_6=m
CONFIG_NLS_ISO8859_7=m
CONFIG_NLS_ISO8859_9=m
CONFIG_NLS_ISO8859_13=m
CONFIG_NLS_ISO8859_14=m
CONFIG_NLS_ISO8859_15=m
CONFIG_NLS_KOI8_R=m
CONFIG_NLS_KOI8_U=m
CONFIG_NLS_UTF8=m
CONFIG_DLM=m
# CONFIG_DLM_DEBUG is not set
#
# Kernel hacking
#
CONFIG_TRACE_IRQFLAGS_SUPPORT=y
CONFIG_PRINTK_TIME=y
# CONFIG_ENABLE_WARN_DEPRECATED is not set
# CONFIG_ENABLE_MUST_CHECK is not set
CONFIG_FRAME_WARN=2048
CONFIG_MAGIC_SYSRQ=y
CONFIG_UNUSED_SYMBOLS=y
CONFIG_DEBUG_FS=y
# CONFIG_HEADERS_CHECK is not set
CONFIG_DEBUG_KERNEL=y
# CONFIG_DEBUG_SHIRQ is not set
CONFIG_DETECT_SOFTLOCKUP=y
# CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC is not set
CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC_VALUE=0
CONFIG_SCHED_DEBUG=y
CONFIG_SCHEDSTATS=y
# CONFIG_TIMER_STATS is not set
# CONFIG_DEBUG_OBJECTS is not set
# CONFIG_SLUB_DEBUG_ON is not set
# CONFIG_SLUB_STATS is not set
# CONFIG_DEBUG_RT_MUTEXES is not set
# CONFIG_RT_MUTEX_TESTER is not set
# CONFIG_DEBUG_SPINLOCK is not set
# CONFIG_DEBUG_MUTEXES is not set
# CONFIG_DEBUG_LOCK_ALLOC is not set
# CONFIG_PROVE_LOCKING is not set
# CONFIG_LOCK_STAT is not set
# CONFIG_DEBUG_SPINLOCK_SLEEP is not set
# CONFIG_DEBUG_LOCKING_API_SELFTESTS is not set
CONFIG_STACKTRACE=y
# CONFIG_DEBUG_KOBJECT is not set
CONFIG_DEBUG_BUGVERBOSE=y
# CONFIG_DEBUG_INFO is not set
# CONFIG_DEBUG_VM is not set
# CONFIG_DEBUG_VIRTUAL is not set
# CONFIG_DEBUG_WRITECOUNT is not set
# CONFIG_DEBUG_LIST is not set
# CONFIG_DEBUG_SG is not set
CONFIG_FRAME_POINTER=y
# CONFIG_BOOT_PRINTK_DELAY is not set
# CONFIG_RCU_TORTURE_TEST is not set
# CONFIG_KPROBES_SANITY_TEST is not set
# CONFIG_BACKTRACE_SELF_TEST is not set
# CONFIG_LKDTM is not set
# CONFIG_FAULT_INJECTION is not set
CONFIG_LATENCYTOP=y
CONFIG_HAVE_FTRACE=y
CONFIG_HAVE_DYNAMIC_FTRACE=y
CONFIG_TRACER_MAX_TRACE=y
CONFIG_TRACING=y
CONFIG_FTRACE=y
# CONFIG_IRQSOFF_TRACER is not set
# CONFIG_SYSPROF_TRACER is not set
CONFIG_SCHED_TRACER=y
CONFIG_CONTEXT_SWITCH_TRACER=y
CONFIG_DYNAMIC_FTRACE=y
# CONFIG_FTRACE_STARTUP_TEST is not set
# CONFIG_PROVIDE_OHCI1394_DMA_INIT is not set
# CONFIG_SAMPLES is not set
CONFIG_HAVE_ARCH_KGDB=y
# CONFIG_KGDB is not set
# CONFIG_NONPROMISC_DEVMEM is not set
CONFIG_X86_VERBOSE_BOOTUP=y
CONFIG_EARLY_PRINTK=y
# CONFIG_DEBUG_STACKOVERFLOW is not set
# CONFIG_DEBUG_STACK_USAGE is not set
# CONFIG_DEBUG_PAGEALLOC is not set
# CONFIG_X86_PTDUMP is not set
CONFIG_DEBUG_RODATA=y
# CONFIG_DIRECT_GBPAGES is not set
# CONFIG_DEBUG_RODATA_TEST is not set
# CONFIG_DEBUG_NX_TEST is not set
# CONFIG_IOMMU_DEBUG is not set
# CONFIG_MMIOTRACE is not set
CONFIG_IO_DELAY_TYPE_0X80=0
CONFIG_IO_DELAY_TYPE_0XED=1
CONFIG_IO_DELAY_TYPE_UDELAY=2
CONFIG_IO_DELAY_TYPE_NONE=3
CONFIG_IO_DELAY_0X80=y
# CONFIG_IO_DELAY_0XED is not set
# CONFIG_IO_DELAY_UDELAY is not set
# CONFIG_IO_DELAY_NONE is not set
CONFIG_DEFAULT_IO_DELAY_TYPE=0
# CONFIG_KMEMCHECK is not set
# CONFIG_DEBUG_BOOT_PARAMS is not set
# CONFIG_CPA_DEBUG is not set
#
# Security options
#
CONFIG_KEYS=y
# CONFIG_KEYS_DEBUG_PROC_KEYS is not set
CONFIG_SECURITY=y
CONFIG_SECURITY_NETWORK=y
CONFIG_SECURITY_NETWORK_XFRM=y
CONFIG_SECURITY_CAPABILITIES=y
# CONFIG_SECURITY_FILE_CAPABILITIES is not set
# CONFIG_SECURITY_ROOTPLUG is not set
CONFIG_SECURITY_DEFAULT_MMAP_MIN_ADDR=65536
CONFIG_SECURITY_SELINUX=y
CONFIG_SECURITY_SELINUX_BOOTPARAM=y
CONFIG_SECURITY_SELINUX_BOOTPARAM_VALUE=0
CONFIG_SECURITY_SELINUX_DISABLE=y
CONFIG_SECURITY_SELINUX_DEVELOP=y
CONFIG_SECURITY_SELINUX_AVC_STATS=y
CONFIG_SECURITY_SELINUX_CHECKREQPROT_VALUE=1
# CONFIG_SECURITY_SELINUX_ENABLE_SECMARK_DEFAULT is not set
# CONFIG_SECURITY_SELINUX_POLICYDB_VERSION_MAX is not set
CONFIG_XOR_BLOCKS=m
CONFIG_ASYNC_CORE=m
CONFIG_ASYNC_MEMCPY=m
CONFIG_ASYNC_XOR=m
CONFIG_CRYPTO=y
#
# Crypto core or helper
#
CONFIG_CRYPTO_ALGAPI=y
CONFIG_CRYPTO_AEAD=m
CONFIG_CRYPTO_BLKCIPHER=m
CONFIG_CRYPTO_HASH=y
CONFIG_CRYPTO_MANAGER=y
CONFIG_CRYPTO_GF128MUL=m
CONFIG_CRYPTO_NULL=m
# CONFIG_CRYPTO_CRYPTD is not set
CONFIG_CRYPTO_AUTHENC=m
CONFIG_CRYPTO_TEST=m
#
# Authenticated Encryption with Associated Data
#
CONFIG_CRYPTO_CCM=m
CONFIG_CRYPTO_GCM=m
CONFIG_CRYPTO_SEQIV=m
#
# Block modes
#
CONFIG_CRYPTO_CBC=m
CONFIG_CRYPTO_CTR=m
# CONFIG_CRYPTO_CTS is not set
CONFIG_CRYPTO_ECB=m
CONFIG_CRYPTO_LRW=m
CONFIG_CRYPTO_PCBC=m
# CONFIG_CRYPTO_XTS is not set
#
# Hash modes
#
CONFIG_CRYPTO_HMAC=y
CONFIG_CRYPTO_XCBC=m
#
# Digest
#
CONFIG_CRYPTO_CRC32C=m
CONFIG_CRYPTO_MD4=m
CONFIG_CRYPTO_MD5=y
CONFIG_CRYPTO_MICHAEL_MIC=m
CONFIG_CRYPTO_SHA1=m
CONFIG_CRYPTO_SHA256=m
CONFIG_CRYPTO_SHA512=m
CONFIG_CRYPTO_TGR192=m
CONFIG_CRYPTO_WP512=m
#
# Ciphers
#
CONFIG_CRYPTO_AES=m
CONFIG_CRYPTO_AES_X86_64=m
CONFIG_CRYPTO_ANUBIS=m
CONFIG_CRYPTO_ARC4=m
CONFIG_CRYPTO_BLOWFISH=m
# CONFIG_CRYPTO_CAMELLIA is not set
CONFIG_CRYPTO_CAST5=m
CONFIG_CRYPTO_CAST6=m
CONFIG_CRYPTO_DES=m
# CONFIG_CRYPTO_FCRYPT is not set
CONFIG_CRYPTO_KHAZAD=m
CONFIG_CRYPTO_SALSA20=m
CONFIG_CRYPTO_SALSA20_X86_64=m
# CONFIG_CRYPTO_SEED is not set
CONFIG_CRYPTO_SERPENT=m
CONFIG_CRYPTO_TEA=m
CONFIG_CRYPTO_TWOFISH=m
CONFIG_CRYPTO_TWOFISH_COMMON=m
CONFIG_CRYPTO_TWOFISH_X86_64=m
#
# Compression
#
CONFIG_CRYPTO_DEFLATE=m
CONFIG_CRYPTO_LZO=m
# CONFIG_CRYPTO_HW is not set
CONFIG_HAVE_KVM=y
# CONFIG_VIRTUALIZATION is not set
#
# Library routines
#
CONFIG_BITREVERSE=y
CONFIG_GENERIC_FIND_FIRST_BIT=y
CONFIG_GENERIC_FIND_NEXT_BIT=y
CONFIG_CRC_CCITT=m
CONFIG_CRC16=m
CONFIG_CRC_ITU_T=m
CONFIG_CRC32=y
# CONFIG_CRC7 is not set
CONFIG_LIBCRC32C=m
CONFIG_ZLIB_INFLATE=y
CONFIG_ZLIB_DEFLATE=m
CONFIG_LZO_COMPRESS=m
CONFIG_LZO_DECOMPRESS=m
CONFIG_TEXTSEARCH=y
CONFIG_TEXTSEARCH_KMP=m
CONFIG_TEXTSEARCH_BM=m
CONFIG_TEXTSEARCH_FSM=m
CONFIG_PLIST=y
CONFIG_HAS_IOMEM=y
CONFIG_HAS_IOPORT=y
CONFIG_HAS_DMA=y
Best regards,
--Edwin
On Sat, 12 Jul 2008 16:42:13 +0300
Török Edwin <[email protected]> wrote:
> On 2008-07-12 13:32, Török Edwin wrote:
> > I haven't yet tried ftrace on this box, and neither did I try
> > Roland's patch yet. I will try that now, and hopefuly come back
> > with some numbers shortly.
> >
>
> tip/master of today already includes Roland's patch, so I have tested
> it without even knowing on the 64-bit box.
> [After gathering all the traces, I wanted to repeat everything with
> the patch applied, but there was no need since its already there].
> I can say that I didn't notice any difference in the delays (I still
> get 24 - 30 second delays).
> I also tried Linus's readdir patch, the latency is around 27 seconds
> (was 30 before). I am not sure if it is an improvement, or measurement
> noise.
>
> Also I wasn't able to reproduce the 'Ctrl+Z works faster than Ctrl+C'
> symptom, perhaps it is just a coincidence: by the time I press Ctrl+Z
> to interrupt, enough time has passed (30+ secs), that the same effect
> would have been obtained by another Ctrl+C, or by just waiting.
>
> On 32-bit, I think the 2-3 seconds latency can be attributed to I/O
> delays, and I think its acceptable given the slow disk (5k4 rpm sata).
I see really bad delays on 32 bit as well, but they go away for me if I
do
echo 4096 > /sys/block/sda/queue/nr_requests
--
If you want to reach me at my work email, use [email protected]
For development, discussion and tips for power savings,
visit http://www.lesswatts.org
On Sat, 12 Jul 2008, T?r?k Edwin wrote:
>
> On my 32-bit box (slow disks, SMP, XFS filesystem) 2.6.26-rc9 behaves
> the same as 2.6.26-rc8, I can reliably reproduce a 2-3 second latency
> [1] between pressing ^C the first time, and the shell returning (on the
> text console too).
> Using ftrace available from tip/master, I see up to 3 seconds of delay
> between kill_pgrp and detach_pid (and during that time I can press ^C
> again, leading to 2-3 kill_pgrp calls)
The thing is, it's important to see what happens in between.
In particular, 2-3 second latencies can be entirely _normal_ (although
obviously very annoying) with most log-based filesystems when they decide
they have to flush the log. A lot of filesystems are not designed for
latency - every single filesystem test I have ever seen has always been
either a throughput test, or a "average random-seek latency" kind of test.
The exact behavior will depend on the filesystem, for example. It will
also easily depend on things like whether you update 'atime' or not. Many
ostensibly read-only loads end up writing some data, especially inode
atimes, and that's when they can get caught up in having to wait for a log
to flush (to make up for that atime thing).
You can try to limit the amount of dirty data in flight by tweaking
/proc/sys/vm/dirty*ratio, but from a latency standpoint the thing that
actually matters more is often not the amount of dirty data, but the size
of the requests queues - because you often care about read latency, but if
you have big requests and especially if you have a load that does lots of
big _contiguous_ writes (like your 'dd's would do), then what can easily
happen is that the read ends up being behind a really big write in the
request queues.
And 2-3 second latencies by no means means that each individual IO is 2-3
seconds long. No - it just means that you ended up having to do multiple
reads synchronously, and since the reads depended on each other (think a
pathname lookup - reading each directory entry -> inode -> data), you can
easily have a single system call causing 5-10 reads (bad cases are _much_
more, but 5-10 are perfectly normal for even well-behaved things), and now
if each of those reads end up being behind a fairly big write...
> On my 64-bit box (2 disks in raid-0, UP, reiserfs filesystem) 2.6.25 and
> 2.6.26-rc9 behave the same, and most of the time (10-20 times in a row)
> find responds to ^C instantly.
>
> However in _some_ cases find doesn't respond to ^C for a very long time
> (~30 seconds), and when this happens I can't do anything else but switch
> consoles, starting another process (latencytop -d) hangs, and so does
> any other external command.
Ok, that is definitel not related to signals at all. You're simply stuck
waiting for IO - or perhaps some fundamental filesystem semaphore which is
held while some IO needs to be flushed. That's why _unrelated_ processes
hang: they're all waiting for a global resource.
And it may be worse on your other box for any number of reasons: raid
means, for example, that you have two different levels of queueing, and
thus effectively your queues are longer. And while raid-0 is better for
throughput, it's not necessarily at all better for latency. The filesystem
also makes a difference, as does the amount of dirty data under write-back
(do you also have more memory in your x86-64 box, for example? That makes
the kernel do bigger writeback buffers by default)
> I haven't yet tried ftrace on this box, and neither did I try Roland's
> patch yet. I will try that now, and hopefuly come back with some numbers
> shortly.
Trust me, roland's patch will make no difference what-so-ever. It's purely
a per-thread thing, and your behaviour is clearly not per-thread.
Signals are _always_ delayed until non-blocking system calls are done, and
that means until the end of IO.
This is also why your trace on just 'kill_pgrp' and 'detach_pid' is not
interesting. It's _normal_ to have a delay between them. It can happen
because the process blocks (or catches) signals, but it will also happen
if some system call waits for disk.
(The waiting for disk may be indirect - it might be due to needing more
memory and needing to write out dirty studd. So it's not necessarily doing
IO per se, although it's quite likely that that is part of it).
You could try 'noatime' and see if that helps behaviour a bit.
Linus
On Sat, 12 Jul 2008, Andi Kleen wrote:
>
> Linus, the fact that is not explained by your theory is why
> Ctrl-Z+kill works but Ctrl-C doesn't.
Umm. Read the reports more carefully. Many of the complaints are about ^Z
too.
More-over, ^Z and ^C are very different from an app standpoint. A _lot_ of
applications block or catch ^C (SIGINT), while doing the same for ^Z
(SIGTSTP) and SIGTERM is much less common (but possible: a tty sends
SIGTSTP, which is blockable, not SIGSTOP, which is not).
The reason? It's quite common to catch ^C in order to do cleanup. The
common handler case is that you catch SIGINT, do cleanup, and then kill
yourself with SIGINT in order to make it _look_ like you reacted directly
to the ^C.
So there are a _lot_ of loads where ^C will not kill things immediately,
but ^Z + SIGKILL (or SIGTERM, the default for "sig") will.
Try doing a "yum update", for example.
Or use 'git', for that matter. git does exactly that kind of thing:
catches SIGINT and removes temp-files etc. But if you kill it manually
using some "stronger" signal (and using ^Z to get the shell back before
you do that is obviously one way), that won't happen.
And trust me, git is not at all unusual in this respect.
Linus
On Sat, 12 Jul 2008, Arjan van de Ven wrote:
>
> I see really bad delays on 32 bit as well, but they go away for me if I
> do
> echo 4096 > /sys/block/sda/queue/nr_requests
Hmm. I think the default is 128, and in many cases latencies should
actually go up with bigger requests queues - especially if it means that
you can have a lot more writes in front of the read. You see the opposite
behaviour.
That could easily happen if the scheduler is crazy and lets writes use up
all of the request queue, or if the limited queue means that it cannot
effectively merge requests. But request merging should happen trivially
for the contiguous 'dd' case almost regardless of queue size, so I wonder
if something else is going on.
Ahh.. I see something _very_ suspicious.
Look at block/blk-core.c: get_request(). It starts throttling and batching
requests when it gets
if (rl->count[rw]+1 >= queue_congestion_on_threshold(q)) {
and notice how this is independent of whether it's a read or a write (but
it does count them separately). But on the wakeup path, it uses different
limits for reads than for writes.
That batching looks pretty bogus for reads to begin with, and then
behaving similarly on throttling but differently on wakup sounds bogus.
The blk_alloc_request() also ends up allocating all requests from one
mempool, so if that mempool runs out (due to writes having used them all
up), then those writes will block reads too, even though reads should have
much higher priority.
I dunno. But there _has_ been a lot of churn in the different block queues
over the last few months. I wouldn't be surprised at all if something got
broken in the process. And as with filesystems, almost all performance
tests are for throughput, not "bad latency" in the presense of other
heavy IO.
Linus
On Sat, 12 Jul 2008 11:00:06 -0700 (PDT)
Linus Torvalds <[email protected]> wrote:
>
>
> On Sat, 12 Jul 2008, Arjan van de Ven wrote:
> >
> > I see really bad delays on 32 bit as well, but they go away for me
> > if I do
> > echo 4096 > /sys/block/sda/queue/nr_requests
>
> Hmm. I think the default is 128, and in many cases latencies should
> actually go up with bigger requests queues - especially if it means
> that you can have a lot more writes in front of the read. You see the
> opposite behaviour.
well... so far my assumption on this has been this:
CFQ is good at fairness, and manages to control latency between
processes in some fair way (eg if one guy is slamming the disk and
someone else needs to page in a 4Kb page but is otherwise not doing
IO... CFQ will let the pagefault skip ahead to a large degree) .
However... the 128 limit happens BEFORE CFQ gets involved and suddenly
the 4kb pagefault has to wait for everything that's pending
> Look at block/blk-core.c: get_request(). It starts throttling and
> batching requests when it gets
>
> if (rl->count[rw]+1 >= queue_congestion_on_threshold(q)) {
>
> and notice how this is independent of whether it's a read or a write
> (but it does count them separately). But on the wakeup path, it uses
> different limits for reads than for writes.
>
> That batching looks pretty bogus for reads to begin with, and then
> behaving similarly on throttling but differently on wakup sounds
> bogus.
>
> The blk_alloc_request() also ends up allocating all requests from one
> mempool, so if that mempool runs out (due to writes having used them
> all up), then those writes will block reads too, even though reads
> should have much higher priority.
>
> I dunno. But there _has_ been a lot of churn in the different block
> queues over the last few months. I wouldn't be surprised at all if
> something got broken in the process. And as with filesystems, almost
> all performance tests are for throughput, not "bad latency" in the
> presense of other heavy IO.
what I'm seeing is not super new; even 2.6.25 has the behavior already
(and with latencytop it's very visible behavior; before changing the
tunable I see 2+ second delays in user apps, after the tunable change
they're down significantly)
--
If you want to reach me at my work email, use [email protected]
For development, discussion and tips for power savings,
visit http://www.lesswatts.org
On Sat, 12 Jul 2008, Arjan van de Ven wrote:
>
> what I'm seeing is not super new; even 2.6.25 has the behavior already
> (and with latencytop it's very visible behavior; before changing the
> tunable I see 2+ second delays in user apps, after the tunable change
> they're down significantly)
Yeah, I think people said it's more visible, and pointed back to 2.6.23 or
something being better. So no, I don't think anybody really claimed this
as a regression from 25.
Linus
On 2008-07-12 20:29, Linus Torvalds wrote:
> On Sat, 12 Jul 2008, T?r?k Edwin wrote:
>
>> On my 32-bit box (slow disks, SMP, XFS filesystem) 2.6.26-rc9 behaves
>> the same as 2.6.26-rc8, I can reliably reproduce a 2-3 second latency
>> [1] between pressing ^C the first time, and the shell returning (on the
>> text console too).
>> Using ftrace available from tip/master, I see up to 3 seconds of delay
>> between kill_pgrp and detach_pid (and during that time I can press ^C
>> again, leading to 2-3 kill_pgrp calls)
>>
>
> The thing is, it's important to see what happens in between.
>
> In particular, 2-3 second latencies can be entirely _normal_ (although
> obviously very annoying) with most log-based filesystems when they decide
> they have to flush the log.
A bit off-topic, but something I noticed during the tests:
In my original test I have rm-ed the files right after launching dd in
the background, yet it still continued to write to the disk.
I can understand that if the file is opened O_RDWR, you might seek back
and read what you wrote, so Linux needs to actually do the write,
but why does it insist on writing to the disk, on a file opened with
O_WRONLY, after the file itself got unlinked?
> A lot of filesystems are not designed for
> latency - every single filesystem test I have ever seen has always been
> either a throughput test, or a "average random-seek latency" kind of test.
>
> The exact behavior will depend on the filesystem, for example. It will
> also easily depend on things like whether you update 'atime' or not. Many
> ostensibly read-only loads end up writing some data, especially inode
> atimes, and that's when they can get caught up in having to wait for a log
> to flush (to make up for that atime thing).
>
I have my filesystems mounted as noatime already.
But yes, I am using different filesystems, the x86-64 box has reiserfs,
and the x86-32 box has xfs.
> You can try to limit the amount of dirty data in flight by tweaking
> /proc/sys/vm/dirty*ratio
I have these in my /etc/rc.local:
echo 5 > /proc/sys/vm/dirty_background_ratio
echo 10 >/proc/sys/vm/dirty_ratio
> , but from a latency standpoint the thing that
> actually matters more is often not the amount of dirty data, but the size
> of the requests queues - because you often care about read latency, but if
> you have big requests and especially if you have a load that does lots of
> big _contiguous_ writes (like your 'dd's would do), then what can easily
> happen is that the read ends up being behind a really big write in the
> request queues.
>
I will try tweaking these (and /sys/block/sda/queue/nr_requests that
Arjan suggested).
> And 2-3 second latencies by no means means that each individual IO is 2-3
> seconds long. No - it just means that you ended up having to do multiple
> reads synchronously, and since the reads depended on each other (think a
> pathname lookup - reading each directory entry -> inode -> data), you can
> easily have a single system call causing 5-10 reads (bad cases are _much_
> more, but 5-10 are perfectly normal for even well-behaved things), and now
> if each of those reads end up being behind a fairly big write...
>
I'll try blktrace tomorrow, it should tell me when I/Os are queued /
completed.
>
>> On my 64-bit box (2 disks in raid-0, UP, reiserfs filesystem) 2.6.25 and
>> 2.6.26-rc9 behave the same, and most of the time (10-20 times in a row)
>> find responds to ^C instantly.
>>
>> However in _some_ cases find doesn't respond to ^C for a very long time
>> (~30 seconds), and when this happens I can't do anything else but switch
>> consoles, starting another process (latencytop -d) hangs, and so does
>> any other external command.
>>
>
> Ok, that is definitel not related to signals at all. You're simply stuck
> waiting for IO - or perhaps some fundamental filesystem semaphore which is
> held while some IO needs to be flushed.
AFAICT reiserfs still uses the BKL, could that explain why one I/O
delays another?
> That's why _unrelated_ processes
> hang: they're all waiting for a global resource.
>
> And it may be worse on your other box for any number of reasons: raid
> means, for example, that you have two different levels of queueing, and
> thus effectively your queues are longer. And while raid-0 is better for
> throughput, it's not necessarily at all better for latency. The filesystem
> also makes a difference, as does the amount of dirty data under write-back
> (do you also have more memory in your x86-64 box, for example? That makes
> the kernel do bigger writeback buffers by default)
>
>
Yes, I have more memory on x86-64 (2G), and x86-32 has 1G.
>> I haven't yet tried ftrace on this box, and neither did I try Roland's
>> patch yet. I will try that now, and hopefuly come back with some numbers
>> shortly.
>>
>
> Trust me, roland's patch will make no difference what-so-ever. It's purely
> a per-thread thing, and your behaviour is clearly not per-thread.
>
Indeed.
[Roland's patch is included in tip/master so I actually tried it even if
I didn't know I did]
> Signals are _always_ delayed until non-blocking system calls are done, and
> that means until the end of IO.
>
> This is also why your trace on just 'kill_pgrp' and 'detach_pid' is not
> interesting. It's _normal_ to have a delay between them. It can happen
> because the process blocks (or catches) signals, but it will also happen
> if some system call waits for disk.
>
Is there a way to trace what happens between those 2 functions?
Maybe if I don't use the trace filter (and thus trace all functions),
and modify the kernel sources to start tracing on kill_pgrp, and stop
tracing on detach_pid.
Would that provide useful info?
Best regards,
--Edwin
On Sat, 12 Jul 2008, T?r?k Edwin wrote:
>
> A bit off-topic, but something I noticed during the tests:
> In my original test I have rm-ed the files right after launching dd in
> the background, yet it still continued to write to the disk.
> I can understand that if the file is opened O_RDWR, you might seek back
> and read what you wrote, so Linux needs to actually do the write,
> but why does it insist on writing to the disk, on a file opened with
> O_WRONLY, after the file itself got unlinked?
Linux itself doesn't insist on writing to disk. In fact, at least with
traditional UNIX filesystems (eg minix, ext2) the deleted writes would be
undone.
But some filesystems can't just invalidate dirty buffers (some won't do it
for meta-data, others won't do it for _any_ data). So again, this
behaviour depends on the filesystem. And sadly, the more "advanced"
filesystem, the worse it usually behaves here.
> I have my filesystems mounted as noatime already.
> But yes, I am using different filesystems, the x86-64 box has reiserfs,
> and the x86-32 box has xfs.
>
> > You can try to limit the amount of dirty data in flight by tweaking
> > /proc/sys/vm/dirty*ratio
>
> I have these in my /etc/rc.local:
> echo 5 > /proc/sys/vm/dirty_background_ratio
> echo 10 >/proc/sys/vm/dirty_ratio
That matches the modern defaults. You can try playing with them if you
want to. And yes, it's worth testing nr_requests too.
> > Ok, that is definitel not related to signals at all. You're simply stuck
> > waiting for IO - or perhaps some fundamental filesystem semaphore which is
> > held while some IO needs to be flushed.
>
> AFAICT reiserfs still uses the BKL, could that explain why one I/O
> delays another?
The BKL should be ok in this respect - it gets automatically dropped when
doing synchronous waiting (this is somethign that will possibly go away as
we try to convince people to get rid of the BKL, but it certainly hasn't
happened yet).
So it actually gets worse with other locks - semaphores or mutexes - that
stay held over IO. And reiserfs has a journal lock (and a "commit" lock),
but I don't know how they are held and whether this could be part of the
issue.
> > This is also why your trace on just 'kill_pgrp' and 'detach_pid' is not
> > interesting. It's _normal_ to have a delay between them. It can happen
> > because the process blocks (or catches) signals, but it will also happen
> > if some system call waits for disk.
>
> Is there a way to trace what happens between those 2 functions?
You could try to trace not just those functions, but scheduling events
too. Or yes, do something special-caed.
Trying to figure out latencies in the block trace is likely also going to
be interesting (although you won't see any signal issues there - but any
long read latencies will automatically tend to imply latency issues not
just for signals, but for pretty much any operations).
Linus
On Saturday 12 July 2008 22:26, T?r?k Edwin wrote:
> A bit off-topic, but something I noticed during the tests:
> In my original test I have rm-ed the files right after launching dd in
> the background, yet it still continued to write to the disk.
> I can understand that if the file is opened O_RDWR, you might seek back
> and read what you wrote, so Linux needs to actually do the write,
> but why does it insist on writing to the disk, on a file opened with
> O_WRONLY, after the file itself got unlinked?
Because process can do
fcntl(fd, F_SETFL, fcntl(fd, F_GETFL) | O_RDWR)
at any time.
--
vda
Linus Torvalds wrote:
>
> On Sat, 12 Jul 2008, Andi Kleen wrote:
>> Linus, the fact that is not explained by your theory is why
>> Ctrl-Z+kill works but Ctrl-C doesn't.
>
> Umm. Read the reports more carefully. Many of the complaints are about ^Z
> too.
At least the original report was about Ctrl-C only versus Ctrl-Z.
I see the problem regularly myself that Ctrl-C doesn't work,
but Ctrl-Z+kill does (although I unfortunately cannot
reproduce it on demand). But it was with programs who shouldn't catch Ctrl-C,
like find. I confirmed with ltrace that at least my version of find doesn't
catch any signals.
Take the original report for example:
http://marc.info/?l=linux-kernel&m=121464952615807&w=2
It also had trouble interrupting find:
$ find / >/dev/null
find: `/boot/lost+found': Permission denied
^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C
but Ctrl-Z+kill worked immediately.
> The reason? It's quite common to catch ^C in order to do cleanup.
Yes I understand that, but I don't think it's the case here.
I'm afraid it's more a "has a simple explanation that is wrong"
case here.
Or we might be talking about multiple different bugs.
-Andi
On 07/12, Denys Vlasenko wrote:
>
> On Saturday 12 July 2008 22:26, T?r?k Edwin wrote:
> > A bit off-topic, but something I noticed during the tests:
> > In my original test I have rm-ed the files right after launching dd in
> > the background, yet it still continued to write to the disk.
> > I can understand that if the file is opened O_RDWR, you might seek back
> > and read what you wrote, so Linux needs to actually do the write,
> > but why does it insist on writing to the disk, on a file opened with
> > O_WRONLY, after the file itself got unlinked?
>
> Because process can do
>
> fcntl(fd, F_SETFL, fcntl(fd, F_GETFL) | O_RDWR)
Is it?
SETFL_MASK doesn't have O_RDWR, and in any case setfl() changes ->f_flags,
not ->f_mode.
Oleg.
On Sunday 13 July 2008 12:46, Oleg Nesterov wrote:
> On 07/12, Denys Vlasenko wrote:
> >
> > On Saturday 12 July 2008 22:26, T?r?k Edwin wrote:
> > > A bit off-topic, but something I noticed during the tests:
> > > In my original test I have rm-ed the files right after launching dd in
> > > the background, yet it still continued to write to the disk.
> > > I can understand that if the file is opened O_RDWR, you might seek back
> > > and read what you wrote, so Linux needs to actually do the write,
> > > but why does it insist on writing to the disk, on a file opened with
> > > O_WRONLY, after the file itself got unlinked?
> >
> > Because process can do
> >
> > fcntl(fd, F_SETFL, fcntl(fd, F_GETFL) | O_RDWR)
>
> Is it?
>
> SETFL_MASK doesn't have O_RDWR, and in any case setfl() changes ->f_flags,
> not ->f_mode.
Just tested it and you are right.
I distinctly remember seeing such code somewhere. Interesting.
Now I wonder whether it was a bug, or those were not file descriptors,
but sockets?...
--
vda
On Sun, 13 Jul 2008, Andi Kleen wrote:
>
> At least the original report was about Ctrl-C only versus Ctrl-Z.
Yes, and I explained why.
> I see the problem regularly myself that Ctrl-C doesn't work,
> but Ctrl-Z+kill does (although I unfortunately cannot
> reproduce it on demand).
There's no way you _can_ reproduce it.
> But it was with programs who shouldn't catch Ctrl-C, like find.
No, it was not. I think you misread it.
Two facts:
- ^Z and ^C are both going to be equally fast if they aren't blocked.
This is just how things are. They are handled by the same codepaths.
- the report you point to does mention 'find', but *not* in the place
where it talks about "^Z+kill". There it's much less specific, and I
bet that Edwin _had_ seen the "^Z works" behavior, and knew it to be
true, but hadn't realized that it depended on the command, so when he
gave an example of something where disk IO makes signals slow, he
happened to pick one where ^Z and ^C were equally slow.
In other words, I can pretty much _guarantee_ that if ^Z+kill is faster
than ^C, then it's because the process blocked or caught ^C.
And I can pretty much also guarantee that when Edwin talked about 'find'
and did that ^C cut-and-paste, he didn't try the ^Z thing (that had
happened with _other_ programs), or if he did, it was just happenstance
(because the latency isn't _always_ 10-30 seconds - that's just the worst
case).
So yes, ^C often works slower. If the process catches it and removes
temporary files as a result (for example - it's what things like gcc do),
then of _course_ it's going to be slower. It's going to be very noticeably
slower, especially since removing a temp-file is going to be even more
likely to hit a filesystem lock if the write-queues are full than just a
normal read is.
But ^C isn't always slower, and if 'find' doesn't block/ignore it (and I
certainly believe you when you say that it doesn't), then for the case of
'find', it won't be true that ^Z+kill is any faster.
In fact, Edwin later in this thread reported:
"Also I wasn't able to reproduce the 'Ctrl+Z works faster than Ctrl+C'
symptom, perhaps it is just a coincidence: by the time I press Ctrl+Z to
interrupt, enough time has passed (30+ secs), that the same effect would
have been obtained by another Ctrl+C, or by just waiting."
and I bet that is exactly because now he's been testing programs that
don't block ^C (ie maybe he's now testing _just_ that "find /" thing).
So before, Edwin was probably doing random things when doing "normal
work", and noticed that ^Z was sometimes faster. Now, he's trying to chase
things down, and he creates this specific test-case, and for that
particular test-case, ^Z _isn't_ faster. He just didn't pick up on the
pattern that it depends on which particular command he works on.
Try "yum list", and you'll certainly see that ^Z works a *lot* faster than
^C (and the latter often needs multiple ^C's even when there is no load
at all)
Linus
On Sun, 13 Jul 2008, Oleg Nesterov wrote:
>
> SETFL_MASK doesn't have O_RDWR, and in any case setfl() changes ->f_flags,
> not ->f_mode.
I think you can still read a write-only file descriptor on some systems:
doing a mmap( PROT_WRITE ) on it can sometimes do it.
The reason - PROT_WRITE often implies PROT_READ (because on many CPU's you
have to do unaligned writes with a read-modify-write cycle - on some like
old alphas you have to do even normal byte and word writes that way).
So if the OS checks just the asked-for protections against the file
descriptor protections, you'll essentially get read access for free when
you do the mmap.
At least current versions of Linux won't ever allow a file mapping of a
non-readable fd, but I won't guarantee we always did that, nor that other
OS's always do it.
Anyway, I misunderstood Edwins idea to actually throw away the writes if
the file isn't linked anywhere and is only open for writing. I guess we
could optimize that, but it's such an unrealistic specialc case that it
really isn't worth it except for some benchmark-gaming kind of thing
(which I don't like doing anyway).
Linus
In article <[email protected]> you wrote:
> Anyway, I misunderstood Edwins idea to actually throw away the writes if
> the file isn't linked anywhere and is only open for writing. I guess we
I don't understand it. The process can still clone itself and its
children may want to look at what it has written (maybe there's too much
file to fit in an mmap all at once) via the same file descriptor. Come
to that, the process itself may want to look at what it's written!
Saving memory is the idea behind writing to disk sometimes. Unlinking
your target is sensible if you want to keep your thoughts private ..
Peter
Linus Torvalds wrote:
>
> On Sun, 13 Jul 2008, Andi Kleen wrote:
>> At least the original report was about Ctrl-C only versus Ctrl-Z.
>
> Yes, and I explained why.
>
>> I see the problem regularly myself that Ctrl-C doesn't work,
>> but Ctrl-Z+kill does (although I unfortunately cannot
>> reproduce it on demand).
>
> There's no way you _can_ reproduce it.
That's not how I remember it from seeing it here, but ok it's possible my
memory is fuzzy and I'm misremembering. I'll continue to watch it.
While the bit about color ls (which I use here) catching signals was
also interesting I wouldn't expect the color ls to take longer to
process Ctrl-C even if it hits user space because it shouldn't
do anything block here (unless the terminal is in flow control,
but is unlikely)
> Two facts:
>
> - ^Z and ^C are both going to be equally fast if they aren't blocked.
>
> This is just how things are. They are handled by the same codepaths.
Yes I know that, that is why the behavior always puzzled me, because
it apparently contradicts that. I think I know reasonably well how
signals work, but cannot say the same about tty, so my guess was
in that area. But we'll see.
-Andi
On Sun, 13 Jul 2008, Andi Kleen wrote:
>
> While the bit about color ls (which I use here) catching signals was
> also interesting I wouldn't expect the color ls to take longer to
> process Ctrl-C even if it hits user space because it shouldn't
> do anything block here (unless the terminal is in flow control,
> but is unlikely)
I didn't take a look at the source, but I literally think that the 'ls'
SIGINT handler is something like
static void sigint(int signr)
{
exit_with_sigint = 1;
}
and then in the output routine it does a
if (exit_with_sigint) {
signal(SIGINT, SIG_DFL);
kill(-1, SIGINT);
}
at the end because that's the only thing that explains that it always
exits after printing the _first_ line of output (ignoring the header - it
obviously doesn't have that "if (exit_with_sigint)" test in that
code-path).
Sad. Horrible crap. It means that it totally disables the kernels ability
to make fatal signals break out of disk wait etc. It's also totally
_unnecessary_, because 'ls' shouldn't even bother to block signals until
just before it starts doing the printout.
Oh well. You can't expect too much of user level programmers.
Linus