There appears to be a bug in 2.6.27 and inotify where it causes
processes in running state to eat 100% CPU and they can't be killed
with kill -9, requiring a reboot to get rid of them.
3 people (Cc'd) so far have reported this to me, all of them last
week. The problem shows up when running my Dovecot IMAP server. Its
imap processes start hanging, apparently it can happen after running
for only a couple of minutes:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6717 arno 20 0 2964 1608 1192 R 100 0.2 1158:05 imap
Only the imap processes hang (Dovecot has a few others). The only
thing special about it is that it uses inotify. After disabling
Dovecot's inotify support at least one of the people (Arno) reported
that he can't reprocuce the bug anymore. He also couldn't reproduce
the bug with 2.6.26 kernel.
I don't think Dovecot does anything special with its inotify code. It
just listens on two directories. You can find the code here: http://hg.dovecot.org/dovecot-1.1/file/f9a000ee400d/src/lib/ioloop-notify-inotify.c
Kernels causing the hang:
2.6.27-9.slh.1-sidux-686
Arch Linux, kernel 2.6.27.8
Gentoo with 2.6.27-gentoo-r5, amd64, 2 dual core opterons
Not causing the hang:
2.6.26-6.slh.1-sidux-686
On Mon, Dec 15, 2008 at 01:31:29PM +0200, Timo Sirainen wrote:
> There appears to be a bug in 2.6.27 and inotify where it causes processes
> in running state to eat 100% CPU and they can't be killed with kill -9,
> requiring a reboot to get rid of them.
> Kernels causing the hang:
>
> 2.6.27-9.slh.1-sidux-686
> Arch Linux, kernel 2.6.27.8
> Gentoo with 2.6.27-gentoo-r5, amd64, 2 dual core opterons
>
> Not causing the hang:
>
> 2.6.26-6.slh.1-sidux-686
Could you try to bisect that? Particulary interesting commits:
8f7b0ba1c853919b85b54774775f567f30006107
6ff2d39b91aec3dcae951afa982059e3dd9b49dc
711a49a07f84f914aac26a52143f6e7526571143
On Mon, 15 Dec 2008 12:05:12 +0000, Al Viro wrote:
> On Mon, Dec 15, 2008 at 01:31:29PM +0200, Timo Sirainen wrote:
>> There appears to be a bug in 2.6.27 and inotify where it causes
>> processes in running state to eat 100% CPU and they can't be killed with
>> kill -9, requiring a reboot to get rid of them.
>
>> Kernels causing the hang:
>>
>> 2.6.27-9.slh.1-sidux-686
>> Arch Linux, kernel 2.6.27.8
>> Gentoo with 2.6.27-gentoo-r5, amd64, 2 dual core opterons
>>
>> Not causing the hang:
>>
>> 2.6.26-6.slh.1-sidux-686
>
> Could you try to bisect that? Particulary interesting commits:
Please see this posting, which also contains a pointer to the
necessary patch:
http://lkml.indiana.edu/hypermail/linux/kernel/0812.1/02305.html
According to GregKH it will be in -stable.10, but in the meantime you can
also apply the patch from http://tinyurl.com/5gpqkj on top stable .8
(which introduced the regression) or .9.
At this point it is not clear whether the changes to inotify caused this
or just fell victim to the parallel regression in idr.
-h