Hi Trond.
I observed a problem with NFS root in 2.6 kernel
(actually it was a 2.5 back then).
It is still present in 2.6.9, thus I am resending
my previous mail:
- - - - - - - - - - - - - - - - - - - - - - - -
I am using NFS root. At shutdown, when I kill
all processes with killall5 -9, NFS temporarily
misbehaves. I narrowed it down to rpciod feeling
bad when signalled with SIGKILL:
# killall5 -9;ps -AH e;read junk;ps -AH e
bash: /bin/ps: Input/output error
<--- I press [Enter] here
PID TTY STAT TIME COMMAND
1 ? S 0:05 /bin/sh /init.vda
2 ? SWN 0:00 [ksoftirqd/0]
3 ? SW< 0:00 [events/0]
4 ? SW< 0:00 [kblockd/0]
5 ? SW 0:00 [pdflush]
6 ? SW 0:00 [pdflush]
8 ? SW< 0:00 [aio/0]
7 ? SW 0:00 [kswapd0]
49 ? SW 0:00 [rpciod]
50 ? SW 0:00 [lockd]
812 vc/2 S 0:00 -bash HOME=/home/vda PATH=/sbin:/bin:/usr/sbin:/usr
1369 tty2 R 0:00 ps -AH e PWD=/app/shutdown-0.0.5/script GROFF_NO_
1368 ? S 0:00 sleep 32000
# kill -9 49;ps -AH e;read junk;ps -AH e
bash: /bin/ps: Input/output error
<--- I press [Enter] here
PID TTY STAT TIME COMMAND
1 ? S 0:05 /bin/sh /init.vda
2 ? SWN 0:00 [ksoftirqd/0]
3 ? SW< 0:00 [events/0]
4 ? SW< 0:00 [kblockd/0]
5 ? SW 0:00 [pdflush]
6 ? SW 0:00 [pdflush]
8 ? SW< 0:00 [aio/0]
7 ? SW 0:00 [kswapd0]
49 ? SW 0:00 [rpciod]
50 ? SW 0:00 [lockd]
812 vc/2 S 0:00 -bash HOME=/home/vda PATH=/sbin:/bin:/usr/sbin:/usr
1369 tty2 R 0:00 ps -AH e PWD=/app/shutdown-0.0.5/script GROFF_NO_
1368 ? S 0:00 sleep 32000
# cat /proc/mounts
rootfs / rootfs rw 0 0
172.16.42.75:/.rootfs/.std / nfs ro,v3,rsize=8192,wsize=8192,hard,udp,lock,addr=172.16.42.75 0 0
none /dev devfs rw 0 0
none /proc proc rw 0 0
none /sys sysfs rw 0 0
172.16.42.75:/.share /.share nfs rw,v3,rsize=8192,wsize=8192,hard,intr,udp,nolock,addr=172.16.42.75 0 0
172.16.42.75:/.1 /.1 nfs rw,v3,rsize=8192,wsize=8192,hard,intr,udp,nolock,addr=172.16.42.75 0 0
172.16.42.75:/.share /.local nfs rw,v3,rsize=8192,wsize=8192,hard,intr,udp,nolock,addr=172.16.42.75 0 0
automount(pid684) /.local/mnt/auto autofs rw 0 0
/dev/hda1 /.local/mnt/auto/vfat.hda1 vfat rw,noatime,nodiratime,nosuid,noexec,fmask=0022,dmask=0022,codepage=cp866,iocharset=koi8-r,shortname=win95,quiet,uni_xlate 0 0
none /dev/pts devpts rw 0 0
# lsof -nP
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
init.vda 1 root cwd DIR 0,12 1024 65282 / (172.16.42.75:/.rootfs/.std)
init.vda 1 root rtd DIR 0,12 1024 65282 / (172.16.42.75:/.rootfs/.std)
init.vda 1 root txt REG 0,12 437844 71410 /bin/bash (172.16.42.75:/.rootfs/.std)
init.vda 1 root mem REG 0,12 832636 8248 /app/glibc-2.3/lib/ld-2.3.so (172.16.42.75:/.rootfs/.std)
init.vda 1 root mem REG 0,12 14831 47164 /lib/libtermcap.so.2.0.8 (172.16.42.75:/.rootfs/.std)
init.vda 1 root mem REG 0,12 50088 8252 /app/glibc-2.3/lib/libdl-2.3.so (172.16.42.75:/.rootfs/.std)
init.vda 1 root mem REG 0,12 16153080 8249 /app/glibc-2.3/lib/libc-2.3.so (172.16.42.75:/.rootfs/.std)
init.vda 1 root mem REG 0,12 312114 8268 /app/glibc-2.3/lib/libnss_compat-2.3.so (172.16.42.75:/.rootfs/.std)
init.vda 1 root mem REG 0,12 1164667 8265 /app/glibc-2.3/lib/libnsl-2.3.so (172.16.42.75:/.rootfs/.std)
init.vda 1 root mem REG 0,12 360212 8257 /app/glibc-2.3/lib/libnss_files-2.3.so (172.16.42.75:/.rootfs/.std)
init.vda 1 root 3r REG 0,12 678 65407 /init.vda (172.16.42.75:/.rootfs/.std)
ksoftirqd 2 root cwd DIR 0,12 1024 65282 / (172.16.42.75:/.rootfs/.std)
ksoftirqd 2 root rtd DIR 0,12 1024 65282 / (172.16.42.75:/.rootfs/.std)
events/0 3 root cwd DIR 0,12 1024 65282 / (172.16.42.75:/.rootfs/.std)
events/0 3 root rtd DIR 0,12 1024 65282 / (172.16.42.75:/.rootfs/.std)
kblockd/0 4 root cwd DIR 0,12 1024 65282 / (172.16.42.75:/.rootfs/.std)
kblockd/0 4 root rtd DIR 0,12 1024 65282 / (172.16.42.75:/.rootfs/.std)
pdflush 5 root cwd DIR 0,12 1024 65282 / (172.16.42.75:/.rootfs/.std)
pdflush 5 root rtd DIR 0,12 1024 65282 / (172.16.42.75:/.rootfs/.std)
pdflush 6 root cwd DIR 0,12 1024 65282 / (172.16.42.75:/.rootfs/.std)
pdflush 6 root rtd DIR 0,12 1024 65282 / (172.16.42.75:/.rootfs/.std)
kswapd0 7 root cwd DIR 0,12 1024 65282 / (172.16.42.75:/.rootfs/.std)
kswapd0 7 root rtd DIR 0,12 1024 65282 / (172.16.42.75:/.rootfs/.std)
aio/0 8 root cwd DIR 0,12 1024 65282 / (172.16.42.75:/.rootfs/.std)
aio/0 8 root rtd DIR 0,12 1024 65282 / (172.16.42.75:/.rootfs/.std)
rpciod 49 root cwd DIR 0,12 1024 65282 / (172.16.42.75:/.rootfs/.std)
rpciod 49 root rtd DIR 0,12 1024 65282 / (172.16.42.75:/.rootfs/.std)
lockd 50 root cwd DIR 0,12 1024 65282 / (172.16.42.75:/.rootfs/.std)
lockd 50 root rtd DIR 0,12 1024 65282 / (172.16.42.75:/.rootfs/.std)
bash 812 root cwd DIR 0,12 1024 69388 /app/shutdown-0.0.5/script (172.16.42.75:/.rootfs/.std)
bash 812 root rtd DIR 0,12 1024 65282 / (172.16.42.75:/.rootfs/.std)
bash 812 root txt REG 0,12 437844 71410 /bin/bash (172.16.42.75:/.rootfs/.std)
bash 812 root mem REG 0,12 832636 8248 /app/glibc-2.3/lib/ld-2.3.so (172.16.42.75:/.rootfs/.std)
bash 812 root mem REG 0,12 14831 47164 /lib/libtermcap.so.2.0.8 (172.16.42.75:/.rootfs/.std)
bash 812 root mem REG 0,12 50088 8252 /app/glibc-2.3/lib/libdl-2.3.so (172.16.42.75:/.rootfs/.std)
bash 812 root mem REG 0,12 16153080 8249 /app/glibc-2.3/lib/libc-2.3.so (172.16.42.75:/.rootfs/.std)
bash 812 root mem REG 0,12 360212 8257 /app/glibc-2.3/lib/libnss_files-2.3.so (172.16.42.75:/.rootfs/.std)
bash 812 root mem REG 0,14 15795 619809 /.share/usr/app/samba-2.2.2/lib/libnss_winbind.so (172.16.42.75:/.share)
bash 812 root mem REG 0,12 1207520 93906 /app/gcc-3.2/lib/libgcc_s.so.1 (172.16.42.75:/.rootfs/.std)
bash 812 root 0u CHR 4,2 22 /dev/vc/2
bash 812 root 1u CHR 4,2 22 /dev/vc/2
bash 812 root 2u CHR 4,2 22 /dev/vc/2
bash 812 root 255u CHR 4,2 22 /dev/vc/2
sleep 1368 root cwd DIR 0,12 1024 65282 / (172.16.42.75:/.rootfs/.std)
sleep 1368 root rtd DIR 0,12 1024 65282 / (172.16.42.75:/.rootfs/.std)
sleep 1368 root txt REG 0,12 5200 71631 /bin/sleep (172.16.42.75:/.rootfs/.std)
sleep 1368 root mem REG 0,12 832636 8248 /app/glibc-2.3/lib/ld-2.3.so (172.16.42.75:/.rootfs/.std)
sleep 1368 root mem REG 0,12 69355 8254 /app/glibc-2.3/lib/libcrypt-2.3.so (172.16.42.75:/.rootfs/.std)
sleep 1368 root mem REG 0,12 16153080 8249 /app/glibc-2.3/lib/libc-2.3.so (172.16.42.75:/.rootfs/.std)
lsof 1376 root cwd DIR 0,12 1024 69388 /app/shutdown-0.0.5/script (172.16.42.75:/.rootfs/.std)
lsof 1376 root rtd DIR 0,12 1024 65282 / (172.16.42.75:/.rootfs/.std)
lsof 1376 root txt REG 0,14 87748 572607 /.share/usr/app/lsof-4.57/bin/lsof (172.16.42.75:/.share)
lsof 1376 root mem REG 0,12 832636 8248 /app/glibc-2.3/lib/ld-2.3.so (172.16.42.75:/.rootfs/.std)
lsof 1376 root mem REG 0,12 16153080 8249 /app/glibc-2.3/lib/libc-2.3.so (172.16.42.75:/.rootfs/.std)
lsof 1376 root 0u CHR 4,2 22 /dev/vc/2
lsof 1376 root 1w REG 0,14 1484 309496 /.local/var/KILLbug (172.16.42.75:/.share)
lsof 1376 root 2w REG 0,14 1484 309496 /.local/var/KILLbug (172.16.42.75:/.share)
lsof 1376 root 3r DIR 0,3 0 1 /proc
lsof 1376 root 4r DIR 0,3 0 90177545 /proc/1376/fd
lsof 1376 root 5w FIFO 0,7 4192 pipe
lsof 1376 root 6r FIFO 0,7 4193 pipe
lsof 1377 root cwd DIR 0,12 1024 69388 /app/shutdown-0.0.5/script (172.16.42.75:/.rootfs/.std)
lsof 1377 root rtd DIR 0,12 1024 65282 / (172.16.42.75:/.rootfs/.std)
lsof 1377 root txt REG 0,14 87748 572607 /.share/usr/app/lsof-4.57/bin/lsof (172.16.42.75:/.share)
lsof 1377 root mem REG 0,12 832636 8248 /app/glibc-2.3/lib/ld-2.3.so (172.16.42.75:/.rootfs/.std)
lsof 1377 root mem REG 0,12 16153080 8249 /app/glibc-2.3/lib/libc-2.3.so (172.16.42.75:/.rootfs/.std)
lsof 1377 root 4r FIFO 0,7 4192 pipe
lsof 1377 root 7w FIFO 0,7 4193 pipe
--
vda
>Hi Trond.
>
>I observed a problem with NFS root in 2.6 kernel
>(actually it was a 2.5 back then).
Does it also happen on 2.4?
>I am using NFS root. At shutdown, when I kill
>all processes with killall5 -9, NFS temporarily
>misbehaves. I narrowed it down to rpciod feeling
>bad when signalled with SIGKILL:
I think this has to do that you kill some userspace application that is
necessary for NFS. I am not exactly sure which one it is (if at all), but I
have had problems mounting an NFS volume when started with -b option (mounting
the root however was done ok by the kernel)
Jan Engelhardt
--
Gesellschaft f?r Wissenschaftliche Datenverarbeitung
Am Fassberg, 37077 G?ttingen, http://www.gwdg.de
m? den 25.10.2004 Klokka 17:02 (+0300) skreiv Denis Vlasenko:
> I am using NFS root. At shutdown, when I kill
> all processes with killall5 -9, NFS temporarily
> misbehaves. I narrowed it down to rpciod feeling
> bad when signalled with SIGKILL:
That is a deliberate feature. It is useful when mountpoints hang etc.
Note however that the patches that convert rpciod to use a workqueue
(they can be found in the latest -mm kernels) remove this feature.
Cheers,
Trond
--
Trond Myklebust <[email protected]>
On Monday 25 October 2004 18:20, Jan Engelhardt wrote:
> >That's the point. It works in 2.4
>
> Maybe because there is no rpciod in 2.4?
It is there.
> >Well, let's see. 2.4 works. rpciod in 2.6 shows this erratic behaviour
> >even if I do "kill -9 <pid_of_rpciod>", thus no other process, kernel
> >or userspace, know about this KILL.
>
> Is rpciod (a kthread as I read from your 'ps' output) killable in 2.4 after
> all?
It is not killable, neither 2.4 nor 2.6 one. It is by design I think,
because I *must not* kill it, or else NFS rootfs will fall off
and box will hang.
However, rpciod gets signalled by -KILL as a side effect of killall5 -9
when I shut my system down.
I do not send -KILL to all processes EXCEPT rpciod because:
(a) there is no suitable command to do that from shell and
(b) I don't like special cases
> Maybe the rpciod-26 is missing a sigblock()?
--
vda
>It is not killable, neither 2.4 nor 2.6 one. It is by design I think,
>because I *must not* kill it, or else NFS rootfs will fall off
>and box will hang.
If it's not killable (per kill(2)), why can killall5 (which will probably use
kill(2)!) do it?
Jan Engelhardt
--
Gesellschaft f?r Wissenschaftliche Datenverarbeitung
Am Fassberg, 37077 G?ttingen, http://www.gwdg.de
On Monday 25 October 2004 18:46, Jan Engelhardt wrote:
> >It is not killable, neither 2.4 nor 2.6 one. It is by design I think,
> >because I *must not* kill it, or else NFS rootfs will fall off
> >and box will hang.
>
> If it's not killable (per kill(2)), why can killall5 (which will probably use
> kill(2)!) do it?
I believe by design it is a kernel thread, unkillable even by -KILL
signal (precisely because of NFS root setups). You are right,
it does not matter how do you send -KILL from shell: via kill,
killall, killall5, etc... It simply should be ignored by rpciod.
And it is ignored in 2.4. In 2.6, rpciod becomes non-functional
(NFS reads return -EIO) for a second or two, then it returns
back to normal.
Hope this clears up the confusion.
(BTW, I planned to temporarily work around this by inserting
"sleep 3" right after killall5, but, hehe, reading /bin/sleep
via NFS, of course, did not work also :)
--
vda
On Monday 25 October 2004 17:41, Trond Myklebust wrote:
> m? den 25.10.2004 Klokka 17:02 (+0300) skreiv Denis Vlasenko:
>
> > I am using NFS root. At shutdown, when I kill
> > all processes with killall5 -9, NFS temporarily
> > misbehaves. I narrowed it down to rpciod feeling
> > bad when signalled with SIGKILL:
>
> That is a deliberate feature. It is useful when mountpoints hang etc.
>
> Note however that the patches that convert rpciod to use a workqueue
> (they can be found in the latest -mm kernels) remove this feature.
These?
linux-2.6.8.1-49-rpc_workqueue.patch
linux-2.6.8.1-50-rpc_queue_lock.patch
I will test 'em, thanks!
--
vda
>That's the point. It works in 2.4
Maybe because there is no rpciod in 2.4?
>Well, let's see. 2.4 works. rpciod in 2.6 shows this erratic behaviour
>even if I do "kill -9 <pid_of_rpciod>", thus no other process, kernel
>or userspace, know about this KILL.
Is rpciod (a kthread as I read from your 'ps' output) killable in 2.4 after
all?
Maybe the rpciod-26 is missing a sigblock()?
Jan Engelhardt
--
Gesellschaft f?r Wissenschaftliche Datenverarbeitung
Am Fassberg, 37077 G?ttingen, http://www.gwdg.de
On Monday 25 October 2004 17:24, Jan Engelhardt wrote:
> >Hi Trond.
> >
> >I observed a problem with NFS root in 2.6 kernel
> >(actually it was a 2.5 back then).
>
> Does it also happen on 2.4?
That's the point. It works in 2.4
> >I am using NFS root. At shutdown, when I kill
> >all processes with killall5 -9, NFS temporarily
> >misbehaves. I narrowed it down to rpciod feeling
> >bad when signalled with SIGKILL:
>
> I think this has to do that you kill some userspace application that is
> necessary for NFS. I am not exactly sure which one it is (if at all), but I
> have had problems mounting an NFS volume when started with -b option (mounting
> the root however was done ok by the kernel)
Well, let's see. 2.4 works. rpciod in 2.6 shows this erratic behaviour
even if I do "kill -9 <pid_of_rpciod>", thus no other process, kernel
or userspace, know about this KILL.
These commands and their output were actually shown in my mail.
Consider rereading it:
> PID TTY STAT TIME COMMAND
> 1 ? S 0:05 /bin/sh /init.vda
> 2 ? SWN 0:00 [ksoftirqd/0]
> 3 ? SW< 0:00 [events/0]
> 4 ? SW< 0:00 [kblockd/0]
> 5 ? SW 0:00 [pdflush]
> 6 ? SW 0:00 [pdflush]
> 8 ? SW< 0:00 [aio/0]
> 7 ? SW 0:00 [kswapd0]
> 49 ? SW 0:00 [rpciod]
> 50 ? SW 0:00 [lockd]
> 812 vc/2 S 0:00 -bash HOME=/home/vda PATH=/sbin:/bin:/usr/sbin:/usr
> 1369 tty2 R 0:00 ps -AH e PWD=/app/shutdown-0.0.5/script GROFF_NO_
> 1368 ? S 0:00 sleep 32000
Aha! rpciod has PID=49, lets -KILL it...
> # kill -9 49;ps -AH e;read junk;ps -AH e
> bash: /bin/ps: Input/output error
Yay. ps binary couldn't be read from NFS.
> <--- I press [Enter] here
Second ps runs fine because rpciod recovered by this time:
> PID TTY STAT TIME COMMAND
> 1 ? S 0:05 /bin/sh /init.vda
> 2 ? SWN 0:00 [ksoftirqd/0]
> 3 ? SW< 0:00 [events/0]
> 4 ? SW< 0:00 [kblockd/0]
> 5 ? SW 0:00 [pdflush]
> 6 ? SW 0:00 [pdflush]
> 8 ? SW< 0:00 [aio/0]
> 7 ? SW 0:00 [kswapd0]
> 49 ? SW 0:00 [rpciod]
> 50 ? SW 0:00 [lockd]
> 812 vc/2 S 0:00 -bash HOME=/home/vda PATH=/sbin:/bin:/usr/sbin:/usr
> 1369 tty2 R 0:00 ps -AH e PWD=/app/shutdown-0.0.5/script GROFF_NO_
> 1368 ? S 0:00 sleep 32000
--
vda
On Monday 25 October 2004 17:41, Trond Myklebust wrote:
> m? den 25.10.2004 Klokka 17:02 (+0300) skreiv Denis Vlasenko:
>
> > I am using NFS root. At shutdown, when I kill
> > all processes with killall5 -9, NFS temporarily
> > misbehaves. I narrowed it down to rpciod feeling
> > bad when signalled with SIGKILL:
>
> That is a deliberate feature. It is useful when mountpoints hang etc.
>
> Note however that the patches that convert rpciod to use a workqueue
> (they can be found in the latest -mm kernels) remove this feature.
I've tried 2.6.9-mm1, it does not exhibit this anomaly.
Thanks!
--
vda