Return-Path: linux-nfs-owner@vger.kernel.org Received: from mail2.candelatech.com ([208.74.158.173]:54642 "EHLO mail2.candelatech.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750907AbaGaVUI (ORCPT ); Thu, 31 Jul 2014 17:20:08 -0400 Message-ID: <53DAB307.2000206@candelatech.com> Date: Thu, 31 Jul 2014 14:20:07 -0700 From: Ben Greear MIME-Version: 1.0 To: NeilBrown CC: "linux-nfs@vger.kernel.org" Subject: Re: Killing process in D state on mount to dead NFS server. References: <53DA8443.407@candelatech.com> <20140801064217.01852788@notabene.brown> In-Reply-To: <20140801064217.01852788@notabene.brown> Content-Type: text/plain; charset=ISO-8859-1 Sender: linux-nfs-owner@vger.kernel.org List-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 07/31/2014 01:42 PM, NeilBrown wrote: > On Thu, 31 Jul 2014 11:00:35 -0700 Ben Greear wrote: > >> So, this has been asked all over the interweb for years and years, but the best answer I can find is to reboot the system or create a fake NFS server >> somewhere with the same IP as the gone-away NFS server. >> >> The problem is: >> >> I have some mounts to an NFS server that no longer exists (crashed/powered down). >> >> I have some processes stuck trying to write to files open on these mounts. >> >> I want to kill the process and unmount. >> >> umount -l will make the mount go a way, sort of. But process is still hung. umount -f complains: umount2: Device or resource busy umount.nfs: /mnt/foo: >> device is busy >> >> kill -9 does not work on process. > > Kill -1 should work (since about 2.6.25 or so). That is -[ONE], right? Assuming so, it did not work for me. Kernel is 3.14.4+, with some of extra patches, but probably nothing that influences this particular behaviour. [root@lf1005-14010010 ~]# cat /proc/3805/stack [] sleep_on_page+0x9/0xd [] wait_on_page_bit+0x71/0x78 [] filemap_fdatawait_range+0xa2/0x16d [] filemap_write_and_wait_range+0x3b/0x77 [] nfs_file_fsync+0x37/0x83 [nfs] [] vfs_fsync_range+0x19/0x1b [] vfs_fsync+0x17/0x19 [] nfs_file_flush+0x6b/0x6f [nfs] [] filp_close+0x3f/0x71 [] __close_fd+0x80/0x98 [] SyS_close+0x1c/0x3e [] system_call_fastpath+0x16/0x1b [] 0xffffffffffffffff [root@lf1005-14010010 ~]# kill -1 3805 [root@lf1005-14010010 ~]# cat /proc/3805/stack [] sleep_on_page+0x9/0xd [] wait_on_page_bit+0x71/0x78 [] filemap_fdatawait_range+0xa2/0x16d [] filemap_write_and_wait_range+0x3b/0x77 [] nfs_file_fsync+0x37/0x83 [nfs] [] vfs_fsync_range+0x19/0x1b [] vfs_fsync+0x17/0x19 [] nfs_file_flush+0x6b/0x6f [nfs] [] filp_close+0x3f/0x71 [] __close_fd+0x80/0x98 [] SyS_close+0x1c/0x3e [] system_call_fastpath+0x16/0x1b [] 0xffffffffffffffff Thanks, Ben > If it doesn't please report the kernel version and cat /proc/$PID/stack > > for some processes that cannot be killed. > > NeilBrown > >> >> >> Aside from bringing a fake NFS server back up on the same IP, is there any other way to get these mounts unmounted and the processes killed without >> rebooting? >> >> Thanks, Ben >> > - -- Ben Greear Candela Technologies Inc http://www.candelatech.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.13 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQEcBAEBAgAGBQJT2rLiAAoJELbHqkYeJT4OqPgH/0taKW6Be90c1mETZf9yeqZF YMLZk8XC2wloEd9nVz//mXREmiu18Hc+5p7Upd4Os21J2P4PBMGV6P/9DMxxehwH YX1HKha0EoAsbO5ILQhbLf83cRXAPEpvJPgYHrq6xjlKB8Q8OxxND37rY7kl19Zz sdAw6GiqHICF3Hq1ATa/jvixMluDnhER9Dln3wOdAGzmmuFYqpTsV4EwzbKKqInJ 6C15q+cq/9aYh6usN6z2qJhbHgqM9EWcPL6jOrCwX4PbC1XjKHekpFN0t9oKQClx qSPuweMQ7fP4IBd2Ke8L/QlyOVblAKSE7t+NdrjfzLmYPzyHTyfLABR/BI053to= =/9FJ -----END PGP SIGNATURE-----