Return-Path: In-Reply-To: <7E189B77-1139-4B16-97E5-4841B41B90C7@oracle.com> References: <4A7BCCCA.4020307@panasas.com> <20090807140425.GA18298@fieldses.org> <4974ED30-D8CA-47B0-9D8F-BCD4410132FC@oracle.com> <7E189B77-1139-4B16-97E5-4841B41B90C7@oracle.com> Date: Tue, 11 Aug 2009 23:37:23 -0300 Message-ID: Subject: Re: AutoFS+NFSv4 server down = LOOOOONG timeout. From: =?ISO-8859-1?Q?Carlos_Andr=E9?= To: Chuck Lever Cc: Ian Kent , NFS list , Linux NFSv4 mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="iso-8859-1" Sender: nfsv4-bounces@linux-nfs.org Errors-To: nfsv4-bounces@linux-nfs.org MIME-Version: 1.0 List-ID: Chuck, Since we'll use some NFS servers, and (some) they are not critical to user work (if they're down, user can just sit and wait for them working on something else like a webapp, printing, etc...) and if we lose a server or a router (worst case) acumulative timeouts on workstation login-boot process will make users try kill us (lol). A acceptable timeout(first mount fail timeout) per automount try is something around 10-15 (max) seconds. For us the best is a option to permit us to do adjustments (1 to X secs), and putting this like a new feature/non-default option will not mess with another users.... I dont wanna make a mess on sources by myself just to modify automounter for our needs, I just want the "right" solution in our case... Thanks for help. 2009/8/11 Chuck Lever : > On Aug 11, 2009, at 8:41 AM, Carlos Andr=E9 wrote: >> >> This long timeout is good if workstation need mount a critical >> directory using /etc/fstab on boot (for example).. >> But in my case, using this loooong timeout doesnt make any sense, >> since autofs retry mount directory on-access. This in fact gives me >> alot of headaches, coz user login 'll just hangs if one server goes >> down for any reason, and will again hangs if user try access directory >> pointing to a NFS down server... > > "retry=3D0" means the mount command will fail as soon as the first mount(= 2) > system call fails. =A0When you set SYN retries to 1, this means after 9 > seconds, the connect fails, and that causes the mount(2) system call to > fail. > > Recent conversations with Ian suggested that a long timeout was desired f= or > automounter as well as other cases. =A0Ian, is there something else we ne= ed to > consider to determine the correct retry timeout for NFS/TCP mount points > handled via automounter? =A0How should mount.nfs wait so we don't make ot= her > use cases worse? =A0(Looks like most of the history is intact below). > > How long do you think is appropriate for the automounter to wait if the > server is down, in your case, Carlos? > >> Am losing something or there have was something weirdo...!? >> ------------------------------------------------ >> [root@KSTATION ~]# echo 5 > /proc/sys/net/ipv4/tcp_syn_retries =A0[DEFAU= LT] >> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o >> proto=3Dtcp,retry=3D1 >> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up). >> >> real =A0 =A03m9.000s >> user =A0 =A00m0.002s >> sys =A0 =A0 0m0.001s >> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o >> sec=3Dkrb5p,proto=3Dtcp,retry=3D1 >> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up). >> >> real =A0 =A03m9.000s >> user =A0 =A00m0.000s >> sys =A0 =A0 0m0.002s >> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o >> proto=3Dtcp,retry=3D0 >> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up). >> >> real =A0 =A03m9.001s >> user =A0 =A00m0.000s >> sys =A0 =A0 0m0.003s >> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o >> sec=3Dkrb5p,proto=3Dtcp,retry=3D0 >> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up). >> >> real =A0 =A03m9.001s >> user =A0 =A00m0.002s >> sys =A0 =A0 0m0.001s >> >> [root@KSTATION ~]# echo 1 > /proc/sys/net/ipv4/tcp_syn_retries [ 5 to 1 = ] >> >> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o >> proto=3Dtcp,retry=3D1 >> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying). [x 6] >> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up). >> >> real =A0 =A01m3.002s >> user =A0 =A00m0.000s >> sys =A0 =A0 0m0.002s >> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o >> sec=3Dkrb5p,proto=3Dtcp,retry=3D1 >> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying). [x 13= ] >> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up). >> >> real =A0 =A02m6.000s >> user =A0 =A00m0.000s >> sys =A0 =A0 0m0.002s >> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o >> proto=3Dtcp,retry=3D0 >> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up). >> >> real =A0 =A00m9.003s >> user =A0 =A00m0.001s >> sys =A0 =A0 0m0.002s >> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o >> sec=3Dkrb5p,proto=3Dtcp,retry=3D0 >> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying). [x 13= ] >> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up). >> >> real =A0 =A02m6.001s >> user =A0 =A00m0.001s >> sys =A0 =A0 0m0.002s >> [root@KSTATION ~]# >> ------------------------------------------------ >> max timeout goes to 2m6s changing tcp_syn_retries from 5 to 1... and >> using retry=3D0 without kerberos I got only 9s... >> >> *sigh* >> >> >> >> 2009/8/10 Chuck Lever : >>> >>> On Aug 10, 2009, at 4:05 PM, Carlos Andr=E9 wrote: >>>> >>>> Something funny: Using default tcp_syn_retries (5) i got >>>> "3,6,12,24,48,96" secs interval... but if i change tcp_syn_retries to >>>> 1 i got "3,6,3,6,3,6..." secs interval... >>> >>> Right. =A0Normally the RPC client calls the kernel's socket connect >>> function, >>> which does 6 SYN retries. =A0That one call usually takes longer than th= e >>> RPC >>> client's connect timeout, so it only makes one connect call, and then >>> fails. >>> >>> Reducing the number of SYN retries per connect attempt causes the RPC >>> client >>> to retry the connect call until its connect timeout expires. =A0Each >>> connect >>> call resets the SYN timeout to 3 seconds. >>> >>>> [root@KSERVER mnt]# time mount 1.2.3.4:/blabla tmp/ -t nfs4 -o >>>> sec=3Dkrb5p,proto=3Dtcp >>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up). >>>> >>>> real =A0 =A03m9.000s >>>> user =A0 =A00m0.000s >>>> sys =A0 =A0 0m0.002s >>>> >>>> [root@KSERVER /]# echo 1 > /proc/sys/net/ipv4/tcp_syn_retries >>>> [root@KSERVER mnt]# time mount 1.2.3.4:/blabla tmp/ -t nfs4 -o >>>> sec=3Dkrb5p,proto=3Dtcp =A0("retry=3D1" =3D no change) >>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying). >>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying). >>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying). >>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying). >>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying). >>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying). >>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying). >>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying). >>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying). >>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying). >>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying). >>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying). >>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying). >>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up). >>>> >>>> real =A0 =A02m6.004s >>>> user =A0 =A00m0.000s >>>> sys =A0 =A0 0m0.004s >>>> >>>> (3,6,3,6... secs interval) >>>> >>>> >>>> >>>> >>>> 2009/8/10 Carlos Andr=E9 : >>>>> >>>>> No, i'm just using packages from CentOS repo... >>>>> >>>>> And u're right about expo retries... with tcpdump i've monitored >>>>> traffic and i got SYN retries in 3, 6, 12, 24, 48, 96 secs on port >>>>> 2049... >>>>> I tried use "retry=3D1" option on mount without any change... I dont >>>>> want change source or tcp timers... just NFSv4 client. >>>>> >>>>> 2009/8/10 Chuck Lever : >>>>>> >>>>>> On Aug 10, 2009, at 2:29 PM, Carlos Andr=E9 wrote: >>>>>>> >>>>>>> Bruce, no... you're right. =A0I'm describing a situation where my >>>>>>> server >>>>>>> died... i need mount fail faster (10 or 15 secs max) than 3 minutes >>>>>>> and 9 seconds... >>>>>> >>>>>> The 189 second timeout is likely how long it takes the kernel to giv= e >>>>>> up >>>>>> trying to connect a TCP socket to the server (6 SYN attempts with >>>>>> exponential retries, or something like that). =A0For stock CentOS 5.= 3, I >>>>>> think >>>>>> user space does only a DNS lookup for normal NFSv4 mounts -- the >>>>>> kernel >>>>>> just >>>>>> tries to connect a TCP socket to port 2049, with no preceding rpcbin= d >>>>>> request. >>>>>> >>>>>> Carlos, let us know if you have replaced any NFS-related CentOS >>>>>> components >>>>>> (kernel, nfs-utils) with something you've built yourself. >>>>>> >>>>>>> 2009/8/7 J. Bruce Fields : >>>>>>>> >>>>>>>> On Fri, Aug 07, 2009 at 09:42:18AM +0300, Benny Halevy wrote: >>>>>>>>> >>>>>>>>> On Aug. 07, 2009, 3:18 +0300, Carlos Andr=E9 >>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> Anyone ? >>>>>>>>>> >>>>>>>>>> 2009/7/29 Carlos Andr=E9 : >>>>>>>>>>> >>>>>>>>>>> PPL, I need put a CentOS 5.3 (updated) NFSv4 server to work wit= h >>>>>>>>>>> Kerberos >>>>>>>>>>> and AutoFS, but i got a problem: If NFS server goes down i get = a >>>>>>>>>>> LOOOOOOONG >>>>>>>>>>> mount timeout on CentOS 5.3 (updated) NFSv4 client... >>>>>>>>>>> >>>>>>>>>>> Since i need mount some (3 to 6) dirs at user logon process, if >>>>>>>>>>> mount >>>>>>>>>>> hangs, >>>>>>>>>>> user logon hangs. Then i want configure it to timeout (if serve= r >>>>>>>>>>> down) >>>>>>>>>>> after >>>>>>>>>>> 10-15 secs (MAX) on each mount attempt. >>>>>>>>>>> >>>>>>>>>>> I already make a lab and tried a LOT of combinations, there my >>>>>>>>>>> findings >>>>>>>>>>> (server DOWN IP: 172.16.0.10 / client IP: 172.16.1.10) using >>>>>>>>>>> basic >>>>>>>>>>> command >>>>>>>>>>> (time mount 172.16.0.10:/remotedir /localdir/ -t nfs4 -o >>>>>>>>>>> sec=3Dkrb5,proto=3D) from NFS client: >>>>>>>>>>> >>>>>>>>>>> - Once i try access mount point using AutoFS (proto=3Dtcp OR >>>>>>>>>>> proto=3Dudp) >>>>>>>>>>> it >>>>>>>>>>> hangs for 189 secs (3m9s: real =A03m9.001s) =A0until show error >>>>>>>>>>> (mount: >>>>>>>>>>> mount to >>>>>>>>>>> NFS server '172.16.0.10' failed: timed out (giving up)) >>>>>>>>> >>>>>>>>> Sounds like you're hitting the server's grace period. >>>>>>>> >>>>>>>> I thought he was describing a situation where the server the serve= r >>>>>>>> is completely gone and isn't coming back, and wondering how to mak= e >>>>>>>> the >>>>>>>> mount fail faster. =A0But I may be misunderstanding. >>>>>>>> >>>>>>>> --b. >>>>>>>> >>>>>>> -- >>>>>>> To unsubscribe from this list: send the line "unsubscribe linux-nfs= " >>>>>>> in >>>>>>> the body of a message to majordomo@vger.kernel.org >>>>>>> More majordomo info at =A0http://vger.kernel.org/majordomo-info.htm= l >>>>>> >>>>>> -- >>>>>> Chuck Lever >>>>>> chuck[dot]lever[at]oracle[dot]com >>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>> >>> -- >>> Chuck Lever >>> chuck[dot]lever[at]oracle[dot]com >>> >>> >>> >>> > > -- > Chuck Lever > chuck[dot]lever[at]oracle[dot]com > > > > _______________________________________________ NFSv4 mailing list NFSv4@linux-nfs.org http://linux-nfs.org/cgi-bin/mailman/listinfo/nfsv4