Return-Path: Received: from mail-ig0-f173.google.com ([209.85.213.173]:34490 "EHLO mail-ig0-f173.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S932189AbbFSUal (ORCPT ); Fri, 19 Jun 2015 16:30:41 -0400 Received: by igboe5 with SMTP id oe5so24665490igb.1 for ; Fri, 19 Jun 2015 13:30:40 -0700 (PDT) Message-ID: <1434745818.8838.1.camel@primarydata.com> Subject: Re: [REGRESSION] NFS is creating a hidden port (left over from xs_bind() ) From: Trond Myklebust To: Jeff Layton Cc: Steven Rostedt , Eric Dumazet , Anna Schumaker , Linux NFS Mailing List , Linux Network Devel Mailing List , LKML , Andrew Morton , Bruce James Fields Date: Fri, 19 Jun 2015 16:30:18 -0400 In-Reply-To: <20150619155226.7c5d6637@synchrony.poochiereds.net> References: <20150611234929.7b48d314@gandalf.local.home> <1434120035.27504.77.camel@edumazet-glaptop2.roam.corp.google.com> <20150612113420.5c114183@gandalf.local.home> <20150612115038.44e20ccd@gandalf.local.home> <20150617230819.0a11e47a@grimm.local.home> <20150618154914.57424225@gandalf.local.home> <20150618185051.2b2828c3@tlielax.poochiereds.net> <20150618210843.4b09c31c@grimm.local.home> <20150618213702.33f64b20@synchrony.poochiereds.net> <20150619122553.43d2d86c@gandalf.local.home> <20150619131759.7b38d9de@gandalf.local.home> <20150619155226.7c5d6637@synchrony.poochiereds.net> Content-Type: text/plain; charset="UTF-8" Mime-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org List-ID: On Fri, 2015-06-19 at 15:52 -0400, Jeff Layton wrote: > On Fri, 19 Jun 2015 13:39:08 -0400 > Trond Myklebust wrote: > > > On Fri, Jun 19, 2015 at 1:17 PM, Steven Rostedt < > > rostedt@goodmis.org> wrote: > > > On Fri, 19 Jun 2015 12:25:53 -0400 > > > Steven Rostedt wrote: > > > > > > > > > > I don't see that 55201 anywhere. But then again, I didn't look > > > > for it > > > > before the port disappeared. I could reboot and look for it > > > > again. I > > > > should have saved the full netstat -tapn as well :-/ > > > > > > Of course I didn't find it anywhere, that's the port on my wife's > > > box > > > that port 947 was connected to. > > > > > > Now I even went over to my wife's box and ran > > > > > > # rpcinfo -p localhost > > > program vers proto port service > > > 100000 4 tcp 111 portmapper > > > 100000 3 tcp 111 portmapper > > > 100000 2 tcp 111 portmapper > > > 100000 4 udp 111 portmapper > > > 100000 3 udp 111 portmapper > > > 100000 2 udp 111 portmapper > > > 100024 1 udp 34243 status > > > 100024 1 tcp 34498 status > > > > > > which doesn't show anything. > > > > > > but something is listening to that port... > > > > > > # netstat -ntap |grep 55201 > > > tcp 0 0 0.0.0.0:55201 0.0.0.0:* > > > LISTEN > > > > > > Hang on. This is on the client box while there is an active NFSv4 > > mount? Then that's probably the NFSv4 callback channel listening > > for > > delegation callbacks. > > > > Can you please try: > > > > echo "options nfs callback_tcpport=4048" > /etc/modprobe.d/nfs > > -local.conf > > > > and then either reboot the client or unload and then reload the nfs > > modules before reattempting the mount. If this is indeed the > > callback > > channel, then that will move your phantom listener to port 4048... > > > > Right, it was a little unclear to me before, but it now seems clear > that the callback socket that the server is opening to the client is > the one squatting on the port. > > ...and that sort of makes sense, doesn't it? That rpc_clnt will stick > around for the life of the client's lease, and the rpc_clnt binds to > a > particular port so that it can reconnect using the same one. > > Given that Stephen has done the legwork and figured out that > reverting > those commits fixes the issue, then I suspect that the real culprit > is > caf4ccd4e88cf2. > > The client is likely closing down the other end of the callback > socket when it goes idle. Before that commit, we probably did an > xs_close on it, but now we're doing a xs_tcp_shutdown and that leaves > the port bound. > Agreed. I've been looking into whether or not there is a simple fix. Reverting those patches is not an option, because the whole point was to ensure that the socket is in the TCP_CLOSED state before we release the socket. Steven, how about something like the following patch? 8<-----------------------------------------------------------------