Date: Wed, 23 Jul 2008 10:26:43 +0200
From: Ingo Molnar <mingo@elte.hu>
To: Willy Tarreau <w@1wt.eu>
Cc: David Newall <davidn@davidnewall.com>,
       Linus Torvalds <torvalds@linux-foundation.org>,
       David Miller <davem@davemloft.net>, akpm@linux-foundation.org,
       netdev@vger.kernel.org, linux-kernel@vger.kernel.org,
       Stefan Richter <stefanr@s5r6.in-berlin.de>
Subject: Re: [TCP bug] stuck distcc connections in latest -git
Message-ID: <20080723082643.GA992@elte.hu>
References: <20080721134506.GA27598@elte.hu> <20080721182318.GA20940@elte.hu> <alpine.LFD.1.10.0807211131390.31863@woody.linux-foundation.org> <20080721184616.GA8442@elte.hu> <20080722112133.GA6575@elte.hu> <4885E482.5020502@davidnewall.com> <20080722135723.GA23077@elte.hu> <4885F496.3010305@davidnewall.com> <20080722153443.GB18757@elte.hu> <20080722211256.GB29189@1wt.eu>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20080722211256.GB29189@1wt.eu>
User-Agent: Mutt/1.5.18 (2008-05-17)
Sender: linux-kernel-owner@vger.kernel.org
Content-Length: 2454
Lines: 57


* Willy Tarreau <w@1wt.eu> wrote:

> On Tue, Jul 22, 2008 at 05:34:43PM +0200, Ingo Molnar wrote:
> > * David Newall <davidn@davidnewall.com> wrote:
> > 
> > > Ingo Molnar wrote:
> > > > * David Newall <davidn@davidnewall.com> wrote:
> > > >   
> > > >> You really should start that capture, and on both client and server. 
> > > >> You don't need to dump everything, only traffic to or from 
> > > >> server:distcc.
> > > >>     
> > > >
> > > > It's not feasible. That box did in excess of 200 GB of network traffic 
> > > > in the past 7 hours alone.
> > > 
> > > You only need distcc traffic, and perhaps only after it's hung.  With 
> > > 250k outstanding per socket, are you certain that no traffic was sent? 
> > > Is it certain that one packet wasn't being sent each three minutes?  I 
> > > suppose you're right and the stack really is stuck, but this is such 
> > > an easy thing to check and eliminate that you should do so.  I 
> > > suppose, too, that you should trace the server-side processes and 
> > > confirm that they are waiting for socket input.  You should dump tcp 
> > > (for the distcc port) next time the problem recurs and also check that 
> > > the server processes are waiting for socket input.
> > 
> > ok, will do that if it happens again.
> 
> Ingo,
> 
> if it can help, I have a "capture" script which allows you to define a 
> size and will rotate captures within that size. That's what I'm using 
> to troubleshoot rarely occuring problems in datacenters, so it's 
> horrible but efficient :-)
> 
> You just have to stop it once the problem has happened again. Ping me 
> if you're interested (I'm lazy to start my laptop right just for it 
> now in fact).

yeah, that would be handy, thanks.

Alas, the problem has not reoccured since then - more than a thousand 
kernel builds down the line. Yesterday it triggered so quickly when i 
updated the buildbox to the new kernel, and happened repeatedly when i 
tried to build a new kernel, that i didnt assume it was something hard 
to reproduce - but it went poof after i restarted distccd on the server.

So i'd suggest we do not count this as a regression, i've got no way at 
the moment of reproducing it reliably.

	Ingo
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/