Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755745AbYGVVOr (ORCPT ); Tue, 22 Jul 2008 17:14:47 -0400 Received: (majordomo@vger.kernel.org) by vger.kernel.org id S1755489AbYGVVOQ (ORCPT ); Tue, 22 Jul 2008 17:14:16 -0400 Received: from 1wt.eu ([62.212.114.60]:1906 "EHLO 1wt.eu" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752912AbYGVVON (ORCPT ); Tue, 22 Jul 2008 17:14:13 -0400 Date: Tue, 22 Jul 2008 23:12:56 +0200 From: Willy Tarreau To: Ingo Molnar Cc: David Newall , Linus Torvalds , David Miller , akpm@linux-foundation.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org, Stefan Richter Subject: Re: [TCP bug] stuck distcc connections in latest -git Message-ID: <20080722211256.GB29189@1wt.eu> References: <20080721133059.GA30637@elte.hu> <20080721134506.GA27598@elte.hu> <20080721182318.GA20940@elte.hu> <20080721184616.GA8442@elte.hu> <20080722112133.GA6575@elte.hu> <4885E482.5020502@davidnewall.com> <20080722135723.GA23077@elte.hu> <4885F496.3010305@davidnewall.com> <20080722153443.GB18757@elte.hu> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20080722153443.GB18757@elte.hu> User-Agent: Mutt/1.5.11 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 1833 Lines: 44 On Tue, Jul 22, 2008 at 05:34:43PM +0200, Ingo Molnar wrote: > * David Newall wrote: > > > Ingo Molnar wrote: > > > * David Newall wrote: > > > > > >> You really should start that capture, and on both client and server. > > >> You don't need to dump everything, only traffic to or from > > >> server:distcc. > > >> > > > > > > It's not feasible. That box did in excess of 200 GB of network traffic > > > in the past 7 hours alone. > > > > You only need distcc traffic, and perhaps only after it's hung. With > > 250k outstanding per socket, are you certain that no traffic was sent? > > Is it certain that one packet wasn't being sent each three minutes? I > > suppose you're right and the stack really is stuck, but this is such > > an easy thing to check and eliminate that you should do so. I > > suppose, too, that you should trace the server-side processes and > > confirm that they are waiting for socket input. You should dump tcp > > (for the distcc port) next time the problem recurs and also check that > > the server processes are waiting for socket input. > > ok, will do that if it happens again. Ingo, if it can help, I have a "capture" script which allows you to define a size and will rotate captures within that size. That's what I'm using to troubleshoot rarely occuring problems in datacenters, so it's horrible but efficient :-) You just have to stop it once the problem has happened again. Ping me if you're interested (I'm lazy to start my laptop right just for it now in fact). Willy -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/