Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1756213Ab0HXWj0 (ORCPT ); Tue, 24 Aug 2010 18:39:26 -0400 Received: from mail.vyatta.com ([76.74.103.46]:60910 "EHLO mail.vyatta.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753283Ab0HXWjY (ORCPT ); Tue, 24 Aug 2010 18:39:24 -0400 Date: Tue, 24 Aug 2010 15:39:20 -0700 From: Stephen Hemminger To: Ben Hutchings Cc: Marc Aurele La France , linux-kernel@vger.kernel.org, netdev@vger.kernel.org, "David S. Miller" , Alexey Kuznetsov , "Pekka Savola (ipv6)" , James Morris , Hideaki YOSHIFUJI , Patrick McHardy Subject: Re: RFC: MTU for serving NFS on Infiniband Message-ID: <20100824153920.63360072@s6510> In-Reply-To: <1282688441.22839.34.camel@localhost> References: <20100823080543.319143e3@nehalam> <1282672647.2302.15.camel@achroite.uk.solarflarecom.com> <1282688441.22839.34.camel@localhost> Organization: Vyatta X-Mailer: Claws Mail 3.7.5 (GTK+ 2.20.1; i486-pc-linux-gnu) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 2794 Lines: 64 On Tue, 24 Aug 2010 23:20:41 +0100 Ben Hutchings wrote: > On Tue, 2010-08-24 at 13:49 -0600, Marc Aurele La France wrote: > > On Tue, 24 Aug 2010, Ben Hutchings wrote: > > > On Tue, 2010-08-24 at 09:14 -0600, Marc Aurele La France wrote: > > >> On Mon, 23 Aug 2010, Stephen Hemminger wrote: > > >>> On Mon, 23 Aug 2010 08:44:37 -0600 (MDT) > > >>> Marc Aurele La France wrote: > > >>>> In regrouping for my next tack at this, I noticed that all stack traces go > > >>>> through ip_append_data(). This would be ipv6_append_data() in the IPv6 case. > > >>>> A _very_ rough draft that would have ip_append_data() temporarily drop down > > >>>> to a smaller fake MTU follows ... > > > > >>> Why doesn't NFS generate page size fragments? Does Infiniband or your > > >>> device not support this? Any thing that requires higher order allocation > > >>> is going to unstable under load. Let's fix the cause not the apply bandaid > > >>> solution to the symptom. > > > > >> From what I can tell, IP fragmentation is done centrally. > > > [...] > > > > > Stephen and I are not talking about IP fragmentation, but about the > > > ability to append 'fragments' to an skb rather than putting the entire > > > packet payload in a linear buffer. See > > > . > > > > Any payload has to either fit in the MTU, or has to be broken up into > > MTU-sized (or less) fragments, come hell or high water. That this is done > > centrally is a good thing. > > Not necessarily. Offloading it to hardware, where possible, is usually > a performance win. > > > It is the "(or less)" part that I am working towards here. > > The inability to allocate large linear buffers is not a good reason to > generate packets smaller than the MTU. You are working around the real > problem. IF NFS server is smart enough to generate: Header (skb) + one or more pages in fragment list then IP fragmentation could do fragmentation by allocating new headers skb (small) and assigning the same pages to multiple skb's using page ref count. It obviously isn't working that way. The whole problem is moot because NFS over UDP has known data corruption issues in the face of packet loss. The sequence number of the IP fragment can easily wrap around causing old data to be grouped with new data and the UDP checksum is so weak that the resulting UDP packet will be consumed by the NFS client ans passed to the user application as corrupted disk block. DON'T USE NFS OVER UDP! -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/