Return-Path: Received: from mail-fx0-f46.google.com ([209.85.161.46]:33343 "EHLO mail-fx0-f46.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752935Ab0FHGy6 (ORCPT ); Tue, 8 Jun 2010 02:54:58 -0400 Received: by fxm8 with SMTP id 8so2604331fxm.19 for ; Mon, 07 Jun 2010 23:54:56 -0700 (PDT) Message-ID: <4C0DE93D.2050503@panasas.com> Date: Tue, 08 Jun 2010 09:54:53 +0300 From: Benny Halevy To: "J. Bruce Fields" CC: Boaz Harrosh , sfaibish , NFS list Subject: Re: Performance results with exofs References: <4C0D195B.8030401@panasas.com> <4C0D1AAB.4070304@panasas.com> <20100607182948.GF25257@fieldses.org> <4C0D3D59.4000305@panasas.com> <20100607184902.GI25257@fieldses.org> In-Reply-To: <20100607184902.GI25257@fieldses.org> Content-Type: text/plain; charset=ISO-8859-1 Sender: linux-nfs-owner@vger.kernel.org List-ID: MIME-Version: 1.0 On 2010-06-07 21:49, J. Bruce Fields wrote: > On Mon, Jun 07, 2010 at 09:41:29PM +0300, Boaz Harrosh wrote: >> On 06/07/2010 09:29 PM, J. Bruce Fields wrote: >>>>> On 06/07/2010 07:07 PM, Boaz Harrosh wrote: >>>>>> I did not yet publish the Document. It's stuck behind my dis-talent for >>>>>> writing and the pnfs bugs de jur. >>> >>> Untalented writing we can fix, as long as the details are there! >>> >>>>>> >>>>>> Basically all machines: >>>>>> - connected by a 1 GBit link. >>>>>> - All clients doing a dd write of 8GB file from /dev/zero >>>>>> - 3of8 is the special raid-groups arrangement of exofs && objlayout >>>>>> where out of 8 devices each file is striped over 3 devices in a >>>>>> round robin fashion. (*With a small dirty trick) >>> >>> Random stupid questions: >>> >>> - why do you think the 3of8 arrangement is scaling better than >>> the 8of8? >> >> It's a know problem with a network storage cluster. What happens is >> that with 8of8 all the clients exercise all of the nodes at the same >> time so they are clashing on the network. > > OK, so if two clients are both trying to send a stripe of data to the > same OSD data at the same time, absent a switch that could somehow > afford to queue up a full stripe-unit's worth of data, packets get lost? > > (Also, out of curiosity: do you know of any papers or documentation that > describe that problem in more detail?) > A good place to start would be http://www.pdl.cmu.edu/Incast/ Benny > --b. > --