Received: by 2002:a05:6a10:16a7:0:0:0:0 with SMTP id gp39csp3391053pxb; Mon, 16 Nov 2020 13:23:29 -0800 (PST) X-Google-Smtp-Source: ABdhPJycbZ5yz8b/pQN/ri4sc3dk4EijwTNPWBIbyAQU8dCXoQL129IJLufQJVR5LGqShg4zhsCT X-Received: by 2002:a17:906:6414:: with SMTP id d20mr15888993ejm.82.1605561809469; Mon, 16 Nov 2020 13:23:29 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1605561809; cv=none; d=google.com; s=arc-20160816; b=z0gUdPRVn136uAGomMXOjbxIwokSB9+4EtGKhtQxyAkSQOh8TwwLtXZZaezxXilw6n 2CzFWA2/u43Yy+zMFuPxif3nj9s0hl2AzydAurOxlL9/rotrWtGo2ZS8KXbrrtRvIXFP ZGWwH5JwpEJRKijjUTnfWHF+s5LHrrvFKvJZeEgmYVsSmIQ4CdPifO0LmSr8ALgje2Tt eoKjj8KC8i26rxr74gj8H3rP3sZkLTWoDHkGKNbpsch/12cQMuRBFx2ZgEVMl7sIp1LN tUDgmN3t8MV5ccu1U50bhKo2+xiUe6qPmrYeBs502UWf0jCbNhaU3RhaA3yVOipwWAxf KsYQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:thread-index:thread-topic :content-transfer-encoding:mime-version:subject:references :in-reply-to:message-id:cc:to:from:date; bh=V4JeWiMRSNcuIlRLKve4bLWpHQWHChUCjTfVWsR+EjI=; b=ggimL7+RHe/jZzFzq8etOs4VcF5+CAB99kjCXUfcLRqdYKMlNwHrRlRYxlEgRHPzBL IoI4iXCAvkGUhE4bC+Arr+14JNQ3Y0Cy9zXgPxD1QitZ6uA+gjFKKY5or/nKxIzEpUmg DVnIgOCgaBYYx8xbWBsHZp9qxU/gPy/Vrpgoq6ewCDGxaGeJ0fofzOnNiB9Su+y9n95B MLaYnfR3IyCUKakDhzCJVgpewgXygnhPVS+5phRZPdmi2wva0JlfdLLl6U38wvCny+hZ Dp0LD9PCyA30V2kVJpqp0+sNlfn59PIk95ZxrV9zibn/b7YkmPooTJS5jVs1IfsBwj/A vw1w== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-nfs-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id a15si13943248eju.565.2020.11.16.13.23.05; Mon, 16 Nov 2020 13:23:29 -0800 (PST) Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-nfs-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729187AbgKPTVx (ORCPT + 99 others); Mon, 16 Nov 2020 14:21:53 -0500 Received: from natter.dneg.com ([193.203.89.68]:46042 "EHLO natter.dneg.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725379AbgKPTVx (ORCPT ); Mon, 16 Nov 2020 14:21:53 -0500 Received: from localhost (localhost [127.0.0.1]) by natter.dneg.com (Postfix) with ESMTP id 8B5537E12CF8; Mon, 16 Nov 2020 19:21:51 +0000 (GMT) X-Virus-Scanned: amavisd-new at mx-dneg Received: from natter.dneg.com ([127.0.0.1]) by localhost (natter.dneg.com [127.0.0.1]) (amavisd-new, port 10024) with LMTP id sLdL-rdGJHSw; Mon, 16 Nov 2020 19:21:51 +0000 (GMT) Received: from zrozimbrai.dneg.com (zrozimbrai.dneg.com [10.11.20.12]) by natter.dneg.com (Postfix) with ESMTPS id 6FD6E7E12CE7; Mon, 16 Nov 2020 19:21:51 +0000 (GMT) Received: from localhost (localhost [127.0.0.1]) by zrozimbrai.dneg.com (Postfix) with ESMTP id 60D158150D4F; Mon, 16 Nov 2020 19:21:51 +0000 (GMT) Received: from zrozimbrai.dneg.com ([127.0.0.1]) by localhost (zrozimbrai.dneg.com [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id Qn-EqBm74RJx; Mon, 16 Nov 2020 19:21:51 +0000 (GMT) Received: from localhost (localhost [127.0.0.1]) by zrozimbrai.dneg.com (Postfix) with ESMTP id 469B88150D5F; Mon, 16 Nov 2020 19:21:51 +0000 (GMT) X-Virus-Scanned: amavisd-new at zimbra-dneg Received: from zrozimbrai.dneg.com ([127.0.0.1]) by localhost (zrozimbrai.dneg.com [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id X_ohQm0wFXkB; Mon, 16 Nov 2020 19:21:51 +0000 (GMT) Received: from zrozimbra1.dneg.com (zrozimbra1.dneg.com [10.11.16.16]) by zrozimbrai.dneg.com (Postfix) with ESMTP id EF4F2814E1B7; Mon, 16 Nov 2020 19:21:47 +0000 (GMT) Date: Mon, 16 Nov 2020 19:21:47 +0000 (GMT) From: Daire Byrne To: bfields Cc: Trond Myklebust , linux-cachefs , linux-nfs Message-ID: <1688437957.87985749.1605554507783.JavaMail.zimbra@dneg.com> In-Reply-To: <20201116155329.GE898@fieldses.org> References: <943482310.31162206.1599499860595.JavaMail.zimbra@dneg.com> <444227972.86442677.1605206025305.JavaMail.zimbra@dneg.com> <20201112205524.GI9243@fieldses.org> <883314904.86570901.1605222357023.JavaMail.zimbra@dneg.com> <20201113145050.GB1299@fieldses.org> <20201113222600.GC1299@fieldses.org> <217712894.87456370.1605358643862.JavaMail.zimbra@dneg.com> <20201116155329.GE898@fieldses.org> Subject: Re: Adventures in NFS re-exporting MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Mailer: Zimbra 8.7.11_GA_1854 (ZimbraWebClient - GC78 (Linux)/8.7.11_GA_1854) Thread-Topic: Adventures in NFS re-exporting Thread-Index: 8LqwOtJALJh7qY93Jll76WzyFc0fTw== Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org ----- On 16 Nov, 2020, at 15:53, bfields bfields@fieldses.org wrote: > On Sat, Nov 14, 2020 at 12:57:24PM +0000, Daire Byrne wrote: >> Now if anyone has any ideas why all the read calls to the originating >> server are limited to a maximum of 128k (with rsize=1M) when coming >> via the re-export server's nfsd threads, I see that as the next >> biggest performance issue. Reading directly on the re-export server >> with a userspace process issues 1MB reads as expected. It doesn't >> happen for writes (wsize=1MB all the way through) but I'm not sure if >> that has more to do with async and write back caching helping to build >> up the size before commit? > > I'm not sure where to start with this one.... > > Is this behavior independent of protocol version and backend server? It seems to the case for all combinations of backend versions and re-export versions. But it does look like it is related to readahead somehow. The default for a client mount is 128k .... I just increased it to 1024 on the client mount of the originating server on the re-export server and now it's doing the expected 1MB (rsize) read requests back to onprem from the clients all the way through. i.e. echo 1024 > /sys/class/bdi/0:52/read_ahead_kb So, there is a difference in behaviour when reading from the client mount with user space processes or the knfsd threads on the re-export server. Daire