Received: by 2002:a25:c593:0:0:0:0:0 with SMTP id v141csp339068ybe; Wed, 18 Sep 2019 18:28:08 -0700 (PDT) X-Google-Smtp-Source: APXvYqxcUnoIq7FqArCkJ70w4/1eGQsWwEHFSohf1gP3xLr8vMEof2Qfz6RPHhMXzQgnqPJniN9g X-Received: by 2002:a50:a939:: with SMTP id l54mr13451297edc.214.1568856488275; Wed, 18 Sep 2019 18:28:08 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1568856488; cv=none; d=google.com; s=arc-20160816; b=nR5Uck+xpLJKNrp5Tl33mF/+upDRxgYvWZ1+dy5Ek1Pynjtgq7aOPXHkKlW3im8Wra t3AagpgnZ/82QPtSbEy6E9KQNTnw54ssgkd5rppow2cWsI0RBLcfz1bryyafwGEKLLzL aiy/5YB5SMGjQjiX9cKQRYmeyb2qI9bebhT8lylpxzkPYV0ouBpX7tBpCycn+i0Y9QbC lpCznfknypGY2CBN9DNY2esLSk4cOWLH6h3kew83OwgVwf9wGBOzXuBf64QcbByUOQcy B2YkzQWPVRTVhXtu0Cc8xWz/PS+f7NzZNFvcrlWG6oDVUPKJ8L7f/Ub9/rWU5nlTAY+O 7AQg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:mime-version:message-id:references :in-reply-to:subject:date:to:from; bh=r2PRAjo/jZAT1iYEFxSePhZxlG4mtQttbsKBYTbYHb0=; b=ryLiVivR06pEgqJAWYwbhmobrnFTGGVVad+p81W3bsnd1GbiLdlQa0oEfVxjTNWQAZ R6///JZjEeWaCOj8FWzEzf/SejN6v/tdt+7Uxghez4Xtk6Bvfd65KGyHOCf6Yn6Pphxy Gp9l0oHfDe1Bt+92IKJ+0M4EbVI8gMnKv7VoG2HiwlzzT0SrsPZU5SvuxJ5MqO/WfIcU 837q5IkZ4CmszfsiEUAVlycLcTx53oKxvUU4Gujf1nXY6CW2ZFyVADhhsvAhUrVTKedP a8tjnlJPpx081/WqNOxTfHeEp82avC+6PQ/JCQ0EN8j2ux0tUGhNBLQiGGndYnG3ltcb BggQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-nfs-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-nfs-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id i12si4334266edb.133.2019.09.18.18.27.20; Wed, 18 Sep 2019 18:28:08 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-nfs-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-nfs-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-nfs-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731118AbfISBIU (ORCPT + 99 others); Wed, 18 Sep 2019 21:08:20 -0400 Received: from mx2.suse.de ([195.135.220.15]:36520 "EHLO mx1.suse.de" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726359AbfISBIU (ORCPT ); Wed, 18 Sep 2019 21:08:20 -0400 X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay2.suse.de (unknown [195.135.220.254]) by mx1.suse.de (Postfix) with ESMTP id 07114AC31; Thu, 19 Sep 2019 01:08:17 +0000 (UTC) From: NeilBrown To: "J. Bruce Fields" , linux-nfs@vger.kernel.org Date: Thu, 19 Sep 2019 11:08:10 +1000 Subject: Re: [PATCH 2/2] nfsd: give out fewer session slots as limit approaches In-Reply-To: <1506345704-9486-3-git-send-email-bfields@redhat.com> References: <1506345704-9486-1-git-send-email-bfields@redhat.com> <1506345704-9486-3-git-send-email-bfields@redhat.com> Message-ID: <87d0fx9jph.fsf@notabene.neil.brown.name> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org --=-=-= Content-Type: text/plain Content-Transfer-Encoding: quoted-printable On Mon, Sep 25 2017, J. Bruce Fields wrote: > From: "J. Bruce Fields" > > Instead of granting client's full requests until we hit our DRC size > limit and then failing CREATE_SESSIONs (and hence mounts) completely, > start granting clients smaller slot tables as we approach the limit. > > The factor chosen here is pretty much arbitrary. Hi Bruce.... I've been looking at this patch - and the various add-ons that fix it :-( There seems to be another problem though. Prior to this patch, avail would never exceed nfsd_drc_max_mem - nfsd_drc_mem_used since this patch, avail will never be less than slotsize, so it could exceed the above. This means that 'num' will never be less than 1 (i.e. never zero). num * slotsize might exceed nfsd_drc_max_mem - nfsd_drc_mem_used and then nfsd_drc_mem_used would exceed nfsd_drc_max_mem When that happens, the next call to nfsd4_get_drc_mem() will evaluate total_avail =3D nfsd_drc_max_mem - nfsd_drc_mem_used; which will be very large (unsigned long) indeed. Maybe not the intention. I would have sent a patch to fix this, except that it bothers me that nfsd4_get_drc_mem() could ever return 0 (it cannot at the moment, but would after a "fix"). That would result in check_forechannel_attrs() returning nfserr_jukebox, and the client retrying indefinitely (which is exactly the symptom I have reported by a customer with a 4.12 kernel). This isn't nice behaviour. Given that the server makes no attempt to reclaim slot memory for clients, would NFS4ERR_RESOURCE be a better error here? Also, I'd like to suggest that the '1/3' heuristic be change to 1/16. Assuming 30 slots get handed out normally (which my testing shows - about 2k each, with an upper limit of 64k): When 90 slots left, we hand out 30 (now 60 left) 20 (now 40 left) 13 (now 27 left) 9 (now 18 left) 6 (now 12 left) 4 (now 8 left) 2 (now 6 left) 2 (now 4 left) 1 1 1 1 which is a rapid decline as clients are added. With 16, we hand out 30 at a time until 480 slots are left (30Meg) then: 30 28 26 24 23 21 20 19 18 6 15 15 14 13 12 11 10 10 9 9 8 8 7 7 6 6 5 5 5 5 4 4 4 3 3 3 3 3 3 2 2 2 2 2 2 2 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 slots per session Am I convincing? To make it more concrete: this is what I'm thinking of. Which bits do you like? Thanks, NeilBrown diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c index 7857942c5ca6..5d11ceaee998 100644 =2D-- a/fs/nfsd/nfs4state.c +++ b/fs/nfsd/nfs4state.c @@ -1573,11 +1573,15 @@ static u32 nfsd4_get_drc_mem(struct nfsd4_channel_a= ttrs *ca) total_avail =3D nfsd_drc_max_mem - nfsd_drc_mem_used; avail =3D min((unsigned long)NFSD_MAX_MEM_PER_SESSION, total_avail); /* =2D * Never use more than a third of the remaining memory, + * Never use more than a 1/16 of the remaining memory, * unless it's the only way to give this client a slot: */ =2D avail =3D clamp_t(unsigned long, avail, slotsize, total_avail/3); + avail =3D clamp_t(unsigned long, avail, slotsize, total_avail/16); num =3D min_t(int, num, avail / slotsize); + if (nfsd_drc_mem_used + num * slotsize > nfsd_drc_max_mem) + /* Completely out of space - sorry */ + num =3D 0; + nfsd_drc_mem_used +=3D num * slotsize; spin_unlock(&nfsd_drc_lock); =20 @@ -3172,7 +3176,7 @@ static __be32 check_forechannel_attrs(struct nfsd4_ch= annel_attrs *ca, struct nfs */ ca->maxreqs =3D nfsd4_get_drc_mem(ca); if (!ca->maxreqs) =2D return nfserr_jukebox; + return nfserr_resource; =20 return nfs_ok; } --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEG8Yp69OQ2HB7X0l6Oeye3VZigbkFAl2C1PsACgkQOeye3VZi gblz4g//dVzoLD+R92bYp+s28+K1LjFLNcNjUiqYf6IRThpwWkn1Kwmk2eNhg1aJ +r7hndV9mxML2SCNv/U2ARfG7xVdtoryu8uAds8swnUhk6QRBsWYNPjyByw+tyvC z3EIvtJQdDRyHP7YeJtYgYDsWQFO8394kz24M1/HiJJcM/aCOv0ZZ+dW/7uhWwUC CI9YGKaVQlEuHce/L4csQTqrsBbjXVfssnPOqUEjNa8IQD38KIvv/m0n9lz0Zeot It6mWoD+QE6XqZJBUVn2po6Du0v9AEmBXZNZZRJl5bnzYzbbwLEaJacqYzqoMkc5 7v+WOHcwV9vD83ym/bTUUyD+/7y+3LLnF9j6S3KOXfwj017EkYbZR/fvaOXrz6XG D9z23PnbufYBeBqnyk50gmfQ8BZhTTm2V4yHZRM+pMbU9r17RxOq4jFbyhqAhgTC RWKD8dr2JrLRoLxk3fzgKkm/BTpoYJNgii+8qMQ5zAvelcVJ/3Ongpe+hrgmP0hW eWxbY0qC6Kohx2OiKs+P2/7vaFXcQIh+ZZy/cOECaz4dmY6rq6toEmU1KEBdR3C5 KHs/8ijcNYPiBAtO/DdSvv5VAfUCvY8eXVUZqv2JKW78SDVoYGF+Vfq1n4HaHy5i YefqC0R+/B+Qa5HFFJ04wDADpQrcOCb5rKLaLtaStjOp7fZZM7U= =dgaK -----END PGP SIGNATURE----- --=-=-=--