Received-SPF: pass (google.com: domain of linux-nfs+bounces-2569-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1;
Date: Sat, 30 Mar 2024 12:26:35 -0400
From: Chuck Lever <chuck.lever@oracle.com>
To: Jan Schunk <scpcom@gmx.de>
Cc: Benjamin Coddington <bcodding@redhat.com>,
        Jeff Layton <jlayton@kernel.org>, Neil Brown <neilb@suse.de>,
        Olga Kornievskaia <kolga@netapp.com>, Dai Ngo <dai.ngo@oracle.com>,
        Tom Talpey <tom@talpey.com>,
        Linux NFS Mailing List <linux-nfs@vger.kernel.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        David Howells <dhowells@redhat.com>
Subject: Re: Re: [External] : nfsd: memory leak when client does many file
 operations
Message-ID: <Zgg9OzeFZUTc4hck@tissot.1015granger.net>
References: <567BBF54-D104-432C-99C0-1A7EE7939090@oracle.com>
 <trinity-66047013-4d84-4eef-b5d3-d710fe6be805-1711316386382@msvc-mesg-gmx005>
 <6F16BCCE-3000-4BCB-A3B4-95B4767E3577@oracle.com>
 <trinity-ad0037c0-1060-4541-a8ca-15f826a5b5a2-1711396545958@msvc-mesg-gmx024>
 <088D9CC3-C5B0-4646-A85D-B3B9ACE8C532@oracle.com>
 <trinity-77720d9d-4d5b-48c6-8b1f-0b7205ea3c2b-1711398394712@msvc-mesg-gmx021>
 <51CAACAB-B6CC-4139-A350-25CF067364D3@oracle.com>
 <trinity-db344068-bb4b-4d0b-9772-ff701a2c70dd-1711663407957@msvc-mesg-gmx026>
 <C14AC427-BD99-4A87-A311-F6FB87FFC134@oracle.com>
 <trinity-157de7e0-d394-47fa-bb44-2621045a5b6e-1711812369391@msvc-mesg-gmx004>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <trinity-157de7e0-d394-47fa-bb44-2621045a5b6e-1711812369391@msvc-mesg-gmx004>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: 
	=?us-ascii?Q?06upQWKLBnDuQbOc9bgoeFn3ZtOSUP8Og9pCbaEWyvUqibPaRJQJJ5NLd8I6?=
 =?us-ascii?Q?Ekk+Mf0EDUBWk9VU+f1lJ7if/iYAg6zXmhYWKY8b6eO+oSuy6paCk4Cbogz2?=
 =?us-ascii?Q?8iPvDlSlnHlC1E48I5iKjs2SlgShl2dovU/oOOCU4nrPJY5I9ROPdidTF1Px?=
 =?us-ascii?Q?7aPJJ+o46+tu07z71jGTM3IF6uyE08h/s8ASbL88G0N7jOIhmWWA438hahIx?=
 =?us-ascii?Q?FjL5rX3pJlEpOPKVaJHu+og3z1Acl9hbENzEL88HMP4fSszdxKZC32TtJK2T?=
 =?us-ascii?Q?RbAh1bLQD38YRtMM7NBo657Ln6e6QrcG6cgE9S/qqinBUXJvLFXHRxRgC3+1?=
 =?us-ascii?Q?WpQwuhP3r+RpLhOTbDqG9fEfwgnZ10xfl4ghG4qNIby9imjN5JN2FNxAyASB?=
 =?us-ascii?Q?S8TAD9uuIVCMAaVOqa3SvTDLKp3C+CNXq1orODcVXao2XMWjg/+Ly1XlkuqM?=
 =?us-ascii?Q?7WKL4nt4XLeExgrJ8r/FXd5Rl1IjIwoiWigUPKiNgclyCb/Fgl58VjQHPi9b?=
 =?us-ascii?Q?zDGqjJaQLTMjO5XoNaNKUmu3QqEhFGR/Jrw1zyzmii/EQoaaOcjRyEwWmrSM?=
 =?us-ascii?Q?Y96Bg7opoYhi9oaap7gPJlqQs9FlHrEhpegYWzylSWyTesUoADLQ6rBjmIm8?=
 =?us-ascii?Q?dZZZX5JVFZLuV7XxezQLpKA8/n6o1h6H09DKA/Mls+5YzdSei2f0Gu3tgxox?=
 =?us-ascii?Q?jE9fCPQ3ynwfam2ElbFI6P/NRLo4zHyqLe0KCx7Mxh26g5EcFvuaszYZGqfR?=
 =?us-ascii?Q?z6MuBidu7cmO+2Xj5uRkVhixmi5ALtCW7UHL/fzVo3eP+t00tin5bPZBwD1j?=
 =?us-ascii?Q?/ya4g/L523I6kaENOkZ5zL6X2Uatl7Wr+anxl2pdvl/8jnbQLN9ws83PKRNU?=
 =?us-ascii?Q?TlLGDsYZ339HkmrKIGzZAErANxtjrFBD7Jsz3pcuZME64WztR6mAwkDPahpc?=
 =?us-ascii?Q?xBULRm9eJyAVVEyWNRLMTrUPKD/1F4CbMtImCXcOa3Z+xAyI8bTE+ZUAPnot?=
 =?us-ascii?Q?HblPYXmYvz+hlTRh+ohDc1NC4PGcxmFA6+1dXoOPfvfNVLdiCz7BOQWkTVdP?=
 =?us-ascii?Q?uN2S83d4TUvBgb7VU49vT7VH6pDA/adI9MI4GC/b+87Fy14fDnJk+v5WZreJ?=
 =?us-ascii?Q?sqorzMLAI5J5HSqwZTso5WmOWtA1E8ZyivKADJf2DrN+iQ2TpjnzrRfaVR4k?=
 =?us-ascii?Q?rU1n1YFm/Cfps3iAVxe6W8dDoqDgO+bkgx1630uAOj9jBAKNneNqV3b2kVE9?=
 =?us-ascii?Q?JWalarPcun03AqnNwd4tBhsIjk2P9BDChFvSxqk+oc/Hz6trMiv7wnnHuD5F?=
 =?us-ascii?Q?YmUxfDb75mDXU8bRqUjnxf2BC/yfpozYE26TK4CoyXoeFElujREBcvM9mpqs?=
 =?us-ascii?Q?ZHo6Dlk/dsVzrVbscIAkKEJ/RORlpKq6rFwnbsaGXm3DrBqsJ++otBbIu3qO?=
 =?us-ascii?Q?SFv47PaQt7YXeOmIZvCsNcJ1QzhY/fFKx+9YoiOLlZeRAqRMLlFhb+9D+4Bf?=
 =?us-ascii?Q?8LYG1XWtpl0DftyWQZMUx9j4+mxzMz3g0pPFASYs6Vq112emmeEcPhbEMfc/?=
 =?us-ascii?Q?+45mm55VIY9xAluizg4oCux1uBaA3GVGmwYazrvSOaDk1ddSLFjW7EaUxOmT?=
 =?us-ascii?Q?Lg=3D=3D?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 
	7r6QwW7/bTS+SiFWaosWErb2xIYuMe1XTxRgHzppiAOzlz73dEafzCnzAg3HinKN/N9QeOzvvv34rhDfsK/HPvun2EWV2e5RkKsqKzY/39et9IrclCO8o2Ar2TBEnkkBdusAemkBJU9VOaoGg33F0QKCtN3BsutpPn4Sl2ngLxBR0PM4/AcAKlTVfGFCBDqSKfUJ/uDfam7IEuINATczZ2ivmgVhkZ2cppCtvWpIst4PiFNp9g/0BDY5shTJdZ65OZGhWJY2t2WxGK0Ieq4NzDX3q95FbCIv36dYwG197IMUeShj6n4yW94mXCCY51jkjzvfKYGw/XCemUBem1Q15NuSvQt43aTlnJ+fUgacsI+Zog8XzuvYWc48pGMMZEyjrnfyLU+R+X5O10J1upVxTDkNn108QypOrC8ldP5pjbKy0CkkRec0eI4G0T8tub+3j3BADDsorjaHtQZ9hAMunsLEjRunJZaqbKXxywL+YQsy2cL2h3b6MrJnRSo9K6llT30S2AHuFCbWIOgOi+GZDfD5+FPMeKOwn4/gBfGAOs5LGziUxbS15Km3ObfKy/pDrAoNsj0ivF0oiNS4WWELTEMZrQskhPPPfGmnby44X+g=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 9ef65959-5167-457d-34e8-08dc50d62cc3
X-MS-Exchange-CrossTenant-AuthSource: BN0PR10MB5128.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 Mar 2024 16:26:38.6246
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: udFpAWtnjNc23OQN9DnRyK+dR/OdL/bYGo2cblkrNBautecixsCl2oO4srZDp3as4fHl/TkVzDPADQPZuY+ICw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR10MB4164
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.272,Aquarius:18.0.1011,Hydra:6.0.619,FMLib:17.11.176.26
 definitions=2024-03-30_11,2024-03-28_01,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 malwarescore=0
 mlxlogscore=999 adultscore=0 bulkscore=0 phishscore=0 mlxscore=0
 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2403210000 definitions=main-2403300134
X-Proofpoint-ORIG-GUID: nk0UnUne2m7iGUfGtIu1WZM9-bdUeBhX
X-Proofpoint-GUID: nk0UnUne2m7iGUfGtIu1WZM9-bdUeBhX

On Sat, Mar 30, 2024 at 04:26:09PM +0100, Jan Schunk wrote:
> Full test result:
> 
> $ git bisect start v6.6 v6.5
> Bisecting: 7882 revisions left to test after this (roughly 13 steps)
> [a1c19328a160c80251868dbd80066dce23d07995] Merge tag 'soc-arm-6.6' of git://git.kernel.org/pub/scm/linux/kernel/git/soc/soc
> --
> $ git bisect good
> Bisecting: 3935 revisions left to test after this (roughly 12 steps)
> [e4f1b8202fb59c56a3de7642d50326923670513f] Merge tag 'for_linus' of git://git.kernel.org/pub/scm/linux/kernel/git/mst/vhost
> --
> $ git bisect bad
> Bisecting: 2014 revisions left to test after this (roughly 11 steps)
> [e0152e7481c6c63764d6ea8ee41af5cf9dfac5e9] Merge tag 'riscv-for-linus-6.6-mw1' of git://git.kernel.org/pub/scm/linux/kernel/git/riscv/linux
> --
> $ git bisect bad
> Bisecting: 975 revisions left to test after this (roughly 10 steps)
> [4a3b1007eeb26b2bb7ae4d734cc8577463325165] Merge tag 'pinctrl-v6.6-1' of git://git.kernel.org/pub/scm/linux/kernel/git/linusw/linux-pinctrl
> --
> $ git bisect good
> Bisecting: 476 revisions left to test after this (roughly 9 steps)
> [4debf77169ee459c46ec70e13dc503bc25efd7d2] Merge tag 'for-linus-iommufd' of git://git.kernel.org/pub/scm/linux/kernel/git/jgg/iommufd
> --
> $ git bisect good
> Bisecting: 237 revisions left to test after this (roughly 8 steps)
> [e7e9423db459423d3dcb367217553ad9ededadc9] Merge tag 'v6.6-vfs.super.fixes.2' of git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs
> --
> $ git bisect good
> Bisecting: 141 revisions left to test after this (roughly 7 steps)
> [8ae5d298ef2005da5454fc1680f983e85d3e1622] Merge tag '6.6-rc-ksmbd-fixes-part1' of git://git.samba.org/ksmbd
> --
> $ git bisect good
> Bisecting: 61 revisions left to test after this (roughly 6 steps)
> [99d99825fc075fd24b60cc9cf0fb1e20b9c16b0f] Merge tag 'nfs-for-6.6-1' of git://git.linux-nfs.org/projects/anna/linux-nfs
> --
> $ git bisect bad
> Bisecting: 39 revisions left to test after this (roughly 5 steps)
> [7b719e2bf342a59e88b2b6215b98ca4cf824bc58] SUNRPC: change svc_recv() to return void.
> --
> $ git bisect bad
> Bisecting: 19 revisions left to test after this (roughly 4 steps)
> [e7421ce71437ec8e4d69cc6bdf35b6853adc5050] NFSD: Rename struct svc_cacherep
> --
> $ git bisect good
> Bisecting: 9 revisions left to test after this (roughly 3 steps)
> [baabf59c24145612e4a975f459a5024389f13f5d] SUNRPC: Convert svc_udp_sendto() to use the per-socket bio_vec array
> --
> $ git bisect bad
> Bisecting: 4 revisions left to test after this (roughly 2 steps)
> [be2be5f7f4436442d8f6bffbb97a6f438df2896b] lockd: nlm_blocked list race fixes
> --
> $ git bisect good
> Bisecting: 2 revisions left to test after this (roughly 1 step)
> [d424797032c6e24b44037e6c7a2d32fd958300f0] nfsd: inherit required unset default acls from effective set
> --
> $ git bisect good
> Bisecting: 0 revisions left to test after this (roughly 1 step)
> [e18e157bb5c8c1cd8a9ba25acfdcf4f3035836f4] SUNRPC: Send RPC message on TCP with a single sock_sendmsg() call
> --
> $ git bisect bad
> Bisecting: 0 revisions left to test after this (roughly 0 steps)
> [2eb2b93581813b74c7174961126f6ec38eadb5a7] SUNRPC: Convert svc_tcp_sendmsg to use bio_vecs directly
> --
> $ git bisect good
> e18e157bb5c8c1cd8a9ba25acfdcf4f3035836f4 is the first bad commit
> commit e18e157bb5c8c1cd8a9ba25acfdcf4f3035836f4

This is a plausible bisect result for this behavior, so nice work.

David (cc'd), can you have a brief look at this? What did we miss?
I'm guessing it's a page reference count issue that might occur
only when the XDR head and tail buffers are in the same page. Or
it might occur if two entries in the XDR page array point to the
same page...?

/me stabs in the darkness


> I found the memory loss inside /proc/meminfo only on MemAvailable
>  MemTotal:         346948 kB
> On a bad test run in looks like this:
> -MemAvailable:     210820 kB
> +MemAvailable:      26608 kB
> On a good test run it looks like this:
> -MemAvailable:     215872 kB
> +MemAvailable:     221128 kB