Received-SPF: pass (google.com: domain of linux-nfs+bounces-2475-linux.lists.archive=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99;
From: Benjamin Coddington <bcodding@redhat.com>
To: Chuck Lever III <chuck.lever@oracle.com>
Cc: Jan Schunk <scpcom@gmx.de>, Jeff Layton <jlayton@kernel.org>,
 Neil Brown <neilb@suse.de>, Olga Kornievskaia <kolga@netapp.com>,
 Dai Ngo <dai.ngo@oracle.com>, Tom Talpey <tom@talpey.com>,
 Linux NFS Mailing List <linux-nfs@vger.kernel.org>,
 linux-kernel@vger.kernel.org
Subject: Re: [External] : nfsd: memory leak when client does many file
 operations
Date: Tue, 26 Mar 2024 07:15:16 -0400
Message-ID: <F594EBB2-5F92-40A9-86FB-CFD58E9CE516@redhat.com>
In-Reply-To: <088D9CC3-C5B0-4646-A85D-B3B9ACE8C532@oracle.com>
References: <trinity-068f55c9-6088-418d-bf3a-c2778a871e98-1711310237802@msvc-mesg-gmx120>
 <E3109CAA-8261-4F66-9D1B-3546E8B797DF@oracle.com>
 <trinity-bfafb9db-d64f-4cad-8cb1-40dac0a2c600-1711313314331@msvc-mesg-gmx105>
 <567BBF54-D104-432C-99C0-1A7EE7939090@oracle.com>
 <trinity-66047013-4d84-4eef-b5d3-d710fe6be805-1711316386382@msvc-mesg-gmx005>
 <6F16BCCE-3000-4BCB-A3B4-95B4767E3577@oracle.com>
 <trinity-ad0037c0-1060-4541-a8ca-15f826a5b5a2-1711396545958@msvc-mesg-gmx024>
 <088D9CC3-C5B0-4646-A85D-B3B9ACE8C532@oracle.com>
Precedence: bulk
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

On 25 Mar 2024, at 16:11, Chuck Lever III wrote:

>> On Mar 25, 2024, at 3:55=E2=80=AFPM, Jan Schunk <scpcom@gmx.de> wrote:=

>>
>> The VM is now running 20 hours with 512MB RAM, no desktop, without the=
 "noatime" mount option and without the "async" export option.
>>
>> Currently there is no issue, but the memory usage is still contantly g=
rowing. It may just take longer before something happens.
>>
>> top - 00:49:49 up 3 min,  1 user,  load average: 0,21, 0,19, 0,09
>> Tasks: 111 total,   1 running, 110 sleeping,   0 stopped,   0 zombie
>> %CPU(s):  0,2 us,  0,3 sy,  0,0 ni, 99,5 id,  0,0 wa,  0,0 hi,  0,0 si=
,  0,0 st
>> MiB Spch:    467,0 total,    302,3 free,     89,3 used,     88,1 buff/=
cache
>> MiB Swap:    975,0 total,    975,0 free,      0,0 used.    377,7 avail=
 Spch
>>
>> top - 15:05:39 up 14:19,  1 user,  load average: 1,87, 1,72, 1,65
>> Tasks: 104 total,   1 running, 103 sleeping,   0 stopped,   0 zombie
>> %CPU(s):  0,2 us,  4,9 sy,  0,0 ni, 53,3 id, 39,0 wa,  0,0 hi,  2,6 si=
,  0,0 st
>> MiB Spch:    467,0 total,     21,2 free,    147,1 used,    310,9 buff/=
cache
>> MiB Swap:    975,0 total,    952,9 free,     22,1 used.    319,9 avail=
 Spch
>>
>> top - 20:48:16 up 20:01,  1 user,  load average: 5,02, 2,72, 2,08
>> Tasks: 104 total,   5 running,  99 sleeping,   0 stopped,   0 zombie
>> %CPU(s):  0,2 us, 46,4 sy,  0,0 ni, 11,9 id,  2,3 wa,  0,0 hi, 39,2 si=
,  0,0 st
>> MiB Spch:    467,0 total,     16,9 free,    190,8 used,    271,6 buff/=
cache
>> MiB Swap:    975,0 total,    952,9 free,     22,1 used.    276,2 avail=
 Spch
>
> I don't see anything in your original memory dump that
> might account for this. But I'm at a loss because I'm
> a kernel developer, not a support guy -- I don't have
> any tools or expertise that can troubleshoot a system
> without rebuilding a kernel with instrumentation. My
> first instinct is to tell you to bisect between v6.3
> and v6.4, or at least enable kmemleak, but I'm guessing
> you don't build your own kernels.
>
> My only recourse at this point would be to try to
> reproduce it myself, but unfortunately I've just
> upgraded my whole lab to Fedora 39, and there's a grub
> bug that prevents booting any custom-built kernel
> on my hardware.
>
> So I'm stuck until I can nail that down. Anyone else
> care to help out?

Sure - I can throw some stuff..

Can we dig into which memory slabs might be growing?  Something like:

watch -d "cat /proc/slabinfo | grep nfsd"

=2E. for a bit might show what is growing.

Then use a systemtap script like the one below to trace the allocations -=
 use:

stap -v --all-modules kmem_alloc.stp <slab_name>

Ben


8<---------------------------- save as kmem_alloc.stp -------------------=
---------

# This script displays the number of given slab allocations and the backt=
races leading up to it.

global slab =3D @1
global stats, stacks
probe kernel.function("kmem_cache_alloc") {
        if (kernel_string($s->name) =3D=3D slab) {
                stats[execname()] <<< 1
                stacks[execname(),kernel_string($s->name),backtrace()] <<=
< 1
        }
}
# Exit after 10 seconds
# probe timer.ms(10000) { exit () }
probe end {
        printf("Number of %s slab allocations by process\n", slab)
        foreach ([exec] in stats) {
                printf("%s:\t%d\n",exec,@count(stats[exec]))
        }
        printf("\nBacktrace of processes when allocating\n")
        foreach ([proc,cache,bt] in stacks) {
                printf("Exec: %s Name: %s  Count: %d\n",proc,cache,@count=
(stacks[proc,cache,bt]))
                print_stack(bt)
                printf("\n-----------------------------------------------=
--------\n\n")
        }
}