Hi,
When I doing the follow test in kernel-5.7-rc2, I found mem-free is
decreased
#!/bin/sh
cd /sys/fs/cgroup/memory/
for((i=0;i<45;i++))
do
for((j=0;j<60000;j++))
do
mkdir /sys/fs/cgroup/memory/yyl-cg$j
done
sleep 1
ls /sys/fs/cgroup/memory/ | grep yyl | xargs rmdir
done
before test the /proc/meminfo is:
MemTotal: 493554824 kB
MemFree: 491240912 kB
MemAvailable: 489424520 kB
Buffers: 4112 kB
Cached: 65400 kB
SwapCached: 0 kB
Active: 156016 kB
Inactive: 37720 kB
Active(anon): 128372 kB
Inactive(anon): 7188 kB
Active(file): 27644 kB
Inactive(file): 30532 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 4194300 kB
SwapFree: 4194300 kB
Dirty: 112 kB
Writeback: 0 kB
AnonPages: 124356 kB
Mapped: 53724 kB
Shmem: 11036 kB
KReclaimable: 93488 kB
Slab: 599660 kB
SReclaimable: 93488 kB
SUnreclaim: 506172 kB
KernelStack: 23008 kB
PageTables: 4340 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 250971712 kB
Committed_AS: 1834448 kB
VmallocTotal: 135290159040 kB
VmallocUsed: 229284 kB
VmallocChunk: 0 kB
Percpu: 80896 kB
HardwareCorrupted: 0 kB
AnonHugePages: 43008 kB
ShmemHugePages: 0 kB
ShmemPmdMapped: 0 kB
FileHugePages: 0 kB
FilePmdMapped: 0 kB
CmaTotal: 65536 kB
CmaFree: 40480 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
Hugetlb: 0 kB
after test:
MemTotal: 493554824 kB
MemFree: 484492920 kB
MemAvailable: 482801124 kB
Buffers: 21984 kB
Cached: 151380 kB
SwapCached: 0 kB
Active: 230000 kB
Inactive: 68068 kB
Active(anon): 130108 kB
Inactive(anon): 13804 kB
Active(file): 99892 kB
Inactive(file): 54264 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 4194300 kB
SwapFree: 4194300 kB
Dirty: 36 kB
Writeback: 0 kB
AnonPages: 125080 kB
Mapped: 55520 kB
Shmem: 19220 kB
KReclaimable: 246696 kB
Slab: 5381572 kB
SReclaimable: 246696 kB
SUnreclaim: 5134876 kB
KernelStack: 27360 kB
PageTables: 4172 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 250971712 kB
Committed_AS: 1588600 kB
VmallocTotal: 135290159040 kB
VmallocUsed: 230836 kB
VmallocChunk: 0 kB
Percpu: 1827840 kB
HardwareCorrupted: 0 kB
AnonHugePages: 43008 kB
ShmemHugePages: 0 kB
ShmemPmdMapped: 0 kB
FileHugePages: 0 kB
FilePmdMapped: 0 kB
CmaTotal: 65536 kB
CmaFree: 40480 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
Hugetlb: 0 kB
after echo 3 > /proc/sys/vm/drop_caches
MemTotal: 493554824 kB
MemFree: 485104048 kB
MemAvailable: 483358392 kB
Buffers: 6168 kB
Cached: 79904 kB
SwapCached: 0 kB
Active: 165348 kB
Inactive: 45780 kB
Active(anon): 130528 kB
Inactive(anon): 13800 kB
Active(file): 34820 kB
Inactive(file): 31980 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 4194300 kB
SwapFree: 4194300 kB
Dirty: 8 kB
Writeback: 0 kB
AnonPages: 125236 kB
Mapped: 55516 kB
Shmem: 19220 kB
KReclaimable: 226332 kB
Slab: 5353952 kB
SReclaimable: 226332 kB
SUnreclaim: 5127620 kB
KernelStack: 23040 kB
PageTables: 4212 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 250971712 kB
Committed_AS: 1672424 kB
VmallocTotal: 135290159040 kB
VmallocUsed: 230436 kB
VmallocChunk: 0 kB
Percpu: 1379840 kB
HardwareCorrupted: 0 kB
AnonHugePages: 43008 kB
ShmemHugePages: 0 kB
ShmemPmdMapped: 0 kB
FileHugePages: 0 kB
FilePmdMapped: 0 kB
CmaTotal: 65536 kB
CmaFree: 40480 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
Hugetlb: 0 kB
after test and drop caches, the /proc/cgroups is:
#subsys_name hierarchy num_cgroups enabled
cpuset 11 1 1
cpu 2 1 1
cpuacct 2 1 1
blkio 8 1 1
memory 5 83 1
devices 3 41 1
freezer 6 1 1
net_cls 9 1 1
perf_event 10 1 1
net_prio 9 1 1
hugetlb 4 1 1
pids 7 51 1
rdma 12 1 1
All the dir that created by the script is already removed, but I got:
- MemFree is decreased about 6.7G
- SUnreclaim is increased about 4.6G
- Percpu is increased about 1.7G
It seems we have memory leak in cgroup ?
+cc Roman who has been looking the most at this area
On Mon, Apr 27, 2020 at 03:48:13PM +0800, Yang Yingliang wrote:
> +cc [email protected] <mailto:[email protected]>
>
> On 2020/4/26 19:21, Yang Yingliang wrote:
> > Hi,
> >
> > When I doing the follow test in kernel-5.7-rc2, I found mem-free is
> > decreased
> >
> > #!/bin/sh
> > cd /sys/fs/cgroup/memory/
> >
> > for((i=0;i<45;i++))
> > do
> > ??????? for((j=0;j<60000;j++))
> > ??????? do
> > ??????????????? mkdir /sys/fs/cgroup/memory/yyl-cg$j
> > ??????? done
> > ??????? sleep 1
> > ??????? ls /sys/fs/cgroup/memory/ | grep yyl | xargs rmdir
> > done
Should be easy enough to reproduce, thanks for the report. I'll try to
take a look later this week, unless Roman beats me to it.
Is this a new observation in 5.7-rc2?
Can you provide /sys/fs/cgroup/unified/cgroup.stat after the test?
On Mon, Apr 27, 2020 at 01:13:04PM -0400, Johannes Weiner wrote:
> +cc Roman who has been looking the most at this area
>
> On Mon, Apr 27, 2020 at 03:48:13PM +0800, Yang Yingliang wrote:
> > +cc [email protected] <mailto:[email protected]>
> >
> > On 2020/4/26 19:21, Yang Yingliang wrote:
> > > Hi,
> > >
> > > When I doing the follow test in kernel-5.7-rc2, I found mem-free is
> > > decreased
> > >
> > > #!/bin/sh
> > > cd /sys/fs/cgroup/memory/
> > >
> > > for((i=0;i<45;i++))
> > > do
> > > ??????? for((j=0;j<60000;j++))
> > > ??????? do
> > > ??????????????? mkdir /sys/fs/cgroup/memory/yyl-cg$j
> > > ??????? done
> > > ??????? sleep 1
> > > ??????? ls /sys/fs/cgroup/memory/ | grep yyl | xargs rmdir
> > > done
>
> Should be easy enough to reproduce, thanks for the report. I'll try to
> take a look later this week, unless Roman beats me to it.
>
> Is this a new observation in 5.7-rc2?
>
> Can you provide /sys/fs/cgroup/unified/cgroup.stat after the test?
I'm actually trying to reproduce it now, but so far haven't found any issues.
Yang, can you, please, attach the config you're using?
And also confirm that you're giving the system some time before looking
at the memory statistics? Reclaim of internal cgroup structure is a complex
process which might take some time to finish.
Is dmesg also clean?
Thanks!
Hi,
On 2020/4/28 1:24, Roman Gushchin wrote:
> On Mon, Apr 27, 2020 at 01:13:04PM -0400, Johannes Weiner wrote:
>> +cc Roman who has been looking the most at this area
>>
>> On Mon, Apr 27, 2020 at 03:48:13PM +0800, Yang Yingliang wrote:
>>> +cc [email protected] <mailto:[email protected]>
>>>
>>> On 2020/4/26 19:21, Yang Yingliang wrote:
>>>> Hi,
>>>>
>>>> When I doing the follow test in kernel-5.7-rc2, I found mem-free is
>>>> decreased
>>>>
>>>> #!/bin/sh
>>>> cd /sys/fs/cgroup/memory/
>>>>
>>>> for((i=0;i<45;i++))
>>>> do
>>>> for((j=0;j<60000;j++))
>>>> do
>>>> mkdir /sys/fs/cgroup/memory/yyl-cg$j
>>>> done
>>>> sleep 1
>>>> ls /sys/fs/cgroup/memory/ | grep yyl | xargs rmdir
>>>> done
>> Should be easy enough to reproduce, thanks for the report. I'll try to
>> take a look later this week, unless Roman beats me to it.
>>
>> Is this a new observation in 5.7-rc2?
>>
>> Can you provide /sys/fs/cgroup/unified/cgroup.stat after the test?
I re-tested in 5.7-rc3, it has same problem and the
/sys/fs/cgroup/unified/cgroup.stat afther test is:
nr_descendants 50
nr_dying_descendants 0
> I'm actually trying to reproduce it now, but so far haven't found any issues.
>
> Yang, can you, please, attach the config you're using?
>
> And also confirm that you're giving the system some time before looking
> at the memory statistics? Reclaim of internal cgroup structure is a complex
> process which might take some time to finish.
>
> Is dmesg also clean?
config and dmesg are attached.
>
> Thanks!
On Tue, Apr 28, 2020 at 05:10:33PM +0800, Yang Yingliang wrote:
> Hi,
>
> On 2020/4/28 1:24, Roman Gushchin wrote:
> > On Mon, Apr 27, 2020 at 01:13:04PM -0400, Johannes Weiner wrote:
> > > +cc Roman who has been looking the most at this area
> > >
> > > On Mon, Apr 27, 2020 at 03:48:13PM +0800, Yang Yingliang wrote:
> > > > +cc [email protected] <mailto:[email protected]>
> > > >
> > > > On 2020/4/26 19:21, Yang Yingliang wrote:
> > > > > Hi,
> > > > >
> > > > > When I doing the follow test in kernel-5.7-rc2, I found mem-free is
> > > > > decreased
> > > > >
> > > > > #!/bin/sh
> > > > > cd /sys/fs/cgroup/memory/
> > > > >
> > > > > for((i=0;i<45;i++))
> > > > > do
> > > > > ??????? for((j=0;j<60000;j++))
> > > > > ??????? do
> > > > > ??????????????? mkdir /sys/fs/cgroup/memory/yyl-cg$j
> > > > > ??????? done
> > > > > ??????? sleep 1
> > > > > ??????? ls /sys/fs/cgroup/memory/ | grep yyl | xargs rmdir
> > > > > done
> > > Should be easy enough to reproduce, thanks for the report. I'll try to
> > > take a look later this week, unless Roman beats me to it.
> > >
> > > Is this a new observation in 5.7-rc2?
> > >
> > > Can you provide /sys/fs/cgroup/unified/cgroup.stat after the test?
> I re-tested in 5.7-rc3, it has same problem and the
> /sys/fs/cgroup/unified/cgroup.stat afther test is:
>
> nr_descendants 50
> nr_dying_descendants 0
>
> > I'm actually trying to reproduce it now, but so far haven't found any issues.
> >
> > Yang, can you, please, attach the config you're using?
> >
> > And also confirm that you're giving the system some time before looking
> > at the memory statistics? Reclaim of internal cgroup structure is a complex
> > process which might take some time to finish.
> >
> > Is dmesg also clean?
> config and dmesg are attached.
Interesting...
I've tried hard to reproduce, but haven't managed to get anything so far.
You've a huge machine with a non-trivial hardware setup (e.g. those cpuset warnings
in dmesg), so there must be something special about it. Could be anything:
percpu, rcu, some incorrectly handled ENOMEM case.
I've several questions, which might help the investigation:
1) Is there any known good revision, which doesn't leak?
2) Can you, please, check that all those cgroups were actually created? I mean
if some mkdir calls returned an error, it could be a huge hint for us
where to look.
3) Can you, please, dump /proc/slabinfo before and after the experiment?
4) Can you, please, repeat the experiment creating cgroups in /sys/fs/cgroup/unified
instead of /sys/fs/cgroup/memory ?
5) If you're familiar with bcc tools (https://github.com/iovisor/bcc), can you, please,
run memleak.py during the experiment?
Thank you!