Received: by 2002:a25:1104:0:0:0:0:0 with SMTP id 4csp231810ybr; Fri, 22 May 2020 05:24:07 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxwVNLylHFffcAq8lkEDIufz/v4bi7hFpAKFuzwlj2E9WaWJVf6/tZcEX8W35DU9kDwtfNx X-Received: by 2002:a17:906:6a18:: with SMTP id o24mr7922139ejr.66.1590150247288; Fri, 22 May 2020 05:24:07 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1590150247; cv=none; d=google.com; s=arc-20160816; b=tXQL2kzejRQbZWItrk8FhIyJMplHusOUbUVDAp87cRLX3fZuV4AUOiRPgUlMabdPGH pgxIKenAskLeMjHxoFGAdfIRJyHNwSGTu+C7iYTAdk5MND6RO7v27Bp3oMdKFcF2BVH0 Osi4PREhqW1YB6J3Nn6hB6PEDsyQzet1pYoBrb+AnYYD/EJpfFPOkDuX6C4ZpBqompSh 4rVPtp9PDegOR1rhqLYtlixDNtKWaIfVo+zkhPYsqTtsyCwlcSzC3i2sbh6nk/9tpwYF KSqnpTKPJEpYyJVCXCpz2MaiNYWdK3VS/u8Ie6jJEctvCk/tc3kKi5YFFhbkk0MWxdVk +u8Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:dkim-signature; bh=vi3v5ntKjOlKgI0yDUgqAwldtjZE2XpACiJPHu/ftMo=; b=opGkMO8BRobYVzXVvP/Ty8+PSXR817AUkaYVt3Mj+mkN0+4dLL3poSmhVRKYNyOzSc eiSUymXQAa2bW7CR7KH4FQFgIFbaec8dNqexi1+PFwqXcp3+1YwKYCoakKd2Se/yhnXX Zke+2zITJNlEpsx2HWeIWnxTq5WesM0s5PBOTPgf4wC3vDeC84EY9Of6NwzaNk2cecAc P6UWmhpeJNDruTliZUZsXD2Qo58v3urYqG9BDh+hNfwOnJVnEIz9WUdd9b4cT4jmZ8vT qUhediJrJX6Dsf1s6avJm+XgtKym0aVu+PhX1IGZjXPSR55rmj5Ii0IUmclDbmltZUIv CUrg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=dHY5S+ti; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id f12si4441122edy.546.2020.05.22.05.23.42; Fri, 22 May 2020 05:24:07 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=dHY5S+ti; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729535AbgEVMTY (ORCPT + 99 others); Fri, 22 May 2020 08:19:24 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33426 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728772AbgEVMTW (ORCPT ); Fri, 22 May 2020 08:19:22 -0400 Received: from mail-io1-xd43.google.com (mail-io1-xd43.google.com [IPv6:2607:f8b0:4864:20::d43]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 770D4C061A0E; Fri, 22 May 2020 05:19:22 -0700 (PDT) Received: by mail-io1-xd43.google.com with SMTP id o5so11101806iow.8; Fri, 22 May 2020 05:19:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=vi3v5ntKjOlKgI0yDUgqAwldtjZE2XpACiJPHu/ftMo=; b=dHY5S+ti8NPOiZvQZhd7m7DkWZzFxApxJFw6LryiupScIIQkXXGmJkS52WjK8ije1W ZNHSqjzOdCzNCft6t1pPlYRfz6NxDbYgHfLbosfHhrWLNp0uscBcqe+FQtmF0eMU6sx3 AK1iEjPc5zepV0uz0FfN/gzEGAy16E3vzM+B78OeTfWhWREFfNe3idej92k3MjmhicNX WX225wsUFzYvs2rYYclgxXlhU0KqYlRSq3Wop4u+RK+VEQNpdh4Crr9MJAL6WmucTHlw b+U6qmG82EgcbGA9gsLhJFAB2mFzdGq9wbXhpEItyrmKr96Taty/Ixc63c+W4vFmwqnH e+7w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=vi3v5ntKjOlKgI0yDUgqAwldtjZE2XpACiJPHu/ftMo=; b=QJSFnstbUCA4wLEq2PWTcZKqD0sL3S6PuDrD16DpQFAwI8vl8qCFrMJyu1C9GG3jxJ viZ4c9obsLQ9fUXO4Dfn8XL35AER0ChcO4tEF/PZ8Vz29y9rr+lEEXnhOBKgu9xtPjIM OlB24Y+gOPWp+iFxPH6fghDOCeD6gGrLxgy5ivKGbY9r3TtzNV9oh1wnYGT2L/oXA2dE pZUAzA+bEplhdCirPYW6SFfbbgBJnHUDO4cj4fp8GRo43LoEptGhSEpoQG7hmz8Fbhec WyIiHHV2JUP9SsMOcyZgkx2r63L0FTiIqwhe7x+JCHqt0/wXKOXRzAMU/uDCuCbqXM5s y0/Q== X-Gm-Message-State: AOAM530eRQ0qzyDAb1Pe1t39f/ef4KGY3itrXLGkQkldprg+zVoFLEYN iC0oEaPAhyi044A+f5fArIc9k3W2Wj48VpxRpn4= X-Received: by 2002:a05:6602:2dd4:: with SMTP id l20mr2836500iow.13.1590149961329; Fri, 22 May 2020 05:19:21 -0700 (PDT) MIME-Version: 1.0 References: <20200505084127.12923-1-laoar.shao@gmail.com> <20200505084127.12923-3-laoar.shao@gmail.com> In-Reply-To: From: Yafang Shao Date: Fri, 22 May 2020 20:18:44 +0800 Message-ID: Subject: Re: [PATCH v3 2/2] mm, memcg: Decouple e{low,min} state mutations from protection checks To: Naresh Kamboju Cc: Chris Down , Michal Hocko , Andrew Morton , Johannes Weiner , Roman Gushchin , linux-mm , Cgroups , open list , lkft-triage@lists.linaro.org Content-Type: text/plain; charset="UTF-8" Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Fri, May 22, 2020 at 7:01 PM Naresh Kamboju wrote: > > On Tue, 5 May 2020 at 14:12, Yafang Shao wrote: > > > > From: Chris Down > > > > mem_cgroup_protected currently is both used to set effective low and min > > and return a mem_cgroup_protection based on the result. As a user, this > > can be a little unexpected: it appears to be a simple predicate function, > > if not for the big warning in the comment above about the order in which > > it must be executed. > > > > This change makes it so that we separate the state mutations from the > > actual protection checks, which makes it more obvious where we need to be > > careful mutating internal state, and where we are simply checking and > > don't need to worry about that. > > This patch is causing oom-killer while running mkfs -t ext4 on i386 kernel > running on x86_64 machine version linux-next 5.7.0-rc6-next-20200521. > Hi Narash, Thanks for your report. My suggestion to the issue found by you is reverting this bad commit. As I have explained earlier in another mail thread [1] that the usage around memcg->{emin, elow} is very buggy. We shouldn't use memcg->{emin, elow} in the reclaim context directly, because these two values can be modified by many reclaimers, so the good usage of it is storing the protection value into the scan_control. IOW, different reclaimers have different protection. But unfortunately my suggestion is ignored. We can set them to 0 before using them to workaround the issue found by you, but the fact is that we will introduce a new issue once we fix an old issue. [1]. https://lore.kernel.org/linux-mm/20200425152418.28388-1-laoar.shao@gmail.com/ > + mkfs -t ext4 /dev/disk/by-id/ata-TOSHIBA_MG04ACA100N_Y8RQK14KF6XF > mke2fs 1.43.8 (1-Jan-2018) > Creating filesystem with 244190646 4k blocks and 61054976 inodes > Filesystem UUID: 5309a579-5f29-4e9e-959d-0cb1da3490a8 > Superblock backups stored on blocks: > 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, > 4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968, > 102400000, 214990848 > Allocating group tables: 0/7453 done > Writing inode tables: 0/7453 done > Creating journal (262144 blocks): [ 35.419377] mkfs.ext4 invoked > oom-killer: gfp_mask=0x101cc0(GFP_USER|__GFP_WRITE), order=0, > oom_score_adj=0 > [ 35.429138] CPU: 2 PID: 391 Comm: mkfs.ext4 Not tainted > 5.7.0-rc6-next-20200521 #1 > [ 35.436707] Hardware name: Supermicro SYS-5019S-ML/X11SSH-F, BIOS > 2.2 05/23/2018 > [ 35.444099] Call Trace: > [ 35.446558] dump_stack+0x54/0x6e > [ 35.449874] dump_header+0x3d/0x1d2 > [ 35.453364] ? oom_badness.part.0+0x11/0x120 > [ 35.457635] ? ___ratelimit+0x8f/0xdc > [ 35.461301] oom_kill_process.cold+0x9/0xe > [ 35.465390] out_of_memory+0x1b1/0x260 > [ 35.469135] __alloc_pages_nodemask+0xe1d/0xed0 > [ 35.473658] ? alloc_page_buffers+0x81/0x130 > [ 35.477923] pagecache_get_page+0xae/0x260 > [ 35.482014] grab_cache_page_write_begin+0x1c/0x30 > [ 35.486798] block_write_begin+0x1e/0x90 > [ 35.490716] blkdev_write_begin+0x1e/0x20 > [ 35.494719] ? bdev_evict_inode+0xd0/0xd0 > [ 35.498722] generic_perform_write+0x97/0x180 > [ 35.503073] __generic_file_write_iter+0x140/0x1f0 > [ 35.507857] blkdev_write_iter+0xc0/0x190 > [ 35.511860] __vfs_write+0x132/0x1e0 > [ 35.515431] vfs_write+0xa1/0x1a0 > [ 35.518742] ksys_pwrite64+0x50/0x80 > [ 35.522315] __ia32_sys_ia32_pwrite64+0x16/0x20 > [ 35.526845] do_syscall_32_irqs_on+0x3d/0x250 > [ 35.531195] ? do_user_addr_fault+0x1a0/0x3c0 > [ 35.535548] ? __prepare_exit_to_usermode+0x50/0x1a0 > [ 35.540504] do_fast_syscall_32+0x39/0xb0 > [ 35.544508] entry_SYSENTER_32+0xa5/0xf8 > [ 35.548426] EIP: 0xb7efb549 > [ 35.551218] Code: 03 74 c0 01 10 05 03 74 b8 01 10 06 03 74 b4 01 > 10 07 03 74 b0 01 10 08 03 74 d8 01 00 00 00 00 00 51 52 55 89 e5 0f > 34 cd 80 <5d> 5a 59 c3 90 90 90 90 8d 76 00 58 b8 77 00 00 00 cd 80 90 > 8d 76 > [ 35.569962] EAX: ffffffda EBX: 00000003 ECX: b77ef010 EDX: 00400000 > [ 35.576217] ESI: 38400000 EDI: 00000074 EBP: 07438400 ESP: bfdff180 > [ 35.582476] DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 007b EFLAGS: 00000246 > [ 35.589260] Mem-Info: > [ 35.591552] active_anon:5403 inactive_anon:2167 isolated_anon:0 > [ 35.591552] active_file:4174 inactive_file:212518 isolated_file:0 > [ 35.591552] unevictable:0 dirty:16699 writeback:6001 unstable:0 > [ 35.591552] slab_reclaimable:5868 slab_unreclaimable:3494 > [ 35.591552] mapped:6285 shmem:2236 pagetables:176 bounce:0 > [ 35.591552] free:264050 free_pcp:1186 free_cma:0 > [ 35.625311] Node 0 active_anon:21612kB inactive_anon:8668kB > active_file:16696kB inactive_file:850072kB unevictable:0kB > isolated(anon):0kB isolated(file):0kB mapped:25140kB dirty:62704kB > writeback:22764kB shmem:8944kB writeback_tmp:0kB unstable:0kB > all_unreclaimable? yes > [ 35.649169] DMA free:3360kB min:68kB low:84kB high:100kB > reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB > active_file:0kB inactive_file:12012kB unevictable:0kB > writepending:12012kB present:15964kB managed:15876kB mlocked:0kB > kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB > free_cma:0kB > [ 35.677015] lowmem_reserve[]: 0 824 1947 824 > [ 35.681286] Normal free:4300kB min:7732kB low:8640kB high:9548kB > reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB > active_file:1132kB inactive_file:786452kB unevictable:0kB > writepending:64032kB present:884728kB managed:845292kB mlocked:0kB > kernel_stack:1112kB pagetables:0kB bounce:0kB free_pcp:2392kB > local_pcp:456kB free_cma:0kB > [ 35.711109] lowmem_reserve[]: 0 0 8980 0 > [ 35.715051] HighMem free:1048740kB min:512kB low:1748kB high:2984kB > reserved_highatomic:0KB active_anon:21612kB inactive_anon:8668kB > active_file:15564kB inactive_file:51556kB unevictable:0kB > writepending:0kB present:1149540kB managed:1149540kB mlocked:0kB > kernel_stack:0kB pagetables:704kB bounce:0kB free_pcp:2216kB > local_pcp:200kB free_cma:0kB > [ 35.745418] lowmem_reserve[]: 0 0 0 0 > [ 35.749117] DMA: 7*4kB (UE) 1*8kB (U) 2*16kB (UE) 1*32kB (E) 1*64kB > (E) 1*128kB (E) 0*256kB 0*512kB 1*1024kB (E) 1*2048kB (E) 0*4096kB = > 3364kB > [ 35.762006] Normal: 19*4kB (U) 18*8kB (UE) 18*16kB (UE) 12*32kB > (UE) 6*64kB (U) 2*128kB (U) 2*256kB (UE) 1*512kB (E) 0*1024kB 1*2048kB > (U) 0*4096kB = 4604kB > [ 35.776020] HighMem: 1*4kB (U) 0*8kB 0*16kB 1*32kB (M) 0*64kB > 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 256*4096kB (M) = > 1048740kB > [ 35.788093] Node 0 hugepages_total=0 hugepages_free=0 > hugepages_surp=0 hugepages_size=4096kB > [ 35.796533] 218903 total pagecache pages > [ 35.800503] 0 pages in swap cache > [ 35.803873] Swap cache stats: add 0, delete 0, find 0/0 > [ 35.809145] Free swap = 0kB > [ 35.812083] Total swap = 0kB > [ 35.815021] 512558 pages RAM > [ 35.817957] 287385 pages HighMem/MovableOnly > [ 35.822221] 9881 pages reserved > [ 35.825368] Tasks state (memory values in pages): > [ 35.830124] [ pid ] uid tgid total_vm rss pgtables_bytes > swapents oom_score_adj name > [ 35.838791] [ 225] 0 225 3425 1294 24576 > 0 0 systemd-journal > [ 35.847868] [ 240] 0 240 3326 868 20480 > 0 -1000 systemd-udevd > [ 35.856785] [ 243] 994 243 3929 450 24576 > 0 0 systemd-timesyn > [ 35.865874] [ 260] 993 260 1569 786 20480 > 0 0 systemd-network > [ 35.874966] [ 278] 992 278 1729 843 20480 > 0 0 systemd-resolve > [ 35.884007] [ 282] 0 282 809 428 16384 > 0 0 crond > [ 35.892229] [ 283] 0 283 2032 1104 20480 > 0 0 haveged > [ 35.900635] [ 284] 0 284 922 194 16384 > 0 0 klogd > [ 35.908820] [ 285] 0 285 5638 1481 32768 > 0 0 thermald > [ 35.917304] [ 286] 995 286 1182 664 20480 > 0 0 avahi-daemon > [ 35.926133] [ 287] 996 287 1174 835 24576 > 0 -900 dbus-daemon > [ 35.934880] [ 288] 995 288 1182 58 20480 > 0 0 avahi-daemon > [ 35.943710] [ 289] 0 289 921 409 16384 > 0 0 syslogd > [ 35.952108] [ 290] 0 290 1468 998 20480 > 0 0 systemd-logind > [ 35.961060] [ 303] 0 303 594 15 16384 > 0 0 acpid > [ 35.969275] [ 304] 0 304 11786 2635 45056 > 0 0 NetworkManager > [ 35.978282] [ 305] 0 305 677 429 16384 > 0 0 atd > [ 35.986280] [ 323] 0 323 1347 321 20480 > 0 0 systemd-hostnam > [ 35.995318] [ 334] 65534 334 729 32 16384 > 0 0 dnsmasq > [ 36.003708] [ 335] 0 335 666 443 16384 > 0 0 agetty > [ 36.012019] [ 336] 0 336 947 710 20480 > 0 0 login > [ 36.020246] [ 337] 0 337 666 419 16384 > 0 0 agetty > [ 36.028555] [ 348] 998 348 19521 2782 65536 > 0 0 polkitd > [ 36.036901] [ 356] 0 356 1892 1183 20480 > 0 0 systemd > [ 36.045300] [ 357] 0 357 2320 316 20480 > 0 0 (sd-pam) > [ 36.053731] [ 361] 0 361 971 721 16384 > 0 0 sh > [ 36.061698] [ 365] 0 365 920 610 16384 > 0 0 su > [ 36.069682] [ 366] 0 366 971 689 16384 > 0 0 sh > [ 36.077664] [ 371] 0 371 903 580 16384 > 0 0 lava-test-runne > [ 36.086761] [ 381] 0 381 903 520 16384 > 0 0 lava-test-shell > [ 36.095852] [ 382] 0 382 903 581 16384 > 0 0 sh > [ 36.103816] [ 391] 0 391 1976 1688 24576 > 0 0 mkfs.ext4 > [ 36.112336] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=polkitd,pid=348,uid=998 > [ 36.125003] Out of memory: Killed process 348 (polkitd) > total-vm:78084kB, anon-rss:2980kB, file-rss:8148kB, shmem-rss:0kB, > UID:998 pgtables:64kB oom_score_adj:0 > [ 36.139810] oom_reaper: reaped process 348 (polkitd), now > anon-rss:0kB, file-rss:0kB, shmem-rss:0kB > [ 36.148879] mkfs.ext4 invoked oom-killer: > gfp_mask=0x101cc0(GFP_USER|__GFP_WRITE), order=0, oom_score_adj=0 > [ 36.158626] CPU: 1 PID: 391 Comm: mkfs.ext4 Not tainted > 5.7.0-rc6-next-20200521 #1 > [ 36.166220] Hardware name: Supermicro SYS-5019S-ML/X11SSH-F, BIOS > 2.2 05/23/2018 > [ 36.173604] Call Trace: > [ 36.176050] dump_stack+0x54/0x6e > [ 36.179360] dump_header+0x3d/0x1d2 > [ 36.182844] ? oom_badness.part.0+0x11/0x120 > [ 36.187110] ? ___ratelimit+0x8f/0xdc > [ 36.190773] oom_kill_process.cold+0x9/0xe > [ 36.194863] out_of_memory+0x1b1/0x260 > [ 36.198610] __alloc_pages_nodemask+0xe1d/0xed0 > [ 36.203141] ? alloc_page_buffers+0x81/0x130 > [ 36.207406] pagecache_get_page+0xae/0x260 > [ 36.211496] grab_cache_page_write_begin+0x1c/0x30 > [ 36.216281] block_write_begin+0x1e/0x90 > [ 36.220197] blkdev_write_begin+0x1e/0x20 > [ 36.224201] ? bdev_evict_inode+0xd0/0xd0 > [ 36.228205] generic_perform_write+0x97/0x180 > [ 36.232557] __generic_file_write_iter+0x140/0x1f0 > [ 36.237340] blkdev_write_iter+0xc0/0x190 > [ 36.241344] __vfs_write+0x132/0x1e0 > [ 36.244915] vfs_write+0xa1/0x1a0 > [ 36.248225] ksys_pwrite64+0x50/0x80 > [ 36.251797] __ia32_sys_ia32_pwrite64+0x16/0x20 > [ 36.256319] do_syscall_32_irqs_on+0x3d/0x250 > [ 36.260693] ? do_user_addr_fault+0x1a0/0x3c0 > [ 36.265045] ? __prepare_exit_to_usermode+0x50/0x1a0 > [ 36.270004] do_fast_syscall_32+0x39/0xb0 > [ 36.274006] entry_SYSENTER_32+0xa5/0xf8 > [ 36.277926] EIP: 0xb7efb549 > [ 36.280728] Code: 03 74 c0 01 10 05 03 74 b8 01 10 06 03 74 b4 01 > 10 07 03 74 b0 01 10 08 03 74 d8 01 00 00 00 00 00 51 52 55 89 e5 0f > 34 cd 80 <5d> 5a 59 c3 90 90 90 90 8d 76 00 58 b8 77 00 00 00 cd 80 90 > 8d 76 > [ 36.299471] EAX: ffffffda EBX: 00000003 ECX: b77ef010 EDX: 00400000 > [ 36.305728] ESI: 38400000 EDI: 00000074 EBP: 07438400 ESP: bfdff180 > [ 36.311986] DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 007b EFLAGS: 00000246 > [ 36.318783] Mem-Info: > [ 36.321070] active_anon:4659 inactive_anon:2167 isolated_anon:0 > [ 36.321070] active_file:4639 inactive_file:212058 isolated_file:0 > [ 36.321070] unevictable:0 dirty:14286 writeback:23 unstable:0 > [ 36.321070] slab_reclaimable:5883 slab_unreclaimable:3313 > [ 36.321070] mapped:5709 shmem:2236 pagetables:176 bounce:0 > [ 36.321070] free:264784 free_pcp:1341 free_cma:0 > [ 36.354669] Node 0 active_anon:18636kB inactive_anon:8668kB > active_file:18556kB inactive_file:848232kB unevictable:0kB > isolated(anon):0kB isolated(file):0kB mapped:22836kB dirty:57144kB > writeback:92kB shmem:8944kB writeback_tmp:0kB unstable:0kB > all_unreclaimable? yes > [ 36.378296] DMA free:3364kB min:68kB low:84kB high:100kB > reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB > active_file:0kB inactive_file:12124kB unevictable:0kB > writepending:12132kB present:15964kB managed:15876kB mlocked:0kB > kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB > free_cma:0kB > [ 36.406167] lowmem_reserve[]: 0 824 1947 824 > [ 36.410467] Normal free:4552kB min:7732kB low:8640kB high:9548kB > reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB > active_file:1132kB inactive_file:786524kB unevictable:0kB > writepending:45100kB present:884728kB managed:845292kB mlocked:0kB > kernel_stack:1112kB pagetables:0kB bounce:0kB free_pcp:2548kB > local_pcp:1432kB free_cma:0kB > [ 36.440330] lowmem_reserve[]: 0 0 8980 0 > [ 36.444274] HighMem free:1051220kB min:512kB low:1748kB high:2984kB > reserved_highatomic:0KB active_anon:18636kB inactive_anon:8668kB > active_file:17424kB inactive_file:49696kB unevictable:0kB > writepending:0kB present:1149540kB managed:1149540kB mlocked:0kB > kernel_stack:0kB pagetables:704kB bounce:0kB free_pcp:2812kB > local_pcp:516kB free_cma:0kB > [ 36.474680] lowmem_reserve[]: 0 0 0 0 > [ 36.478401] DMA: 3*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (UE) > 1*128kB (E) 0*256kB 0*512kB 1*1024kB (E) 1*2048kB (E) 0*4096kB = > 3364kB > [ 36.490847] Normal: 19*4kB (U) 22*8kB (UE) 19*16kB (UE) 13*32kB > (UE) 6*64kB (U) 2*128kB (U) 2*256kB (UE) 1*512kB (E) 0*1024kB 1*2048kB > (U) 0*4096kB = 4684kB > [ 36.504851] HighMem: 119*4kB (UM) 48*8kB (M) 12*16kB (UM) 11*32kB > (M) 8*64kB (UM) 6*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB > 256*4096kB (M) = 1051260kB > [ 36.518695] Node 0 hugepages_total=0 hugepages_free=0 > hugepages_surp=0 hugepages_size=4096kB > [ 36.527151] 218965 total pagecache pages > [ 36.531095] 0 pages in swap cache > [ 36.534430] Swap cache stats: add 0, delete 0, find 0/0 > [ 36.539697] Free swap = 0kB > [ 36.542602] Total swap = 0kB > [ 36.545531] 512558 pages RAM > [ 36.548436] 287385 pages HighMem/MovableOnly > [ 36.552729] 9881 pages reserved > [ 36.555897] Tasks state (memory values in pages): > [ 36.560621] [ pid ] uid tgid total_vm rss pgtables_bytes > swapents oom_score_adj name > [ 36.569230] [ 225] 0 225 3425 1294 28672 > 0 0 systemd-journal > [ 36.578295] [ 240] 0 240 3326 868 20480 > 0 -1000 systemd-udevd > [ 36.587186] [ 243] 994 243 3929 450 24576 > 0 0 systemd-timesyn > [ 36.596249] [ 260] 993 260 1569 786 20480 > 0 0 systemd-network > [ 36.605307] [ 278] 992 278 1729 843 20480 > 0 0 systemd-resolve > [ 36.614363] [ 282] 0 282 809 428 16384 > 0 0 crond > [ 36.622551] [ 283] 0 283 2032 1104 20480 > 0 0 haveged > [ 36.630918] [ 284] 0 284 922 194 16384 > 0 0 klogd > [ 36.639114] [ 285] 0 285 5638 1481 32768 > 0 0 thermald > [ 36.647563] [ 286] 995 286 1182 664 20480 > 0 0 avahi-daemon > [ 36.656362] [ 287] 996 287 1174 835 24576 > 0 -900 dbus-daemon > [ 36.665081] [ 288] 995 288 1182 58 20480 > 0 0 avahi-daemon > [ 36.673884] [ 289] 0 289 921 409 16384 > 0 0 syslogd > [ 36.682249] [ 290] 0 290 1468 998 20480 > 0 0 systemd-logind > [ 36.691228] [ 303] 0 303 594 15 16384 > 0 0 acpid > [ 36.699425] [ 304] 0 304 11786 2635 45056 > 0 0 NetworkManager > [ 36.708395] [ 305] 0 305 677 429 16384 > 0 0 atd > [ 36.716394] [ 323] 0 323 1347 321 20480 > 0 0 systemd-hostnam > [ 36.725435] [ 334] 65534 334 729 32 16384 > 0 0 dnsmasq > [ 36.733781] [ 335] 0 335 666 443 16384 > 0 0 agetty > [ 36.742041] [ 336] 0 336 947 710 20480 > 0 0 login > [ 36.750237] [ 337] 0 337 666 419 16384 > 0 0 agetty > [ 36.758514] [ 356] 0 356 1892 1183 20480 > 0 0 systemd > [ 36.766877] [ 357] 0 357 2320 316 20480 > 0 0 (sd-pam) > [ 36.775326] [ 361] 0 361 971 721 16384 > 0 0 sh > [ 36.783257] [ 365] 0 365 920 610 16384 > 0 0 su > [ 36.791170] [ 366] 0 366 971 689 16384 > 0 0 sh > [ 36.799082] [ 371] 0 371 903 580 16384 > 0 0 lava-test-runne > [ 36.808122] [ 381] 0 381 903 520 16384 > 0 0 lava-test-shell > [ 36.817161] [ 382] 0 382 903 581 16384 > 0 0 sh > [ 36.825073] [ 391] 0 391 1976 1688 24576 > 0 0 mkfs.ext4 > [ 36.833591] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=NetworkManager,pid=304,uid=0 > [ 36.846621] Out of memory: Killed process 304 (NetworkManager) > total-vm:47144kB, anon-rss:1396kB, file-rss:9144kB, shmem-rss:0kB, > UID:0 pgtables:44kB oom_score_adj:0 > [ 36.861729] oom_reaper: reaped process 304 (NetworkManager), now > anon-rss:0kB, file-rss:0kB, shmem-rss:0kB > [ 36.861738] klogd invoked oom-killer: > gfp_mask=0xc2cc0(GFP_KERNEL|__GFP_NOWARN|__GFP_COMP|__GFP_NOMEMALLOC), > order=0, oom_score_adj=0 > [ 36.883368] CPU: 2 PID: 284 Comm: klogd Not tainted > 5.7.0-rc6-next-20200521 #1 > [ 36.890607] Hardware name: Supermicro SYS-5019S-ML/X11SSH-F, BIOS > 2.2 05/23/2018 > [ 36.897992] Call Trace: > [ 36.900438] dump_stack+0x54/0x6e > [ 36.903763] dump_header+0x3d/0x1d2 > [ 36.907247] ? oom_badness.part.0+0x11/0x120 > [ 36.911510] ? ___ratelimit+0x8f/0xdc > [ 36.915167] oom_kill_process.cold+0x9/0xe > [ 36.919258] out_of_memory+0x1b1/0x260 > [ 36.923005] __alloc_pages_nodemask+0xe1d/0xed0 > [ 36.927537] alloc_slab_page+0x13d/0x2c0 > [ 36.931453] new_slab+0x22e/0x280 > [ 36.934766] ? cpumask_next_and+0x18/0x20 > [ 36.938777] ___slab_alloc.constprop.0+0x248/0x4a0 > [ 36.943563] ? __alloc_skb+0x64/0x1a0 > [ 36.947228] ? __alloc_skb+0x41/0x1a0 > [ 36.950885] __slab_alloc.constprop.0+0xf/0x20 > [ 36.955321] ? __slab_alloc.constprop.0+0xf/0x20 > [ 36.959932] __kmalloc_track_caller+0x23e/0x270 > [ 36.964456] ? __alloc_skb+0x64/0x1a0 > [ 36.968114] ? _cond_resched+0x12/0x30 > [ 36.971857] __kmalloc_reserve.isra.0+0x1d/0x70 > [ 36.976380] __alloc_skb+0x64/0x1a0 > [ 36.979866] ? update_rq_clock.part.0+0x16/0xe0 > [ 36.984389] alloc_skb_with_frags+0x3b/0x1a0 > [ 36.988665] sock_alloc_send_pskb+0x1c7/0x1f0 > [ 36.993040] ? selinux_inet_conn_established+0x40/0x40 > [ 36.998179] unix_dgram_sendmsg+0x12a/0x690 > [ 37.002365] ? selinux_socket_recvmsg+0x20/0x20 > [ 37.006897] ? unix_dgram_connect+0x2f0/0x2f0 > [ 37.011247] sock_sendmsg+0x5c/0x60 > [ 37.014730] __sys_sendto+0xe1/0x120 > [ 37.018301] ? msg_print_text+0x80/0x1c0 > [ 37.022219] ? do_syslog+0x19c/0x940 > [ 37.025788] ? do_syslog+0x878/0x940 > [ 37.029358] ? do_syslog+0x878/0x940 > [ 37.032931] ? kunmap_atomic_high+0x4e/0x58 > [ 37.037109] ? handle_mm_fault+0x25c/0xc60 > [ 37.041209] __ia32_sys_socketcall+0x29f/0x330 > [ 37.045665] do_syscall_32_irqs_on+0x3d/0x250 > [ 37.050038] ? do_user_addr_fault+0x1a0/0x3c0 > [ 37.054388] ? __prepare_exit_to_usermode+0x50/0x1a0 > [ 37.059346] do_fast_syscall_32+0x39/0xb0 > [ 37.063351] entry_SYSENTER_32+0xa5/0xf8 > [ 37.067276] EIP: 0xb7f66549 > > ref: > https://lkft.validation.linaro.org/scheduler/job/1444120#L1220 > > -- > Linaro LKFT > https://lkft.linaro.org -- Thanks Yafang