Received: by 2002:a05:7412:8d1c:b0:fa:4c10:6cad with SMTP id bj28csp335489rdb; Wed, 17 Jan 2024 03:40:27 -0800 (PST) X-Google-Smtp-Source: AGHT+IEfYsimu72qUJSUFFcJCkGPJ198f282pQ3ZWgX5KgzYjRG9X7O3XvTA9bvI7sZeNqCSBgSp X-Received: by 2002:ac8:7d07:0:b0:429:8aa4:9813 with SMTP id g7-20020ac87d07000000b004298aa49813mr12152908qtb.111.1705491627251; Wed, 17 Jan 2024 03:40:27 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1705491627; cv=pass; d=google.com; s=arc-20160816; b=E8nX6GG1cMX38X6yupoa7zxP/Z3yxA7uJDy0jPphTksDxVPKr/xHHSGJFyFh3x2SDC g60vkMOsmivEq8GkW5zF16lzoeiXG9So1bhDOnkbKxjPtVggvftydt3PoVxl4fQV3/qC dJELJdS6CbANm+tODPWzHgQNRPWfTGo1+I5ACDqK2N2txj5kxRPsf7E0G5/N6kN00DPS b23ZpXr1WjN6Ltk3O8/r3ne8rArKS00b6CxPgar8guUzIWwp6wBpMZ2d7gi9W9Xum0aQ GIR9va88IJsW9RPb9hIIInxkcC2HtWv9xFJX/PbdMw4UDXQEr1j1MTsLrrnvL8uabZ06 BCYg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:subject:message-id:date:from:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:dkim-signature; bh=5TFYs1NHQSNe09ooyq6JqoLvFuR9S1Ao6SPcZ/zAQqo=; fh=xexEV3H19JsPCKmzaS4eOsSwkVtrXnuK8b7ThoQwHTQ=; b=I6jfOHLOkQuDuiFmKY2ZTLcF4fLvCCD0QV7EE0DEZGDNQD938WeEPl8anOJosp82Wm DM/ivxj6Iq0YaPsi5aDyPki/2ghHVHD0QWN7oaTZunzkJ2p/5ukLdUqPIFMYOu8LLqfX QcS18CCJCg/vjLow2Fu7VEcratTg9EiIMlGwQFINFpC6IpZUBz6TbYNLVY0y/7X/+8TN FRFQuzNtEEYF6t1M8DDA8jjOrGDm7McSUAVXNwleaKbgBCNlz6L/zsWn3YnIFIS/n2V1 PiwdDwXsoOQWk9Wi2exEzIEDws0mC+8q5tsY7b7dxm24ap3tdo2jGrKUJ6dAUTSA/419 FC3w== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@gmail.com header.s=20230601 header.b=CMK6IYAC; arc=pass (i=1 spf=pass spfdomain=gmail.com dkim=pass dkdomain=gmail.com dmarc=pass fromdomain=gmail.com); spf=pass (google.com: domain of linux-kernel+bounces-28920-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) smtp.mailfrom="linux-kernel+bounces-28920-linux.lists.archive=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [147.75.199.223]) by mx.google.com with ESMTPS id n4-20020ac85a04000000b0042a128c039esi337231qta.58.2024.01.17.03.40.27 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 17 Jan 2024 03:40:27 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-28920-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20230601 header.b=CMK6IYAC; arc=pass (i=1 spf=pass spfdomain=gmail.com dkim=pass dkdomain=gmail.com dmarc=pass fromdomain=gmail.com); spf=pass (google.com: domain of linux-kernel+bounces-28920-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) smtp.mailfrom="linux-kernel+bounces-28920-linux.lists.archive=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id 86AF11C242E6 for ; Wed, 17 Jan 2024 11:40:26 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id A3F7A1DFC8; Wed, 17 Jan 2024 11:40:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="CMK6IYAC" Received: from mail-ed1-f42.google.com (mail-ed1-f42.google.com [209.85.208.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CEB171DDE9 for ; Wed, 17 Jan 2024 11:40:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.42 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1705491615; cv=none; b=CohXp7cb6ZaCz1I3C3OVhR6XoYKcbLlnFRC2yw5bnuFgFX+jHIe0KotnO0P11heukc6sEB3twtwlhOTfk3XpGSMzM/4jYA84TARjKYi913yk2ocCc4Mx4WUuqHq7KZqyqYzXSRlJoBVz2z+33fiH2SmzqTxCGSlFSXT+fyXVTOo= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1705491615; c=relaxed/simple; bh=1Z8lC1uBCnXPaoD8hYA4tvQshRgYck9kKl4amZcO9fI=; h=Received:DKIM-Signature:X-Google-DKIM-Signature: X-Gm-Message-State:X-Google-Smtp-Source:X-Received:MIME-Version: From:Date:Message-ID:Subject:To:Cc:Content-Type; b=prghx8FQjsf+yTtLkvduHLAdkSoj4Ts6bJA+h8/DKqjddsRe4grkcuDdW2bHtfJxMgCzTXEf8nmDmEQ5u3Gy2xFczr9HPaU3CE7xLtARz2Fl692BSXcW0X1LGFgP/Sb5YOuSej+u704hK3aQ69wpu7XIbXk2GKQoWc+J0vH+TG0= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=CMK6IYAC; arc=none smtp.client-ip=209.85.208.42 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Received: by mail-ed1-f42.google.com with SMTP id 4fb4d7f45d1cf-55369c59708so2282951a12.1 for ; Wed, 17 Jan 2024 03:40:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1705491609; x=1706096409; darn=vger.kernel.org; h=cc:to:subject:message-id:date:from:mime-version:from:to:cc:subject :date:message-id:reply-to; bh=5TFYs1NHQSNe09ooyq6JqoLvFuR9S1Ao6SPcZ/zAQqo=; b=CMK6IYACviLQAZVUqi9r5Outw3B8bDPWLNZgtYRNJLK6xnqdRB1V5ge4JW/DiQoZdJ VmcODI0HUOCKEmbao3lJvchxOyk3r8wKxhT0+B0go8yf3JVWFPGmmVqjBuMDI2UlILtV o24IWz3vuDrYkSBkmo7J93r9bWlnJoU5kCvD0bOBsxrDmM23KzYFh4tB9CqqhMzQeJRx lFfSr+Pmc3xVUeT/Zolng2Ti5B5EF0uCrchnfFpz/IRQriR1fb6tpvODBLkkNB/YMFuG cYq7GnOh0rcub+w36swx2kPz+opIiuGw0XI3VEPLgtt14d5QQve8mTajvd+2x+EjJvdm ElGQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1705491609; x=1706096409; h=cc:to:subject:message-id:date:from:mime-version:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=5TFYs1NHQSNe09ooyq6JqoLvFuR9S1Ao6SPcZ/zAQqo=; b=jDzIDYmpGyXAhi88grYq3mkLUAVURA6I3PksVJxxSpdnx08mssCr5lzG2W4XNl0p/4 5RPPunVZrXyRwRiQkxTY0cLyuNZFjh57LdEjpKDU6ID16q+zHtbHucurcCJUmBUxQN8/ PK/T+x1fRU6SrCXKJ24B02VwNfgZw9zJPFgZPuRXbNBqzu4w+CsSoo1l/6aLyGL2Yq2b BbU6SDbbvN89xNiStsS97Y2hyBQ0i9aZikkl67Sgc9p1wXxHNUhvgVOV/L2SEJkiYMoi 1IT9jF3PVBghMlv8EC8182yEOIDx8JBD9GA16jCeDZXYUw5qU7xa2WrJfLPQaYqgVaPA mSuQ== X-Gm-Message-State: AOJu0YypC7ywGLSCJiKY3gRiO8FAGGHL08/XYV4CGgUeEFoXkzruxUog pl9Q5il9iEmli6M6RYsTLUa/RrInIQ51YAXqSbB/5MFYNoo= X-Received: by 2002:aa7:d655:0:b0:552:29d9:92c9 with SMTP id v21-20020aa7d655000000b0055229d992c9mr8472043edr.3.1705491608689; Wed, 17 Jan 2024 03:40:08 -0800 (PST) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Karim Manaouil Date: Wed, 17 Jan 2024 11:39:56 +0000 Message-ID: Subject: Debug: mm: meminfo reports 222GiB of AnonPages but sum of RSS in ps is barely 400MiB To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org Content-Type: text/plain; charset="UTF-8" Hi, My system says that 231GiB is used out of 251GiB, but when I try to find the processes using this memory in htop or ps, I find nothing. In fact, the sum of RSS memory across all processes listed in ps (both user and kernel process) is barely 400MiB. A look at Meminfo shows that 222GiB of memory is AnonPages. With the help of `numastat -m`, it is inactive anonymous, more precisely. I am really baffled and I have no idea who is using my memory?!! Here is what `free -h` is reporting. karim@luna:~/mem_debug$ free -h total used free shared buff/cache available Mem: 251Gi 231Gi 1.1Gi 52Mi 21Gi 20Gi Swap: 119Gi 75Gi 43Gi And this is meminfo: karim@luna:~/mem_debug$ grep AnonPages /proc/meminfo AnonPages: 222998596 kB Finally, I include the output of `ps aux --sort=-rss` ---- USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 3285 0.3 0.0 7140704 56612 ? Ssl Jan16 4:02 /usr/bin/dockerd -H fd:// -H tcp://0.0.0.0:2375 --containerd=/run/containerd/containerd.sock root 2876 0.0 0.0 39912 22528 ? Ss Jan16 0:03 /usr/bin/python3 /usr/bin/supervisord -n -c /etc/supervisor/supervisord.conf root 2880 0.0 0.0 5413104 19292 ? Ssl Jan16 0:55 /usr/bin/containerd root 3488 0.0 0.0 83532 16384 ? Ss Jan16 0:02 /usr/sbin/smbd --foreground --no-process-group redis 2870 0.0 0.0 90864 15556 ? Ssl Jan16 0:54 /usr/bin/redis-server 127.0.0.1:16379 root 101045 0.0 0.0 49336 14388 ? Ss 07:44 0:01 /lib/systemd/systemd-journald debian-+ 3411 0.0 0.0 38160 13768 ? Ss Jan16 0:00 /usr/bin/tor --defaults-torrc /usr/share/tor/tor-service-defaults-torrc -f /etc/tor/torrc --RunAsDaemon 0 root 3460 0.0 0.0 70576 12288 ? Ss Jan16 0:07 /usr/sbin/nmbd --foreground --no-process-group root 1 0.0 0.0 169020 10240 ? Ss Jan16 0:22 /sbin/init karim 13264 0.0 0.0 19240 9216 ? Ss 05:56 0:00 /lib/systemd/systemd --user root 3055 0.0 0.0 15404 8192 ? Ss Jan16 0:00 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups root 116282 0.0 0.0 17692 8192 ? Ss 10:03 0:00 sshd: karim [priv] root 116298 0.0 0.0 17692 8192 ? Ss 10:03 0:00 sshd: karim [priv] root 3555 0.0 0.0 81524 7176 ? S Jan16 0:00 /usr/sbin/smbd --foreground --no-process-group root 1435 0.0 0.0 25756 7168 ? Ss Jan16 0:01 /lib/systemd/systemd-udevd systemd+ 2205 0.0 0.0 21080 7168 ? Ss Jan16 0:02 /lib/systemd/systemd-networkd polkitd 3131 0.0 0.0 310436 6144 ? Ssl Jan16 0:00 /usr/lib/polkit-1/polkitd --no-debug karim 116289 0.0 0.0 17924 5336 ? S 10:03 0:00 sshd: karim@pts/0 karim 116305 0.0 0.0 17924 5300 ? S 10:03 0:01 sshd: karim@pts/3 systemd+ 2216 0.0 0.0 90384 5120 ? Ssl Jan16 0:00 /lib/systemd/systemd-timesyncd root 2654 0.0 0.0 11984 5120 ? Ss Jan16 0:00 /usr/sbin/smartd -n root 2665 0.0 0.0 17144 5120 ? Ss Jan16 0:00 /lib/systemd/systemd-logind root 2675 0.0 0.0 16688 5120 ? Ss Jan16 0:00 /lib/systemd/systemd-machined karim 119269 1500 0.0 11780 5120 pts/4 R+ 11:22 0:00 ps aux --sort=-rss root 3556 0.0 0.0 81516 4104 ? S Jan16 0:00 /usr/sbin/smbd --foreground --no-process-group message+ 2564 0.0 0.0 9600 4096 ? Ss Jan16 0:01 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only root 2574 0.0 0.0 221776 4096 ? Ssl Jan16 0:00 /usr/sbin/rsyslogd -n -iNONE root 2790 0.0 0.0 16512 4096 ? Ss Jan16 0:00 /sbin/wpa_supplicant -u -s -O DIR=/run/wpa_supplicant GROUP=netdev karim 13343 0.0 0.0 8988 4096 ? Ss 05:56 0:00 /usr/bin/dbus-daemon --session --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only karim 13447 0.0 0.0 9768 4096 pts/2 Ss+ 06:03 0:00 -bash karim 14941 0.0 0.0 9896 4096 pts/4 Ss 06:20 0:00 -bash karim 116290 0.0 0.0 8252 4096 pts/0 Ss 10:03 0:00 -bash karim 116306 0.0 0.0 8252 4096 pts/3 Ss 10:03 0:00 -bash karim 13265 0.0 0.0 170080 3660 ? S 05:56 0:00 (sd-pam) avahi 2553 0.0 0.0 8384 3072 ? Ss Jan16 0:00 avahi-daemon: running [luna.local] karim 13440 1.5 0.0 10292 3072 ? Ss 06:03 5:00 tmux karim 13441 0.0 0.0 9808 3072 pts/1 Ss+ 06:03 0:01 -bash karim 116332 0.0 0.0 8008 3072 pts/3 S+ 10:03 0:00 tmux attach -t 1 karim 116333 0.0 0.0 8008 3072 pts/0 S+ 10:03 0:00 tmux attach -t 0 root 2147 0.0 0.0 6100 2048 ? Ss Jan16 0:00 /usr/sbin/rdma-ndd --systemd _rpc 2213 0.0 0.0 7876 2048 ? Ss Jan16 0:00 /sbin/rpcbind -f -w root 2253 0.0 0.0 5204 2048 ? Ss Jan16 0:00 /usr/sbin/nfsdcld root 2340 0.0 0.0 13120 2048 ? S