Date:   Tue, 11 Jan 2022 12:00:25 -0800
From:   Roman Gushchin <guro@fb.com>
To:     Muchun Song <songmuchun@bytedance.com>
CC:     <willy@infradead.org>, <akpm@linux-foundation.org>,
        <hannes@cmpxchg.org>, <mhocko@kernel.org>,
        <vdavydov.dev@gmail.com>, <shakeelb@google.com>,
        <shy828301@gmail.com>, <alexs@kernel.org>,
        <richard.weiyang@gmail.com>, <david@fromorbit.com>,
        <trond.myklebust@hammerspace.com>, <anna.schumaker@netapp.com>,
        <jaegeuk@kernel.org>, <chao@kernel.org>,
        <kari.argillander@gmail.com>, <linux-fsdevel@vger.kernel.org>,
        <linux-kernel@vger.kernel.org>, <linux-mm@kvack.org>,
        <linux-nfs@vger.kernel.org>, <zhengqi.arch@bytedance.com>,
        <duanxiongchun@bytedance.com>, <fam.zheng@bytedance.com>,
        <smuchun@gmail.com>
Subject: Re: [PATCH v5 10/16] mm: list_lru: allocate list_lru_one only when
 needed
Message-ID: <Yd3h2YwGIZs1A+2s@carbon.dhcp.thefacebook.com>
References: <20211220085649.8196-1-songmuchun@bytedance.com>
 <20211220085649.8196-11-songmuchun@bytedance.com>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <20211220085649.8196-11-songmuchun@bytedance.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?Qxd9HYN66bOS0O1JalOWUXsbvedp81X9hdcKxUUclLlugrq2WnPl5UXEssFd?=
 =?us-ascii?Q?M/5QABebkdwYDxoasgQc64L5DsLHgeOW1uz/QfeWv1drevSpGBPwfJsuwwep?=
 =?us-ascii?Q?vRMMHXXyIK5QbMEJF6x2rIJR5sLmR2WJdA/d47+Mu0LI6B/OB0ADvhFSAwSB?=
 =?us-ascii?Q?OCzJRVw73Z0052U5ewFzP0KvZlvP0zCcWWAhb9UXU+e5I2sSQ7InsJsnRt2r?=
 =?us-ascii?Q?Xn9Ar7Biuv0tCs3PWII/qFCnlbS4MxqDXl5cUlkDsEdj8Mr4Ly7rNEajXTGR?=
 =?us-ascii?Q?lEK6iwW7wlnSlYzxhw6uBMOdbJdo+k11Y6ukDz6VjwylArYkZhWSYbFJ3Z6c?=
 =?us-ascii?Q?uyfDdEGW7htaap8Le1X2lp311ttU/ozR+SUynArzS/dEaBHzelAai0glZnE1?=
 =?us-ascii?Q?wpuK/7QlWDJUlmuo8T5VKjtsHfCU27RSKl6k9YSMZ0kJShuv+wRJY+PdYZDc?=
 =?us-ascii?Q?/rl/VzitEtstl9Zia/JFu5ocm9v/os6DjEWJPzelU5jYm35nNm/RzHBMioyt?=
 =?us-ascii?Q?LZio2EzAaNs6WVHb7GXhH8TpjY6jn2Z5826bsdn+zt36lk628x1mvHJ4pwZm?=
 =?us-ascii?Q?bum9HFQpIMXeUP/TU85sjuhAed1baqHqByeJn2Mtpx8MMyLFPT0Ero/CUAg9?=
 =?us-ascii?Q?yQkVjvdUyO8iG6WnUNFTMqnpMqCurJv5znPnS3aRztb9mrJLAe9ZGwO2FKCF?=
 =?us-ascii?Q?SM3qN6dq+UMpFoeoO0VxEX+vARvb4tMTg8EGsph82REMXFkfh00J/0NSCnPz?=
 =?us-ascii?Q?XHA76aNn5a61EwpR+4lwJW7cn4qE/ujpn8D7e1bmn7f86sMN33ttggXtPYa+?=
 =?us-ascii?Q?IQICw1sIq9QXiS+aZoiDZ1cmAfaA4wqdsSe0PxOmEeQJRkW60ikFwqQFMDnt?=
 =?us-ascii?Q?oIw6cFbxw6Je0qHtI4WHEX/HoZFYSdIWgi56SXNbUKfiXN0/5VrLkMhhaeT3?=
 =?us-ascii?Q?ZfUo6Fj0xB7XIKjxI5r1B24uz235l3j2KwZH5yaU3Z7zTINCqthmE+9CHnVe?=
 =?us-ascii?Q?dLbk0V3YBQnKSjf8DlgYqLKsrLTjVRSHO+c1GEwnITU1VaN/PzG1l5sME9TL?=
 =?us-ascii?Q?fTI6fe1H5rWeB3zj01KuY7jJY+tZfSZBKsqOIe8Q5nPHYj2c9x+NoKSQ+QfI?=
 =?us-ascii?Q?Y64hZjQisLTA6Mu/8HjosdhctoFPW2LxztwdMjjqfAALkupK7ISZUdPTpo9X?=
 =?us-ascii?Q?FXiRFrPl8ve2LkLea4nbWOJb62qRWDH+cEOPH8KDyW3s+909qqgi+YbozG1T?=
 =?us-ascii?Q?RB4pZmmQmNvYqiB9t8e80S/q916R2Vm0x97weAJl5aNOUHAb3/U/mH53Qmqj?=
 =?us-ascii?Q?4EA7Y/V6iigym1/I7G0jZ3mEOZs42EBn5CJ4eeI77KRaMrTDzOp5Uorzk0Aq?=
 =?us-ascii?Q?0pdasKs2i8OMbhGYyDvKjiCG0EDedfMH6TKMKZ/ZaBTffR2PUQtJJwIJ+KAB?=
 =?us-ascii?Q?VvoSdPeHUXAHPCu4xMZufno8anPMVZlx/MtT5EY+hnMSFhpiC6NkivLigbHG?=
 =?us-ascii?Q?zW4GvUqWgzB5yA6AKtG951aXZepj7AA8zq5m3zqVBAs5UMODWdoPr2POAnh4?=
 =?us-ascii?Q?WUXwaEJVU3655Ih4voU4BwA/iNozxIffuZm+vuK+?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 63165696-02de-4a48-4afa-08d9d53d0564
X-MS-Exchange-CrossTenant-AuthSource: BYAPR15MB4136.namprd15.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 11 Jan 2022 20:00:31.2940
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 8ae927fe-1255-47a7-a2af-5f3a069daaa2
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: IHHJILAXVzfqFWszHVRfwX8gz3PRttP9Od5FY5Xeu3Sc2o8Un3tVLsL6kPn11v6f
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR15MB3125
X-OriginatorOrg: fb.com
X-Proofpoint-ORIG-GUID: bFokckKKrQ5YAOzxicQTugdD-CDDwhhA
X-Proofpoint-GUID: bFokckKKrQ5YAOzxicQTugdD-CDDwhhA
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.205,Aquarius:18.0.790,Hydra:6.0.425,FMLib:17.11.62.513
 definitions=2022-01-11_04,2022-01-11_01,2021-12-02_01
X-Proofpoint-Spam-Details: rule=fb_outbound_notspam policy=fb_outbound score=0 lowpriorityscore=0
 bulkscore=0 impostorscore=0 mlxscore=0 spamscore=0 malwarescore=0
 phishscore=0 mlxlogscore=740 priorityscore=1501 clxscore=1015 adultscore=0
 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2110150000 definitions=main-2201110105
X-FB-Internal: deliver
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org

On Mon, Dec 20, 2021 at 04:56:43PM +0800, Muchun Song wrote:
> In our server, we found a suspected memory leak problem. The kmalloc-32
> consumes more than 6GB of memory. Other kmem_caches consume less than
> 2GB memory.
> 
> After our in-depth analysis, the memory consumption of kmalloc-32 slab
> cache is the cause of list_lru_one allocation.
> 
>   crash> p memcg_nr_cache_ids
>   memcg_nr_cache_ids = $2 = 24574
> 
> memcg_nr_cache_ids is very large and memory consumption of each list_lru
> can be calculated with the following formula.
> 
>   num_numa_node * memcg_nr_cache_ids * 32 (kmalloc-32)
> 
> There are 4 numa nodes in our system, so each list_lru consumes ~3MB.
> 
>   crash> list super_blocks | wc -l
>   952
> 
> Every mount will register 2 list lrus, one is for inode, another is for
> dentry. There are 952 super_blocks. So the total memory is 952 * 2 * 3
> MB (~5.6GB). But the number of memory cgroup is less than 500. So I
> guess more than 12286 containers have been deployed on this machine (I
> do not know why there are so many containers, it may be a user's bug or
> the user really want to do that). And memcg_nr_cache_ids has not been
> reduced to a suitable value. This can waste a lot of memory.

But on the other side you increase the size of struct list_lru_per_memcg,
so if number of cgroups is close to memcg_nr_cache_ids, we can actually
waste more memory. I'm not saying the change is not worth it, but would be
nice to add some real-world numbers.

Or it's all irrelevant and is done as a preparation to the conversion to xarray?
If so, please, make it clear.

Thanks!