Received: by 2002:a05:6500:1b41:b0:1fb:d597:ff75 with SMTP id cz1csp407347lqb; Tue, 4 Jun 2024 15:25:50 -0700 (PDT) X-Forwarded-Encrypted: i=3; AJvYcCWxZQUjMHn8jTu4yI3lIgAE0emaXUHOS9DHc6CqUW+Z7U4qFJti5XJDS9qtL0CuKtM+WeuoGchgoDpD/7BX3zDO688Q/IKDNXnI9GjOtQ== X-Google-Smtp-Source: AGHT+IHis8avM5xpDYQbsLjpX9RsTU4jF3TQmyW6NcaqhymVPHnaOScekcreusFw7+7Tdi01tVbK X-Received: by 2002:a05:6a20:914b:b0:1af:f497:8225 with SMTP id adf61e73a8af0-1b2b6fbd251mr1257490637.31.1717539949771; Tue, 04 Jun 2024 15:25:49 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1717539949; cv=pass; d=google.com; s=arc-20160816; b=KdtI8uBai1ToWJgIPCvuA1D2UJJpt6jSKCocBE0KcWL9OSeVI/0SC0LlEgNtRb1O1u UrYw9GyLYmyK7UaOgO1wQo4HC+wVlfQyvZO/yptRq5UB7SWopZcwUAFBzM9/m16ZSA5K Z68yNMBWbJmtpzX9TvcZ8HWd/nLysrwR7TuhxRrZ2x3L5YEvPqoBk/YjGdGqrRvfrM8K Ayg5/xIs9DY/J80F77ReCKDv5urtwVeSHZlu4wwdbK+ORPwFRicba+yMhAadC33lsRP7 TUz0RBj37l2ovVGLXSCtxo3ZD1E/cfoYeT7pYFhwAv4B1xfyb7zt/uqXmFX1pMJBMQlU 8uvw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :subject:cc:to:from:date:dkim-signature; bh=guuxTy+CZVyMI/LzvDTj+NobCvDyE9ykfdNc4s7xxKU=; fh=3xOvTPqcmA/bWLzK4IrAqOX4WYVPwnmbJKYsoP0Tu6g=; b=ecXrxJLYzPmMilT40STABBtyVDEFGS3k2WPTzB7FCU7BKxedj0KokvM+3xObYOwo/R KQ9TaIrXk3vnnKrVBB2DqADimkSSfBMnZRynsl5eDTBCwK49IceygW/rAzXmgXIBslql EBfmac1TFn+NRTVmnf8uSnS4xzNAK01/9DafZKc524/CWQPA6WM9mpDElvXsQWfO7d+0 BusFFsNkgbBArLx9IYxVbMRjFunQSyg49wAudMn2UigLiPczD4PV6UBb/AvghHY5A1GH iNL0J4vmL+1xKplEhkcYICCIxuUvyig/yhCXubuaRV1ufhHPsyKm1ZAcYLbLH128jwtE EBUg==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@mailbox.org header.s=mail20150812 header.b=Mz4zL+fk; arc=pass (i=1 spf=pass spfdomain=mailbox.org dkim=pass dkdomain=mailbox.org dmarc=pass fromdomain=mailbox.org); spf=pass (google.com: domain of linux-kernel+bounces-201454-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-201454-linux.lists.archive=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=mailbox.org Return-Path: Received: from sy.mirrors.kernel.org (sy.mirrors.kernel.org. [2604:1380:40f1:3f00::1]) by mx.google.com with ESMTPS id 41be03b00d2f7-6c35cdf4518si3033052a12.788.2024.06.04.15.25.49 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 04 Jun 2024 15:25:49 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel+bounces-201454-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) client-ip=2604:1380:40f1:3f00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@mailbox.org header.s=mail20150812 header.b=Mz4zL+fk; arc=pass (i=1 spf=pass spfdomain=mailbox.org dkim=pass dkdomain=mailbox.org dmarc=pass fromdomain=mailbox.org); spf=pass (google.com: domain of linux-kernel+bounces-201454-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-201454-linux.lists.archive=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=mailbox.org Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sy.mirrors.kernel.org (Postfix) with ESMTPS id 015FAB2717B for ; Tue, 4 Jun 2024 22:19:09 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 538CB14D2BD; Tue, 4 Jun 2024 22:18:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=mailbox.org header.i=@mailbox.org header.b="Mz4zL+fk" Received: from mout-p-201.mailbox.org (mout-p-201.mailbox.org [80.241.56.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1607514D2B2 for ; Tue, 4 Jun 2024 22:17:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=80.241.56.171 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717539479; cv=none; b=IkooezudTCtoEpYYK1b5tpZk2e7pPrUvVHq2OBEpBP2uRna02d7WcwzYY+tVmR3y/ocLAKbEqxs4whpekL/fHjILxIezZ/p/7ZoGKf0RYcbJT630rL44w3zWxItTZAql6Cj0bhhJ0XP2Ozo8Ds/UXmOHNaB8Za46hW09RCaXMlg= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717539479; c=relaxed/simple; bh=DK4CPWNPGd3mRzWeANhjnOCdP3YfV6KKDwPWj2sjMFE=; h=Date:From:To:Cc:Subject:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=hYzHewmcyTf/8A4xuHUSV34hIhQTfwzos79JSSTCK0Jgb08oDso6QolAYzLF26+fDNz8GPPBziStkrbef8eiNFaUGMbyR5vgVdtGhgXooBSJlorrw+XZu0X6lEIDT7EtjwxGK34oKocbKCShRcEdqL8MCutvlLGmqzVKWxYrVUg= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=mailbox.org; spf=pass smtp.mailfrom=mailbox.org; dkim=pass (2048-bit key) header.d=mailbox.org header.i=@mailbox.org header.b=Mz4zL+fk; arc=none smtp.client-ip=80.241.56.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=mailbox.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=mailbox.org Received: from smtp2.mailbox.org (smtp2.mailbox.org [IPv6:2001:67c:2050:b231:465::2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mout-p-201.mailbox.org (Postfix) with ESMTPS id 4Vv4jD2Rznz9sSN; Wed, 5 Jun 2024 00:17:52 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mailbox.org; s=mail20150812; t=1717539472; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=guuxTy+CZVyMI/LzvDTj+NobCvDyE9ykfdNc4s7xxKU=; b=Mz4zL+fkVn0xahQamRTxt5EodXqPnJdgpFxGEsfrP3tDTxMI+LHndT9Z8o1KAcwBPp1B+l VU4Ka0eWCs5sSk2DCHoMRZ9YG/UMSnfp+i/eI0pTcb4jNdwKT8hV9RltP54v1PsltDNWnW 6yOJkScrh8ZisDXePiSziqOVyxJXk9+kY2vGYc6mtMtDzXUlLFkRvrxB8Xxcj4QmBcmW4n O8e/JjOBVk/A3feAyjNQev4hhGBaBkOZpIQihowArQFMCW77Un3k+b82g91js6V+DOQPW2 zV4WyuOhiKTPb711dsqV9bAIuljb1AvFtHgRM85SqodWENpIC8QQPhSwrj1JlQ== Date: Wed, 5 Jun 2024 00:17:47 +0200 From: Erhard Furtner To: Yu Zhao Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, Johannes Weiner , Nhat Pham , Chengming Zhou , Sergey Senozhatsky , Minchan Kim , Yosry Ahmed Subject: Re: kswapd0: page allocation failure: order:0, mode:0x820(GFP_ATOMIC), nodemask=(null),cpuset=/,mems_allowed=0 (Kernel v6.5.9, 32bit ppc) Message-ID: <20240605001747.2d3e13fa@yea> In-Reply-To: References: <20240508202111.768b7a4d@yea> <20240515224524.1c8befbe@yea> <20240602200332.3e531ff1@yea> <20240604001304.5420284f@yea> <20240604134458.3ae4396a@yea> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-MBO-RS-META: ik41k7m3xcnx6y18u6tq8497mnd3iouf X-MBO-RS-ID: e7369f9761aeea9ca46 On Tue, 4 Jun 2024 11:18:25 -0600 Yu Zhao wrote: > Hi Erhard, > > If it's not too much trouble, could you "grep nr_zspages /proc/vmstat" > on kernels before and after the bad commit? It'd be great if you could > run the grep command right before the OOM kills. > > The overall internal fragmentation of multiple zsmalloc pools might be > higher than a single one. I suspect this might be the cause. > > Thank you. I used watch -n1 'grep nr_zspages /proc/vmstat' to get the readings and repeated this 3 times to check whether the reported values differ much. Bad commit was b8cf32dc6e8c75b712cbf638e0fd210101c22f17 "mm: zswap: multiple zpools support", next commit in git log after the bad one would be 42c06a0e8ebe95b81e5fb41c6556ff22d9255b0c "mm: kill frontswap". With this kernel I got 2440, 2337, 3245 as nr_zspages and the kswapd0: page allocation error. Commit in git log before the bad one would be bfaa4a0ce1bbc1b2b67de7e4c2e1679495f7b905 "scsi: gvp11: Remove unused gvp11_setup() function". With this kernel I got 25537, 11321, 16087 as nr_zspages and the OOM reaper. Tomorrow I could also check the .config changes Vlastimil suggested (CONFIG_SLAB_MERGE_DEFAULT=y, no CONFIG_RANDOM_KMALLOC_CACHES) and report back if that's of interest. Regards, Erhard