Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   Naohiro Aota <Naohiro.Aota@wdc.com>
To:     Sweet Tea Dorminy <sweettea-kernel@dorminy.me>
CC:     Chris Mason <clm@fb.com>, Josef Bacik <josef@toxicpanda.com>,
        David Sterba <dsterba@suse.com>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "linux-btrfs@vger.kernel.org" <linux-btrfs@vger.kernel.org>,
        "kernel-team@fb.com" <kernel-team@fb.com>,
        David Sterba <dsterba@suse.cz>
Subject: Re: [PATCH] btrfs: wait between incomplete batch allocations
Thread-Topic: [PATCH] btrfs: wait between incomplete batch allocations
Thread-Index: AQHYSichAkdsxoKIM0Khpq5R5KWda6zqUmYA
Date:   Mon, 11 Apr 2022 07:11:24 +0000
Message-ID: <20220411071124.zwtcarqngqqkdd6q@naota-xeon>
References: <07d6dbf34243b562287e953c44a70cbb6fca15a1.1649268923.git.sweettea-kernel@dorminy.me>
In-Reply-To: <07d6dbf34243b562287e953c44a70cbb6fca15a1.1649268923.git.sweettea-kernel@dorminy.me>
Accept-Language: ja-JP, en-US
Content-Language: en-US
wdcipoutbound: EOP-TRUE
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?KQOz6msJXgvd0qdtm4bSOIz2Eb9YSQoJUOpOZ/wG+EYDJtZO/yYt0tqxQ7zo?=
 =?us-ascii?Q?gQh8xRSGfg4qJYfes01HEpAD+s0hbA9PRiuZ7cPdti6Xugrd9JlNILiLLpRg?=
 =?us-ascii?Q?ejOine9MT3/5IXijg2xUmQZGKeDRc//B5Sk1DAmUdajkHMn/XC2UcXM4IWaf?=
 =?us-ascii?Q?3DOEHSQf2SDZ3YN1p9CBsL/oqrdQX6LvUvUz9K9EZUX94wRcTKZ5pdPlQYla?=
 =?us-ascii?Q?aKHrXuEteqjuLOfJfAlyL7asejKxTyxae7Y8XCBtVJb6/Mm1tb0MCZF4dcgR?=
 =?us-ascii?Q?d2w1dIUD0Aes5HNGrG2PMqWr2swunXx0/w3rMiQ3u4Rb7cj7CnCybDQAqxJF?=
 =?us-ascii?Q?qFaB3k7suzm5yOg2tG8xYQsKylZFAgqE40rbMPnY8neyEp+roZ4OQDLSo8//?=
 =?us-ascii?Q?JPFKADnNVwj2K7SbOVsYaPOUX7c2OiSiI7atOL2Gkgy57wCG6rgEmVpFVnqf?=
 =?us-ascii?Q?mLCDGH+2tD08Eg/Mxxb7dmKKY9SxC2t7eNnkRFDS0fKv3hp05LvE6gm7kxyP?=
 =?us-ascii?Q?bJInx3orv9FbvKh9ldRjp1+AoCVjav4uE3NVjT+wqfdW1R4pBqJWVT4f8/WR?=
 =?us-ascii?Q?84q/o4gkUU7rvTMOStg0nwANuYfPT7DQSKIU0YrTnU5gGK4ecgCONtjou6ky?=
 =?us-ascii?Q?OFQOiZj+cjznuMDZUqu1hbXmwbjsnSzKO8nrjJnlp99vEttrr8ToUMepCD7t?=
 =?us-ascii?Q?a4bKc9NATFIUSXCS7kq4qWby5pmLmCUokAJjmOmN0U31WeKBZlVl0m39x0BP?=
 =?us-ascii?Q?Vj/mpC/96Oh98KczSIN3PbFUfuHQfXSBB2UuOION7lvjlJSNW3vFwe9MbxFh?=
 =?us-ascii?Q?1i8wneMPRSmuuHiqrx2HNwzbilyGqFul7uDZRMhqXhTMNVz9IpGl5X+e7Z13?=
 =?us-ascii?Q?5mQYbbT3iazRiogH+VmYQkJrVvsXlpShJDXS8LIkV6PLHuGtTDzvsSULeX/r?=
 =?us-ascii?Q?f7yMWpnJZJMHSPT6bjpgDtlbWwQ00eWTx97VhhcXKbTV1MkoG3lmy5hdAuwL?=
 =?us-ascii?Q?b6pmcpamosOD9YHFG2XwiIPSysQ7f/TN/IVayWDNM4XAC8asqKOfeR3P46Dn?=
 =?us-ascii?Q?u2KlACO94REJ5IpwvWW0e4+MhzxpaSOKvPqE8ECBl2DPV33Q15iCgrQcLDum?=
 =?us-ascii?Q?MgDiL8vLy5CGEA25wr8hfkxM/XPQF4fpWgg45/63U78qCVyjqFNQ0V47+jYX?=
 =?us-ascii?Q?QU39JqKMb7DttWuky2MSRZwbkTAtU9lrmw7n7nksHERMvQVoLnR/8PbE14oO?=
 =?us-ascii?Q?onCJ+sedJ/4PfAoEipigVK3pVqCPrcqpTbEyeoqyPVOLGWkK3BVvatfhyAtQ?=
 =?us-ascii?Q?xwp7kyt5KiojLsSxQHtZ6pc0LeVfSnkcuNuabR1aB4P9O0otclDhtcj/5pxL?=
 =?us-ascii?Q?xiPddXpDx72dsTUDDLDq7xgw/zey2pOcBQXKCq4BLc4RkaDS/SoXi627DPdG?=
 =?us-ascii?Q?T8YYjjhTzqW9IpvJrDa9Y/iR0MoFojJQo+cvdFHqWIEElykFqDEIoClUT5Z1?=
 =?us-ascii?Q?RNeXC6qE6+HaUmQqowZkIZTqpg9+t73XvQjOPRg+gQBmovuh5lbn3+gzdPl7?=
 =?us-ascii?Q?i3p/6h2QpOLTQd7Lxqkph6UPGbpS8BqBc0ZjbAHo2U3j/aQQZaFvHu2oTn1W?=
 =?us-ascii?Q?ZEtlnVAcOuweRl7EJYAF0wDUsnrFgdIqR0K4tYWWloHD6yDmB2b3pE5TEpUl?=
 =?us-ascii?Q?YgIW1Rch9dtkwFsz9DdVCfWnc08I/qk5oYkk71XvNGZ0nH0bb0G9GgZZyxLq?=
 =?us-ascii?Q?ZCz9N3NjAIjCkZuyquQVtOuxlmDSojg=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-ID: <CF0374E0302ACB40AB70EB25D8C1BBDF@namprd04.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: wdc.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: SJ0PR04MB7776.namprd04.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: adfe9a0e-4f11-4759-b914-08da1b8a7d5e
X-MS-Exchange-CrossTenant-originalarrivaltime: 11 Apr 2022 07:11:24.7893
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: b61c8803-16f3-4c35-9b17-6f65f441df86
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: HIKzBCNgbv8ubgl4IYopk7mnP4sOexe6bamXdHPILmG+mGsCfTFCRTNUn9bcBhJDI8oSrk86zlAzwpBIS+RjHg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY4PR0401MB3587
X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,RCVD_IN_DNSWL_MED,SPF_HELO_PASS,SPF_NONE,
        T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Wed, Apr 06, 2022 at 02:24:18PM -0400, Sweet Tea Dorminy wrote:
> When allocating memory in a loop, each iteration should call
> memalloc_retry_wait() in order to prevent starving memory-freeing
> processes (and to mark where allcoation loops are). ext4, f2fs, and xfs
> all use this function at present for their allocation loops; btrfs ought
> also.
>=20
> The bulk page allocation is the only place in btrfs with an allocation
> retry loop, so add an appropriate call to it.
>=20
> Suggested-by: David Sterba <dsterba@suse.cz>
> Signed-off-by: Sweet Tea Dorminy <sweettea-kernel@dorminy.me>

The fstests btrfs/187 becomes incredibly slow with this patch applied.

For example, on a nvme ZNS SSD (zoned) device, it takes over 10 hours to
finish the test case. It only takes 765 seconds if I revert this commit
from the misc-next branch.

I also confirmed the same slowdown occurs on regular btrfs. For the
baseline, with this commit reverted, it takes 335 seconds on 8GB ZRAM
device running on QEMU (8GB RAM), and takes 768 seconds on a (non-zoned)
HDD running on a real machine (128GB RAM). The tests on misc-next with the
same setup above is still running, but it already took 2 hours.

The test case runs full btrfs sending 5 times and incremental btrfs sending
10 times at the same time. Also, dedupe loop and balance loop is running
simultaneously while all the send commands finish.

The slowdown of the test case basically comes from slow "btrfs send"
command. On the HDD run, it takes 25 minutes to run a full btrfs sending
command and 1 hour 18 minutes to run a incremental btrfs sending
command. Thus, we will need 78 minutes x 5 =3D 6.5 hours to finish all the
send commands, making the test case incredibly slow.

> ---
>  fs/btrfs/extent_io.c | 3 +++
>  1 file changed, 3 insertions(+)
>=20
> diff --git a/fs/btrfs/extent_io.c b/fs/btrfs/extent_io.c
> index 9f2ada809dea..4bcc182744e4 100644
> --- a/fs/btrfs/extent_io.c
> +++ b/fs/btrfs/extent_io.c
> @@ -6,6 +6,7 @@
>  #include <linux/mm.h>
>  #include <linux/pagemap.h>
>  #include <linux/page-flags.h>
> +#include <linux/sched/mm.h>
>  #include <linux/spinlock.h>
>  #include <linux/blkdev.h>
>  #include <linux/swap.h>
> @@ -3159,6 +3160,8 @@ int btrfs_alloc_page_array(unsigned int nr_pages, s=
truct page **page_array)
>  		 */
>  		if (allocated =3D=3D last)
>  			return -ENOMEM;
> +
> +		memalloc_retry_wait(GFP_NOFS);

And, I just noticed this is because we are waiting for the retry even if we
successfully allocated all the pages. We should exit the loop if (allocated
=3D=3D nr_pages).

>  	}
>  	return 0;
>  }
> --=20
> 2.35.1
> =