Received: by 2002:a05:6358:3188:b0:123:57c1:9b43 with SMTP id q8csp16900631rwd; Mon, 26 Jun 2023 17:18:18 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ6MAgPqtpfpCQoB2nj5DanyQT0B1bVcVnUh9MPqzGmQfAsyDm7lGyqsgaRmtnaOeBH2GiaJ X-Received: by 2002:a17:907:940e:b0:988:3ce8:c5bf with SMTP id dk14-20020a170907940e00b009883ce8c5bfmr19075002ejc.33.1687825098243; Mon, 26 Jun 2023 17:18:18 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1687825098; cv=none; d=google.com; s=arc-20160816; b=grjNBDtcZZu/Nas1nbrtf6eOQK7mL0mfjjTnL2icaEuUPimiwyvDsNDEfKT3znq/he n4zydB+SEpQRIwQU4YLGgR3x9ketnJgSZTFAmP6/c2+Kcj+mqon8tn6/AeJcaR2wIUXI hsRnkz1GmxGBEMchPcBdRj0JDZedqlGGffkmNVKgUFBu0cmvBzkk76y1GFGyDcCOkW6W YTbDqeqwzyLI8xii3JC0XyRPC/6bjmjK4DY2qBXnaEefbRb5fDf0Qq3KNMBLpqIbkviT 5FMjpjogZ0A0oz0GWnXh2241ulvgOMrfpsjd8ezw4z9+XTAVMH1sPmPBAxBE5Cb9Z4cM 6dog== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:in-reply-to:content-disposition:mime-version :references:message-id:subject:cc:to:from:date:dkim-signature; bh=U/tKJDeC9Ma6SKor5x7zCPydbUPIhxnfEKbFjUGAIjU=; fh=9vQoXMadWu5h3FxsPHSGhPFdWHcGvAAUtStLELiydqQ=; b=UQBI/o48ddP8/6RV2+5JqdmHgQ7IZP3PH674OVzdXxtumilHFSYPjtCowDUPCdikkr 8fGj2AztK45rBwUoI+GhYX2isJU/JQDUnQleP15f/5VKACp+zyoiNxXfDQmK9ttGsCMP kAG68Q1dxKc5IJh5tSJGvRedm5jZe33GCZFY5NCG54EM518lvrpwkMWDCXMwLpRT7nBO +gxoUiQnVKhOP/fUnw0b1MID0JaroZmKDQAGVm9ZUodRk5lJSBf9XrUuecT06LNOvyEs AFuhK5pgK8iyEk16mXSfwQn35YcQrsGX+mMTaT97sMEctWRUD6b37BHNijPZtU9N5osr ncLg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@infradead.org header.s=casper.20170209 header.b=vB3uUOC0; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id r16-20020a170906549000b00988c3881c33si3695454ejo.234.2023.06.26.17.17.53; Mon, 26 Jun 2023 17:18:18 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@infradead.org header.s=casper.20170209 header.b=vB3uUOC0; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229810AbjF0ANb (ORCPT + 99 others); Mon, 26 Jun 2023 20:13:31 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42094 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229729AbjF0ANa (ORCPT ); Mon, 26 Jun 2023 20:13:30 -0400 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3D19DE5A; Mon, 26 Jun 2023 17:13:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=In-Reply-To:Content-Type:MIME-Version: References:Message-ID:Subject:Cc:To:From:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=U/tKJDeC9Ma6SKor5x7zCPydbUPIhxnfEKbFjUGAIjU=; b=vB3uUOC03Z/e1lfLsyUOHxeQ7y ktAhXuhL6tHs4lLOLjxfGOmXj1JfrwyianGS2N0GVFskczZnGvNLVKqAbqX6qP7dhs5rsApb1CeZB 79U7VdiTafVYNFgjU1r1G68WOzmB8gd5l18sGALhgWVrUSWzyEV+1Gkhn92zbgTSIGk1rSBbnljhb /dKl23kxEFmQrZHe++tdrgO9MLkLSOAFvVWBW3pxAD/NPerkXXlNQ8D2m9w4Tq+hQ6vDxEVVgN2st rUO1dcdPKTT2ljMbUHse4nfZ+HWsFUH1yFtYzPLWyjolGSrdrx/6K7N0Fs9uJkhlu1xCniZK+Agz0 FgUoIkqw==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qDwKj-002BL7-Gf; Tue, 27 Jun 2023 00:13:25 +0000 Date: Tue, 27 Jun 2023 01:13:25 +0100 From: Matthew Wilcox To: Dave Chinner Cc: Marcelo Tosatti , linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, Alexander Viro , Christian Brauner , Christoph Hellwig , Jens Axboe , Frederic Weisbecker , Valentin Schneider , Leonardo Bras , Yair Podemsky , P J P Subject: Re: [PATCH] fs/buffer.c: remove per-CPU buffer_head lookup cache Message-ID: References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_NONE,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, Jun 27, 2023 at 09:30:09AM +1000, Dave Chinner wrote: > On Mon, Jun 26, 2023 at 07:47:42PM +0100, Matthew Wilcox wrote: > > On Mon, Jun 26, 2023 at 03:04:53PM -0300, Marcelo Tosatti wrote: > > > Upon closer investigation, it was found that in current codebase, lookup_bh_lru > > > is slower than __find_get_block_slow: > > > > > > 114 ns per __find_get_block > > > 68 ns per __find_get_block_slow > > > > > > So remove the per-CPU buffer_head caching. > > > > LOL. That's amazing. I can't even see why it's so expensive. The > > local_irq_disable(), perhaps? Your test case is the best possible > > one for lookup_bh_lru() where you're not even doing the copy. > > I think it's even simpler than that. > > i.e. the lookaside cache is being missed, so it's a pure cost and > the code is always having to call __find_get_block_slow() anyway. How does that happen? __find_get_block(struct block_device *bdev, sector_t block, unsigned size) { struct buffer_head *bh = lookup_bh_lru(bdev, block, size); if (bh == NULL) { /* __find_get_block_slow will mark the page accessed */ bh = __find_get_block_slow(bdev, block); if (bh) bh_lru_install(bh); The second (and all subsequent) calls to __find_get_block() should find the BH in the LRU. > IMO, this is an example of how lookaside caches are only a benefit > if the working set of items largely fits in the lookaside cache and > the cache lookup itself is much, much slower than a lookaside cache > miss. But the test code he posted always asks for the same buffer each time. So it should find it in the lookaside cache?