Received-SPF: pass (google.com: domain of linux-kernel+bounces-172611-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) client-ip=147.75.48.161;
Date: Wed, 8 May 2024 09:45:45 +0800
From: Xu Yang <xu.yang_2@nxp.com>
To: Christoph Hellwig <hch@lst.de>
Cc: Jens Axboe <axboe@kernel.dk>, "Darrick J. Wong" <djwong@kernel.org>,
	Andrew Morton <akpm@linux-foundation.org>,
	Matthew Wilcox <willy@infradead.org>,
	Christian Brauner <christian@brauner.io>,
	linux-block@vger.kernel.org, linux-fsdevel@vger.kernel.org,
	linux-xfs@vger.kernel.org, linux-mm@kvack.org,
	linux-kernel@vger.kernel.org, Luis Chamberlain <mcgrof@kernel.org>,
	Pankaj Raghav <p.raghav@samsung.com>,
	Hannes Reinecke <hare@suse.de>, jun.li@nxp.com, haibo.chen@nxp.com
Subject: Re: [PATCH 5/6] block: use iomap for writes to block devices
Message-ID: <20240508014545.mf7pexpctfl44pq3@hippo>
References: <20230801172201.1923299-1-hch@lst.de>
 <20230801172201.1923299-6-hch@lst.de>
 <20240426103727.hzzv4hv54an5jzab@hippo>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20240426103727.hzzv4hv54an5jzab@hippo>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?us-ascii?Q?rt3/TkLxLa52IOv9QkdHBOsryyYIgrH+Z0myHQ74gCs6G/07UvNp90isDCrX?=
 =?us-ascii?Q?D12a7z7Wwd8/UC0UynmStUpg74Ca7o3m++JHvIegOg+RpktPLqoP1z+0lFlm?=
 =?us-ascii?Q?v6opp2O9C5upsKCo1m/Y+fOke5cowwAo8p9zhCKBdR+vZQv/PRk/c5MzRJfg?=
 =?us-ascii?Q?H3NaphdUD4kfEoHVJrupm91dx0smYnMe8sSo5xQVTPA9rqATD9yYRGOu9W8Q?=
 =?us-ascii?Q?sdNkqC77zezdROmPhBg0VPam4Q6iR4TskdMdL4P1rhjQcnmQJ/9fMMXWaDHs?=
 =?us-ascii?Q?c+w+0qIJSR0k/vflAVnIZOFAfsD3ZJyuKoKdUJY9xfAfbtlo5WpZnLEjTgq9?=
 =?us-ascii?Q?Jyk6YTJZerNeISsBtH27ym6t3pF7Po+we1LQAyxfPD4q+CJu12z9tryMXG00?=
 =?us-ascii?Q?dLclk/lioHxSQb4ZGSRxfypq0oCxMp5PUH71D9LolJQDTUL/PB4CVuDQKiEy?=
 =?us-ascii?Q?aMucQM2XLeUt8CgD8/d9TrKCgylBneLneeDu04nkX2xlcVXu+GnMuZFToyvO?=
 =?us-ascii?Q?j64cv9cDB8UH7/NvXEVaFhlE/7oeAI0lZ2RMklWWIxHvQCwP+d8EJVcWdsR3?=
 =?us-ascii?Q?r9JgZ6VgUI4j8jIiahTqwaJIDrbMCa9H56VpiO5ZTv5LswLLGYiKvG+sZaVB?=
 =?us-ascii?Q?Coh+gQbbvDt4BNSih9Ybbp1fpgjie3JnhEjajnGiL6Af3uT7Yqi5HE/lYuOE?=
 =?us-ascii?Q?NkebrqVwI8nWecOB6TSGrzzyUVS/uPoK/xJNbOXoUxcj7I3vlp8un7kLJGGz?=
 =?us-ascii?Q?EA2ChG94hbEnfdsAPoAA/FbSv0TSyVbPnQuNZ74xC2cgoat1VyIjOZ/w8yAv?=
 =?us-ascii?Q?K04xBifHv0D0QIiYDmsY1NBBi26mzJdjv0ijEOR7UUvLqR18LlLIQ6T+Ievn?=
 =?us-ascii?Q?5k/Oc1XrCshgkTrXz4AqwYNjgTpI1vVt3ZihRDGDy5otCiTTP2vatquzR1b1?=
 =?us-ascii?Q?vDQ6YPzdRSARykmm+H5ITNKMBvkwUDXRyaJjCSsJ3c4VPx8GlLkmYAnBPqZP?=
 =?us-ascii?Q?i9HgD8K4cGHOTkdSlPez935E092YRmAT2rQa7YTRc/xOYausvkiovzlygeb1?=
 =?us-ascii?Q?4N4jK7zaFHo7XRtdtTyuoh5Q0xDQpExGlMhNPkgWRJNsbDI+UdET5LR3GQLa?=
 =?us-ascii?Q?rcfE1dP9KlAMIYUqsjyCXBMT2GL8w3M2f0GmcWPeLCooULJDW6kTDEGX5tLK?=
 =?us-ascii?Q?Swy9+HJZy8uUyTuxPocmJWtbVNY2sigZ9i1887FhM3We1/BjU15L4B/EN2jB?=
 =?us-ascii?Q?6JlqtKbcpkqG/ySGmXHCMvsgApAYlRwS7MrArIcB4gCPnkrHtIyxn2E1AuK6?=
 =?us-ascii?Q?NfUNjjVwagUaNWYnmI9mAZ0FhJbU7oAAfGytf0DGDYaGl5KbUK73fCWDlsYR?=
 =?us-ascii?Q?mLAPH7t7Z/X+CHVZY4bT70ET6GILlkL8m7nwwjrSOmzt2DFWACKCHjRFvdXj?=
 =?us-ascii?Q?DIBmqPh4vGUNf80UpZcH0DlRFMU5clAMs7/oDKIwvqiJVxSmayNofX7u4bFR?=
 =?us-ascii?Q?Utu6OKfDlCHOk6UjeRDqRTxfMddZS2hnhI3xGZcsUnbTYNkqnq+KDyvrhXAS?=
 =?us-ascii?Q?RqFgc2GVXQsbuurVKtuFzSUhsMIIWbgIcPAIL/6q?=
X-OriginatorOrg: nxp.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 07a3f267-91ed-43f8-0694-08dc6f00c09f
X-MS-Exchange-CrossTenant-AuthSource: DU2PR04MB8822.eurprd04.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 May 2024 01:47:00.6595
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 686ea1d3-bc2b-4c6f-a92c-d99c5c301635
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: aJoujG8j3qKulQ3GjKDQDpi3qske2C5f64DMklRNX+XddggvLBUNYku980GkpG57SHJeMZHrrsjEh0jy6RqsSg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DU2PR04MB9524

On Fri, Apr 26, 2024 at 06:37:27PM +0800, Xu Yang wrote:
> Hi Christoph,
> 
> On Tue, Aug 01, 2023 at 07:22:00PM +0200, Christoph Hellwig wrote:
> > Use iomap in buffer_head compat mode to write to block devices.
> > 
> > Signed-off-by: Christoph Hellwig <hch@lst.de>
> > Reviewed-by: Luis Chamberlain <mcgrof@kernel.org>
> > Reviewed-by: Pankaj Raghav <p.raghav@samsung.com>
> > Reviewed-by: Hannes Reinecke <hare@suse.de>
> > ---
> >  block/Kconfig |  1 +
> >  block/fops.c  | 31 +++++++++++++++++++++++++++++--
> >  2 files changed, 30 insertions(+), 2 deletions(-)
> > 
> > diff --git a/block/Kconfig b/block/Kconfig
> > index 86122e459fe046..1a13ef0b1ca10c 100644
> > --- a/block/Kconfig
> > +++ b/block/Kconfig
> > @@ -5,6 +5,7 @@
> >  menuconfig BLOCK
> >         bool "Enable the block layer" if EXPERT
> >         default y
> > +       select FS_IOMAP
> >         select SBITMAP
> >         help
> >  	 Provide block layer support for the kernel.
> > diff --git a/block/fops.c b/block/fops.c
> > index f0b822c28ddfe2..063ece37d44e44 100644
> > --- a/block/fops.c
> > +++ b/block/fops.c
> > @@ -15,6 +15,7 @@
> >  #include <linux/falloc.h>
> >  #include <linux/suspend.h>
> >  #include <linux/fs.h>
> > +#include <linux/iomap.h>
> >  #include <linux/module.h>
> >  #include "blk.h"
> >  
> > @@ -386,6 +387,27 @@ static ssize_t blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter)
> >  	return __blkdev_direct_IO(iocb, iter, bio_max_segs(nr_pages));
> >  }
> >  
> > +static int blkdev_iomap_begin(struct inode *inode, loff_t offset, loff_t length,
> > +		unsigned int flags, struct iomap *iomap, struct iomap *srcmap)
> > +{
> > +	struct block_device *bdev = I_BDEV(inode);
> > +	loff_t isize = i_size_read(inode);
> > +
> > +	iomap->bdev = bdev;
> > +	iomap->offset = ALIGN_DOWN(offset, bdev_logical_block_size(bdev));
> > +	if (iomap->offset >= isize)
> > +		return -EIO;
> > +	iomap->type = IOMAP_MAPPED;
> > +	iomap->addr = iomap->offset;
> > +	iomap->length = isize - iomap->offset;
> > +	iomap->flags |= IOMAP_F_BUFFER_HEAD;
> > +	return 0;
> > +}
> > +
> > +static const struct iomap_ops blkdev_iomap_ops = {
> > +	.iomap_begin		= blkdev_iomap_begin,
> > +};
> > +
> >  static int blkdev_writepage(struct page *page, struct writeback_control *wbc)
> >  {
> >  	return block_write_full_page(page, blkdev_get_block, wbc);
> > @@ -556,6 +578,11 @@ blkdev_direct_write(struct kiocb *iocb, struct iov_iter *from)
> >  	return written;
> >  }
> >  
> > +static ssize_t blkdev_buffered_write(struct kiocb *iocb, struct iov_iter *from)
> > +{
> > +	return iomap_file_buffered_write(iocb, from, &blkdev_iomap_ops);
> > +}
> > +
> >  /*
> >   * Write data to the block device.  Only intended for the block device itself
> >   * and the raw driver which basically is a fake block device.
> > @@ -605,9 +632,9 @@ static ssize_t blkdev_write_iter(struct kiocb *iocb, struct iov_iter *from)
> >  		ret = blkdev_direct_write(iocb, from);
> >  		if (ret >= 0 && iov_iter_count(from))
> >  			ret = direct_write_fallback(iocb, from, ret,
> > -					generic_perform_write(iocb, from));
> > +					blkdev_buffered_write(iocb, from));
> >  	} else {
> > -		ret = generic_perform_write(iocb, from);
> > +		ret = blkdev_buffered_write(iocb, from);
> >  	}
> >  
> >  	if (ret > 0)
> 
> I'm testing SSD block device write performance recently. I found the write
> speed descrased greatly on my board (330MB/s -> 130MB/s). Then I spent some
> time to find cause, finally find that it's caused by this patch and if I
> revert this patch, write speed can recover to 330MB/s.
> 
> I'm using below command to test write performance:
> dd if=/dev/zero of=/dev/sda bs=4M count=1024
> 
> And I also do more tests to get more findings. In short, I found write
> speed changes with the "bs=" parameter.
> 
> I totally write 4GB data to sda for each test, the results as below:
> 
>  - dd if=/dev/zero of=/dev/sda bs=400K  count=10485  (334 MB/s)
>  - dd if=/dev/zero of=/dev/sda bs=800K  count=5242   (278 MB/s)
>  - dd if=/dev/zero of=/dev/sda bs=1600K count=2621   (204 MB/s)
>  - dd if=/dev/zero of=/dev/sda bs=2200K count=1906   (170 MB/s)
>  - dd if=/dev/zero of=/dev/sda bs=3000K count=1398   (150 MB/s)
>  - dd if=/dev/zero of=/dev/sda bs=4500K count=932    (139 MB/s)
> 
> When this patch reverted, I got below results:
> 
>  - dd if=/dev/zero of=/dev/sda bs=400K  count=10485  (339 MB/s)
>  - dd if=/dev/zero of=/dev/sda bs=800K  count=5242   (330 MB/s)
>  - dd if=/dev/zero of=/dev/sda bs=1600K count=2621   (332 MB/s)
>  - dd if=/dev/zero of=/dev/sda bs=2200K count=1906   (333 MB/s)
>  - dd if=/dev/zero of=/dev/sda bs=3000K count=1398   (333 MB/s)
>  - dd if=/dev/zero of=/dev/sda bs=4500K count=932    (333 MB/s)
> 
> I just want to know if this results is expected when uses iomap, or it's
> a real issue?
> 
> Many thanks in advance!

A gentle ping.

> 
> Best Regards,
> Xu Yang
> 
> > -- 
> > 2.39.2
> >