Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
IronPort-SDR: 78SL++A2mlZf8upMwhcrXnQ4ONmR9lJvn8lJ+kI5Dk7cl9fFfC/VykY1DRWS26TlAzVr9cErEj
 y1SN7s3ZaT1fFVPoJKY0BnN1WEBuj7mR3BV6JHc0+VACHjeqW0UYTBI++Fz3m+iB8lkka0YMlK
 eEwsAIpRUk5P8XiMui+3yoHyFggXqlRjVqUeWSaApiM9gKN5l4THd5IA9/JdOKg+v97Q/ctcjd
 yTLT2N+BxjO6y2iecg443kVXK44Enkqo2Ohp/FKYE2ASRPvnTTwIBEahzNVo3D7Scp4dGBQTkF
 Ty0=
From:   Damien Le Moal <Damien.LeMoal@wdc.com>
To:     Changheun Lee <nanich.lee@samsung.com>,
        "bvanassche@acm.org" <bvanassche@acm.org>,
        Johannes Thumshirn <Johannes.Thumshirn@wdc.com>,
        "asml.silence@gmail.com" <asml.silence@gmail.com>,
        "axboe@kernel.dk" <axboe@kernel.dk>,
        "gregkh@linuxfoundation.org" <gregkh@linuxfoundation.org>,
        "hch@infradead.org" <hch@infradead.org>,
        "linux-block@vger.kernel.org" <linux-block@vger.kernel.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "ming.lei@redhat.com" <ming.lei@redhat.com>,
        "osandov@fb.com" <osandov@fb.com>,
        "patchwork-bot@kernel.org" <patchwork-bot@kernel.org>,
        "tj@kernel.org" <tj@kernel.org>,
        "tom.leiming@gmail.com" <tom.leiming@gmail.com>
CC:     "jisoo2146.oh@samsung.com" <jisoo2146.oh@samsung.com>,
        "junho89.kim@samsung.com" <junho89.kim@samsung.com>,
        "mj0123.lee@samsung.com" <mj0123.lee@samsung.com>,
        "seunghwan.hyun@samsung.com" <seunghwan.hyun@samsung.com>,
        "sookwan7.kim@samsung.com" <sookwan7.kim@samsung.com>,
        "woosung2.lee@samsung.com" <woosung2.lee@samsung.com>,
        "yt0928.kim@samsung.com" <yt0928.kim@samsung.com>
Subject: Re: [PATCH v8] bio: limit bio max size
Thread-Topic: [PATCH v8] bio: limit bio max size
Thread-Index: AQHXNpXqF7AsXoOwSUG6kQu0CPLJbw==
Date:   Fri, 23 Apr 2021 11:36:44 +0000
Message-ID: <BL0PR04MB65143E72BE89BD60334E994AE7459@BL0PR04MB6514.namprd04.prod.outlook.com>
References: <CGME20210421100544epcas1p13c2c86e84102f0955dd591f72e45756a@epcas1p1.samsung.com>
 <20210421094745.29660-1-nanich.lee@samsung.com>
Accept-Language: en-US
Content-Language: en-US
wdcipoutbound: EOP-TRUE
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-Network-Message-Id: 6484402a-4767-4ead-447d-08d9064c127d
X-MS-Exchange-CrossTenant-originalarrivaltime: 23 Apr 2021 11:36:44.5822
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: b61c8803-16f3-4c35-9b17-6f65f441df86
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: LLiR5yXUwMHX2b/5fjP5MDF8LC39NBD7K/4joy2h2bzIHAr3bGsnvTAf2HTq/0sk2i1Q+tr75Khk7Uw7xQZN2g==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL0PR04MB4740
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On 2021/04/21 19:05, Changheun Lee wrote:=0A=
> bio size can grow up to 4GB when muli-page bvec is enabled.=0A=
> but sometimes it would lead to inefficient behaviors.=0A=
> in case of large chunk direct I/O, - 32MB chunk read in user space -=0A=
> all pages for 32MB would be merged to a bio structure if the pages=0A=
> physical addresses are contiguous. it makes some delay to submit=0A=
> until merge complete. bio max size should be limited to a proper size.=0A=
=0A=
And what is the "proper size" ? You should be specific here and mention=0A=
max_sectors or max_hw_sectors.=0A=
=0A=
> =0A=
> When 32MB chunk read with direct I/O option is coming from userspace,=0A=
> kernel behavior is below now in do_direct_IO() loop. it's timeline.=0A=
> =0A=
>  | bio merge for 32MB. total 8,192 pages are merged.=0A=
>  | total elapsed time is over 2ms.=0A=
>  |------------------ ... ----------------------->|=0A=
>                                                  | 8,192 pages merged a b=
io.=0A=
>                                                  | at this time, first bi=
o submit is done.=0A=
>                                                  | 1 bio is split to 32 r=
ead request and issue.=0A=
>                                                  |--------------->=0A=
>                                                   |--------------->=0A=
>                                                    |--------------->=0A=
>                                                               ......=0A=
>                                                                    |-----=
---------->=0A=
>                                                                     |----=
----------->|=0A=
>                           total 19ms elapsed to complete 32MB read done f=
rom device. |=0A=
> =0A=
> If bio max size is limited with 1MB, behavior is changed below.=0A=
> =0A=
>  | bio merge for 1MB. 256 pages are merged for each bio.=0A=
>  | total 32 bio will be made.=0A=
>  | total elapsed time is over 2ms. it's same.=0A=
>  | but, first bio submit timing is fast. about 100us.=0A=
>  |--->|--->|--->|---> ... -->|--->|--->|--->|--->|=0A=
>       | 256 pages merged a bio.=0A=
>       | at this time, first bio submit is done.=0A=
>       | and 1 read request is issued for 1 bio.=0A=
>       |--------------->=0A=
>            |--------------->=0A=
>                 |--------------->=0A=
>                                       ......=0A=
>                                                  |--------------->=0A=
>                                                   |--------------->|=0A=
>         total 17ms elapsed to complete 32MB read done from device. |=0A=
> =0A=
> As a result, read request issue timing is faster if bio max size is limit=
ed.=0A=
> Current kernel behavior with multipage bvec, super large bio can be creat=
ed.=0A=
> And it lead to delay first I/O request issue.=0A=
=0A=
The above message describes the problem very well, but there is no explanat=
ion=0A=
of the solution implemented. So one cannot check if the implementation matc=
hes=0A=
the intent. Please add a description of the solution, more detailed than ju=
st=0A=
saying "limit bio size".=0A=
=0A=
> =0A=
> Signed-off-by: Changheun Lee <nanich.lee@samsung.com>=0A=
> ---=0A=
>  block/bio.c            | 9 ++++++++-=0A=
>  block/blk-settings.c   | 5 +++++=0A=
>  include/linux/bio.h    | 4 +++-=0A=
>  include/linux/blkdev.h | 2 ++=0A=
>  4 files changed, 18 insertions(+), 2 deletions(-)=0A=
> =0A=
> diff --git a/block/bio.c b/block/bio.c=0A=
> index 50e579088aca..9e5061ecc317 100644=0A=
> --- a/block/bio.c=0A=
> +++ b/block/bio.c=0A=
> @@ -255,6 +255,13 @@ void bio_init(struct bio *bio, struct bio_vec *table=
,=0A=
>  }=0A=
>  EXPORT_SYMBOL(bio_init);=0A=
>  =0A=
> +unsigned int bio_max_size(struct bio *bio)=0A=
> +{=0A=
> +	struct request_queue *q =3D bio->bi_bdev->bd_disk->queue;=0A=
> +=0A=
> +	return q->limits.bio_max_bytes;=0A=
> +}=0A=
> +=0A=
>  /**=0A=
>   * bio_reset - reinitialize a bio=0A=
>   * @bio:	bio to reset=0A=
> @@ -866,7 +873,7 @@ bool __bio_try_merge_page(struct bio *bio, struct pag=
e *page,=0A=
>  		struct bio_vec *bv =3D &bio->bi_io_vec[bio->bi_vcnt - 1];=0A=
>  =0A=
>  		if (page_is_mergeable(bv, page, len, off, same_page)) {=0A=
> -			if (bio->bi_iter.bi_size > UINT_MAX - len) {=0A=
> +			if (bio->bi_iter.bi_size > bio_max_size(bio) - len) {=0A=
>  				*same_page =3D false;=0A=
>  				return false;=0A=
>  			}=0A=
> diff --git a/block/blk-settings.c b/block/blk-settings.c=0A=
> index b4aa2f37fab6..cd3dcb5afe50 100644=0A=
> --- a/block/blk-settings.c=0A=
> +++ b/block/blk-settings.c=0A=
> @@ -37,6 +37,7 @@ EXPORT_SYMBOL_GPL(blk_queue_rq_timeout);=0A=
>   */=0A=
>  void blk_set_default_limits(struct queue_limits *lim)=0A=
>  {=0A=
> +	lim->bio_max_bytes =3D UINT_MAX;=0A=
>  	lim->max_segments =3D BLK_MAX_SEGMENTS;=0A=
>  	lim->max_discard_segments =3D 1;=0A=
>  	lim->max_integrity_segments =3D 0;=0A=
> @@ -168,6 +169,10 @@ void blk_queue_max_hw_sectors(struct request_queue *=
q, unsigned int max_hw_secto=0A=
>  				 limits->logical_block_size >> SECTOR_SHIFT);=0A=
>  	limits->max_sectors =3D max_sectors;=0A=
>  =0A=
> +	if (check_shl_overflow(max_sectors, SECTOR_SHIFT,=0A=
> +				&limits->bio_max_bytes))=0A=
> +		limits->bio_max_bytes =3D UINT_MAX;=0A=
=0A=
	limits->bio_max_bytes =3D min_t(sector_t,=0A=
			(sector_t)max_sectors << SECTOR_SHIFT, UINT_MAX);=0A=
=0A=
is easier to understand in my opinion.=0A=
=0A=
> +=0A=
>  	q->backing_dev_info->io_pages =3D max_sectors >> (PAGE_SHIFT - 9);=0A=
>  }=0A=
>  EXPORT_SYMBOL(blk_queue_max_hw_sectors);=0A=
> diff --git a/include/linux/bio.h b/include/linux/bio.h=0A=
> index d0246c92a6e8..e5add63da3af 100644=0A=
> --- a/include/linux/bio.h=0A=
> +++ b/include/linux/bio.h=0A=
> @@ -106,6 +106,8 @@ static inline void *bio_data(struct bio *bio)=0A=
>  	return NULL;=0A=
>  }=0A=
>  =0A=
> +extern unsigned int bio_max_size(struct bio *bio);=0A=
=0A=
No need for extern.=0A=
=0A=
> +=0A=
>  /**=0A=
>   * bio_full - check if the bio is full=0A=
>   * @bio:	bio to check=0A=
> @@ -119,7 +121,7 @@ static inline bool bio_full(struct bio *bio, unsigned=
 len)=0A=
>  	if (bio->bi_vcnt >=3D bio->bi_max_vecs)=0A=
>  		return true;=0A=
>  =0A=
> -	if (bio->bi_iter.bi_size > UINT_MAX - len)=0A=
> +	if (bio->bi_iter.bi_size > bio_max_size(bio) - len)=0A=
>  		return true;=0A=
>  =0A=
>  	return false;=0A=
> diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h=0A=
> index 158aefae1030..c205d60ac611 100644=0A=
> --- a/include/linux/blkdev.h=0A=
> +++ b/include/linux/blkdev.h=0A=
> @@ -312,6 +312,8 @@ enum blk_zoned_model {=0A=
>  };=0A=
>  =0A=
>  struct queue_limits {=0A=
> +	unsigned int		bio_max_bytes;=0A=
=0A=
Please move this below in the structure together with all other fields that=
 are=0A=
unsigned int too.=0A=
=0A=
> +=0A=
>  	unsigned long		bounce_pfn;=0A=
>  	unsigned long		seg_boundary_mask;=0A=
>  	unsigned long		virt_boundary_mask;=0A=
> =0A=
=0A=
As far as I can tell, bio_max_bytes is UINT_MAX by default, becomes equal t=
o=0A=
max_hw_sectors when that limit is set by a driver and cannot take any other=
=0A=
value. So why introduce this new limit at all ? Why not use max_hw_sectors=
=0A=
directly as a bio size limit ?=0A=
=0A=
=0A=
=0A=
-- =0A=
Damien Le Moal=0A=
Western Digital Research=0A=