Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) client-ip=23.128.96.37;
Message-ID: <36ee54b4-b8d5-4b3c-81a0-cc824b6ef68e@oracle.com>
Date:   Wed, 13 Dec 2023 09:13:48 +0000
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH v2 01/16] block: Add atomic write operations to
 request_queue limits
To:     Ming Lei <ming.lei@redhat.com>
Cc:     axboe@kernel.dk, kbusch@kernel.org, hch@lst.de, sagi@grimberg.me,
        jejb@linux.ibm.com, martin.petersen@oracle.com, djwong@kernel.org,
        viro@zeniv.linux.org.uk, brauner@kernel.org, dchinner@redhat.com,
        jack@suse.cz, linux-block@vger.kernel.org,
        linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org,
        linux-xfs@vger.kernel.org, linux-fsdevel@vger.kernel.org,
        tytso@mit.edu, jbongio@google.com, linux-scsi@vger.kernel.org,
        jaswin@linux.ibm.com, bvanassche@acm.org,
        Himanshu Madhani <himanshu.madhani@oracle.com>
References: <20231212110844.19698-1-john.g.garry@oracle.com>
 <20231212110844.19698-2-john.g.garry@oracle.com> <ZXkIEnQld577uHqu@fedora>
Content-Language: en-US
From:   John Garry <john.g.garry@oracle.com>
Organization: Oracle Corporation
In-Reply-To: <ZXkIEnQld577uHqu@fedora>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?bHJQcyszSFQ5VlZKdjdoUENjOWUyVmE5RTlFN1NPcmZobGdMTThwZng1bzVr?=
 =?utf-8?B?a2cvbnlOcm1KdWRoV0dIbk5YOVRaWWJ1L09NSE9TenFxYVhUQnZKOXphOXN5?=
 =?utf-8?B?SEtTSzd5bXRxb0NYNFJKRDZXQkczd3ZUeTVsUm9vNS9zMkZybDVNQmN6Ukdn?=
 =?utf-8?B?MmRpVXk0YU5IdkI0bHhidmRzZ1lHU2lFbFZkMDdUenl2aU1PMlhqSUZmU3dO?=
 =?utf-8?B?a3RyZ1l0RS94T2hRYUc2VmRtclcrTTFWcnlSVjdkdmpYQzlpekxhVi9jZXpD?=
 =?utf-8?B?dEZHSmhadlFrNThSZFFoQlY2S0I2ZGFZQXZmYTQwcTdqaXVuV3R0dElVaUdu?=
 =?utf-8?B?cm1iU3dXWEUxUXE4c2U4enpidzlQMWVTbmFocDdnT3dkWlhWK21sMzJkUkN6?=
 =?utf-8?B?QXlOOG9Sd0J3K3JXdnB2SXZrTUZUalk3MDUrajRVMlVqc2Vjc0k5cTIwMDF4?=
 =?utf-8?B?NDRRNGp1TnVnUmxtVFRkNjNBUWw2MTNDdDlJUFJnUjdHZTBGM1N5cmM0cVVX?=
 =?utf-8?B?MnhMdlJkekVkN2U1S3I3QmpSVjY4N3U2REx0Y0NZdVpuRmRqNVNnNUQ1T2o1?=
 =?utf-8?B?MmVaODZwK01EYzF5T2J4NDM3K0pQbGg4d1I4eGI4ZjI1NWZKclNVa1EyQ0F3?=
 =?utf-8?B?Zk9jek9WSnNYcXRGZXJzZGk5N00rMHVxZUVraGNuck1hWnBadnNJamZnS2Fj?=
 =?utf-8?B?MnRhRGVWZU13L1pibkUwdU1MMWh6cC9GcmNNaENTZjRCZW5ETzcvOTd2SFpU?=
 =?utf-8?B?bW9pKy9PVlhBaGE4SEYxdzZzVUJPbW8yS0xndUxDWUlYaXJDRDBWZHFnZHZR?=
 =?utf-8?B?bGtqdzZIcmEyOG1nQ2J4bm5oZ2I3OENVbTd6WDZlODk4RXZoQXZDTktpejEv?=
 =?utf-8?B?RE0rc0w5TitSRmtsQU9kTE5RNVloWUc4M1VwbmY4cmllbFRjOXZ6TkJ4R3Nt?=
 =?utf-8?B?eHJYMFZxNkJKOC95UFRwOGh2cytqckhFNDNYdVMzWk93YTR4VURGZ3k0Y202?=
 =?utf-8?B?NGVRWU5abjlPdXdEYVU0V1R1aWIwL2FDVjF3SjdXbjVCcVg2RGNIdlU5QWp5?=
 =?utf-8?B?eUZDWTFyMEQrWHpDZTZBRHhyYkgrSDdVMHg3S3JxcGp4Tzc0dkptWW1TVjhx?=
 =?utf-8?B?MDJaQmRBVjU2MVErc0JMTUlmckFpK05mWVdzbktZbkJ6NHhyc3VQNHpjcHdB?=
 =?utf-8?B?U3ZhU3pKYXU0eWdITWZHMlR2SzBWdG1rYXFVTjhNMXhoZjcwb1lQckpqbWZm?=
 =?utf-8?B?eUM1NGUwbFFkL3E2VW9oQUZzdVN0NVV0QVo2cXNSMzF0T3hZTWdraENFWkY2?=
 =?utf-8?B?UU9TbXFTcjA3dHJPb2svQjE5T00reVF0MGx0cGROWTI4cE5uTy9OVjZIMDdH?=
 =?utf-8?B?eGZsUVgwemVjZzN1RjAwZDl3cDQrdlZEcjY4SjBxSEo4S1dEd0poVDhjTC9Y?=
 =?utf-8?B?dHFoc3EraHhuQzk0bEh5MzRwVXE2d0F6aGppU2djZ0MwcmdqQWM3ckdWT2l2?=
 =?utf-8?B?Szl3MjVhMllrajNpanFxaTRMeEZ4Lzdodm5oRDZkaE5rcTIrb0REZzVhd093?=
 =?utf-8?B?MUtxSTlGVVJUWkNYUDdPYzQzOGpxcWxwTmRlRlN0blovbnJMaDZHYjJmS3Zr?=
 =?utf-8?B?dWhPQy9RUzMwayt5bmR2Y0pWaUtBU2svK0FjeWx2TlJoNVo5YWJCcDdoYmVL?=
 =?utf-8?B?UkdzZUR3Z0hEaFovK2V2S0Y2OXdVZkg3TTZpeEFzaERYcWFXczJGOWVFSlla?=
 =?utf-8?B?bzZtMURzMjRmMXFpSlpwdUVSbCtieGFnYjRZaGpBd2dQaFRMZUJPMUQwcjhY?=
 =?utf-8?B?N2RYMEhLMmNnUFROUTN1Um1LbWpla1FpTmNsSUROWWlCN2JJdlRuT1UrTlhH?=
 =?utf-8?B?bW8vTUNaU1lxaEZQMnZMaVdrOExkeUc4M2FIY1lyVzZqcVRlc0k5L1RYVGYx?=
 =?utf-8?B?MU8rZXZZalpzZjhqdTg1blY2NEQ3K3RlWEhkZmlKNTBwN2tRVUlRcTluQ1Zs?=
 =?utf-8?B?cFVXZ1AvaVVRaXBwRVRJODA2M2QxYWc5Q3hsUXUyVUhMVEVKWW5HaU1tODRI?=
 =?utf-8?B?bFFsbWRYSjJaWHF5cVh5dUg0OVdrSnJrc05sUUltYk9FQUhYTW1LdzBlcDVL?=
 =?utf-8?B?WnI3a0lXa0xoRHd5WVNvN0REcCtmNEZYeExTWFIwUUJ6Uk1uNG5Pb1RLamli?=
 =?utf-8?B?Ync9PQ==?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: dPUoLZjIvWy0e5F0dPxxYcfJM5mTvexgy5McQDEva84gE70ehdA3SiBfx/rKkuoJv4ETagPQobxwFakTxp8Fc8IKfGZG2M7daouSIKS8fQeWhtza+rrC86ZxT2bVV1xsvU7tHLCd7ZS56BhdnuaobfNkT3BHKJgWopnRueogGSkR3va1JvtYl/zoLjAlKCuweWjcwzAjnTeQaelLJib9hY5pywm+H02X6wdCkJhjczQRU7X+LRY1wBAfG/g/IkNsZmg1HclyYMoHR4gDdfNoC7d1eglpaJOkIpHJy0cOPsWt/x2Cxh24NxWHVMyWj7UOYLoLxK3Pxvyv3LVyas7fVmZmxmYV2FxCG08wNzQn1hp3cO5BbRsIXmLdr+xBqC21Yfv6BxMqAl49tqtMd4d16sD3mHFTfKBpKiFiv9lbpKj1UMAqyjm5NQFJLZLE7U5z9io6gnz7IU4wUr7by42TNaeVJcSAyoS39C5gYEhqNNesAus87D1E29jDfxuST8vDjovwg9B9dT2AhdkK0r21Q/EeOaj3mcxzaumQtidIEWO7YDtbKW404i3rUnWrqWY7Tmo+vYyw15QNdGoXb5h6+OMul0WNlAHoi8Gdp9ZyY0g=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 23674b0d-c8ab-464f-2719-08dbfbbbd36a
X-MS-Exchange-CrossTenant-AuthSource: DM6PR10MB4313.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 13 Dec 2023 09:13:52.9769
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: h2q338Y3EIo0w4MJ8PmPkBH/BCEc5q+p8Hn6PfYECEui1tx4MUafJ8VUXWpydkh6dmFC9K2aH/x9uLLahm49nQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CO6PR10MB5537
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26
 definitions=2023-12-13_01,2023-12-12_01,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=999 bulkscore=0 spamscore=0
 mlxscore=0 adultscore=0 phishscore=0 malwarescore=0 suspectscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000
 definitions=main-2312130066
X-Proofpoint-GUID: R4rGgWxaitsYiI58cDy50FIn_8O7zQPt
X-Proofpoint-ORIG-GUID: R4rGgWxaitsYiI58cDy50FIn_8O7zQPt
X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_LOW,
        RCVD_IN_MSPIKE_H5,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE,
        T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Wed, 13 Dec 2023 01:14:26 -0800 (PST)

>> +
>>   
>>   What:		/sys/block/<disk>/diskseq
>>   Date:		February 2021
>> diff --git a/block/blk-settings.c b/block/blk-settings.c
>> index 0046b447268f..d151be394c98 100644
>> --- a/block/blk-settings.c
>> +++ b/block/blk-settings.c
>> @@ -59,6 +59,10 @@ void blk_set_default_limits(struct queue_limits *lim)
>>   	lim->zoned = BLK_ZONED_NONE;
>>   	lim->zone_write_granularity = 0;
>>   	lim->dma_alignment = 511;
>> +	lim->atomic_write_unit_min_sectors = 0;
>> +	lim->atomic_write_unit_max_sectors = 0;
>> +	lim->atomic_write_max_sectors = 0;
>> +	lim->atomic_write_boundary_sectors = 0;
> 
> Can we move the four into single structure

There is no precedent for a similar structure in struct queue_limits. So 
would only passing a structure to the blk-settings.c API be ok?

> and setup them in single
> API? Then cross-validation can be done in this API.

I suppose so, if you think that it is better.

We rely on the driver to provide sound values. I suppose that we can 
sanitize them also (in a single API).

> 
>>   }
>>   
>>   /**
>> @@ -183,6 +187,62 @@ void blk_queue_max_discard_sectors(struct request_queue *q,
>>   }
>>   EXPORT_SYMBOL(blk_queue_max_discard_sectors);
>>   
>> +/**
>> + * blk_queue_atomic_write_max_bytes - set max bytes supported by
>> + * the device for atomic write operations.
>> + * @q:  the request queue for the device
>> + * @size: maximum bytes supported
>> + */
>> +void blk_queue_atomic_write_max_bytes(struct request_queue *q,
>> +				      unsigned int bytes)
>> +{
>> +	q->limits.atomic_write_max_sectors = bytes >> SECTOR_SHIFT;
>> +}
>> +EXPORT_SYMBOL(blk_queue_atomic_write_max_bytes);
> 
> What if driver doesn't call it but driver supports atomic write?

We rely on the driver to do this. Any basic level of testing will show 
an issue if they don't.

> 
> I guess the default max sectors should be atomic_write_unit_max_sectors
> if the feature is enabled.

Sure. If we have a single API to set all values, then we don't need to 
worry about this (assuming the values are filled in properly).

> 
>> +
>> +/**
>> + * blk_queue_atomic_write_boundary_bytes - Device's logical block address space
>> + * which an atomic write should not cross.
>> + * @q:  the request queue for the device
>> + * @bytes: must be a power-of-two.
>> + */
>> +void blk_queue_atomic_write_boundary_bytes(struct request_queue *q,
>> +					   unsigned int bytes)
>> +{
>> +	q->limits.atomic_write_boundary_sectors = bytes >> SECTOR_SHIFT;
>> +}
>> +EXPORT_SYMBOL(blk_queue_atomic_write_boundary_bytes);
> 
> Default atomic_write_boundary_sectors should be
> atomic_write_unit_max_sectors in case of atomic write?

Having atomic_write_boundary_sectors default to 
atomic_write_unit_max_sectors is effectively same as a default of 0.

> 
>> +
>> +/**
>> + * blk_queue_atomic_write_unit_min_sectors - smallest unit that can be written
>> + * atomically to the device.
>> + * @q:  the request queue for the device
>> + * @sectors: must be a power-of-two.
>> + */
>> +void blk_queue_atomic_write_unit_min_sectors(struct request_queue *q,
>> +					     unsigned int sectors)
>> +{
>> +	struct queue_limits *limits = &q->limits;
>> +
>> +	limits->atomic_write_unit_min_sectors = sectors;
>> +}
>> +EXPORT_SYMBOL(blk_queue_atomic_write_unit_min_sectors);
> 
> atomic_write_unit_min_sectors should be >= (physical block size >> 9)
> given the minimized atomic write unit is physical sector for all disk.

For SCSI, we have a granularity VPD value, and when set we pay attention 
to that. If not, we use the phys block size.

For NVMe, we use the logical block size. For physical block size, that 
can be greater than the logical block size for npwg set, and I don't 
think it's suitable use that as minimum atomic write unit.

Anyway, I am not too keen on sanitizing this value in this way.

> 
>> +
>> +/*
>> + * blk_queue_atomic_write_unit_max_sectors - largest unit that can be written
>> + * atomically to the device.
>> + * @q: the request queue for the device
>> + * @sectors: must be a power-of-two.
>> + */
>> +void blk_queue_atomic_write_unit_max_sectors(struct request_queue *q,
>> +					     unsigned int sectors)
>> +{
>> +	struct queue_limits *limits = &q->limits;
>> +
>> +	limits->atomic_write_unit_max_sectors = sectors;
>> +}
>> +EXPORT_SYMBOL(blk_queue_atomic_write_unit_max_sectors);
> 
> atomic_write_unit_max_sectors should be >= atomic_write_unit_min_sectors.
> 

Again, we rely on the driver to provide sound values. However, as 
mentioned, we can sanitize.

Thanks,
John