Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) client-ip=23.128.96.36;
Message-ID: <bd639010-2ad7-4379-ba0a-64b5f6ebec41@oracle.com>
Date:   Mon, 4 Dec 2023 13:13:55 +0000
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH 10/21] block: Add fops atomic write support
To:     Ming Lei <ming.lei@redhat.com>
Cc:     axboe@kernel.dk, kbusch@kernel.org, hch@lst.de, sagi@grimberg.me,
        jejb@linux.ibm.com, martin.petersen@oracle.com, djwong@kernel.org,
        viro@zeniv.linux.org.uk, brauner@kernel.org,
        chandan.babu@oracle.com, dchinner@redhat.com,
        linux-block@vger.kernel.org, linux-kernel@vger.kernel.org,
        linux-nvme@lists.infradead.org, linux-xfs@vger.kernel.org,
        linux-fsdevel@vger.kernel.org, tytso@mit.edu, jbongio@google.com,
        linux-api@vger.kernel.org
References: <20230929102726.2985188-1-john.g.garry@oracle.com>
 <20230929102726.2985188-11-john.g.garry@oracle.com> <ZW05th/c0sNbM2Zf@fedora>
 <03a87103-0721-412c-92f5-9fd605dc0c74@oracle.com> <ZW3DracIEH7uTyEA@fedora>
Content-Language: en-US
From:   John Garry <john.g.garry@oracle.com>
Organization: Oracle Corporation
In-Reply-To: <ZW3DracIEH7uTyEA@fedora>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?eDM1bFJvKzNnTXRlQjB6NTJPNjh2NUxZdlRueDYwQVJYcTkxbktxYmdWcTI1?=
 =?utf-8?B?VTdxOFcrTFZ4WW5HV0dQU0FIekdHZ1p3S2MxclN0WnB2Rkl4bDJFby9ZUWR3?=
 =?utf-8?B?dDlrTkZUTWpqcUNlRllzM2tWaHlWKzhGMjRrSTgwODZaWmhsdC92N1NiZVBO?=
 =?utf-8?B?aGpKUEwyNmNscVErKzFnb29hN0JTMGJ6aWxkOVZQSkxVWisxdllQbElZb0dY?=
 =?utf-8?B?U01hc3BUZjBITzl6VHZXSUI5TW5KL0NXWGt4dGU5a0l4SFhJUTFOM1R1Rm1G?=
 =?utf-8?B?Y2dxYUtHcmcrN1laenlXVjcvMkRCS2FNN2s3azJyY29wciswRFo2UmFOWFRY?=
 =?utf-8?B?VjZIYU16VnNMUWJsaUtFbVhGSWwvekVpT2RoQnFiNkYvTmFtY2tJTmsvTkdC?=
 =?utf-8?B?UThtQnpiQUluZllhYVFremxRaUtoZWNtSW44ZW1majFNaVd3ZVRGUFU0MEhP?=
 =?utf-8?B?aVFuWGV3cXpkY1hWMEhyOHJOaWIxSkpseGtRMElnQ3pOYW9DQ3NrNm5ST295?=
 =?utf-8?B?OTU0UUJhNlBxS2FXMmNidXJkTFhrMGlCUmtVVmZYdGp6djVzYzYyUGd3cHNm?=
 =?utf-8?B?eW0yN0hlTnRtNlJLL1lDSWVwV0NBeVNqWjZaTHNxaUNZTTdsMytVbWcvZkor?=
 =?utf-8?B?aVdPYm94TmpaMzNhR25jcVptSG40T3NxRUJiWVpuUWFRNjkvaU8xamZNQjQz?=
 =?utf-8?B?OFY3cStER2NKTmhXbFJBcnBTK3o0L2JscGhUM3NFcCtFczRZNkc5SnVYR2hV?=
 =?utf-8?B?UXZsNVZUSjVUQ0NEK0hYNkJHSmJoRUEzOGVPMTAxSVdCcCtQd213KzJ6RnJy?=
 =?utf-8?B?MzBrTXFpKytBMm9YOTAwVkxtd3JramlTWDM1UlVGNFFIK1lrN05zb0dsYXRt?=
 =?utf-8?B?WW9aV1hWbTNOMXFtM1hibjBkWTFRQ3dZdkZYRThCZWZETnBnY2xTMHFTSnd0?=
 =?utf-8?B?cmoxS2IzSk1UdlpHdTZNNkE5SERzQXdvd1M3ZnVyZ2ZQaUJ1bU9wUWFlbTJi?=
 =?utf-8?B?eE51YTdJSitSeGtWQTB2a3F4QzNNR1VCV2JLUkREZ0NNNDhiK2FRSjZodEJB?=
 =?utf-8?B?S0tjRGRheWZMUnliQkI2SlRtTnNTbnZKalV3eXJMOFc2TmtZQUFKT0pwWjMx?=
 =?utf-8?B?cWpPZ3Z4NFV3TjRHZGFXWG1QajRBWGNsWTB4aytOZ2g5VFY4dklQaFFrZHFP?=
 =?utf-8?B?S0RxaG5GNGd5VlVadHZFdmgrU3h3RGlha05YQS9OYjBDcEY2bVdsQ0dwUmVB?=
 =?utf-8?B?STNVbEZFVUdMbFVGL1hMYU5WT3VYdFZzSVBxV3Q3ZU1tdnNOdDFlZE9LRXY0?=
 =?utf-8?B?UjJMUlI3UjFUUEY3OUE4clFBVU12aEdSRFA4ZDIwMkswd2dwenJqZTRPT21K?=
 =?utf-8?B?V2cyaUEvNUJ6bEdka2lTUldzb1B0cms4U2tKSVJDSVlsaVRKUUFMZ0N6SHN2?=
 =?utf-8?B?eUFjU3hTNGR1OWpxQ1YzMndmdlJZNFJoWWF1N0ZlMmlpYnN0eTFJRnYwOFNa?=
 =?utf-8?B?MnNvcWNkRkdqb2Q0WGNyNnhKSG90RVRUOE56Qmprd09qMWFCWDBkL1F4VjRB?=
 =?utf-8?B?SUZuclFmRjBGZUFuWGJrcmhXcFExMHRMQnNBakNkVWdBSjFSM0VyZ0xoWWFi?=
 =?utf-8?B?c1RiQW8rNWlZZHRNS1kwdjFzL2VPcVZMcWw1YXlTTnY4ejJ1aWRzaG43NHI2?=
 =?utf-8?B?S0U1QmZuVWJFRXJMckl1NTdUSUhNL1p4U2JpUkRRWVM4UDFuTXlhSjlhN24x?=
 =?utf-8?B?SHl0TCtFSEtJUkhLTXdPOGtmZkxJb0JUVlN6alNTS3JISXVZRVZsZnJvVnB2?=
 =?utf-8?B?ZThsRjBqdXlaWjFGSGVWcUt4VURHWFd6dXR3ZWluVDlUWVJhR3RCY0t3TFF6?=
 =?utf-8?B?VTNIaFBkUXE3ODlGbk00dlVRVGFTVmlwK0ZjVXhkWTNKQnYrQkhKOXZyd1k0?=
 =?utf-8?B?cnI1RE41UUN5ZmRwZmowTWFXOCtDdUg5M09ESDBGSUd5bWM3K3JRdm4rRk40?=
 =?utf-8?B?UjdlVmdzazN2dTgwM0lSZHY2Y0xoMzJ1elRSeW9TTmR0TS9kMnRNc0V0UjN2?=
 =?utf-8?B?SnpyWHhhd1k5NkdNN1FaZW5IOE54UTZ4M2dENVFtazVYRkI4RHNiRUs0dzRj?=
 =?utf-8?Q?CyqElXZUW+OkQ2O3k/7zxl/DB?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: =?utf-8?B?QkJxNEw4ZzVuQjFvN1grL3lYK0FRZUlGNWxkMVBaaVliNGcxV2M5R09weVQr?=
 =?utf-8?B?Sk1lWm1EQjhJQjZvTzkrMk82bHVxYW5HMys0MnBPdDdUVjgxRktBSXRFcWRu?=
 =?utf-8?B?REJnbFNaOFVyS2NUdmYxbnd3am80dWN2SDBuVUtjaHNabDkxUVZBYWMvM2kz?=
 =?utf-8?B?L0wwSjlscm5BS1NaYWlxWGpwY0piRnYrQURxeExhWnl5K0FyUVllSnRkZ1Fk?=
 =?utf-8?B?RzZjUi92V3hXRHVVNnFVSTc0cnFBSmRYeVdIOFROSEZQTXJXNnQ5bC92ZGxo?=
 =?utf-8?B?M2Rmemg1M3RYWlloaFJ5NlZKakZvbi9QK1I3dDRBWmY4Vnc2WWZzQ3p0clVu?=
 =?utf-8?B?SFdTbUJ6Q1ZyMVlvTFJZM1gwam9lTmk2QUIycXV0QzZ3M2Q0cEp1R2swbVB0?=
 =?utf-8?B?Qy9vT2pXSG1EY0pTMVFmSG8vOVVKZ0o1QjQraDVWdkhEM3AxL01ZOVJaWFln?=
 =?utf-8?B?TXI4a3grTGZIVDVuUzAwK1hmUEJaUkhvYmxIZ0V2V25TVy90T2xjWnN6Q3FT?=
 =?utf-8?B?VklJdVJ4WDNETEFuY1pnQlNES2NVRllIZFZVMk1IcFBsdDM0a0lrSGVmK0ky?=
 =?utf-8?B?Z0pGajBpMFgvNXVEUDN2OFNWZWYyOURQSk0zVW41SW05Q3pPTzdhajhZK2dJ?=
 =?utf-8?B?N3E0dzFqdzdQbFlSVDVnT3VJRTMreU44b0xZNkVVbndKdEdTY0RBSEoydWhw?=
 =?utf-8?B?NTU4SXdLTTJQUXVlclBhWlorcHNYZ2ZrRlZPeHJaQW1Ja3Vkc0VxRnd5SjlP?=
 =?utf-8?B?N1grVGxJQ0IvWFowVXE3Uzk4K0F2MUFoWUpXVGdhWEpNTjFKV3BtSEE3cXJi?=
 =?utf-8?B?VHFzdTBseFRTdHNpYW1JamxTR1VLeE5RV0lKMDlyVWNJWkFqcjNJekRBUjMy?=
 =?utf-8?B?ekpaTjhvZUljYkpKOFZXWkZmc1IydUprZDlZNm1IMHovVS8zdnlVeVFsejhO?=
 =?utf-8?B?UEJLZkhDQW1tNlZEM3RRSGgrVVdEWkNmMW5GdDlPUXkrWElRekNhRi9YeHdp?=
 =?utf-8?B?NG0wMU5kQSt1TmhiOEREWUF5TklCTXBoV1FBTE5nUyt5dFpWREFSVGlKNUls?=
 =?utf-8?B?VlhJOW50MUpUeGpYK1NJUnl0VHR2eFQ3YnpjMnRWVzBaempCU0Mrb3dKTXVq?=
 =?utf-8?B?TWZqWjlLckxGQjJPblp4YnVQbkRRbjJkaUdjd0RnZFBlb2NQTEVjSjhNVkRL?=
 =?utf-8?B?NThWcThhZUQxVXhPaVY3cHA5eVhwbnFkUlllVFNPSmVKb092K0dvZGdzYUpi?=
 =?utf-8?B?bjdSOUpnM2FrWHp2UUxTY1pMSUNHU1VrdzB3V0ZtSEVOaHZka3dWSHN5WlF1?=
 =?utf-8?B?elVzZjZ5L0NJS2NUVE51S0lWTTJsaVN3eFJaR2wwS2NZajYvd2M3bXFUdTQ3?=
 =?utf-8?B?M1M3cFFxTE83NDgrNmQ3UkowQ1FJOXhNeVlCM1ZjTDROOTBBaUhCdVNNUHY5?=
 =?utf-8?B?cGVjNzNEVkx6VW5HWXFrdHRsNkpzUFZ3RmFLT293PT0=?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 94e4349b-2d7f-4349-1608-08dbf4cae685
X-MS-Exchange-CrossTenant-AuthSource: DM6PR10MB4313.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Dec 2023 13:14:09.4242
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: WIXlHHz6wMqETQ2kVFlZ6c8qCzaOkLJi/2lC6NcQQEHpSsmwdsuZr9mAcCy7q/zLxJyI0YqH7ez7bhykqp6UVA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR10MB5855
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26
 definitions=2023-12-04_11,2023-12-04_01,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 suspectscore=0
 mlxlogscore=999 adultscore=0 phishscore=0 malwarescore=0 spamscore=0
 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2311060000 definitions=main-2312040099
X-Proofpoint-GUID: elMMzkmnQdMeBEHJ9Jdlda51XxvUS9jm
X-Proofpoint-ORIG-GUID: elMMzkmnQdMeBEHJ9Jdlda51XxvUS9jm
X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID,
	DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,
	SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable
	autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on pete.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (pete.vger.email [0.0.0.0]); Mon, 04 Dec 2023 05:14:56 -0800 (PST)


>>
>> I added this here (as opposed to the caller), as I was not really worried
>> about speeding up the failure path. Are you saying to call even earlier in
>> submission path?
> atomic_write_unit_min is one hardware property, and it should be checked
> in blk_queue_atomic_write_unit_min_sectors() from beginning, then you
> can avoid this check every other where.

ok, but we still need to ensure in the submission path that the block 
device actually supports atomic writes - this was the initial check.

> 
>>>> +	if (pos % atomic_write_unit_min_bytes)
>>>> +		return false;
>>>> +	if (iov_iter_count(iter) % atomic_write_unit_min_bytes)
>>>> +		return false;
>>>> +	if (!is_power_of_2(iov_iter_count(iter)))
>>>> +		return false;
>>>> +	if (iov_iter_count(iter) > atomic_write_unit_max_bytes)
>>>> +		return false;
>>>> +	if (pos % iov_iter_count(iter))
>>>> +		return false;
>>> I am a bit confused about relation between atomic_write_unit_max_bytes and
>>> atomic_write_max_bytes.
>> I think that naming could be improved. Or even just drop merging (and
>> atomic_write_max_bytes concept) until we show it to improve performance.
>>
>> So generally atomic_write_unit_max_bytes will be same as
>> atomic_write_max_bytes, however it could be different if:
>> a. request queue nr hw segments or other request queue limits needs to
>> restrict atomic_write_unit_max_bytes
>> b. atomic_write_unit_max_bytes does not need to be a power-of-2 and
>> atomic_write_max_bytes does. So essentially:
>> atomic_write_unit_max_bytes = rounddown_pow_of_2(atomic_write_max_bytes)
>>
> plug merge often improves sequential IO perf, so if the hardware supports
> this way, I think 'atomic_write_max_bytes' should be supported from the
> beginning, such as:
> 
> - user space submits sequential N * (4k, 8k, 16k, ...) atomic writes, all can
> be merged to single IO request, which is issued to driver.
> 
> Or
> 
> - user space submits sequential 4k, 4k, 8k, 16K, 32k, 64k atomic writes, all can
> be merged to single IO request, which is issued to driver.

Right, we do expect userspace to use a fixed block size, but we give 
scope in the API to use variable size.

> 
> The hardware should recognize unit size by start LBA, and check if length is
> valid, so probably the interface might be relaxed to:
> 
> 1) start lba is unit aligned, and this unit is in the supported unit
> range(power_2 in [unit_min, unit_max])
> 
> 2) length needs to be:
> 
> - N * this_unit_size
> - <= atomic_write_max_bytes

Please note that we also need to consider:
- any atomic write boundary (from NVMe)
- virt boundary (from NVMe)

And, as I mentioned elsewhere, I am still not 100% comfortable that we 
don't pay attention to regular max_sectors_kb...

> 
> 
>>> Here the max IO length is limited to be <= atomic_write_unit_max_bytes,
>>> so looks userspace can only submit IO with write-atomic-unit naturally
>>> aligned IO(such as, 4k, 8k, 16k, 32k, ...),
>> correct
>>
>>> but these user IOs are
>>> allowed to be merged to big one if naturally alignment is respected and
>>> the merged IO size is <= atomic_write_max_bytes.
>> correct, but the resultant merged IO does not have have to be naturally
>> aligned.
>>
>>> Is my understanding right?
>> Yes, but...
>>
>>> If yes, I'd suggest to document the point,
>>> and the last two checks could be change to:
>>>
>>> 	/* naturally aligned */
>>> 	if (pos % iov_iter_count(iter))
>>> 		return false;
>>>
>>> 	if (iov_iter_count(iter) > atomic_write_max_bytes)
>>> 		return false;
>> .. we would not be merging at this point as this is just IO submission to
>> the block layer, so atomic_write_max_bytes does not come into play yet. If
>> you check patch 7/21, you will see that we limit IO size to
>> atomic_write_max_bytes, which is relevant merging.
> I know the motivation of atomic_write_max_bytes, and now I am wondering
> atomic_write_max_bytes may be exported to userspace for the sake of
> atomic write performance.

It is available from sysfs for the request queue, but in an earlier 
series Dave Chinner suggested doing more to expose to the application 
programmer. So here that would mean a statx member. I'm still not 
sure... it just didn't seem like a detail which the user would need to 
know or be able to do much with.

Thanks,
John