Received-SPF: pass (google.com: domain of linux-kernel+bounces-138174-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3;
Message-ID: <b0789a97-7dcf-48aa-9980-8525942dabfa@oracle.com>
Date: Wed, 10 Apr 2024 09:34:36 +0100
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH v6 00/10] block atomic writes
To: Luis Chamberlain <mcgrof@kernel.org>
Cc: Matthew Wilcox <willy@infradead.org>,
        Pankaj Raghav
 <p.raghav@samsung.com>,
        Daniel Gomez <da.gomez@samsung.com>,
        =?UTF-8?Q?Javier_Gonz=C3=A1lez?= <javier.gonz@samsung.com>,
        axboe@kernel.dk, kbusch@kernel.org, hch@lst.de, sagi@grimberg.me,
        jejb@linux.ibm.com, martin.petersen@oracle.com, djwong@kernel.org,
        viro@zeniv.linux.org.uk, brauner@kernel.org, dchinner@redhat.com,
        jack@suse.cz, linux-block@vger.kernel.org,
        linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org,
        linux-fsdevel@vger.kernel.org, tytso@mit.edu, jbongio@google.com,
        linux-scsi@vger.kernel.org, ojaswin@linux.ibm.com, linux-aio@kvack.org,
        linux-btrfs@vger.kernel.org, io-uring@vger.kernel.org,
        nilay@linux.ibm.com, ritesh.list@gmail.com
References: <20240326133813.3224593-1-john.g.garry@oracle.com>
 <ZgOXb_oZjsUU12YL@casper.infradead.org>
 <c4c0dad5-41a4-44b4-8f40-2a250571180b@oracle.com>
 <Zg7Z4aJtn3SxY5w1@casper.infradead.org>
 <f3c1d321-0dfc-466f-9f6a-fe2f0513d944@oracle.com>
 <ZhQud1NbO4aMt0MH@bombadil.infradead.org>
Content-Language: en-US
From: John Garry <john.g.garry@oracle.com>
Organization: Oracle Corporation
In-Reply-To: <ZhQud1NbO4aMt0MH@bombadil.infradead.org>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: 
	=?utf-8?B?MnU5OVFHc0h5TzgrVzF2WDhoTjZTZVh4QUM1cEZPNkVGODJSRkkxRkd5SUxW?=
 =?utf-8?B?dXdiZ04wRXA1Mi9IS2lDUVdVZUQyYklHb2lwRlBMd0ZoZFBJT0JtcFNnSFhL?=
 =?utf-8?B?TDZ2T1pIdVlMYzgvbU9NZmNQcmlCUWN0aURVUHFmZXVpZWZFM3Y4ZW9JYzgy?=
 =?utf-8?B?Tk1aUzMxNldENmZTc1pFWE5PQnBUYkJQcC9VelpxRnpSbGRUN1d1bnZSMi9P?=
 =?utf-8?B?eWJyRzdCa2ZnVmlXT01iNWtLMTRWWENRd1ZVQXluMm1yb29OdjVhSFRsTDNR?=
 =?utf-8?B?VXU4UVA3RnBNZUZGbGJhZEFtU216Q0FWekdFTmFyYkhySVY1YjAxdENjSURG?=
 =?utf-8?B?a2ZaS1RJeU9aVDE0d0xsUkd1Zk9PM2FkTE43TDV2R2N0c2djRFMwTm9lY0FM?=
 =?utf-8?B?U3lGVnN2UEJmUFFDc1dpczVDNUdIYkQzTHFJQXNyVk8vSXFGakE1cHpFTEdC?=
 =?utf-8?B?QUtXWjVQRFk4UVB4Qk9ta0c1b2VoZm1pUGltNWNjZU1HSldhTVU0eVVxTE9N?=
 =?utf-8?B?dDNadlBLaktucXZPUDJscTJxcVZQR0x1L3NKU3BNTmNkTFRRT0ZFb0FGV3FZ?=
 =?utf-8?B?aDROdVltMmhFTzUrQndKT0ZWM0RLN3d6K0xIbFdHZW5hdHQwMFN0M214Vk1r?=
 =?utf-8?B?VW5vc1pDc05NZXhwR2s0S0JlVFBQVkFlUGNPbFJCQWZXd05IblNSbnNTVmdR?=
 =?utf-8?B?cTB6TlgrRm9TVEdVWXZNWExtR2ZxVEJkRm1QQ1dtcm9HUlV3dUpOdVNqT0Yz?=
 =?utf-8?B?UVJPajhWekp4WGxwbTNCYkhTWm1rOFlPeC9GRVhUTkg2VWNvR1pBbkhwaklh?=
 =?utf-8?B?b3NSZStRMXJjTndJbHBjOGU5MUV5YnhhYXVHNE9qNmJJN1hCNlZCUy9Bdm1H?=
 =?utf-8?B?UExJL2ZLVDl2bGdJRHZhSjFkeDRTU1ljMXdPRnZLdStJNHhYV3ljRTE1Mit6?=
 =?utf-8?B?OVlkTjA1Y09wUzBwb202NHp4RTd4MzM5TmRlZjFlM3lIcEczV3J0Rk1VL0d6?=
 =?utf-8?B?MzRGMVpZeUFzc3B4cDlNUURUcXVuVFkwMVNqdmg5ekRIN0FXVFdPNlNsc0tl?=
 =?utf-8?B?KzQ3dVhGN3BVVldycWl1RlRLNkt0d3d0d1k1T2IwbmZEUVN1M3dBVzg5Y2Rn?=
 =?utf-8?B?VlJ5ZGd0dFdyWjRlQXFxdE1zKzdHd25BdjdFUE1KcmUvTXBjeDJNOURpMWth?=
 =?utf-8?B?YnZiMXJMb3N3TG92QTlITW1JREMrbjYzZFJxWXJVZWYvUVArRm5iNWpEYmtK?=
 =?utf-8?B?SEVRTlNOS3B3WTBmSDh6S2FQSy9BUi9TRlJLWFRoeDdxZWFheXJpejBxdU9B?=
 =?utf-8?B?Tzg4Mm0rcmpld2x4MjVGZnRFR2tCa09rT1g0bWsxTURRRTIxWXhOWWh2S3o5?=
 =?utf-8?B?UTh0aElvcTgydGdHb0RBNm11TlRtU3o2bnppTXUzell0VitjQVNFK280Y0hP?=
 =?utf-8?B?Vk84bEsxL25OVUplVFowTVlNaCtZOFo5b0daN2FXQUpodDBVMnhFOWxhOWtF?=
 =?utf-8?B?WGVtYm1vNlJ6bUtFc1dTbnErN2xEclZjZ1J1aHp2SUwyQXlCTWVTNXVhaVFo?=
 =?utf-8?B?NVgwNGN4UWNpQVFvbFZUZmpBWmFiUU52aGZHelNEcFo2UDQ2empIL2lSM0E4?=
 =?utf-8?B?Wk1zVmhxQXYrYW5EcFM1RWhQNy9neEpoUGpBRjFiazVWeGd1OGphSzFRVVVU?=
 =?utf-8?B?N1FPNlI2TUpqeUpiSGk1Z210OC9PQjhvbWVyUGNDQmNhNmlTc245VnhQb1Bw?=
 =?utf-8?B?WWFzckJEc3BrT0FDM3ArT0FSWEZIeElmdW0wd2pyZ2dSSlNHWFJ2MGFuVnQx?=
 =?utf-8?B?eFpkVTNmcWltMDRYYzVWNk9uWjh2TVM1OTRVZTVvOG40NjNuY25NT2lFRnl5?=
 =?utf-8?B?a3NJWW5uaERLa1ZHbmdvMzF2T2NJbnREMGFlYUFzaUtsTElYZlhRN1RnYkhZ?=
 =?utf-8?B?TlpLZ3VCZEx5TkxrMXNEcG5xVkptYnpHN1UyVnpmZkw4ZVI2SkRoeUsvMnZP?=
 =?utf-8?B?MEFPcTd2Q0VJVWNrMW1BT3NLbFZpVlhHNDlJZVpORDJTR3dCNi85T3lDMzEx?=
 =?utf-8?B?NXJTWDQrNCtZTnBNdjFlNk4vVzlpVW9WcXJJMUtlTlV1aVhyeXFTNkRLL01R?=
 =?utf-8?Q?/KefMqG/h4yBYQFptCacLp4nc?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 
	7TKa53HbKp+clWf8gE+5yRVQXQ24Ww4AZ+OxjIf5G1FFj4GQUZ6qdFZbujnucoqxQJKK/OPGUcsK1x9G5ruzeRJHbB3gcCI5wRGHlFz6DxaNxxXHUMNwpARq36NjVpyaxVGRC6kVCAgY4gW5fIHUv7yCA5/M96j//XvSG2oZnaMAYiOKbMyWYXRLYRemXMnntB/4pLkMnK1nQbqWe3GkG2cAIz++0SuFx8WkBdyoO+XK2Z4Birfb9e7lke5OX1DhheEVq/1vKr1J4fRprXkT0/eGo7uTDbGHjQVVtKxMv7sAws/2oKw8H3ku2k808DgEnydZfSpqgS4ctFv9GTMKYHbL9w2AsqiBqYVon44N/pqHWmulZY9ycXN0IbEQT9mapbdNiwOF+qkqkPqJBGDfg9SN3DSIsb6gJmKVgzmmtb1Eood3AL0zCKY7Jw9/DCYTGS/ZGNDkKId+e/C3MVWaTJ5ke/MqQlAZEjwu7dTYVoBgKiipsUNHp0BmuP90mMHuhkvHmb22ok2alFrG5KZgznDoQmVYGKRtGnnlt2mIFxTK0GeQHkQbTN/A5C+pQ05N4QaSa+q7YDE6a3UiHvf/XYcjfIdRzYNkp4pvF1LjuiY=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 702d0fa3-a805-4741-d856-08dc593911dd
X-MS-Exchange-CrossTenant-AuthSource: DM6PR10MB4313.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 Apr 2024 08:34:42.9979
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: HsvxryLXUhu7O+hTLUyzgyRfw2Pm+T08PPZinXk4MN8KOMtQ35T5f90WeQZIESvDkuoaRPXR342eWlHPMt/CUQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH0PR10MB4908
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.272,Aquarius:18.0.1011,Hydra:6.0.619,FMLib:17.11.176.26
 definitions=2024-04-10_03,2024-04-09_01,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=999 phishscore=0
 adultscore=0 malwarescore=0 bulkscore=0 suspectscore=0 mlxscore=0
 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2404010000 definitions=main-2404100061
X-Proofpoint-GUID: RQVKpZfZ5Q0lxVyRjatHfpJWPWEtNvVq
X-Proofpoint-ORIG-GUID: RQVKpZfZ5Q0lxVyRjatHfpJWPWEtNvVq

On 08/04/2024 18:50, Luis Chamberlain wrote:
> I agree that when you don't set the sector size to 16k you are not forcing the
> filesystem to use 16k IOs, the metadata can still be 4k. But when you
> use a 16k sector size, the 16k IOs should be respected by the
> filesystem.
> 
> Do we break BIOs to below a min order if the sector size is also set to
> 16k?  I haven't seen that and its unclear when or how that could happen.

AFAICS, the only guarantee is to not split below LBS.

> 
> At least for NVMe we don't need to yell to a device to inform it we want
> a 16k IO issued to it to be atomic, if we read that it has the
> capability for it, it just does it. The IO verificaiton can be done with
> blkalgn [0].
> 
> Does SCSI*require*  an 16k atomic prep work, or can it be done implicitly?
> Does it need WRITE_ATOMIC_16?

physical block size is what we can implicitly write atomically. So if 
you have a 4K PBS and 512B LBS, then WRITE_ATOMIC_16 would be required 
to write 16KB atomically.

> 
> [0]https://urldefense.com/v3/__https://github.com/dagmcr/bcc/tree/blkalgn__;!!ACWV5N9M2RV99hQ!I0tfdPsEq9vdHMSC7JVmVDHCb5w6invjudW7pZW50v3mZ7dWMMf0cBtY_BQlZZmYSjLzPQDZoLO7-K6MQQ$  
> 
>> So just increasing the inode block size / FS block size does not
>> really change anything, in itself.
> If we're breaking up IOs when a min order is set for an inode, that
> would need to be looked into, but we're not seeing that.

In practice you won't see it, but I am talking about guarantees not to 
see it.

> 
>>> Do untorn writes actually exist in SCSI?  I was under the impression
>>> nobody had actually implemented them in SCSI hardware.
>> I know that some SCSI targets actually atomically write data in chunks >
>> LBS. Obviously atomic vs non-atomic performance is a moot point there, as
>> data is implicitly always atomically written.
>>
>> We actually have an mysql/innodb port of this API working on such a SCSI
>> target.
> I suspect IO verification with the above tool should prove to show the
> same if you use a filesystem with a larger sector size set too, and you
> just would not have to do any changes to userspace other than the
> filesystem creation with say mkfs.xfs params of -b size=16k -s size=16k

Ok, I see

> 
>> However I am not sure about atomic write support for other SCSI targets.
> Good to know!
> 
>>>> We saw untorn writes as not being a property of the file or even the inode
>>>> itself, but rather an attribute of the specific IO being issued from the
>>>> userspace application.
>>> The problem is that keeping track of that is expensive for buffered
>>> writes.  It's a model that only works for direct IO.  Arguably we
>>> could make it work for O_SYNC buffered IO, but that'll require some
>>> surgery.
>> To me, O_ATOMIC would be required for buffered atomic writes IO, as we want
>> a fixed-sized IO, so that would mean no mixing of atomic and non-atomic IO.
> Would using the same min and max order for the inode work instead?

Maybe, I would need to check further.

Thanks,
John