Received-SPF: pass (google.com: domain of linux-kernel+bounces-121373-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249;
Message-ID: <c4c0dad5-41a4-44b4-8f40-2a250571180b@oracle.com>
Date: Wed, 27 Mar 2024 13:37:41 +0000
User-Agent: Mozilla Thunderbird
From: John Garry <john.g.garry@oracle.com>
Subject: Re: [PATCH v6 00/10] block atomic writes
To: Matthew Wilcox <willy@infradead.org>
Cc: axboe@kernel.dk, kbusch@kernel.org, hch@lst.de, sagi@grimberg.me,
        jejb@linux.ibm.com, martin.petersen@oracle.com, djwong@kernel.org,
        viro@zeniv.linux.org.uk, brauner@kernel.org, dchinner@redhat.com,
        jack@suse.cz, linux-block@vger.kernel.org,
        linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org,
        linux-fsdevel@vger.kernel.org, tytso@mit.edu, jbongio@google.com,
        linux-scsi@vger.kernel.org, ojaswin@linux.ibm.com, linux-aio@kvack.org,
        linux-btrfs@vger.kernel.org, io-uring@vger.kernel.org,
        nilay@linux.ibm.com, ritesh.list@gmail.com
References: <20240326133813.3224593-1-john.g.garry@oracle.com>
 <ZgOXb_oZjsUU12YL@casper.infradead.org>
Content-Language: en-US
Organization: Oracle Corporation
In-Reply-To: <ZgOXb_oZjsUU12YL@casper.infradead.org>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: 
	=?utf-8?B?bVVNQjcwK0FJZ2EvSUlBeFNBdzViWmJ3SXlwYXJDdVV3Z01EakliRW1BSTZ3?=
 =?utf-8?B?SlR5U0kzbVdTSnpnTXpwYkdxMWtvTnl3TTVkWndqUGZ2NExRZWZ0NjlhZWpq?=
 =?utf-8?B?c1hHVlJ2M0VDY0hDNnU5cE4vY3o5TSs1ZnNvQm1SWnFwV2ZBMWJ3QlBCUEpw?=
 =?utf-8?B?WFFaRE9IQkNhaitZZ3ZHbExEaWtjUjRBWnVxU0ZvdjVkVCtnSmUzQ05LanBZ?=
 =?utf-8?B?Vi8xbkh5cFdwYkR4Q2hGdld0bG5Hblo5TWVhN0xldVhtRDdHSndJanA3a2Zy?=
 =?utf-8?B?MVp2M08xd05iSURYNHlWWFhGSjl2WW5CelE5dGI0MFNPbWZicWdCZ0g2UWVq?=
 =?utf-8?B?aWxlVDQ5RXpJc3dPNEpYMno3RmZmcEtoZ2ZhMGZidmhhWk1CM29PaVIxQkcr?=
 =?utf-8?B?U3YvcDB5ZkhRaEdFTy9FZEZ6dXNMV2MwUC9Sa1p5TnFvbTh5UExYVFhwWmFX?=
 =?utf-8?B?c2pGU2N0RlhCQ2xPVVlOcHY2ZE9FMHkwK2VBUk9LNm9ObWZTN09Ycm1VeHVm?=
 =?utf-8?B?a0h5dkQ0aEd6SlFGMWRTTUpNNDhVaWFTeWZCTHZXM0N3SEZ0ZWk5dkN3NUxv?=
 =?utf-8?B?Zy9RQTdLVDhuWHFudnlaU0I0L0FGNm9rZDNGV2VhSXJzM0w3a05sRDZTZGgy?=
 =?utf-8?B?OE5nbzhvOFpOY1doRUZTQnE4cTcxUnMxdDh1bzhSZ3BEUk9XbzNGUlM2NUwv?=
 =?utf-8?B?SSsrcXhxN01RT01kci92SjJhTjFZTDE5ZVo0MXA1a1RVd2l4amxIZWpybDhr?=
 =?utf-8?B?Y0FwQjlSemhUdFc0WnNtc0x5ekwyd3Y1a1hWSVp4aTdubVdXQVNjaVlhRXdl?=
 =?utf-8?B?RURvbDNNend3US9XUnBmUXY0R3c0eExGMXBuK2NFUHdZTVd2QjkzVzg2bXFt?=
 =?utf-8?B?Q0dDWFlReGZYY25oYjBEc3RGMkJ1NnZJcWZScXpubzNzZDJXRGpUc0dWT0pY?=
 =?utf-8?B?NmR2b0tHdFJkNC9zYlZSVGZrU0FyQzU2NDRadXh0T2c2UEpiRGtOQU0zS3pm?=
 =?utf-8?B?VTlXSkVnUzFoeThrNFNMNzdlalNPK2dsZVRJOWdvQTA3RzIxM1lZQXZ2a3F0?=
 =?utf-8?B?TzZHVDJ5UDhzQXQ2YmNDRS9FbTFUMG8rNFRvZEQrU0dlZklYSEY1YlF5QkQx?=
 =?utf-8?B?WWRIODlZaTNnalFJM1VvTlc3ZGpScWRHeHphaTFSLzZTS2s4aWtiY1RNc2sz?=
 =?utf-8?B?YStzRE1ERUY0dCtNdWFnRGxqcnU5Skh0elI1Y2RLWHQ4R2FJb1dHV1VwZUZZ?=
 =?utf-8?B?V3Jid0ExSDFKNlJVSjdPYzNZOGNjYUhlcWFqYU5yWFROcStjS0IxRHdtYmc4?=
 =?utf-8?B?SWRycDlsU0NjZG0yaTkra2paQnJJSmZ6OHVjK1pGVHMvcWI2bmVkWG9DRllv?=
 =?utf-8?B?eUlaaXEwN2VQYWZiRnF5cXhzaXdRbXhJVUtKTGhsWWFIaVRCdlo3bzJoRE8v?=
 =?utf-8?B?UjFyUmlyei81Ly8yR25IRmJXeHFEckRXL0hxeGwvNmNZR2FNZmhhWXJaaEcw?=
 =?utf-8?B?NXJQTEhBWHVKa01tdGdGcEMrSUNPVkg1aDFWbGRkU3kweWNEL2pZbTZlZjRR?=
 =?utf-8?B?ZllwTWtEWm4zOWtwTzdDM0FVcWMxSkZUQlJuSTJYL2lOOFYySXFGNXVicTBJ?=
 =?utf-8?B?MHRrZmJZYU5HVVdRSVJReURiQzdzcFlKT20zYi92bG1EYzZHTTBkZ0J0eElS?=
 =?utf-8?B?VHY2czRBaU40K0VrWWFiRHljczBRSnBzK3dlcU1WUjNFZE5DMTJRRjgrWDV2?=
 =?utf-8?B?WUtOa2NVNWVsNVIvVXc0QTdVRWsvQS9EeTNDMDJiYlZXVXAzNzF1WDRiVHNJ?=
 =?utf-8?B?MmtESHhZeUgrV2JqZkVPTGxjUXlFNVFtckthbE93TmZDVWEyS1EvMEJUOEc2?=
 =?utf-8?B?ZDRMa1FPbWtKeU5TREh6eURER0xYcUhYbFlvbUlzc25MYlN2RDh6ZS9jS1FG?=
 =?utf-8?B?RERqNFBmWEtXK0FUTlo0Q1ZMcCtxTTFpdHIzaVVoOTV0azJoeDdGU1p4KzRo?=
 =?utf-8?B?aXd5OEdLQWRjMHk0MmhlZDQvSHBFOWpZZ0FKamJqY3U3eUg1ckNRUTQ3YWo0?=
 =?utf-8?B?NHoyNkF2SXpjOXFjTkVOdFhJbU9ZNkd1SkJSR0oxZlNkMGE5VlhxTSsvMjR5?=
 =?utf-8?Q?H13biPNiUB/JUvmG4CDfOqa2/?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 
	adTmNkHVpaTjAO44HJ6W/yCSKfF4+N9fbZmpXXBfPaXtcbcPI+culcdZzfEML68EXbDVKCcAW7zeF242xhrxVtr3W9ZE4NFFYlS3UR57N+i0zfr9MLoamGV62+ITwiUSnK7eeKBeQsP1HHN7WqzDGHIFw26xy8WNnGqZmWuGVNs6mkZNRWhf9YqGB88R8lC9M/aSUUlh8ETL3oLWemsH9eyret9jAbg/sxosUJVdK747bUar5zxsO0lcn9lIgy8ldtuRq2Y5TNvm0t9eseDI2moIRfiMhS2UyFglDnRMsVitcXkswqMrFZywgOrgre3mLFDTrIaETBCq5qru1WDNXioZbj9nTfXWJNsXSBILTMJYNMjY8kadDlk8Bc3RdCrNJkMZmFg+9iOwwi/6JH8eU/IKba6zW2FOEKrzo02Qnb/eBJlGoiQttFQDgpm1c3hCWwav0/btsxYHxXWl8vRIM28WoUw5ecaHi5CChF64oqqQ7RyBGNLw7WulbxbSJ7PgtxypHdj71x0vyTUFIwpGekyc1F41ce3Re0B2Uyo54qIRzdMJQJXSAPQj3JgpYaYmHznVDglpqboI5wPnbasGdET0v4vRdDWx7mtz2ZVBIvI=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 30c9f5d6-5431-412a-9f54-08dc4e631e10
X-MS-Exchange-CrossTenant-AuthSource: DM6PR10MB4313.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Mar 2024 13:37:59.6339
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: +gN5fyjQrNl/hjQhTrNHFphfdw57XEhr5nmKgebvrQYZertXjSf2g/zz6K6XpHDB3FESaR22kEsSad0ylzaL3w==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV8PR10MB7800
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.272,Aquarius:18.0.1011,Hydra:6.0.619,FMLib:17.11.176.26
 definitions=2024-03-27_08,2024-03-27_01,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 mlxlogscore=999
 suspectscore=0 adultscore=0 mlxscore=0 spamscore=0 malwarescore=0
 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2403210000 definitions=main-2403270092
X-Proofpoint-GUID: Y5DHeqOIUvzwdcDn5DinBb4ekGs5LkrR
X-Proofpoint-ORIG-GUID: Y5DHeqOIUvzwdcDn5DinBb4ekGs5LkrR

On 27/03/2024 03:50, Matthew Wilcox wrote:
> On Tue, Mar 26, 2024 at 01:38:03PM +0000, John Garry wrote:
>> The goal here is to provide an interface that allows applications use
>> application-specific block sizes larger than logical block size
>> reported by the storage device or larger than filesystem block size as
>> reported by stat().
>>
>> With this new interface, application blocks will never be torn or
>> fractured when written. For a power fail, for each individual application
>> block, all or none of the data to be written. A racing atomic write and
>> read will mean that the read sees all the old data or all the new data,
>> but never a mix of old and new.
>>
>> Three new fields are added to struct statx - atomic_write_unit_min,
>> atomic_write_unit_max, and atomic_write_segments_max. For each atomic
>> individual write, the total length of a write must be a between
>> atomic_write_unit_min and atomic_write_unit_max, inclusive, and a
>> power-of-2. The write must also be at a natural offset in the file
>> wrt the write length. For pwritev2, iovcnt is limited by
>> atomic_write_segments_max.
>>
>> There has been some discussion on supporting buffered IO and whether the
>> API is suitable, like:
>> https://lore.kernel.org/linux-nvme/ZeembVG-ygFal6Eb@casper.infradead.org/
>>
>> Specifically the concern is that supporting a range of sizes of atomic IO
>> in the pagecache is complex to support. For this, my idea is that FSes can
>> fix atomic_write_unit_min and atomic_write_unit_max at the same size, the
>> extent alignment size, which should be easier to support. We may need to
>> implement O_ATOMIC to avoid mixing atomic and non-atomic IOs for this. I
>> have no proposed solution for atomic write buffered IO for bdev file
>> operations, but I know of no requirement for this.
> 
> The thing is that there's no requirement for an interface as complex as
> the one you're proposing here.  I've talked to a few database people
> and all they want is to increase the untorn write boundary from "one
> disc block" to one database block, typically 8kB or 16kB.
> 
> So they would be quite happy with a much simpler interface where they
> set the inode block size at inode creation time,

We want to support untorn writes for bdev file operations - how can we 
set the inode block size there? Currently it is based on logical block size.

> and then all writes to
> that inode were guaranteed to be untorn.  This would also be simpler to
> implement for buffered writes.

We did consider that. Won't that lead to the possibility of breaking 
existing applications which want to do regular unaligned writes to these 
files? We do know that mysql/innodb does have some "compressed" mode of 
operation, which involves regular writes to the same file which wants 
untorn writes.

Furthermore, untorn writes in HW are expensive - for SCSI anyway. Do we 
always want these for such a file?

We saw untorn writes as not being a property of the file or even the 
inode itself, but rather an attribute of the specific IO being issued 
from the userspace application.

> 
> Who's asking for this more complex interface?

It's not a case of someone specifically asking for this interface. This 
is just a proposal to satisfy userspace requirement to do untorn writes 
in a generic way.

 From a user point-of-view, untorn writes for a regular file can be 
enabled for up to a specific size* with FS_IOC_SETFLAGS API. Then they 
need to follow alignment and size rules for issuing untorn writes, but 
they would always need to do this. In addition, the user may still issue 
regular (tearable) writes to the file.

* I think that we could change this to only allow writes for that 
specific size, which was my proposal for buffered IO.

Thanks,
John