Received-SPF: pass (google.com: domain of linux-kernel+bounces-180334-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249;
Message-ID: <9e230104-4fb8-44f1-ae5a-a940f69b8d45@oracle.com>
Date: Wed, 15 May 2024 13:54:39 -0600
User-Agent: Mozilla Thunderbird
Subject: Re: [LSF/MM/BPF TOPIC] untorn buffered writes
To: Theodore Ts'o <tytso@mit.edu>, lsf-pc@lists.linux-foundation.org
Cc: linux-fsdevel@vger.kernel.org, linux-mm <linux-mm@kvack.org>,
        Luis Chamberlain <mcgrof@kernel.org>,
        "Martin K. Petersen" <martin.petersen@oracle.com>,
        Matthew Wilcox <willy@infradead.org>,
        Dave Chinner <david@fromorbit.com>, linux-kernel@vger.kernel.org
References: <20240228061257.GA106651@mit.edu>
Content-Language: en-US
From: John Garry <john.g.garry@oracle.com>
Organization: Oracle Corporation
In-Reply-To: <20240228061257.GA106651@mit.edu>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: 
	=?utf-8?B?d1dKMXMySG94K1BPVXB0UnhtWWl5WEtNUDRvZXE4dGJ0TGZ6RmxneGtvZFpV?=
 =?utf-8?B?Qjhib3BuK2tLOVlsa1ZXTWg3ektsUUxKTXdLdEg5N1VQVU5TWE9NZVNWa1RI?=
 =?utf-8?B?OG1pNUlLY3lsMW1kbWhEYUxPc1FFL2c0VUptb0drbHlCYkFRUDRnRTZoNno5?=
 =?utf-8?B?TzBvb3Z3N2dTYktTZ3IwODh3UDhpaXA2RHBINlRVQWFKVERIUmxGM1J4bGpX?=
 =?utf-8?B?UGZvb2xMSi8zRVNzL0hYMVZESVh3VVoxczZtY3VBS3BwZEMzOUx5VXZuU0Ey?=
 =?utf-8?B?Vmh6NWRVRjFpN0M2ZERQeWZzQ3F4anhjdUJLN0pQRW54T1ZXbWZ6YkNPdGlK?=
 =?utf-8?B?aGxNVERMVHpLcmR4SFdzR2VQOUxCS09ZeDhYQnJ1TzJQMTM4VHJabDhzZWZZ?=
 =?utf-8?B?VDI1TnlCWHM2OU9VUzJneUp2eWlPKzhCN0hFTVFFL21RaVJNY1RHVWFDT0l0?=
 =?utf-8?B?UENldC8rZUppQzVPVlF5OS9DMTQzL0JaaHprMTNpT0RDUHVKL2o1Z3VIcnBQ?=
 =?utf-8?B?UldtdkFXUnZPTjRaTURoV1dKMnJRanJla0xFRHRwWDdZZGYrWkkwdTNDdmtB?=
 =?utf-8?B?NmJTSHRFdU9pYkFYNUc1NTgyLzE1cndPV2hSYmpzbWlUS2h0MXJ2dEEwakpM?=
 =?utf-8?B?VTJKOFladmV0Z0RWK2FsVjgzajV0bEwzQUtvUzZOWlNnejFSMXNPeUM4OEhK?=
 =?utf-8?B?YUJRWVVBbXdydmhla2pSNWsvdEU5Myt1U2pZSEl3aUZZMHlJaC9uOVBXeGcz?=
 =?utf-8?B?c1E3REtSSTZ6SVZoVnQwTDVFUUg2aGQrY2U1a2ErdWl4Z1loODVhT1pJa3p3?=
 =?utf-8?B?ZWs0ZjJDT1NSbkFpREhnWnFKUlZEYTJLY3dpNkVYWkxad095Sm1YK0hVYjZC?=
 =?utf-8?B?Z3RCbmRxQmU0d0gyYmdzdWRDaFJGRmdmTDdOanVYcHg5ZTYxMFNveFBCUG9y?=
 =?utf-8?B?ZDhjRUphNzZ0TEZ5TENuSXlQU3pSTjUzaXR6SFdaSW9SQkhnN3kzTnE5NUhS?=
 =?utf-8?B?TGRrdTJVcS9uZXNyc1lORnFsMG1oZ2pvN3JnZXRlU0R2dVVqY1ZJU0VyT3l6?=
 =?utf-8?B?cHFFcVdDekFuRkFteWNCcThwYWhFZnJaeDVHWG5ySHFYT0pMSVFCbXFnQWc3?=
 =?utf-8?B?MWVPdi9xU3pibjNjWElCT0tJc25sL2dGNVVHZDVNZWJyUkEyYjVNMmY2QTR1?=
 =?utf-8?B?MDNLM0pVYXFJelpKdG81VC9veEFJd1hCS3g5ZzVHVUtjbnJ2bjR0c3Q1MXNp?=
 =?utf-8?B?Y3hNOUszc3pZMGVoVURjMkE2RWRTRFlhaEZVTGNVSzZYZ2hLKzI4Uko0OE9F?=
 =?utf-8?B?Y3Zvdnk4TmdZNkJvc042OExrWC9mT2RuS1h1VmFVcWEvQ1J0K3NpdTh4YXhK?=
 =?utf-8?B?UU0zSVJlN0YwbW0rTGlJV3F2eFdxeWlOZW83QWpnalVYM1hvMmFFc1ljMkp1?=
 =?utf-8?B?MHJ6Tm1nMDdKNHl4ZjFPVnZlVkVLcENYWklSU25tWkhRdTloKzRVWHVKQW8y?=
 =?utf-8?B?S3BsS3huQi9IaUN3RzJ3c3hvWjU1RFRGUVVOc3BUUEpFbGYwZ09tak9GTENk?=
 =?utf-8?B?Mm9uK0NaK1BJblVmY21UMEFNWFV3Ym41VDBvMDRUZTRoTVl2N1JjZXNnWmdC?=
 =?utf-8?B?cHY4dmhFQnRib3YwQW1NQWRnYmVPdEpLaEZDa05Nb3RWVlRQeDA3UXMwUVg0?=
 =?utf-8?B?TG1icy9MbTU3UmJOL1BwQU9XTG9adFRRdVlGSmdHbXpPWVJPRUZaaW81NDJt?=
 =?utf-8?B?NmRWZ1dldC8vS0VNSWVsV3Q3WlQ4MnFDdFREVHpyVndVRDJ5TTRQUEljbEtZ?=
 =?utf-8?B?a3B1alpGZTE1KzF6OGFacmswV0FGM2tsMkFxRHBraUhQakFTM1p2ZUlORlk0?=
 =?utf-8?B?cDVIQmFiTkNBbTBpSTdqamdyTGcwVkRDVTV1ZENleUtuTFMvcG5rNGFjTWJE?=
 =?utf-8?B?V0Nqek5hcStCd0xtNmdUcmZOeHl0QVhNc1dsdndScFQwaDZVRVpQSWltd3Vt?=
 =?utf-8?B?M3JSUVIxWnVoVldBa2RiUnpHcDgyektTck9DeGZpVWY5UDVwSnhsKzIrMWpo?=
 =?utf-8?B?VlhFSVBGR0ZnTjFQVXduelFvdGtKY0ZuQSs4eGNUa0kvNk9jaEh4cU5QNlZF?=
 =?utf-8?Q?0OgLoTW+SSVZE625nogszP4hN?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 
	qiwcZqugvgYog35mAtwlZlO+ZCARes0j4pXlmaYjJWtEXOSdw/7Qx3wTLxdgC5aZFqZqk/T3qpiiUxP6NHnDTDnzqi21aing1wJzfLCr0Fs+qlFPW/nvSS9eXL/I+aLakkN3XwxJQaaAQNSC6UZQecybXH9K330iNJOkmXwuzgi5ydQ0EyB2SHLv3EFxsh8Py5PdnZVftTKEBvGV341hceb/qPS04+126YoizDWdHVY8wS51IWNu7uK627+YTPeIxfrWIWZN9UbeKU8X4Ev0QtpBGoWBcFJDA6wB8rXDP6LoHdxfYoo+9AYvGu+rhTX1l3m3O23RH0Tkk4PYUffLX8T3mKAhaKCLnJBVraG7EptRUwiXW1PbUzIE34Qp3/iGrTM3OSCC8RTvvTI5SbNj7+gKswLYU3fRhMDnbJ2Mtfuss60tL8DKCG/jqCFvCwnsY3G/GJYAJu9b0PMp8PNkKaxOBy3O3JYcS0dkebFcS94UxMczohcS46aqUTr4HxudHZvOkS6f3B16f2EuOZG79/SmD7f99zb6B3uQApR76T5uEZ1AEOMscCkmVUpf5QbJ5YiGIhVePPpA3DQ80phRPh8KxqvXVPO6mPGkMXuh/0U=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 64647efa-01fc-4d82-1512-08dc7518dc40
X-MS-Exchange-CrossTenant-AuthSource: DM6PR10MB4313.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 May 2024 19:54:41.6236
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 6ZMH85UGiizkcipk2QkGXOi0oWrbVy3EVvxmaBqyEBcPZ6p5YzAKMF+rIg8EokhBsVkI0WcVkxk+4kshPidclQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR10MB7699
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.293,Aquarius:18.0.1039,Hydra:6.0.650,FMLib:17.11.176.26
 definitions=2024-05-15_12,2024-05-15_01,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 mlxlogscore=999
 adultscore=0 mlxscore=0 spamscore=0 suspectscore=0 phishscore=0
 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2405010000 definitions=main-2405150142
X-Proofpoint-ORIG-GUID: StZrP23CJUj5kJycqu2WPUZ5gUEWVxm3
X-Proofpoint-GUID: StZrP23CJUj5kJycqu2WPUZ5gUEWVxm3

On 27/02/2024 23:12, Theodore Ts'o wrote:
> Last year, I talked about an interest to provide database such as
> MySQL with the ability to issue writes that would not be torn as they
> write 16k database pages[1].
> 
> [1] https://urldefense.com/v3/__https://lwn.net/Articles/932900/__;!!ACWV5N9M2RV99hQ!Ij_ZeSZrJ4uPL94Im73udLMjqpkcZwHmuNnznogL68ehu6TDTXqbMsC4xLUqh18hq2Ib77p1D8_4mV5Q$
> 

After discussing this topic earlier this week, I would like to know if 
there are still objections or concerns with the untorn-writes userspace 
API proposed in 
https://lore.kernel.org/linux-block/20240326133813.3224593-1-john.g.garry@oracle.com/

I feel that the series for supporting direct-IO only, above, is stuck 
because of this topic of buffered IO.

So I sent an RFC for buffered untorn-writes last month in 
https://lore.kernel.org/linux-fsdevel/20240422143923.3927601-1-john.g.garry@oracle.com/, 
which did leverage the bs > ps effort. Maybe it did not get noticed due 
to being an RFC. It works on the following principles:

- A buffered atomic write requires RWF_ATOMIC flag be set, same as
   direct IO. The same other atomic writes rules apply.
- For an inode, only a single size of buffered write is allowed. So for
   statx, atomic_write_unit_min = atomic_write_unit_max always for
   buffered atomic writes.
- A single folio maps to an atomic write in the pagecache. So inode
   address_space folio min order = max order = atomic_write_unit_min/max
- A folio is tagged as "atomic" when atomically written and written back
   to storage "atomically", same as direct-IO method would do for an
   atomic write.
- If userspace wants to guarantee a buffered atomic write is written to
   storage atomically after the write syscall returns, it must use
   RWF_SYNC or similar (along with RWF_ATOMIC).

This is all along the lines of what I described on Monday.

There are no concrete semantics for buffered untorn-writes ATM - like 
mixing RWF_ATOMIC write with non-RWF_ATOMIC writes in the pagecache - 
but I don't think that this needs to be formalized yet. Or, if it really 
does, let me know.

There was also talk in the "limits of buffered IO.. " session - as I 
understand - that RWF_ATOMIC for buffered IO should be writethough. If 
anyone wants to discuss that further or describe that issue, then please do.

Anyway, I plan to push the direct IO series for merging in the next 
cycle, so let me know of what else to discuss and get conclusion on.


> There is a patch set being worked on by John Garry which provides
> stronger guarantees than what is actually required for this use case,
> called "atomic writes".  The proposed interface for this facility
> involves passing a new flag to pwritev2(2), RWF_ATOMIC, which requests
> that the specific write be written to the storage device in an
> all-or-nothing fashion, and if it can not be guaranteed, that the
> write should fail.  In this interface, if the userspace sends an 128k
> write with the RWF_ATOMIC flag, if the storage device will support
> that an all-or-nothing write with the given size and alignment the
> kernel will guarantee that it will be sent as a single 128k request
> --- although from the database perspective, if it is using 16k
> database pages, it only needs to guarantee that if the write is torn,
> it only happen on a 16k boundary.  That is, if the write is split into
> 32k and 96k request, that would be totally fine as far as the database
> is concerned --- and so the RWF_ATOMIC interface is a stronger
> guarantee than what might be needed.
> 
> So far, the "atomic write" patchset has only focused on Direct I/O,
> where this stronger guarantee is mostly harmless, even if it is
> unneeded for the original motivating use case.  Which might be OK,
> since perhaps there might be other future use cases where they might
> want some 32k writes to be "atomic", while other 128k writes might
> want to be "atomic" (that is to say, persisted with all-or-nothing
> semantics), and the proposed RWF_ATOMIC interface might permit that
> --- even though no one can seem top come up with a credible use case
> that would require this.
> 
> 
> However, this proposed interface is highly problematic when it comes
> to buffered writes, and Postgress database uses buffered, not direct
> I/O writes.   Suppose the database performs a 16k write, followed by a
> 64k write, followed by a 128k write --- and these writes are done
> using a file descriptor that does not have O_DIRECT enable, and let's
> suppose they are written using the proposed RWF_ATOMIC flag.   In
> order to provide the (stronger than we need) RWF_ATOMIC guarantee, the
> kernel would need to store the fact that certain pages in the page
> cache were dirtied as part of a 16k RWF_ATOMIC write, and other pages
> were dirtied as part of a 32k RWF_ATOMIC write, etc, so that the
> writeback code knows what the "atomic" guarantee that was made at
> write time.   This very quickly becomes a mess.
> 
> Another interface that one be much simpler to implement for buffered
> writes would be one the untorn write granularity is set on a per-file
> descriptor basis, using fcntl(2).  We validate whether the untorn
> write granularity is one that can be supported when fcntl(2) is
> called, and we also store in the inode the largest untorn write
> granularity that has been requested by a file descriptor for that
> inode.  (When the last file descriptor opened for writing has been
> closed, the largest untorn write granularity for that inode can be set
> back down to zero.)
> 
> The write(2) system call will check whether the size and alignment of
> the write are valid given the requested untorn write granularity.  And
> in the writeback path, the writeback will detect if there are
> contiguous (aligned) dirty pages, and make sure they are sent to the
> storage device in multiples of the largest requested untorn write
> granularity.  This provides only the guarantees required by databases,
> and obviates the need to track which pages were dirtied by an
> RWF_ATOMIC flag, and the size of the RWF_ATOMIC write.
> 
> I'd like to discuss at LSF/MM what the best interface would be for
> buffered, untorn writes (I am deliberately avoiding the use of the
> word "atomic" since that presumes stronger guarantees than what we
> need, and because it has led to confusion in previous discussions),
> and what might be needed to support it.
> 
> 						- Ted
>