Received-SPF: pass (google.com: domain of linux-kernel+bounces-210147-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) client-ip=2604:1380:40f1:3f00::1;
Message-ID: <fc768320-f4a2-43d9-a7de-4441b60ced28@oracle.com>
Date: Tue, 11 Jun 2024 16:23:22 +0100
User-Agent: Mozilla Thunderbird
Subject: Re: [LSF/MM/BPF TOPIC] untorn buffered writes
To: Theodore Ts'o <tytso@mit.edu>
Cc: Luis Chamberlain <mcgrof@kernel.org>, David Bueso <dave@stgolabs.net>,
        lsf-pc@lists.linux-foundation.org, linux-fsdevel@vger.kernel.org,
        linux-mm <linux-mm@kvack.org>,
        "Martin K. Petersen" <martin.petersen@oracle.com>,
        Matthew Wilcox <willy@infradead.org>,
        Dave Chinner <david@fromorbit.com>, linux-kernel@vger.kernel.org,
        catherine.hoang@oracle.com
References: <20240228061257.GA106651@mit.edu>
 <9e230104-4fb8-44f1-ae5a-a940f69b8d45@oracle.com>
 <Zk5qKUJUOjGXEWus@bombadil.infradead.org>
 <bf638db9-c4d3-44bd-a92c-d36e3d95adb6@oracle.com>
 <20240601093325.GC247052@mit.edu>
Content-Language: en-US
From: John Garry <john.g.garry@oracle.com>
Organization: Oracle Corporation
In-Reply-To: <20240601093325.GC247052@mit.edu>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: 
	=?utf-8?B?ZmJHTE0zK3BpaXRWcTg0anJDMy8xUmZMNllxRExIWU5PY3NvK2FTREIvbndF?=
 =?utf-8?B?Y3pNWHlVUW1KMElydTFMOFVLV3ZzM25zeWl0UnRkZlBYUS91S01MRVhpZ3ZL?=
 =?utf-8?B?eEpPZzJIZmwwNURBdU5uV3Z2VEFwcS9iNW9lT01Ld01PQXUyQ3ptbTJWZEt5?=
 =?utf-8?B?ZERPVGVzNFJOUXhlVllOSnJWSVNtdVdiNUdqVVQyd0RpK1FwVWVaTGdIbXZ0?=
 =?utf-8?B?Qk9JWE9TanZyaEFmRGl6a0JsTHVDN1FwREM4d0JkZkxsK0xJN254L2t5bWF3?=
 =?utf-8?B?MWQ2LzYxandwbmxLc3Jmelh2Wjk2ZVo1OWMzTFpUOUh6SGJyVVlFbS9JMEdO?=
 =?utf-8?B?a2tJVkJ0S2Z1NGJhRnQxeDZKMkZaNGpPSjlKS2dkS3pMKzhQTklnYzdIQjdU?=
 =?utf-8?B?VCtTb2hrbzRkaDNmdFJPY2dpSit2b2U3Q2ZqN0JSdlVGUythMDNoTUxuaVR3?=
 =?utf-8?B?T1cxbVcyRW1jVVd6UkszUE9MTEY1aFg5ZUFCMmFiYWNhQ0RmUmFjWHErT1lM?=
 =?utf-8?B?cmNHVjNrdXVDdEcvZTNWMVdKSEhjWEYzRzVqV2xRSnNmcUNzMS9KQitVQ2Vn?=
 =?utf-8?B?M0lMWnBuY3R0ZUlpamZlTkFFaG1FMVFqOWhEaVV5b0FnKzBzZWFGb0FuaWdp?=
 =?utf-8?B?MGlmcFBZclQxTitFdFpmNXFjNEdCbm1HUThPNkRaZlRrU1p4SUFqMis4TVlZ?=
 =?utf-8?B?Q2lVY2hMOWhYb1I4ajlic2FYMjU0em95V2NnbUZueEtGem5KUmswZXhQZ0Q1?=
 =?utf-8?B?MmtiaVFUN0dZS3lqTXZiZFE4SGRkOVVTWkZlcmNySDBjMHdHK3NpR1J2eU1m?=
 =?utf-8?B?YUJzVnZJc1pkNmpHYzdPWjZTd1V6OE5KNzErRWc2ZHhMNWRNZWVTWkZYeUhN?=
 =?utf-8?B?aThxekRYcUNYbVRRK0V0QkludmdFQm9SZkxKM1psejF6TDFQOW95NkFvRThr?=
 =?utf-8?B?MmVhd2J4dGlBUTVuczVKQXh0d3BTbVM2RXJPZFhpKzBndUpKbUZoaEFEN2Y0?=
 =?utf-8?B?TlU3OUtmVmlvbGJraFB4eDNqd0ladFlTWjY1OVBFRGpPbFY1RkNuSWVuYklv?=
 =?utf-8?B?eGhIYit2SFByR25PZEducE1nVU0zQ3o0SVptMUF5ZXJKN0VaMWt6REh5ZlJZ?=
 =?utf-8?B?b2tLZ2IyaW5Uc2VGYllBcGpnbStvcHMzdFpLZlRUSExQNjlzS0FrRnR5cDBu?=
 =?utf-8?B?THRDMzEvc202MDVXeW9aeDAzZ0VQYVIwN25xZ0pyeDFnTVQ3SDBHYWdCQzVC?=
 =?utf-8?B?bGgvV2xsVUlUMnZ5NnUwWE5Kb21iUFhDVFAvRG55eG5zYjhEWmVwbHVyNnBS?=
 =?utf-8?B?OS94NWFmZmUwN0tEMGd1T1huWWptMnQ2d3UwU1Z0WWdxSVVjWkJxVHYvdkQx?=
 =?utf-8?B?dVhkZXIwMlFqQkVDdFA5YmZjaHZCUWp5YUpiQkx3YjcyM2N1RithT3hCQkI1?=
 =?utf-8?B?U3V6WGR4L2dtRTNpcXZQbFJINDVYbktKY0N3U1ZKUWlIZTlOaFpsMndFQVB3?=
 =?utf-8?B?VWRvVU5CR2YvRFRKVmpGYTRlRGlybEhHYUZBVWZpSHowWG95N0szRlJ3dWVr?=
 =?utf-8?B?bENIVlFXRWhzeG9Iem5kZEJLdytlSy9JODZEcWlleCtJdTJjWGxhbWg0Nytn?=
 =?utf-8?B?NWF1V0lRbVR4RFIycEZMOUV1bkVyRmpVb2VmTHNiblZSQlpHZ3NEUnBMWGhO?=
 =?utf-8?B?YzVNNVovU3A0cWxST0tMcUc4SElROXNlOXhvTmh1K29JRDB4TzdheG4wYk1w?=
 =?utf-8?B?L2VBek5JU00xRlhHNWM2YVFCUEFrTEx6WSt5aDNuUlhXa1R2bTV6bEFTY2lz?=
 =?utf-8?B?dG0yd0pRSk80STVtNHUzVnpUa1RUYVJkaG1jRURuMjAzT2crQXVYbGdZSWM3?=
 =?utf-8?B?U1ZaZUtCM3ZhSWVPRjJiZVI0ODJoOERUUGt3emlIeUV2MTZsay9BQUgxd2tN?=
 =?utf-8?B?Z3d1dFhvRUVRUGZWeWM1NG1oRklFbFdIWEtQK3RZQzNHSU03WDJSUFNEYlND?=
 =?utf-8?B?ZmI0LzZPK0R5Rms0VTEva3d0N3BRWElJSlFMVGoyTWtaRExWNG9vMTYrNVZX?=
 =?utf-8?B?K2h2bzZZMDd6bmZTSVVHZE03MmNXMmg5QXdhSHI0bEdNdUpuZGVEbElra3Zj?=
 =?utf-8?B?RU5yTVlHVFRIQXhpeWJXT0srRW50UWRUcTZ1TnUvOGFTNFNJWnhiengyczZ3?=
 =?utf-8?B?M1E9PQ==?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 
	Rp6gWgUXUzTH6NW0Iuu2oiesxDa+pfrqkC1cL1kqv99VjI4ARr0UYAKIxnoupC7TDlodFvCyt2RF/LdyyIRosxgIJZl1K1ab9mTqk/cRc123//1JHzziUFxxKn0KLJ2tM/XVJ53w5a3fEAZqCYdqBUusT3iLvD8J6jsqBbmCSbA7KQKanfJFg9HE3rPLcKR5xoY/nbDKrGu0Q9NmRlzaTCEalsfK8RZYHDe4/yfZvqRSKwS9qSFXmxX8/7lzabkTIOk6Bop23yuuU21D97ujFe7YnqgJGf1YaYM6/pf9QZCWofk48doQdF7oHsaSqmdO7sHK9O7QUUzCa5tJchfKlnneuKI8V77At4xQSxn90G1CoXyPCWdyi2DAGTcgG9knS6ryko6vhMLenfnjqYqp5WoJiMJCUVEDQtbslLeyNRDCt7Swwgn4SYIXvCtNW/OyhILL2sqFsIhZ00gQaTQVtGavmJTuhVcA5bwk5GjzD5O3GfdE9S6Sksxl0xrMpfcPr0s7tOaHjOG6lh3uwPoy8Wiw1pPPT+qq+K19lm3a4atmGko681nZTcKk5DbfpAy1YQHHc37CUXQfo/2tio6tgoLXolZOACEpPeUsFAg4lHY=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 4d651431-1625-442b-4d30-08dc8a2a75f0
X-MS-Exchange-CrossTenant-AuthSource: DM6PR10MB4313.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 11 Jun 2024 15:23:35.4845
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: d4wt7M/lqlz0wN6eOhFvLxHCQaP3+TyScZxIDqOLqibCbmD4d8wK7ICWDHKkMaMbYxjto07hrXawbh1gYZbHAQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR10MB5970
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.293,Aquarius:18.0.1039,Hydra:6.0.680,FMLib:17.12.28.16
 definitions=2024-06-11_09,2024-06-11_01,2024-05-17_01
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 mlxlogscore=942
 bulkscore=0 malwarescore=0 spamscore=0 phishscore=0 mlxscore=0
 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2405010000 definitions=main-2406110111
X-Proofpoint-GUID: __X6WnEfLKdQWb6Zh69vCotQ9trXf75S
X-Proofpoint-ORIG-GUID: __X6WnEfLKdQWb6Zh69vCotQ9trXf75S

On 01/06/2024 10:33, Theodore Ts'o wrote:
> On Thu, May 23, 2024 at 12:59:57PM +0100, John Garry wrote:
>>
>> That's my point really. There were some positive discussion. I put across
>> the idea of implementing buffered atomic writes, and now I want to ensure
>> that everyone is satisfied with that going forward. I think that a LWN
>> report is now being written.
> 
> I checked in with some PostgreSQL developers after LSF/MM, and
> unfortunately, the idea of immediately sending atomic buffered I/O
> directly to the storage device is going to be problematic for them.

This was not my idea (for supporting buffered atomic writes).

As I remember, that was a candidate solution for dealing with the 
problem that is how to tag a buffered write as atomic. Or deal with 
overlapping atomic writes. And that solution is to just write through, 
so we don't need to remember if it was atomic.

For performance reasons, I was not keen on that, and prefer the solution 
I already mentioned earlier.

> The problem is that they depend on the database to coalesce writes for
> them.  So if they are doing a large database commit that involves
> touching hundreds or thousands of 16k database pages, they today issue
> a separate buffered write request for each database page.  So if we
> turn each one into an immediate SCSI/NVMe write request, that would be
> disastrous for performance. 

FWIW, atomic writes support merging in the block layer.

But, that aside, IMHO, talking about performance like this is close to 
speculation.

> Yes, when they migrate to using Direct
> I/O, the database is going to have to figure out how to coalesce write
> requests; but this is why it's going to take at least 3 years to make
> this migration (and some will call this hopelessly optimistic), and
> then users will probably wait another 3 to 5 years before they trust
> that the database rewrite to use Direct I/O will get it right and
> trust their enterprise workloads to it....
> 
> So I think this goes back to either (a) trying to track which writes
> we've promised atomic write semantics, or (b) using a completely
> different API that only promises "untorn writes with a specified
> granulatity" approach for the untorn buffered writes I/O interface,
> instead in addition to, or instead of, the current "atomic write"
> interface which we are currently trying to promulate for Direct I/O.
> 
> Personally, I'd advocate for two separate interfaces; one for "atomic"
> I/O's, and a different one for "untorn writes with a specified
> guaranteed granularity".  And if XFS folks want to turn the atomic I/O
> interface into something where you can do a multi-megabyte atomic
> write into something that requires allocating new blocks and
> atomically mutating the file system metadata to do this kind of
> atomicity --- even though the Database folks Don't Care --- God bless.

At this stage, if people want buffered atomic writes support for 
PostgreSQL - and not prepared to wait for or help with direct io support 
for that DB - then they need to design/extend a kernel API, implement 
that, and then port PostgreSQL. Then the performance figures can be 
seen. And then try to upstream kernel support.

We have already done such a thing for MySQL for direct IO. We know that 
the performance is good, and we want to support it in the kernel today.

> 
> But let's have something which *just* promises the guarantee requested
> by the primary requesteres of this interface, at least for the
> buffered I/O case.
> 

I think that you need decide whether you want to endorse our direct IO 
support today (and give acked-by or similar), or .. live with probably 
no support for any sort of atomic writes in the kernel...

Thanks,
John