Received-SPF: pass (google.com: domain of linux-kernel+bounces-21865-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249;
Message-ID: <aaa33b4f-dea7-4596-82ce-8c7e6cdaa6ef@oracle.com>
Date: Wed, 10 Jan 2024 08:55:06 +0000
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH v2 00/16] block atomic writes
To: Dave Chinner <david@fromorbit.com>
Cc: Christoph Hellwig <hch@lst.de>, "Darrick J. Wong" <djwong@kernel.org>,
        axboe@kernel.dk, kbusch@kernel.org, sagi@grimberg.me,
        jejb@linux.ibm.com, martin.petersen@oracle.com,
        viro@zeniv.linux.org.uk, brauner@kernel.org, dchinner@redhat.com,
        jack@suse.cz, linux-block@vger.kernel.org,
        linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org,
        linux-xfs@vger.kernel.org, linux-fsdevel@vger.kernel.org,
        tytso@mit.edu, jbongio@google.com, linux-scsi@vger.kernel.org,
        ming.lei@redhat.com, bvanassche@acm.org, ojaswin@linux.ibm.com
References: <b8b0a9d7-88d2-45a9-877a-ecc5e0f1e645@oracle.com>
 <20231213154409.GA7724@lst.de>
 <c729b03c-b1d1-4458-9983-113f8cd752cd@oracle.com>
 <20231219051456.GB3964019@frogsfrogsfrogs> <20231219052121.GA338@lst.de>
 <76c85021-dd9e-49e3-80e3-25a17c7ca455@oracle.com>
 <20231219151759.GA4468@lst.de>
 <fff50006-ccd2-4944-ba32-84cbb2dbd1f4@oracle.com>
 <20231221065031.GA25778@lst.de>
 <73d03703-6c57-424a-80ea-965e636c34d6@oracle.com>
 <ZZ3Q4GPrKYo91NQ0@dread.disaster.area>
Content-Language: en-US
From: John Garry <john.g.garry@oracle.com>
Organization: Oracle Corporation
In-Reply-To: <ZZ3Q4GPrKYo91NQ0@dread.disaster.area>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: 
	=?utf-8?B?UE1YTllvejBkOElaazJKa3RTNGl0K01uTzRpbC93ZWN4cXpOOU93Tk40VG9t?=
 =?utf-8?B?T0hVdzVueVNUMG4wMlQ4c1ZWZWtIaGtJT1F1SURTQjE3aHZWMDdnbmkvNUtX?=
 =?utf-8?B?L2tRZGlKMThkbnFUWlZtOTFqYzM2MXFlR09TdklWcTd3WEFHc2VNRXorc3VG?=
 =?utf-8?B?b3hFbkVyNENudXVvSzF0ZWVuUlJRRUsxVlVJL0hINEFTdzhDMUc1azBwWTQ5?=
 =?utf-8?B?eWtCMXRVcmNkTzlWVXpVVTlOcmUvRm9DelJEWmE0OEdQZWFmWEtpMGZYYzBU?=
 =?utf-8?B?UUlQcXhYbGloQmgrOTltZkZNZnBXTFhCdzZYdHlwNklJbzQ4T1B3VlpuM0FT?=
 =?utf-8?B?dk9zTkJRTlFuaGJnY2huWnF2dHE2R2p0ZCtPa1VqQmpNYnFJTkQ3OUwyUVNB?=
 =?utf-8?B?akRzdHh0ZHIzWk50OHlMU3BSdytMWnZ4eWkxYWVzc29MRmwyUncvRDgyOU5p?=
 =?utf-8?B?SHYvMU84OEZCajlxL2hiN3d0Z296YW1Uck1mTnNJbGhMWDVnbHN3d1VlZWZB?=
 =?utf-8?B?OHd2SFB0NVpySjRrMldqbTdrdWEybWFTVXBXalhXYmxFV0xlNmJ3MTA0dE52?=
 =?utf-8?B?UXErTGtVK2ZNUnlnQjIrWFVQNFV6VkR0M0dpV3plNDYvdkdvT20wOUZJY3pB?=
 =?utf-8?B?QWwyd251OHhNRFNRakdzTWFoSmM3MkVyalRMVXg3TlJ4Y3VadUVZZ29ENkt4?=
 =?utf-8?B?ZEhCRkl0NEhuUk1UM1pIVXFpZVkxRnB3ekw1MjZzWmtTSmRsME9jdEJOcEk5?=
 =?utf-8?B?Q0gycytpTDBDR2kyWEo2YTVJS1kwZ1hOLzYrUForT0JaajFzTXdrV3dBVmhM?=
 =?utf-8?B?S05SYkRtNFRHOTNTeGlKTllUVjlIcFh1bzlVR2VCdUxnUjNESWw1TjdJUC9a?=
 =?utf-8?B?eXN4MzlLZTRQeEhJdDI5dkdGb2tIdWNVWTF1b1dzZFZ3NFR0dDIxN3hNN1J6?=
 =?utf-8?B?TlpEd2JYY3pPZ0JFaUVxcHp5b0FFeWNDUm5wQjA4OFF1L203dzFKYmxhaXlR?=
 =?utf-8?B?Z0VBOVZLTEpmM3J5OXV1OTJTazhFdWRMVWxEQmk0K2V6MG1Tc3ZSR0tsRTdu?=
 =?utf-8?B?azFwSGNETk5wQmdxTUJwR2Q4SWtPWkhpbnQza3hyTmo0UWZwTVlqSUluLzh3?=
 =?utf-8?B?eTg0L2FuVGdmZGI4S1hacDV5ZGw0aUxyQklCRHdLRTNRam1FSm96bUdzNHdt?=
 =?utf-8?B?VWY0aXJMbXFRNDFSb2krKzM0V0ZtZ1Zwb1pWakE4ZUxJaUdzWUhUQ3FxSEdD?=
 =?utf-8?B?VkVicGpMdUVnTjBFd00rTVphQVlYZWpBeU03V21sUklTeVNWMFpGSFJML0RK?=
 =?utf-8?B?TUR2YjNvbkt0Mit0L0FYd2ptb01wejk5OUV1QzBHbVQ2a09yT3p1dUpnZG0r?=
 =?utf-8?B?ZUJ0UWJnYk5zd3lUSm5CeStWZGtCdUh3RU1wTU5Pb0x2N2NFVjZoNTlkTTBz?=
 =?utf-8?B?TGpWMWg0dEJscVRGVkVWdlE2UXA4aG00ZmxFUlVhUGVCNE9WTi96WGlZV3lE?=
 =?utf-8?B?TzVyWWxMalJWVlA2N01uY3FLc2JDdTVGWjhuRkdXZ1FPTHBmbFhIUGJ0ZUY5?=
 =?utf-8?B?bE1malZ1cWlwa1FsL2lGVUFwOUI3dDZ1c243dHo4U2M2ZHc1RVdKdjNHVi9k?=
 =?utf-8?B?ZkhqNTgxd2hYNUZjd0J2S0ZrUE9WNThSZUJ3dmNQcC9mSFpiWVpPUXZ5SzI5?=
 =?utf-8?B?b25ObmwzSkRscXd4Z3g5aDBFY1lSQ0hmZm1xK1dOWFg3ajZEMVNmVWh5WTRD?=
 =?utf-8?B?RnNSbFJQUkV0cW15TlRxVXR2bEFWcFI4QmEyWEJkRDRITFFLaDNXa0ZFTjFX?=
 =?utf-8?B?MmV5Ry90U1ludmxJNGVCRmxQOExxUElJVzY3SmNyUGIySlk5a2pjR3hoM1Zn?=
 =?utf-8?B?T045dGJycUJwaVJFOThNRnBsQ3lMUU5zVlRBMHVuM1RuQXFwc1ExTXdWb01P?=
 =?utf-8?B?OW83b0JGOWRKQjMwQ0h0MUhmQTVvTXVKdWJWK2U5WHBKbWRJbStSNDJnQU1v?=
 =?utf-8?B?UFFVYnM4ZXE4WVNCdUtaOHFWdzlBNlozanJFRU5iMXE0ZlE4YUY5SEEvVzkr?=
 =?utf-8?B?eEJ5QjVHM3JjT0g5OFF5NmVEamlWMWEvZXZ3VGs5enR2ZDA3NUVXTVVjYUVt?=
 =?utf-8?B?WlA3VytUVGxGbXBWSUc1TDgzK29sWDlvV3VtZGxWMGxFbWNZdWUyQ3hWVHp1?=
 =?utf-8?B?K3c9PQ==?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 
	8YDyxOM9CdeyP0y3+lVViOeu1KnfQl6NoqUPr8CH0sJ9pmG+IubBcANb+ZnpHFU1rM59F7NjsHLZzQeLi94Ygg7t5FD/J+T0MwNKDlouRoY5Rh31LMcbvxTV3As4/nXwLFRGzeHWjVU65rXf0+ikQC/ZRknpobYPfqlrHLCNCWhYcAb1Y8bK1BC2arRvvwfrnhjVHFy6QyyvrHtvSG4y2rkNzUSLHIXVVf3KNOxaWBMM5w8u+gJsSwnfYja5WQD3tIEza2oiv4LTEjY0XYB3vrtPryE8U3/yD7XsHhsrD5T3FN5TiM4YtGeorea+HfCUkaDWETuqq96mtryhx3Pa+cj9I1zDPhCQZ3FcTHlIElsPJ/j/hSC7JNmQOnWzMn6MJmo9wYg7AohVbCwRfhG+beMxQQ0dl803r8i7KSVF3B7EgHO+62qMNHsgEQCdH5gMTkySpQIM2ZJkMhgZqqzeKzjua9jAjVbdwV2Tpq8hN+teQh8vA6bKxRoxE3NAAMPdwlBXmvKol+hPAANw375GDVfvcAL9iZ0UHhwMRnFrEMxLLbLoXvJ4oOJ3uuvgGiYwZEP60pCxJG+/65vq/yNsa0mfP1IsbJTRRREnGNcKJic=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: a9e378bf-63f3-4550-4db7-08dc11b9dafd
X-MS-Exchange-CrossTenant-AuthSource: DM6PR10MB4313.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 Jan 2024 08:55:12.3023
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: HOS6gHbBqaCjhW8KJRaJ1LqpBg1atkwy1F2d+73qALXnC5RJUvJZhMpICQ6eIeDv/A1Z53Yk/sk9ZSI8KIzLwg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ2PR10MB6992
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26
 definitions=2024-01-10_02,2024-01-09_02,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 mlxscore=0 bulkscore=0
 mlxlogscore=999 adultscore=0 spamscore=0 phishscore=0 suspectscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000
 definitions=main-2401100072
X-Proofpoint-GUID: HweSHLUbVkVvX-qJ7P4uZFxvEWpgBhrE
X-Proofpoint-ORIG-GUID: HweSHLUbVkVvX-qJ7P4uZFxvEWpgBhrE

On 09/01/2024 23:04, Dave Chinner wrote:
>> --- a/include/uapi/linux/fs.h
>> +++ b/include/uapi/linux/fs.h
>> @@ -118,7 +118,8 @@ struct fsxattr {
>>         __u32           fsx_nextents;   /* nextents field value (get)   */
>>         __u32           fsx_projid;     /* project identifier (get/set) */
>>         __u32           fsx_cowextsize; /* CoW extsize field value
>> (get/set)*/
>> -       unsigned char   fsx_pad[8];
>> +       __u32           fsx_atomicwrites_size; /* unit max */
>> +       unsigned char   fsx_pad[4];
>> };
>>
>> /*
>> @@ -140,6 +141,7 @@ struct fsxattr {
>> #define FS_XFLAG_FILESTREAM    0x00004000      /* use filestream allocator
>> */
>> #define FS_XFLAG_DAX           0x00008000      /* use DAX for IO */
>> #define FS_XFLAG_COWEXTSIZE    0x00010000      /* CoW extent size
>> allocator hint */
>> +#define FS_XFLAG_ATOMICWRITES  0x00020000
>> #define FS_XFLAG_HASATTR       0x80000000      /* no DIFLAG for this   */
>>
>> /* the read-only stuff doesn't really belong here, but any other place is
>> lines 1-22/22 (END)
>>
>> Having FS_XFLAG_ATOMICWRITES set will lead to FMODE_CAN_ATOMIC_WRITE being
>> set.
>>
>> So a user can issue:
>>
>>> xfs_io -c "atomic-writes 64K" mnt/file
>>> xfs_io -c "atomic-writes" mnt/file
>> [65536] mnt/file
> Where are you going to store this value in the inode?  It requires a
> new field in the inode and so is a change of on-disk format, right?

It would require an on-disk format change, unless we can find an 
alternative way to store the value, like:
a. re-use pre-existing extsize or even cowextsize fields and 'xfs_io -c 
"atomic-writes $SIZE"' would update those fields and 
FS_XFLAG_ATOMICWRITES would be incompatible with FS_XFLAG_COWEXTSIZE or 
FS_XFLAG_EXTSIZE
b. require FS_XFLAG_EXTSIZE and extsize be also set to enable atomic 
writes, and extsize is used for atomic write unit max

I'm trying to think of ways to avoid requiring a value, but I don't see 
good options, like:
- make atomic write unit max some compile-time option
- require mkfs stripe alignment/width be set and use that as basis for 
atomic write unit max

We could just use the atomic write unit max which HW provides, but that 
could be 1MB or more and that will hardly give efficient data usage for 
small files. But maybe we don't care about that if we expect this 
feature to only be used on DB files, which can be huge anyway. However I 
still have concerns – we require that value to be fixed, but a disk 
firmware update could increase that value and this could mean we have 
what would be pre-existing mis-aligned extents.

> 
> As it is, I really don't see this as a better solution than the
> original generic "force align" flag that simply makes the extent
> size hint alignment a hard physical alignment requirement rather
> than just a hint. This has multiple uses (DAX PMD alignment is
> another), so I just don't see why something that has a single,
> application specific API that implements a hard physical alignment
> is desirable.

I would still hope that we will support forcealign separately for those 
purposes.

> 
> Indeed, the whole reason that extent size hints are so versatile is
> that they implement a generic allocation alignment/size function
> that can be used for anything your imagination extends to. If they
> were implemented as a "only allow RAID stripe aligned/sized
> allocation" for the original use case then that functionality would
> have been far less useful than it has proven to be over the past
> couple of decades.
> 
> Hence history teaches us that we should be designing the API around
> the generic filesystem function required (hard alignment of physical
> extent allocation), not the specific use case that requires that
> functionality.

I understand your concern. However I am not even sure that forcealign 
even gives us everything we want to enable atomic writes. There is an 
issue where we were required to pre-zero a file prior to issuing atomic 
writes to ensure extents are suitably sized, so FS_XFLAG_ATOMICWRITES 
would make the FS do what is required to avoid that pre-zeroing (but 
that pre-zeroing requirement that does sound like a forcealign issue...)

Furthermore, there was some desire to support atomic writes on block 
devices with no HW support by using a CoW-based solution, and forcealign 
would not be relevant there.

Thanks,
John