Received-SPF: pass (google.com: domain of linux-kernel+bounces-52729-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223;
Message-ID: <e61cf382-66bd-4091-b49c-afbb5ce67d8f@oracle.com>
Date: Mon, 5 Feb 2024 13:36:03 +0000
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH RFC 5/6] fs: xfs: iomap atomic write support
Content-Language: en-US
To: "Darrick J. Wong" <djwong@kernel.org>
Cc: hch@lst.de, viro@zeniv.linux.org.uk, brauner@kernel.org,
        dchinner@redhat.com, jack@suse.cz, chandan.babu@oracle.com,
        martin.petersen@oracle.com, linux-kernel@vger.kernel.org,
        linux-xfs@vger.kernel.org, linux-fsdevel@vger.kernel.org,
        tytso@mit.edu, jbongio@google.com, ojaswin@linux.ibm.com
References: <20240124142645.9334-1-john.g.garry@oracle.com>
 <20240124142645.9334-6-john.g.garry@oracle.com>
 <20240202184758.GA6226@frogsfrogsfrogs>
From: John Garry <john.g.garry@oracle.com>
Organization: Oracle Corporation
In-Reply-To: <20240202184758.GA6226@frogsfrogsfrogs>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: 
	=?utf-8?B?bitzNi8xRmJnaDlrSjBmTmZYRDIyWjRiRkt6MEZrSStMQ2hadnVWZjkreFBK?=
 =?utf-8?B?Z1lkNWNyOEFFT1hvdVpMUXdpRmZLSmNEWkJuRWlaWDlJdHZLdW92T2N4V0VT?=
 =?utf-8?B?WU8yb1V0QWdRZi8yNTZDZ0VXMk55K0o5NncwYlhabGJEdkVwTjJJQUdkeitG?=
 =?utf-8?B?TU5qNzZDU1dzNTFlRnFPMDNkUzlFRkxlYWRIY0tLR2plUGdGS05MZnc3cWtx?=
 =?utf-8?B?dTZBM3hrdHBDdDU0UFpLRXc2eEQzSHc3VUxRN1JNdTJvZitDV0RuMzVOV2J0?=
 =?utf-8?B?M1JLWkZwK2hjZ1JKYkdxdWFtVERGdUdrNDRmYkR3NzBsTVJKZ3RXZTBOcWVR?=
 =?utf-8?B?MzllMW9rRlkyYUlHSUV0UDN0Q09OQks4d3dQU29ZZ21rSlVaTmNHa2lFUHB5?=
 =?utf-8?B?V0dISGZnd3MvRXRQdkl6SjJyRitsMGNLaWsveVorMjJnQkxaQWlnQzhxV2Jm?=
 =?utf-8?B?eEhzSDQvc2czbDYxL2FhaUVvcG5ZdVZibWNuNzl0ZUluZ0RoOFBxNlIwcFhB?=
 =?utf-8?B?S0F5NjFQa1pmSW1OU1BHZEJUbnVhc3ZhcDAyK1FjUVZER0dVM3MwWjJWWGhC?=
 =?utf-8?B?K2hUVVAxZXJOQis0ZTlqdkVXRVhRVWxzQWkvdExBSzR1d1NqWVhHbDY3ZHdi?=
 =?utf-8?B?UXRIY3l3MExxSDcvS3RpcG56c3NHUTEzTjc4NVlnNkJZb0ZWenBqbXVjaFdW?=
 =?utf-8?B?UmVDYkVKR0ViL1VEbGdWNFNRTWhPcHM4SWR2cDVnWWdtMTduSlFWYVdscmFX?=
 =?utf-8?B?clRkaS96UmVmWjFpYXk5SS9tUlpjdlVKL0N3a2U2U3VZMW5DVzdnTVhkMGFI?=
 =?utf-8?B?Tnh5S0o0d1p6Y3pCd21pL0VSTGJJQ3NBM0ZLd1dEckFzNEgveTBkZnlTUzRR?=
 =?utf-8?B?bHRRWHJubzhLREFyTVVrRW1MMHJJd2ZkNnExL0R0TzY0amJlTlFFN0ZWcFNJ?=
 =?utf-8?B?ZUIyU2tpdjFzaEZQVUNDSWphQnA4L3Jhdmptc0tHbTBJRnoraGpRcVZGaG1k?=
 =?utf-8?B?NU1MczF5bEczV0FzdHhEQmttazQ0dGtDRTJrQUtGU013eHh0RUhOd2tUMk14?=
 =?utf-8?B?N1MrVmhoVExLOW8vRWlLNmtoWDlHZHMzS00yd25uN2hTYnpUc0cvNFh6bnI5?=
 =?utf-8?B?N2lZYjd6NjVHY3FGZjRZd0ttR0FJNTFMV2gvSS82KzdwejN3WTUzNTBYTXVI?=
 =?utf-8?B?cXZ0am9EN2pmZ3dZQ1Q4VnNtK2RNMkhGMlNHMVRrbmxOWFVjNTRXcGpsNWJj?=
 =?utf-8?B?dlpMQ2FTVmhWamlsbzB5ZEkwc2VzTTRRQ0xNTkFhYVRzRlQ1eXFkRE9EeE5j?=
 =?utf-8?B?MWpISTVSYmVOaVQxcmNzZjRlbGVDMnplSEhzNzdvY0FURkxoWVkzN3p4QlRF?=
 =?utf-8?B?RDlEdmg4SlZZMmQ4WkplcnFFK2NPT1BvSnlhRnIwSldpS3hjQWprekNrUkpv?=
 =?utf-8?B?WjN5VVYwNmVzNmlqd3RSNitUZzFUMlNCQVE0bW1LYlNmZFRSd3NoNlJscDNy?=
 =?utf-8?B?S29EU01BaXh4aVY0M0ZabkRUcjV5azFaVk5sbmU1N1ROcGd4TnQwbjdyRmV4?=
 =?utf-8?B?dTdSZGZ2TS9sVXlrMzVnc1o3YkVzdzJiRzZkbDA4SlhFN2RBVkY0ZnRPcWJo?=
 =?utf-8?B?RUwvTG9RdU42czRMNjZBM3VDK1Q3TXBnSXpXa2dxYVQ4ampFRTUzZHJONVdF?=
 =?utf-8?B?VEZYcDhQNlo2TDJWczl0V00wUkt1alNRMmx4STRhNjBHSzVBdUF3WnEvZllh?=
 =?utf-8?B?MzZIK2NhdDJSUTZseUFuVkQ4Q2pNc29xdGNFaDJEVG5aVmRmMXBSd1E3bjky?=
 =?utf-8?B?dVpCMjJHeDZkclZUSUc0VVBoMzdPOW5aUXVaSzlqK0tJYXNTYzFvUUwzTERq?=
 =?utf-8?B?WEpOV0lwdFlmNS92TjJOQVM1RVB0dU9kSmNVRnVLZGxJV2FVbklUN2x2SFRr?=
 =?utf-8?B?bHdSajgwSitKNWJZN21mMGhJZEVNc2VieUJWdm1PR2VhV3Fza2FWb3NMZlJQ?=
 =?utf-8?B?bGYyTjMzR2hXMWkvZmJzekdMdU14Q2ZKdFgwS2FSaXEvNnJuVVV3eUpUcWEz?=
 =?utf-8?B?dW5XZEw2K1NRSU9uS091VDl6QzR2cFZOVnFlVExHKzFkUEE3Q3pQZHJ4WWl4?=
 =?utf-8?Q?uOkSysLyIDYkQU0j1VhTQsjdq?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 
	YpJejQo8QgiCegeJpD8ayXfz2tlkYl61Wq3g60yd0Si2c19YVs6owGXs5sUZDrp4tA5LoPiP5dtA7sUEhL6dxIWdczR+CahCSxlpZJwwxnq1WhaLS6xYN9dTwgnHhxX1+BVIsWxmhlZ/8sJhLLNsdRYzuLRlOCu2lhYKcBP2HvA95pqRn7nB73Bqbx2uBiHT4KMv8Y3xBpNZ8fGcP5/PkNDAU7zquD7xYZtxS96c4GiZLFsmm20cOkeMgI9zNui5/ZD2gFec1KS+yWIiLelW+aFUWsKOfKRGhSDGLAHeKelKOrTqZAocIG4OaaIQX/2p7g24151TgzNT86Ok+rBL5TP0pZa1qv95CXWIGlgNXFdFcEj4lYkXpZa84HZVKnIFN5aX9Og2bm+ssxvBgeA0AICJnhsOQfA1lUpPfFsKePy0h2Mc1zyl91PgG6pi52zEFc3WKPHXxgKt9IH5isyzZdnCnCv1kuS4a4ZouVsi5axW95j+vpGN/6zwZGT5294PMK+HopKlYPn2EradMhz2p21GnopLaDA0/dD0JIzX1ejWTn8ETAAf9O/2sG2zEDB2KYPvqSYzLhOK9+R+ewPpKCjYuzhSU2+E+OawUzlCsTk=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: f77e0926-306b-453f-1974-08dc264f68f2
X-MS-Exchange-CrossTenant-AuthSource: DM6PR10MB4313.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Feb 2024 13:36:08.6572
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: vPNAo+tbJ+59fWCVhIkv2ZG1atYp2yhf+EZzwt1lzebiqITKqHLkwbvvrvnweWwJN80EpncZ2zmg2+YUX6/IpA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN0PR10MB4919
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.272,Aquarius:18.0.1011,Hydra:6.0.619,FMLib:17.11.176.26
 definitions=2024-02-05_08,2024-01-31_01,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 suspectscore=0
 mlxlogscore=999 bulkscore=0 mlxscore=0 adultscore=0 malwarescore=0
 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2311290000 definitions=main-2402050103
X-Proofpoint-GUID: UNK5v-WCnGAgErvVyDFFHmiBxmuBUnxx
X-Proofpoint-ORIG-GUID: UNK5v-WCnGAgErvVyDFFHmiBxmuBUnxx

On 02/02/2024 18:47, Darrick J. Wong wrote:
> On Wed, Jan 24, 2024 at 02:26:44PM +0000, John Garry wrote:
>> Ensure that when creating a mapping that we adhere to all the atomic
>> write rules.
>>
>> We check that the mapping covers the complete range of the write to ensure
>> that we'll be just creating a single mapping.
>>
>> Currently minimum granularity is the FS block size, but it should be
>> possibly to support lower in future.
>>
>> Signed-off-by: John Garry <john.g.garry@oracle.com>
>> ---
>> I am setting this as an RFC as I am not sure on the change in
>> xfs_iomap_write_direct() - it gives the desired result AFAICS.
>>
>>   fs/xfs/xfs_iomap.c | 41 +++++++++++++++++++++++++++++++++++++++++
>>   1 file changed, 41 insertions(+)
>>
>> diff --git a/fs/xfs/xfs_iomap.c b/fs/xfs/xfs_iomap.c
>> index 18c8f168b153..758dc1c90a42 100644
>> --- a/fs/xfs/xfs_iomap.c
>> +++ b/fs/xfs/xfs_iomap.c
>> @@ -289,6 +289,9 @@ xfs_iomap_write_direct(
>>   		}
>>   	}
>>   
>> +	if (xfs_inode_atomicwrites(ip))
>> +		bmapi_flags = XFS_BMAPI_ZERO;
> 
> Why do we want to write zeroes to the disk if we're allocating space
> even if we're not sending an atomic write?
> 
> (This might want an explanation for why we're doing this at all -- it's
> to avoid unwritten extent conversion, which defeats hardware untorn
> writes.)

It's to handle the scenario where we have a partially written extent, 
and then try to issue an atomic write which covers the complete extent. 
In this scenario, the iomap code will issue 2x IOs, which is 
unacceptable. So we ensure that the extent is completely written 
whenever we allocate it. At least that is my idea.

> 
> I think we should support IOCB_ATOMIC when the mapping is unwritten --
> the data will land on disk in an untorn fashion, the unwritten extent
> conversion on IO completion is itself atomic, and callers still have to
> set O_DSYNC to persist anything. 

But does this work for the scenario above?

> Then we can avoid the cost of
> BMAPI_ZERO, because double-writes aren't free.

About double-writes not being free, I thought that this was acceptable 
to just have this write zero when initially allocating the extent as it 
should not add too much overhead in practice, i.e. it's one off.

> 
>> +
>>   	error = xfs_trans_alloc_inode(ip, &M_RES(mp)->tr_write, dblocks,
>>   			rblocks, force, &tp);
>>   	if (error)
>> @@ -812,6 +815,44 @@ xfs_direct_write_iomap_begin(
>>   	if (error)
>>   		goto out_unlock;
>>   
>> +	if (flags & IOMAP_ATOMIC) {
>> +		xfs_filblks_t unit_min_fsb, unit_max_fsb;
>> +		unsigned int unit_min, unit_max;
>> +
>> +		xfs_get_atomic_write_attr(ip, &unit_min, &unit_max);
>> +		unit_min_fsb = XFS_B_TO_FSBT(mp, unit_min);
>> +		unit_max_fsb = XFS_B_TO_FSBT(mp, unit_max);
>> +
>> +		if (!imap_spans_range(&imap, offset_fsb, end_fsb)) {
>> +			error = -EINVAL;
>> +			goto out_unlock;
>> +		}
>> +
>> +		if ((offset & mp->m_blockmask) ||
>> +		    (length & mp->m_blockmask)) {
>> +			error = -EINVAL;
>> +			goto out_unlock;
>> +		}
>> +
>> +		if (imap.br_blockcount == unit_min_fsb ||
>> +		    imap.br_blockcount == unit_max_fsb) {
>> +			/* ok if exactly min or max */
>> +		} else if (imap.br_blockcount < unit_min_fsb ||
>> +			   imap.br_blockcount > unit_max_fsb) {
>> +			error = -EINVAL;
>> +			goto out_unlock;
>> +		} else if (!is_power_of_2(imap.br_blockcount)) {
>> +			error = -EINVAL;
>> +			goto out_unlock;
>> +		}
>> +
>> +		if (imap.br_startoff &&
>> +		    imap.br_startoff & (imap.br_blockcount - 1)) {
> 
> Not sure why we care about the file position, it's br_startblock that
> gets passed into the bio, not br_startoff.

We just want to ensure that the length of the write is valid w.r.t. to 
the offset within the extent, and br_startoff would be the offset within 
the aligned extent.

> 
> I'm also still not convinced that any of this validation is useful here.
> The block device stack underneath the filesystem can change at any time
> without any particular notice to the fs, so the only way to find out if
> the proposed IO would meet the alignment constraints is to submit_bio
> and see what happens.

I am not sure what submit_bio() would do differently. If the block 
device is changing underneath the block layer, then there is where these 
things need to be checked.

> 
> (The "one bio per untorn write request" thing in the direct-io.c patch
> sound sane to me though.)
> 

ok

Thanks,
John