Received-SPF: pass (google.com: domain of linux-kernel+bounces-153564-linux.lists.archive=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99;
From: John Garry <john.g.garry@oracle.com>
To: axboe@kernel.dk, brauner@kernel.org, djwong@kernel.org,
        viro@zeniv.linux.org.uk, jack@suse.cz, akpm@linux-foundation.org,
        willy@infradead.org, dchinner@redhat.com, tytso@mit.edu, hch@lst.de,
        martin.petersen@oracle.com, nilay@linux.ibm.com, ritesh.list@gmail.com,
        mcgrof@kernel.org
Cc: linux-block@vger.kernel.org, linux-kernel@vger.kernel.org,
        linux-xfs@vger.kernel.org, linux-fsdevel@vger.kernel.org,
        linux-mm@kvack.org, ojaswin@linux.ibm.com, p.raghav@samsung.com,
        jbongio@google.com, okiselev@amazon.com,
        John Garry <john.g.garry@oracle.com>
Subject: [PATCH RFC 5/7] fs: iomap: buffered atomic write support
Date: Mon, 22 Apr 2024 14:39:21 +0000
Message-Id: <20240422143923.3927601-6-john.g.garry@oracle.com>
In-Reply-To: <20240422143923.3927601-1-john.g.garry@oracle.com>
References: <20240422143923.3927601-1-john.g.garry@oracle.com>
Content-Transfer-Encoding: 8bit
Content-Type: text/plain
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: 
	=?us-ascii?Q?w8gviT6NdX1aPPpzDJ+HJITzUAvCC5Vd7mzgfsmPlnBx2ZpeMhSbqaWsL8SQ?=
 =?us-ascii?Q?mKnUGpvgOFiIxhYVH8IopqDLLXnG3lrZ6H2pUAAMLUu5atujv/uYy8n88003?=
 =?us-ascii?Q?DMJkXTX8MHvchtIzc+WqSI3LJWq8HslIOHYKuJB57GlRuayMXKD2VHwbkBAC?=
 =?us-ascii?Q?POyjl6H/p5Qn6pOYsGRvDCYnGr5WZH0Ou+0eUnJtKQ52HYNyea8XAZYBGAiV?=
 =?us-ascii?Q?pkhJR9x8uY+m84+9hNzv2+6k06fKBUETCyZmVmNcORBJicjUbK3hez6V+OXh?=
 =?us-ascii?Q?HZUlI4iaQRgFAGapnDvpwIPXncjuK+ZPWhD1HMtRFUhQEfPkYY39M3rEBn5s?=
 =?us-ascii?Q?0gx/QRRV8inIZXgGE0p0iL4aIOalxIsWiCS2lOVIwyd6Rw566N5UnGTdw3xp?=
 =?us-ascii?Q?RvFndoXx9BMNuqWeKPbvpTpI++W9s3vnlLKpH5Z1+XG9VG8WWEuEPS4PlGya?=
 =?us-ascii?Q?TmyUqOVlEvzdvYqa+PcXJgqxmkdl8bkViPUxHYbV0ZjuGWTc/TPE3kpbTs9W?=
 =?us-ascii?Q?vdiGOc1GlAPnJFdL3V1BegcjRYZHvzHX0q3xsdEby7TB6hO/uvpAbhJQILBT?=
 =?us-ascii?Q?T8/muFLmeaj/v3IhhfMPJxLzdzL0bXutCF9WiYwHoFX0H8+DnPYil+h9tbjr?=
 =?us-ascii?Q?MyVM3UqKXxC9vEX7kvA6qRwuFbrIY2PU2H+nCGD8xam4XIbrj5LBCs5c2m1q?=
 =?us-ascii?Q?Gk8EBVRoPVPow1Idd7azLn3sA7dzQCG3AuPTm3m9M0GGtjkeEyZMybOEVCaB?=
 =?us-ascii?Q?QcBf+gflqG2PAZ2op0DO8TcbmOOLUlIvJM/KBzPqaoVVclesVJBh0c3kRxxy?=
 =?us-ascii?Q?lieYPRQoxnZh2H8E9Uyfp/pihpls7WgtgMVReM8Fe8BzTpO6vojo2LnLsDfo?=
 =?us-ascii?Q?1diwHEEL5Wc7ZO84UdOzVmJsbwhyWamdAP6kkMk+oit7+71L64oqfbgjWxRP?=
 =?us-ascii?Q?ZnAUKVI/TDRQR79SILRvrqf5h/SneWGB6NwPhJfV3t8LxmqLP0buCtK63fBg?=
 =?us-ascii?Q?1hncqxjrmriDmtEbMxA7onO8Qx8BPfJ942fx+Ln/v2h+SsxtHN/oheSV3kzK?=
 =?us-ascii?Q?tva8ftJdVLUBdGCuz0l7rGFDzWOjkAJKpfovd/xcPWl0hInaDRItkj2NCTQu?=
 =?us-ascii?Q?frS/1tna9mJDe77n+kr5tAZHkF6D1MPLme5p+HSA+sb4edBkdxcfo5fGuLjL?=
 =?us-ascii?Q?JSKYPOR6bcsEWXqPKgZ1/FJwun/WbttvgH+IUDN/EpwJCmJ14CSC/OUjlc7R?=
 =?us-ascii?Q?L7POKcC9gry/BGim7GYpbWzLvIpRouzv4NYufntsj6C+wjRTGBsCwwBsie1+?=
 =?us-ascii?Q?rhFtclImPIA1X6Gb2Zqh2hAd6L1JqilhYe3JL0h2f9CdKa71mcErlzTEIOtF?=
 =?us-ascii?Q?3JjozNkb4ZnOs1OfXKBnU7d6zFtI427L/w4K2WsKmMQ22uzep+DPfTHzUjNP?=
 =?us-ascii?Q?MBoRjLU5Y27558Vx01wo6vn+VvNJYf7Y5w8zWlSeVj+8TnJCJWS43iuYsYdd?=
 =?us-ascii?Q?3/LQeSHsddLZICPy4jYmPzklNQMMtZZjHx3xfPlYY6WOnu289W9zL/oie7T8?=
 =?us-ascii?Q?ifzEmJ6xTw+q0AsziHXG93gBfAGnHu9w6Lk4hreJRb4mOYEL2P4wZNO5MXCQ?=
 =?us-ascii?Q?tA=3D=3D?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 
	iMQ1leu0l6zcaMq5gYg8MwdJafGfmO+Tg8rkrUVaokUubVJfp6BvBCGHJAUcSJdggzUrpH69d1HTge/ZT2Y3zvxTm3rydZTtnTEsip83fkpYlGeiep+oecH4RK7PharPcQAf6gRT7rOyEZLg76t1VEADyR0RPTpq0r4JsMubT6kFtpS8sCLnPpDqfg9AKyUwlHbx+j1H8Y3V1xBlHfjJ9rI0Sv1iayPSuDgbMYGuVcG+0gjskz/N+2z1fva85a9Ici9uBjBpkanLb6f6NMPM5ycEzAL0HCOnvjfb8zd8CXk+7CNAqNFK/v2WbDvgOPhbTvkYr7sjK3s1IxGJMb4qAJMnIWq001L7sA+vUcerJDLHQRvJNrrmsZah6/hEMgH6IE6ICdvZW+mnvDGLMh/jkkJiG4B6xGtiEkf9AI5SgpswYxFk06HvN2YJ+Ae+m1oAxLfgC21aOMeLAEK479RKs5v3vOIGiu0Rdiwf4PnvaWFqeAocWaTLlB2zziRLlNoZrxgUNKZKYwlLe4cAf/KCYlFaKaqxlMbo3+1hwTudWUZpyRL1Opb6gCNVXIvSmnUOmKn60stzsX8cLzXtSa9M+wkygRStb2CN0xn1h9vtois=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: eb47ac47-80ae-4058-f4c4-08dc62da2548
X-MS-Exchange-CrossTenant-AuthSource: DM6PR10MB4313.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Apr 2024 14:40:24.9493
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: EyU4QUJSaQShh/uz0X8SuIHChrzKdgWW4JnX4FDZqVPcC9M6VzKos8zWICuVqoH2/i1W16eHhi/cjgc6nppZKg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS7PR10MB5039
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.293,Aquarius:18.0.1011,Hydra:6.0.619,FMLib:17.11.176.26
 definitions=2024-04-22_09,2024-04-22_01,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 mlxlogscore=999
 mlxscore=0 phishscore=0 spamscore=0 malwarescore=0 adultscore=0
 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2404010000 definitions=main-2404220063
X-Proofpoint-ORIG-GUID: de4RcoK0WpeaFunLdETakunvmUYU-xgk
X-Proofpoint-GUID: de4RcoK0WpeaFunLdETakunvmUYU-xgk

Add special handling of PG_atomic flag to iomap buffered write path.

To flag an iomap iter for an atomic write, set IOMAP_ATOMIC.

For a folio associated with a write which has IOMAP_ATOMIC set, set
PG_atomic.

Otherwise, when IOMAP_ATOMIC is unset, clear PG_atomic.

This means that for an "atomic" folio which has not been written back, it
loses it "atomicity". So if userspace issues a write with RWF_ATOMIC set
and another write with RWF_ATOMIC unset and which fully or partially
overwrites that same region as the first write, that folio is not written
back atomically. For such a scenario to occur, it would be considered a
userspace usage error.

To ensure that a buffered atomic write is written back atomically when
the write syscall returns, RWF_SYNC or similar needs to be used (in
conjunction with RWF_ATOMIC).

As a safety check, when getting a folio for an atomic write in
iomap_get_folio(), ensure that the length matches the inode mapping folio
order-limit.

Only a single BIO should ever be submitted for an atomic write. So modify
iomap_add_to_ioend() to ensure that we don't try to write back an atomic
folio as part of a larger mixed-atomicity BIO.

In iomap_alloc_ioend(), handle an atomic write by setting REQ_ATOMIC for
the allocated BIO.

When a folio is written back, again clear PG_atomic, as it is no longer
required. I assume it will not be needlessly written back a second time...

Signed-off-by: John Garry <john.g.garry@oracle.com>
---
 fs/iomap/buffered-io.c | 53 ++++++++++++++++++++++++++++++++++++------
 fs/iomap/trace.h       |  3 ++-
 include/linux/iomap.h  |  1 +
 3 files changed, 49 insertions(+), 8 deletions(-)

diff --git a/fs/iomap/buffered-io.c b/fs/iomap/buffered-io.c
index 4e8e41c8b3c0..ac2a014c91a9 100644
--- a/fs/iomap/buffered-io.c
+++ b/fs/iomap/buffered-io.c
@@ -586,13 +586,25 @@ EXPORT_SYMBOL_GPL(iomap_is_partially_uptodate);
  */
 struct folio *iomap_get_folio(struct iomap_iter *iter, loff_t pos, size_t len)
 {
+	struct address_space *mapping = iter->inode->i_mapping;
 	fgf_t fgp = FGP_WRITEBEGIN | FGP_NOFS;
 
 	if (iter->flags & IOMAP_NOWAIT)
 		fgp |= FGP_NOWAIT;
 	fgp |= fgf_set_order(len);
 
-	return __filemap_get_folio(iter->inode->i_mapping, pos >> PAGE_SHIFT,
+	if (iter->flags & IOMAP_ATOMIC) {
+		unsigned int min_order = mapping_min_folio_order(mapping);
+		unsigned int max_order = mapping_max_folio_order(mapping);
+		unsigned int order = FGF_GET_ORDER(fgp);
+
+		if (order != min_order)
+			return ERR_PTR(-EINVAL);
+		if (order != max_order)
+			return ERR_PTR(-EINVAL);
+	}
+
+	return __filemap_get_folio(mapping, pos >> PAGE_SHIFT,
 			fgp, mapping_gfp_mask(iter->inode->i_mapping));
 }
 EXPORT_SYMBOL_GPL(iomap_get_folio);
@@ -769,6 +781,7 @@ static int iomap_write_begin(struct iomap_iter *iter, loff_t pos,
 {
 	const struct iomap_folio_ops *folio_ops = iter->iomap.folio_ops;
 	const struct iomap *srcmap = iomap_iter_srcmap(iter);
+	bool is_atomic = iter->flags & IOMAP_ATOMIC;
 	struct folio *folio;
 	int status = 0;
 
@@ -786,6 +799,11 @@ static int iomap_write_begin(struct iomap_iter *iter, loff_t pos,
 	if (IS_ERR(folio))
 		return PTR_ERR(folio);
 
+	if (is_atomic)
+		folio_set_atomic(folio);
+	else
+		folio_clear_atomic(folio);
+
 	/*
 	 * Now we have a locked folio, before we do anything with it we need to
 	 * check that the iomap we have cached is not stale. The inode extent
@@ -1010,6 +1028,8 @@ iomap_file_buffered_write(struct kiocb *iocb, struct iov_iter *i,
 
 	if (iocb->ki_flags & IOCB_NOWAIT)
 		iter.flags |= IOMAP_NOWAIT;
+	if (iocb->ki_flags & IOCB_ATOMIC)
+		iter.flags |= IOMAP_ATOMIC;
 
 	while ((ret = iomap_iter(&iter, ops)) > 0)
 		iter.processed = iomap_write_iter(&iter, i);
@@ -1499,8 +1519,10 @@ static void iomap_finish_folio_write(struct inode *inode, struct folio *folio,
 	WARN_ON_ONCE(i_blocks_per_folio(inode, folio) > 1 && !ifs);
 	WARN_ON_ONCE(ifs && atomic_read(&ifs->write_bytes_pending) <= 0);
 
-	if (!ifs || atomic_sub_and_test(len, &ifs->write_bytes_pending))
+	if (!ifs || atomic_sub_and_test(len, &ifs->write_bytes_pending)) {
+		folio_clear_atomic(folio);
 		folio_end_writeback(folio);
+	}
 }
 
 /*
@@ -1679,14 +1701,18 @@ static int iomap_submit_ioend(struct iomap_writepage_ctx *wpc, int error)
 }
 
 static struct iomap_ioend *iomap_alloc_ioend(struct iomap_writepage_ctx *wpc,
-		struct writeback_control *wbc, struct inode *inode, loff_t pos)
+		struct writeback_control *wbc, struct inode *inode, loff_t pos,
+		bool atomic)
 {
+	blk_opf_t opf = REQ_OP_WRITE | wbc_to_write_flags(wbc);
 	struct iomap_ioend *ioend;
 	struct bio *bio;
 
+	if (atomic)
+		opf |= REQ_ATOMIC;
+
 	bio = bio_alloc_bioset(wpc->iomap.bdev, BIO_MAX_VECS,
-			       REQ_OP_WRITE | wbc_to_write_flags(wbc),
-			       GFP_NOFS, &iomap_ioend_bioset);
+			       opf, GFP_NOFS, &iomap_ioend_bioset);
 	bio->bi_iter.bi_sector = iomap_sector(&wpc->iomap, pos);
 	bio->bi_end_io = iomap_writepage_end_bio;
 	wbc_init_bio(wbc, bio);
@@ -1744,14 +1770,27 @@ static int iomap_add_to_ioend(struct iomap_writepage_ctx *wpc,
 {
 	struct iomap_folio_state *ifs = folio->private;
 	size_t poff = offset_in_folio(folio, pos);
+	bool is_atomic = folio_test_atomic(folio);
 	int error;
 
-	if (!wpc->ioend || !iomap_can_add_to_ioend(wpc, pos)) {
+	if (!wpc->ioend || is_atomic || !iomap_can_add_to_ioend(wpc, pos)) {
 new_ioend:
 		error = iomap_submit_ioend(wpc, 0);
 		if (error)
 			return error;
-		wpc->ioend = iomap_alloc_ioend(wpc, wbc, inode, pos);
+		wpc->ioend = iomap_alloc_ioend(wpc, wbc, inode, pos, is_atomic);
+	}
+
+	/* We must not append anything later if atomic, so submit now */
+	if (is_atomic) {
+		if (!bio_add_folio(&wpc->ioend->io_bio, folio, len, poff))
+			return -EINVAL;
+		wpc->ioend->io_size = len;
+		wbc_account_cgroup_owner(wbc, &folio->page, len);
+		if (ifs)
+			atomic_add(len, &ifs->write_bytes_pending);
+
+		return iomap_submit_ioend(wpc, 0);
 	}
 
 	if (!bio_add_folio(&wpc->ioend->io_bio, folio, len, poff))
diff --git a/fs/iomap/trace.h b/fs/iomap/trace.h
index 0a991c4ce87d..4118a42cdab0 100644
--- a/fs/iomap/trace.h
+++ b/fs/iomap/trace.h
@@ -98,7 +98,8 @@ DEFINE_RANGE_EVENT(iomap_dio_rw_queued);
 	{ IOMAP_REPORT,		"REPORT" }, \
 	{ IOMAP_FAULT,		"FAULT" }, \
 	{ IOMAP_DIRECT,		"DIRECT" }, \
-	{ IOMAP_NOWAIT,		"NOWAIT" }
+	{ IOMAP_NOWAIT,		"NOWAIT" }, \
+	{ IOMAP_ATOMIC,		"ATOMIC" }
 
 #define IOMAP_F_FLAGS_STRINGS \
 	{ IOMAP_F_NEW,		"NEW" }, \
diff --git a/include/linux/iomap.h b/include/linux/iomap.h
index f726f0058fd6..2f50abe06f27 100644
--- a/include/linux/iomap.h
+++ b/include/linux/iomap.h
@@ -179,6 +179,7 @@ struct iomap_folio_ops {
 #else
 #define IOMAP_DAX		0
 #endif /* CONFIG_FS_DAX */
+#define IOMAP_ATOMIC	(1 << 9)
 
 struct iomap_ops {
 	/*
-- 
2.31.1