Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:6 as permitted sender) client-ip=2620:137:e000::3:6;
From:   John Garry <john.g.garry@oracle.com>
To:     axboe@kernel.dk, kbusch@kernel.org, hch@lst.de, sagi@grimberg.me,
        jejb@linux.ibm.com, martin.petersen@oracle.com, djwong@kernel.org,
        viro@zeniv.linux.org.uk, brauner@kernel.org, dchinner@redhat.com,
        jack@suse.cz
Cc:     linux-block@vger.kernel.org, linux-kernel@vger.kernel.org,
        linux-nvme@lists.infradead.org, linux-xfs@vger.kernel.org,
        linux-fsdevel@vger.kernel.org, tytso@mit.edu, jbongio@google.com,
        linux-scsi@vger.kernel.org, ming.lei@redhat.com,
        jaswin@linux.ibm.com, bvanassche@acm.org,
        John Garry <john.g.garry@oracle.com>
Subject: [PATCH v2 11/16] block: Add fops atomic write support
Date:   Tue, 12 Dec 2023 11:08:39 +0000
Message-Id: <20231212110844.19698-12-john.g.garry@oracle.com>
In-Reply-To: <20231212110844.19698-1-john.g.garry@oracle.com>
References: <20231212110844.19698-1-john.g.garry@oracle.com>
Content-Transfer-Encoding: 8bit
Content-Type: text/plain
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?nocAZu8PuRVhC8JIMO/WR8shiu3zjDMaZpXqHS9Rhe3y1Ivg+IyQouRs/sKg?=
 =?us-ascii?Q?DAiDRFKwuw+Sf2Xwx7aNg8kIqxpk/AOfYdkLFDdiaipDsrt67HSBnS1V4fh6?=
 =?us-ascii?Q?fyu+C14G3ehYWy6IJALr/iYr4ROpQ/K2igdM1dUpS4+EqE6oNrKpEnfjGYRn?=
 =?us-ascii?Q?+2XjPS56Xjr8aluwilEw6wRH0sVMZiufsqw6R6xTQ+nRy924d4cEk6JTiL/m?=
 =?us-ascii?Q?8KVsJyPVZo/YWKDCxzppsG+LEceUH5VYnrO6TnPtARwMBfPv0hhPpQFtmv1M?=
 =?us-ascii?Q?T5FQKwqg2T2oPJU77fy7Q7tWbTk1nQYLdWUYzkcaleFm3m7ilVOK3FRcyPJ7?=
 =?us-ascii?Q?FYRSWCUJ0ryNUh9kYAEH2Sx5piiiNehPcUNgk3ZIXdZ14qECLO3+bz4yyhgW?=
 =?us-ascii?Q?0UnyeaOGAEG38Mhft+y8zkedJxIiiBrs9MA+QYqLHaP9fy1rvd5/OVPTXtgp?=
 =?us-ascii?Q?N2jLKl0/+7z2Pd1YMvQxYrG2XU3Rqr/xAiA1ZrykcbAz7z2k4SFaEXe2NXAJ?=
 =?us-ascii?Q?3V9W80lXRW9BWvSHDdHPzeTQ2bxOvq7rszABi3nhwdiKQ//u8J86ZP0Fh85j?=
 =?us-ascii?Q?pj/Zs0ww4CU+UX9M3C7BODLbUibdBdNQZAcfgsoUX75aFuX9W2CG/0pJnsFR?=
 =?us-ascii?Q?mzSb47yeL94LdIj7cx925og6UT5H74y6hMzIR16+GRbq69BeeB2WHLk9V7mN?=
 =?us-ascii?Q?6Huiaf/WUF3k12AiErrgaH/pOfU85rF9iB6s288PQBYIf95P7YBYqQIbcYOh?=
 =?us-ascii?Q?tuqawo90ksHQJxFxuWZOm0g5kPbHhZjrtx34+mBQvujfpjTTmIKKG/7KxIcI?=
 =?us-ascii?Q?lGhFQgxd+vXhaWK8KOEXsk6YDJ03v430edpin+XyYkVwnhvp8CfCeWZ5LSTZ?=
 =?us-ascii?Q?T+lFo/ES6OycwmYxBkrDC55pepq25t4m5U2nsXWRL43u3p8WuJq5oUStDslU?=
 =?us-ascii?Q?f9fsaJkrNaW1GDGYr044ygxFNjc2zx2ZEPgskI0XbhK1q/AakhmXjxKg0M8t?=
 =?us-ascii?Q?CQ3fSIWbK8lF4AE9PVN86HSrR3StGK2MG05sf5Vnc6AWXlXHYPpQgRVE1F8d?=
 =?us-ascii?Q?RicWJR5yThOi7iLs0JZotlBeq1D2pC6mAKT2Ctb/y/NRy5MlKaSvi7tQ6Qf3?=
 =?us-ascii?Q?GauFQOH37K9DkHQ3h9VzNE6NzZfrTT4RsuvMUqDNiEuhBPbqkcfZ2o2b6Ojx?=
 =?us-ascii?Q?GMriNs8p53QPQ7nCFFjiXQZ+LLQGnJ1QEYpEtAOeWiOAo7or+1tkDaRxG9x0?=
 =?us-ascii?Q?3isetP1UvKb3+3CjW+TZmEdXIMsVl6zuZGCVW+E+UUWavokq/iUHPEN1BK7E?=
 =?us-ascii?Q?Z/K5BdN5P9ZDDp8zTtB74N+XMSNja4+uVtyfsNp8hZqaVc+zhnZzq0/eWH1F?=
 =?us-ascii?Q?xcVaq7nmkPh9bsSlVDmHrWGblKy4eeRVcOib1ARjzEKsLYE/P/cGeEqYUaJ4?=
 =?us-ascii?Q?ZHEi1I8uHfLlPOyEJ1I2sVm8IxSVHC9q3eOTri8DF3+KD3HVhJopmqu+GoZJ?=
 =?us-ascii?Q?8o9cWxVpI6Tu0oo50w59fazFk6W93CbmCGdXOfDXxSks+ThTvdH4lxLAjsLL?=
 =?us-ascii?Q?cNee0Dhg8gvE8VVGGGCSFVEpoKcTtHPV96cXtdsEGE2CNwd49YQAAZSnzBXr?=
 =?us-ascii?Q?VA=3D=3D?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 87vdRZZ7izbyVbKCniqagp2ZcFTY/jWCiDN9FzlBQ4VBwvz2jFgnR6iYyQN1B73lQnPwdayoH27GTAiIUDAd+PD9bWhmF+Y4BDoX8ZXRYCEXjCVVSTrSTRbsRY9klsCuUONDJU5P24MH2AaaN9yHApEiwrYiZBm8BEaO5ZKHcIKP73eObKS5ct4L1X9nDUdisBX+MqiXQLyewRLQrfiBgQsda0/VqfXAZthuyG//zwjHLrmIpEcD9tG4zz4lbZsVjS52tviDgoDqyu8Mj4TJYQNBvQLoBG6FXQMDgkwiLbK9H+ki5g0ztnA46UnnI7kBKh+3hsSyuw4ukr9IOpVE537dM1KLI/J7SqHbkRdPk/65n/X6Sy3XjhorbaMZwiKt2inxX81fHiXXVIt9AqWOrf9gst/uhcve0MiWeHR3El71fEP8acTesxFvkx6dm7YhxTSfqLLjKzWvDDwpKBIzCMnUnzq5185SDHcNCEceBtILheYEC1oioauzBOg2BF5M8LvNA3TURJBLwhL+cyWNETUIQal6RhLrwzWLJwieMTBa0xfVWLiqlQGxcUrxHFT8Wxg0tawHsWYrKaCGZ/CEyvZSiE7KZTZ6H5J48KdXyeY=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: efa23dcb-72cc-41ed-87f7-08dbfb02cede
X-MS-Exchange-CrossTenant-AuthSource: DM6PR10MB4313.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Dec 2023 11:09:28.3907
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: RENRQV1NfBtHclKPyjucw9jaDVHAk4AMS5wWl8s//g/7cJPQdoHhiluVd8O2zfAT9yQ3aouiHX7h+FOV1Qgftg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR10MB6663
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26
 definitions=2023-12-12_04,2023-12-12_01,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 malwarescore=0
 mlxlogscore=999 adultscore=0 phishscore=0 suspectscore=0 mlxscore=0
 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2311290000 definitions=main-2312120089
X-Proofpoint-ORIG-GUID: D_VDOgAfLjlKB2oZyoQIbz2wV8LnhhOn
X-Proofpoint-GUID: D_VDOgAfLjlKB2oZyoQIbz2wV8LnhhOn
X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID,
	DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,
	SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable
	autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on pete.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (pete.vger.email [0.0.0.0]); Tue, 12 Dec 2023 03:13:23 -0800 (PST)

Add support for atomic writes, as follows:
- Ensure that the IO follows all the atomic writes rules, like must be
  naturally aligned
- Set REQ_ATOMIC

Signed-off-by: John Garry <john.g.garry@oracle.com>
---
 block/fops.c | 40 +++++++++++++++++++++++++++++++++++++++-
 1 file changed, 39 insertions(+), 1 deletion(-)

diff --git a/block/fops.c b/block/fops.c
index 0abaac705daf..ba6a2c5a74b1 100644
--- a/block/fops.c
+++ b/block/fops.c
@@ -41,6 +41,24 @@ static bool blkdev_dio_unaligned(struct block_device *bdev, loff_t pos,
 		!bdev_iter_is_aligned(bdev, iter);
 }
 
+static bool blkdev_atomic_write_valid(struct block_device *bdev, loff_t pos,
+			      struct iov_iter *iter)
+{
+	struct request_queue *q = bdev_get_queue(bdev);
+	unsigned int min_bytes = queue_atomic_write_unit_min_bytes(q);
+	unsigned int max_bytes = queue_atomic_write_unit_max_bytes(q);
+
+	if (iov_iter_count(iter) & (min_bytes - 1))
+		return false;
+	if (!is_power_of_2(iov_iter_count(iter)))
+		return false;
+	if (pos & (iov_iter_count(iter) - 1))
+		return false;
+	if (iov_iter_count(iter) > max_bytes)
+		return false;
+	return true;
+}
+
 #define DIO_INLINE_BIO_VECS 4
 
 static ssize_t __blkdev_direct_IO_simple(struct kiocb *iocb,
@@ -48,6 +66,8 @@ static ssize_t __blkdev_direct_IO_simple(struct kiocb *iocb,
 {
 	struct block_device *bdev = I_BDEV(iocb->ki_filp->f_mapping->host);
 	struct bio_vec inline_vecs[DIO_INLINE_BIO_VECS], *vecs;
+	bool is_read = iov_iter_rw(iter) == READ;
+	bool atomic_write = (iocb->ki_flags & IOCB_ATOMIC) && !is_read;
 	loff_t pos = iocb->ki_pos;
 	bool should_dirty = false;
 	struct bio bio;
@@ -56,6 +76,9 @@ static ssize_t __blkdev_direct_IO_simple(struct kiocb *iocb,
 	if (blkdev_dio_unaligned(bdev, pos, iter))
 		return -EINVAL;
 
+	if (atomic_write && !blkdev_atomic_write_valid(bdev, pos, iter))
+		return -EINVAL;
+
 	if (nr_pages <= DIO_INLINE_BIO_VECS)
 		vecs = inline_vecs;
 	else {
@@ -65,7 +88,7 @@ static ssize_t __blkdev_direct_IO_simple(struct kiocb *iocb,
 			return -ENOMEM;
 	}
 
-	if (iov_iter_rw(iter) == READ) {
+	if (is_read) {
 		bio_init(&bio, bdev, vecs, nr_pages, REQ_OP_READ);
 		if (user_backed_iter(iter))
 			should_dirty = true;
@@ -74,6 +97,8 @@ static ssize_t __blkdev_direct_IO_simple(struct kiocb *iocb,
 	}
 	bio.bi_iter.bi_sector = pos >> SECTOR_SHIFT;
 	bio.bi_ioprio = iocb->ki_ioprio;
+	if (atomic_write)
+		bio.bi_opf |= REQ_ATOMIC;
 
 	ret = bio_iov_iter_get_pages(&bio, iter);
 	if (unlikely(ret))
@@ -167,10 +192,14 @@ static ssize_t __blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter,
 	struct blkdev_dio *dio;
 	struct bio *bio;
 	bool is_read = (iov_iter_rw(iter) == READ), is_sync;
+	bool atomic_write = (iocb->ki_flags & IOCB_ATOMIC) && !is_read;
 	blk_opf_t opf = is_read ? REQ_OP_READ : dio_bio_write_op(iocb);
 	loff_t pos = iocb->ki_pos;
 	int ret = 0;
 
+	if (atomic_write)
+		return -EINVAL;
+
 	if (blkdev_dio_unaligned(bdev, pos, iter))
 		return -EINVAL;
 
@@ -305,6 +334,7 @@ static ssize_t __blkdev_direct_IO_async(struct kiocb *iocb,
 	struct block_device *bdev = I_BDEV(iocb->ki_filp->f_mapping->host);
 	bool is_read = iov_iter_rw(iter) == READ;
 	blk_opf_t opf = is_read ? REQ_OP_READ : dio_bio_write_op(iocb);
+	bool atomic_write = (iocb->ki_flags & IOCB_ATOMIC) && !is_read;
 	struct blkdev_dio *dio;
 	struct bio *bio;
 	loff_t pos = iocb->ki_pos;
@@ -313,6 +343,9 @@ static ssize_t __blkdev_direct_IO_async(struct kiocb *iocb,
 	if (blkdev_dio_unaligned(bdev, pos, iter))
 		return -EINVAL;
 
+	if (atomic_write && !blkdev_atomic_write_valid(bdev, pos, iter))
+		return -EINVAL;
+
 	if (iocb->ki_flags & IOCB_ALLOC_CACHE)
 		opf |= REQ_ALLOC_CACHE;
 	bio = bio_alloc_bioset(bdev, nr_pages, opf, GFP_KERNEL,
@@ -347,6 +380,8 @@ static ssize_t __blkdev_direct_IO_async(struct kiocb *iocb,
 			bio_set_pages_dirty(bio);
 		}
 	} else {
+		if (atomic_write)
+			bio->bi_opf |= REQ_ATOMIC;
 		task_io_account_write(bio->bi_iter.bi_size);
 	}
 
@@ -605,6 +640,9 @@ static int blkdev_open(struct inode *inode, struct file *filp)
 	if (bdev_nowait(handle->bdev))
 		filp->f_mode |= FMODE_NOWAIT;
 
+	if (queue_atomic_write_unit_min_bytes(bdev_get_queue(handle->bdev)))
+		filp->f_mode |= FMODE_CAN_ATOMIC_WRITE;
+
 	filp->f_mapping = handle->bdev->bd_inode->i_mapping;
 	filp->f_wb_err = filemap_sample_wb_err(filp->f_mapping);
 	filp->private_data = handle;
-- 
2.35.3