Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) client-ip=23.128.96.31;
From:   John Garry <john.g.garry@oracle.com>
To:     axboe@kernel.dk, kbusch@kernel.org, hch@lst.de, sagi@grimberg.me,
        jejb@linux.ibm.com, martin.petersen@oracle.com, djwong@kernel.org,
        viro@zeniv.linux.org.uk, brauner@kernel.org, dchinner@redhat.com,
        jack@suse.cz
Cc:     linux-block@vger.kernel.org, linux-kernel@vger.kernel.org,
        linux-nvme@lists.infradead.org, linux-xfs@vger.kernel.org,
        linux-fsdevel@vger.kernel.org, tytso@mit.edu, jbongio@google.com,
        linux-scsi@vger.kernel.org, ming.lei@redhat.com,
        jaswin@linux.ibm.com, bvanassche@acm.org,
        John Garry <john.g.garry@oracle.com>
Subject: [PATCH v2 10/16] block: Add checks to merging of atomic writes
Date:   Tue, 12 Dec 2023 11:08:38 +0000
Message-Id: <20231212110844.19698-11-john.g.garry@oracle.com>
In-Reply-To: <20231212110844.19698-1-john.g.garry@oracle.com>
References: <20231212110844.19698-1-john.g.garry@oracle.com>
Content-Transfer-Encoding: 8bit
Content-Type: text/plain
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?UTI/9KdLqabV5P42TuU6CeTp0FxRgkZm/a5HvQyW6n4f8FS4lCj2JT5aPgPh?=
 =?us-ascii?Q?q2j4kX8l1zvh4o+l9kISOdG5eewp/NsEyobqViMz2D1cG/eU42M9DzVazui3?=
 =?us-ascii?Q?OoMjPQVsybuASv1vN+6aFeOqgGo+uT7N4EZykVJIC4rYAdR2m0qa8zRG3mkk?=
 =?us-ascii?Q?YVBRH5HGvEq3Aozdc/pMbxvQLgMr5cCvvlR/sLmxvJc9xCo7n239ftw219id?=
 =?us-ascii?Q?+eyKpI6aPB3XSLDWWEtc0camBKhfl/GDCbs/SKuSU9W9cAA+JoC5pWdwshXV?=
 =?us-ascii?Q?rkkct1g4BrrlStFcXJm5CninH2WD0y1NF3QBl0Rw31Ptv3fnVHaKnh69CXWh?=
 =?us-ascii?Q?Yfhfr2SlbNOxuf8CAznWA46rSOaBkxw8lcZyRPGS8AquqOSmI+IIF1Ptgb+r?=
 =?us-ascii?Q?aguVYfqPgcWqBmzq+zQnIJzTmynNfZN0TSdPlo0Kw3IzTVF6v/C4S/Y6tFj8?=
 =?us-ascii?Q?Ab5JnnUcSflvuVbdW59cb0NvgU6ddXDGSqA3HKTDFsAOT51H4AzM7BCsE/lq?=
 =?us-ascii?Q?OTW/HByKyGtth5pQ9Di8tFmj/btDs5Z/DqvOpRQCL7e/JCHtrMiCuSN/38xm?=
 =?us-ascii?Q?ojhygUCqhHwbP0Vnn2Cva8MeyPD0mGVwJVO0SRfywkaMbqJs6pwgX4PN/7Np?=
 =?us-ascii?Q?qtJe2F69s/z1WCgg8lQ2cWvDM64HNzpeA3UZa0wSJnjHvDPFbxNthXn+8b+c?=
 =?us-ascii?Q?RMeYPvH66zIIxsHqOnlzlANLm3vgmw3Z+hXkUKPVwKY5nHaYqOXB8X+C8bcQ?=
 =?us-ascii?Q?FNYQt7DzBvEYUcVppi7RzKBncDXEBiwAsAaCOhC4vhrTqksRapUOCXhQYSnd?=
 =?us-ascii?Q?/kNouyHnMXQq1tX+igdvVBito4hYFjp0PkyR/XLlGxLX9s44o0U22ELeQBSC?=
 =?us-ascii?Q?LVVt20St0EEKB6zQ9ml9O2qi8tEWBMAfDekl5GT8vSrZw+P9g2R30LLZgug8?=
 =?us-ascii?Q?GIdohpG6u0rVQPee4AmzzPa7gBouP8b2yhVHgX4MuJ8UchX8QRuj5PncO0bh?=
 =?us-ascii?Q?F9CJvi1F6m4XOOJh+xRSB4GQWvKoAacjgeie9MF1seM+rNpKtZ4mGi00hC5v?=
 =?us-ascii?Q?xlAj3kbR54pfMgObpQ+51qAG8UlUD3ZT36wKMGT7sRxNGGEYFTXkxSHO7zOd?=
 =?us-ascii?Q?LU79f0inRaUTZHhT9vLwQNTkhZMyi+NYB2GydAyUye3ero5ZB5bYo0J4wplm?=
 =?us-ascii?Q?g/oZcVjoAl64CJfdaYn2ewOVfCwGRP0nUpnm+kfvaXR7bVuuHmDmRqqUudby?=
 =?us-ascii?Q?OJBScAvXLsKR30gHbeX5xl/qCPXct7Ana8uu/UCzlwbua8XSzf9JY1YqNYMS?=
 =?us-ascii?Q?Jpb/ilnIIGG/0mkWcBrcBz3DfHCotmCvtVz51O1dyTxhs3c8ACUY8Z4Vsk8X?=
 =?us-ascii?Q?eb66wKzA9NjGWXhA848faUWjW+jzxReRwuaaqiPkbiINVCtoi0yIK7B9mViy?=
 =?us-ascii?Q?9MmAsKqLLFp0+BfTopMaW8EyzUT5X21WStfpQhuSS1jqlxolPjYPcsStGy+N?=
 =?us-ascii?Q?7GphCPA1FKzI5ywcjtn6e+mkjj0aJ03oM31Qn7OJi2wHGvrYiZlPrwrUJ+r2?=
 =?us-ascii?Q?zooI2M0f4moXnq2me/auuPDjloMmj97WvnhfBDqx9KrYOBczYGBheBlMi5f3?=
 =?us-ascii?Q?Sw=3D=3D?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 77E6KHa8qug3QTbUICFLic0eeGeUExrOu/UtBZJoPdgBXjdjh0ttfrRf1ekffveGpbBRt8NpP+ZvDcCwGvlFj7uj7qfPNtL6du5NXTE34Wgg2XuTXo3h5dQdP4qBTTMPowILIgAMNHwhBRvWLu95dVhlwI+FnY1/domKL7LSlMV+oyHAJ/M77Tlwbh9FxgS5I3tpFcUl3ZGfO8ujYLJ/fJ3A3Ug87wHZnA9FW65SmHK584wiVb3JnXx+rgNHxsR2KHArtYhuHzzXQjWwPQR30UBxVeGmoM5qOwS3V4fIiicdHh55gUGPwiBb9/cUcEFAQRLzvG4IUsS1Q90K6+kNQWY9/6DEhzW3NTji5/RXr446QIe3LTsfHUD3u0MbNVBXuuX5he5/txAALua14dUknxZ4FumHjhCbCYsyTvTm47v/XfKI11a4rCGossoKXWFpKsyvx5mqwUxjc6D3GeOvAviaGBzeP7EY9TtCX7i8AXonDOET5N0bqsrvnFvqdNg8i5vG1cDVVOPyVs7/JwkWeVzp0A9qEd8V3K0i5KKbmU6cs97KFawARWs+qF9nCb+fhWWUJ0hqtEXuU5+2chdTE3A0yGpxLNiRfQ5/7tRiDaA=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 4ae97a58-1355-4e84-ba83-08dbfb02cd90
X-MS-Exchange-CrossTenant-AuthSource: DM6PR10MB4313.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Dec 2023 11:09:26.1746
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: V+NeYy0iI1u2doxcR7SnDS87RnlTSV5J7+5tf+sT6FhGWaSTHCaoOr/bdYv2hmJ/5CIhtl84dVqoV+oEwmUtiA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR10MB6663
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26
 definitions=2023-12-12_04,2023-12-12_01,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 suspectscore=0
 phishscore=0 malwarescore=0 spamscore=0 mlxlogscore=999 bulkscore=0
 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2311290000 definitions=main-2312120089
X-Proofpoint-ORIG-GUID: Je6yWMQ4gDSeKJd3lb6811HVLh3-YyCB
X-Proofpoint-GUID: Je6yWMQ4gDSeKJd3lb6811HVLh3-YyCB
X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID,
	DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,
	SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable
	autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on morse.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (morse.vger.email [0.0.0.0]); Tue, 12 Dec 2023 03:10:46 -0800 (PST)

For atomic writes we allow merging, but we must adhere to some additional
rules:
- Only allow merging of atomic writes with other atomic writes
- Ensure that the merged IO would not cross an atomic write boundary, if
  any

We already ensure that we don't exceed the atomic writes size limit in
get_max_io_size().

Signed-off-by: John Garry <john.g.garry@oracle.com>
---
 block/blk-merge.c | 75 +++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 75 insertions(+)

diff --git a/block/blk-merge.c b/block/blk-merge.c
index bc21f8ff4842..05eb227a5644 100644
--- a/block/blk-merge.c
+++ b/block/blk-merge.c
@@ -18,6 +18,42 @@
 #include "blk-rq-qos.h"
 #include "blk-throttle.h"
 
+static bool blk_rq_straddles_atomic_write_boundary(struct request *rq,
+						unsigned int front,
+						unsigned int back)
+{
+	unsigned int boundary = queue_atomic_write_boundary_bytes(rq->q);
+	unsigned int mask, imask;
+	loff_t start, end;
+
+	if (!boundary)
+		return false;
+
+	start = rq->__sector << SECTOR_SHIFT;
+	end = start + rq->__data_len;
+
+	start -= front;
+	end += back;
+
+	/* We're longer than the boundary, so must be crossing it */
+	if (end - start > boundary)
+		return true;
+
+	mask = boundary - 1;
+
+	/* start/end are boundary-aligned, so cannot be crossing */
+	if (!(start & mask) || !(end & mask))
+		return false;
+
+	imask = ~mask;
+
+	/* Top bits are different, so crossed a boundary */
+	if ((start & imask) != (end & imask))
+		return true;
+
+	return false;
+}
+
 static inline void bio_get_first_bvec(struct bio *bio, struct bio_vec *bv)
 {
 	*bv = mp_bvec_iter_bvec(bio->bi_io_vec, bio->bi_iter);
@@ -664,6 +700,13 @@ int ll_back_merge_fn(struct request *req, struct bio *bio, unsigned int nr_segs)
 		return 0;
 	}
 
+	if (req->cmd_flags & REQ_ATOMIC) {
+		if (blk_rq_straddles_atomic_write_boundary(req,
+				0, bio->bi_iter.bi_size)) {
+			return 0;
+		}
+	}
+
 	return ll_new_hw_segment(req, bio, nr_segs);
 }
 
@@ -683,6 +726,13 @@ static int ll_front_merge_fn(struct request *req, struct bio *bio,
 		return 0;
 	}
 
+	if (req->cmd_flags & REQ_ATOMIC) {
+		if (blk_rq_straddles_atomic_write_boundary(req,
+				bio->bi_iter.bi_size, 0)) {
+			return 0;
+		}
+	}
+
 	return ll_new_hw_segment(req, bio, nr_segs);
 }
 
@@ -719,6 +769,13 @@ static int ll_merge_requests_fn(struct request_queue *q, struct request *req,
 	    blk_rq_get_max_sectors(req, blk_rq_pos(req)))
 		return 0;
 
+	if (req->cmd_flags & REQ_ATOMIC) {
+		if (blk_rq_straddles_atomic_write_boundary(req,
+				0, blk_rq_bytes(next))) {
+			return 0;
+		}
+	}
+
 	total_phys_segments = req->nr_phys_segments + next->nr_phys_segments;
 	if (total_phys_segments > blk_rq_get_max_segments(req))
 		return 0;
@@ -814,6 +871,18 @@ static enum elv_merge blk_try_req_merge(struct request *req,
 	return ELEVATOR_NO_MERGE;
 }
 
+static bool blk_atomic_write_mergeable_rq_bio(struct request *rq,
+					      struct bio *bio)
+{
+	return (rq->cmd_flags & REQ_ATOMIC) == (bio->bi_opf & REQ_ATOMIC);
+}
+
+static bool blk_atomic_write_mergeable_rqs(struct request *rq,
+					   struct request *next)
+{
+	return (rq->cmd_flags & REQ_ATOMIC) == (next->cmd_flags & REQ_ATOMIC);
+}
+
 /*
  * For non-mq, this has to be called with the request spinlock acquired.
  * For mq with scheduling, the appropriate queue wide lock should be held.
@@ -833,6 +902,9 @@ static struct request *attempt_merge(struct request_queue *q,
 	if (req->ioprio != next->ioprio)
 		return NULL;
 
+	if (!blk_atomic_write_mergeable_rqs(req, next))
+		return NULL;
+
 	/*
 	 * If we are allowed to merge, then append bio list
 	 * from next to rq and release next. merge_requests_fn
@@ -960,6 +1032,9 @@ bool blk_rq_merge_ok(struct request *rq, struct bio *bio)
 	if (rq->ioprio != bio_prio(bio))
 		return false;
 
+	if (blk_atomic_write_mergeable_rq_bio(rq, bio) == false)
+		return false;
+
 	return true;
 }
 
-- 
2.35.3