Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   luhongfei <luhongfei@vivo.com>
To:     Jens Axboe <axboe@kernel.dk>,
        Pavel Begunkov <asml.silence@gmail.com>,
        io-uring@vger.kernel.org (open list:IO_URING),
        linux-kernel@vger.kernel.org (open list)
Cc:     opensource.kernel@vivo.com, luhongfei <luhongfei@vivo.com>
Subject: [PATCH] io_uring: Optimization of buffered random write
Date:   Wed, 19 Apr 2023 17:22:33 +0800
Message-Id: <20230419092233.56338-1-luhongfei@vivo.com>
Content-Transfer-Encoding: 8bit
Content-Type: text/plain
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?CETpt4tKpaSr4vPDrlSHfVsCjnsUGMhsfQD/x4MZ6dsatx4DsekgzMFwaSBY?=
 =?us-ascii?Q?H+9OirYlhbBXMu6WMVr6PZSzPr9FI19eWdaEJ/Xf7p0uIxDuARLTgHAYlaGV?=
 =?us-ascii?Q?w56VjkTrF8zP6c7rCJvOTgjqoKvWjGpeXm4O4rAr7IYbXEFPWzd7M9h0EydZ?=
 =?us-ascii?Q?ZjzS0eyqegKofSyMShP9zqwr5qBxQgTsP7xrgnGT85u+LX9dtXfwPne5k8So?=
 =?us-ascii?Q?qUcPNXwa7OckyoFaa1JlfkLzgeiBdXGTrxvdINuM4ebzborhuC5L2/QPVcFY?=
 =?us-ascii?Q?eHjDeoIUp2zZ5d03hehYJgL5d0Tu0j2XsS9+bEFrR621BxrOU/s19Hjcymg3?=
 =?us-ascii?Q?GDG9W0yT0NTWAOy+r4DbTpK2WNnI5PPPPW2LAORPN8J+JJGzhePlS0e3hHET?=
 =?us-ascii?Q?9zzT9BJhCTzdV8Ppr0ZPjmZF5GeKK5n7JPs8lz7mPWHE/9JkfhiE8UaxVLqu?=
 =?us-ascii?Q?DhS9vKOzJSvygudXB1wjkLlKvLfJj3IIur0BlzGAl6JzZ+z//2b34HdLWJVb?=
 =?us-ascii?Q?GXuha4vWrs/C/rwAeeBE8/vNgRrTJQMO9ehF7p8kOSSEnOrZ9pUMVQRNCev/?=
 =?us-ascii?Q?0bFL+OBxlG+vbS0MYEeYWC7XtPVelR0jFJYaQgFIPXJeYcQ/fhMQvRjfoHLi?=
 =?us-ascii?Q?o6f4I9QdiNT8Tj+oTKJjxK71/Hf8MMA50EkpPOfVSTaROD9YgSPZvuRls9Nw?=
 =?us-ascii?Q?qO77mgJG5tPH1CCkz21c340cXUElbk/oUO5k8w1WL0hrE4BgfuxEt6s3CDX3?=
 =?us-ascii?Q?pXvLaIOmdRF7MOAt9e12+MGEBnaQ50+KjXVBYNtwiM6qqy5VHrQileiGRXsB?=
 =?us-ascii?Q?Z+5Mb+zP9+yWRJH16LrUW/JP2NlBsI608+pV5o7nrAfnSVdPTor9fGXWtvFg?=
 =?us-ascii?Q?g99en9PZ0mKqKdWL2hBXQDTMMO23tTYiKLpSwTMGnr1Z2+LevEWCpR7i9afz?=
 =?us-ascii?Q?0PSmUJ6VU55YP8U7a55zR2lDN+TejWArzJv55ZpFenoU3Nmp7zbuiyu/s6l9?=
 =?us-ascii?Q?XSD+wEmuvlXrD6loYTTbspaA0P+Lz9SN3/5e5o+nrWdbJGlRMVsyVs4G0ZpX?=
 =?us-ascii?Q?sSOgL0LpjHEoRKecKvxZBLdNLS8X2nSsCShh2KyIn3O33LrSl+RsdP/N2ckV?=
 =?us-ascii?Q?Cw5Qm+jjNrJacNyuB6xZkCjRqHN7Fsp6uKKat9J/Y6ZwK8KwiUFdMxZda6t2?=
 =?us-ascii?Q?qtM1kMj/pK7ydimBfDO5dehALqDNORudpTalJkRCoWZPsB8uMWHMJMNqWnJ9?=
 =?us-ascii?Q?W8YJvldyNYou9/eQgV2MrDXe6yHznN7VbG4EZ8bvwpeOOoPYKhmBqak6hqCH?=
 =?us-ascii?Q?S7wfMgSORSx0l0Uan/bHOk3H3mffitxUsMfCQlyfawgs13WLk2HZB39Vx121?=
 =?us-ascii?Q?nl+qCw4B9CoBQviz8gd4vAAYzBAigrAYakoUJ+PgcK3Mpl6ZfVTODn2+u0rf?=
 =?us-ascii?Q?W3gHeHY1o8/oK3CG9x3aWl5mxBosKsgg8kBcUCMybfBoqhhm7lNAUFkANQB8?=
 =?us-ascii?Q?1KNc2VYBJM4w5nxOvs7KC8q/Q0+LxFnUDHPP3+Ew6lkQ3zB2+vi8o9pNk8hv?=
 =?us-ascii?Q?nyeqWaaClSIhRxyBFhSOIujN7iopXT2/DxUgIrEp?=
X-OriginatorOrg: vivo.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 0de00e67-dd38-43ec-0f69-08db40b7a130
X-MS-Exchange-CrossTenant-AuthSource: SG2PR06MB2169.apcprd06.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Apr 2023 09:22:43.9881
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 923e42dc-48d5-4cbe-b582-1a797a6412ed
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: vhzTWrEbPSeGdA8cSl5zFwr5tVOCCPeWnaMEOL++TMiywL22H4r+C4MZyysbi7WhakVn6nYqEUfq2+cHwJVVYg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: KL1PR0601MB4257
X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE,
        RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_PASS,T_SCC_BODY_TEXT_LINE,
        URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

The buffered random write performance of io_uring is poor
due to the following reason:
By default, when performing buffered random writes, io_sq_thread
will call io_issue_sqe writes req, but due to the setting of
IO_URING_F_NONBLOCK, req is executed asynchronously in iou-wrk,
where io_wq_submit_work calls io_issue_sqe completes the write req,
with issue_flag as IO_URING_F_UNLOCKED | IO_URING_F_IOWQ,
which will reduce performance.
This patch will determine whether this req is a buffered random write,
and if so, io_sq_thread directly calls io_issue_sqe(req, 0)
completes req instead of completing it asynchronously in iou wrk.

Performance results:
  For fio the following results have been obtained with a queue depth of
  8 and 4k block size:

             random writes:
             without patch          with patch      libaio      psync
iops:           287k                   560k          248K        324K
bw:            1123MB/s               2188MB/s       970MB/s    1267MB/s
clat:         52760ns                69918ns       28405ns      2109ns

Signed-off-by: luhongfei <luhongfei@vivo.com>
---
 io_uring/io_uring.c | 17 ++++++++++++++++-
 1 file changed, 16 insertions(+), 1 deletion(-)
 mode change 100644 => 100755 io_uring/io_uring.c

diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c
index 4a865f0e85d0..64bb91beb4d6
--- a/io_uring/io_uring.c
+++ b/io_uring/io_uring.c
@@ -2075,8 +2075,23 @@ static inline void io_queue_sqe(struct io_kiocb *req)
 	__must_hold(&req->ctx->uring_lock)
 {
 	int ret;
+	bool is_write;
 
-	ret = io_issue_sqe(req, IO_URING_F_NONBLOCK|IO_URING_F_COMPLETE_DEFER);
+	switch (req->opcode) {
+	case IORING_OP_WRITEV:
+	case IORING_OP_WRITE_FIXED:
+	case IORING_OP_WRITE:
+		is_write = true;
+		break;
+	default:
+		is_write = false;
+		break;
+	}
+
+	if (!is_write || (req->rw.kiocb.ki_flags & IOCB_DIRECT))
+		ret = io_issue_sqe(req, IO_URING_F_NONBLOCK|IO_URING_F_COMPLETE_DEFER);
+	else
+		ret = io_issue_sqe(req, 0);
 
 	/*
 	 * We async punt it if the file wasn't marked NOWAIT, or if the file
-- 
2.39.0