Received: by 2002:a05:6358:11c7:b0:104:8066:f915 with SMTP id i7csp3273290rwl; Thu, 13 Apr 2023 19:20:35 -0700 (PDT) X-Google-Smtp-Source: AKy350YIr0BfC9srXv+M8aSm5QEzHHs1hVC8cHv93SAr3vMhihlVTsQgK4yU8n36wU92l71V5R0o X-Received: by 2002:a05:6a00:178c:b0:63b:4e99:807d with SMTP id s12-20020a056a00178c00b0063b4e99807dmr5145650pfg.8.1681438835271; Thu, 13 Apr 2023 19:20:35 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1681438835; cv=none; d=google.com; s=arc-20160816; b=0fH3ZhgAXw9ZpCaeDZRZOte/exEtrIu31+uHeLZkTRCJBMt6JtTahH4TV+m/1Sas1k UxWSgu7n0BoypjqK4CXyAewleVKVe9A03rP0cX2N6UHwH2tyTxUBBlHH1DpbgMzit2EB Bw0Ae+HhAo4faaJ5VGOXfnZiaW0QMJJQyvgBTt0xOyBSyBNOrgoi8TPJVPVs1lu2dZ/m xdzGqGSuSWgbRmcE6hiYn8dwyy1QWb6lTlGbr2uY2EzuB4B0gdBZL8MikxLN7xCCXjG+ zdoD3O5hwURiAFO0qDe/O4mfNvfdp3A58IgA+j1VY9/zsTffbFXN6Q43ef4BQj4qnJoD c6Fw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:in-reply-to:content-disposition:mime-version :references:message-id:subject:cc:to:from:date:dkim-signature; bh=xU5po7bo+0f7mtpnyY2arS5aqu7vRvq3JKh8xv5QFh4=; b=JytO2X0kn9xBCs8Tffz4G/dn8Ifa7fQpJw+FbgFBSMkdp6D4ZpH12KY8pFAbyRfpsk oe5r7g0fR+A4jtZ/4nK7vFczQiqL2Da4P1+AL1DvCtOtIyQDJWHV3YEbsqtwDig49kmI J/E70BsdXLAv2shar9U+uDN0taTO4ei/FB6vL/51yFKpdAHjyrdS+S04WGbus0bg3nWZ 8y6YkS+Rv9G+EX0PSISOUZGiFWJu/YYKULXOmPJiAUowKZo2yGY0C1L96QjM6se0YSY0 fbn4tvRQKMW8QFn1ufX4cySblByXebT7XgGwx1n+XrrmHfoMQwUZnVcCU4++PtPOYT/X J9Nw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=LiZE1esw; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Return-Path: Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id x17-20020aa79a51000000b00627dd2b5307si3166797pfj.102.2023.04.13.19.20.21; Thu, 13 Apr 2023 19:20:35 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=LiZE1esw; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229762AbjDNCOH (ORCPT + 99 others); Thu, 13 Apr 2023 22:14:07 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42146 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229815AbjDNCOE (ORCPT ); Thu, 13 Apr 2023 22:14:04 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 327D82D74 for ; Thu, 13 Apr 2023 19:13:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1681438396; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=xU5po7bo+0f7mtpnyY2arS5aqu7vRvq3JKh8xv5QFh4=; b=LiZE1eswDu85yHULt1HyUFEqmcRxOOJfuu0PgCYkeWpYltboKi1/bJ1kkt4Qal/NSIcSNL NkxCs6AsGW9In+s4OdJ7x8pvOKYYsdjPbsJz0cIFUKsLhgrDxGo7XS1u0tDo3/lyLkUVVE Gm0bZRLpUb7oHH6DSd9X8BuWNKKPLLg= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-551-VKX7xa-EOl-aq5sktc-3bA-1; Thu, 13 Apr 2023 22:13:10 -0400 X-MC-Unique: VKX7xa-EOl-aq5sktc-3bA-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 814258996F5; Fri, 14 Apr 2023 02:13:09 +0000 (UTC) Received: from ovpn-8-21.pek2.redhat.com (ovpn-8-21.pek2.redhat.com [10.72.8.21]) by smtp.corp.redhat.com (Postfix) with ESMTPS id F1F9214171B6; Fri, 14 Apr 2023 02:12:58 +0000 (UTC) Date: Fri, 14 Apr 2023 10:12:52 +0800 From: Ming Lei To: Breno Leitao Cc: asml.silence@gmail.com, axboe@kernel.dk, davem@davemloft.net, dccp@vger.kernel.org, dsahern@kernel.org, edumazet@google.com, io-uring@vger.kernel.org, kuba@kernel.org, leit@fb.com, linux-kernel@vger.kernel.org, marcelo.leitner@gmail.com, matthieu.baerts@tessares.net, mptcp@lists.linux.dev, netdev@vger.kernel.org, pabeni@redhat.com, willemdebruijn.kernel@gmail.com, ming.lei@redhat.com Subject: Re: [PATCH RFC] io_uring: Pass whole sqe to commands Message-ID: References: <20230406144330.1932798-1-leitao@debian.org> <20230406165705.3161734-1-leitao@debian.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Scanned-By: MIMEDefang 3.1 on 10.11.54.7 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Thu, Apr 13, 2023 at 09:47:56AM -0700, Breno Leitao wrote: > Hello Ming, > > On Thu, Apr 13, 2023 at 10:56:49AM +0800, Ming Lei wrote: > > On Thu, Apr 06, 2023 at 09:57:05AM -0700, Breno Leitao wrote: > > > Currently uring CMD operation relies on having large SQEs, but future > > > operations might want to use normal SQE. > > > > > > The io_uring_cmd currently only saves the payload (cmd) part of the SQE, > > > but, for commands that use normal SQE size, it might be necessary to > > > access the initial SQE fields outside of the payload/cmd block. So, > > > saves the whole SQE other than just the pdu. > > > > > > This changes slighlty how the io_uring_cmd works, since the cmd > > > structures and callbacks are not opaque to io_uring anymore. I.e, the > > > callbacks can look at the SQE entries, not only, in the cmd structure. > > > > > > The main advantage is that we don't need to create custom structures for > > > simple commands. > > > > > > Suggested-by: Pavel Begunkov > > > Signed-off-by: Breno Leitao > > > --- > > > > ... > > > > > diff --git a/io_uring/uring_cmd.c b/io_uring/uring_cmd.c > > > index 2e4c483075d3..9648134ccae1 100644 > > > --- a/io_uring/uring_cmd.c > > > +++ b/io_uring/uring_cmd.c > > > @@ -63,14 +63,15 @@ EXPORT_SYMBOL_GPL(io_uring_cmd_done); > > > int io_uring_cmd_prep_async(struct io_kiocb *req) > > > { > > > struct io_uring_cmd *ioucmd = io_kiocb_to_cmd(req, struct io_uring_cmd); > > > - size_t cmd_size; > > > + size_t size = sizeof(struct io_uring_sqe); > > > > > > BUILD_BUG_ON(uring_cmd_pdu_size(0) != 16); > > > BUILD_BUG_ON(uring_cmd_pdu_size(1) != 80); > > > > > > - cmd_size = uring_cmd_pdu_size(req->ctx->flags & IORING_SETUP_SQE128); > > > + if (req->ctx->flags & IORING_SETUP_SQE128) > > > + size <<= 1; > > > > > > - memcpy(req->async_data, ioucmd->cmd, cmd_size); > > > + memcpy(req->async_data, ioucmd->sqe, size); > > > > The copy will make some fields of sqe become READ TWICE, and driver may see > > different sqe field value compared with the one observed in io_init_req(). > > This copy only happens if the operation goes to the async path > (calling io_uring_cmd_prep_async()). This only happens if > f_op->uring_cmd() returns -EAGAIN. > > ret = file->f_op->uring_cmd(ioucmd, issue_flags); > if (ret == -EAGAIN) { > if (!req_has_async_data(req)) { > if (io_alloc_async_data(req)) > return -ENOMEM; > io_uring_cmd_prep_async(req); > } > return -EAGAIN; > } > > Are you saying that after this copy, the operation is still reading from > sqe instead of req->async_data? I meant that the 2nd read is on the sqe copy(req->aync_data), but same fields can become different between the two READs(first is done on original SQE during io_init_req(), and second is done on sqe copy in driver). Will this kind of inconsistency cause trouble for driver? Cause READ TWICE becomes possible with this patch. > > If you have an example of the two copes flow, that would be great. Not any example yet, but also not see any access on cmd->sqe(except for cmd_op) in your patches too. Thanks, Ming