Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Fri, 25 Nov 2022 10:16:30 -0400
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     "lizhijian@fujitsu.com" <lizhijian@fujitsu.com>
Cc:     "zyjzyj2000@gmail.com" <zyjzyj2000@gmail.com>,
        "leon@kernel.org" <leon@kernel.org>,
        Bob Pearson <rpearsonhpe@gmail.com>,
        "linux-rdma@vger.kernel.org" <linux-rdma@vger.kernel.org>,
        Mark Bloch <mbloch@nvidia.com>, Tom Talpey <tom@talpey.com>,
        "tomasz.gromadzki@intel.com" <tomasz.gromadzki@intel.com>,
        Dan Williams <dan.j.williams@intel.com>,
        "yangx.jy@fujitsu.com" <yangx.jy@fujitsu.com>,
        "Yasunori Gotou (Fujitsu)" <y-goto@fujitsu.com>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
Subject: Re: [for-next PATCH v6 09/10] RDMA/cm: Make QP FLUSHABLE
Message-ID: <Y4DOPjDKM64ryuP3@nvidia.com>
References: <20221116081951.32750-1-lizhijian@fujitsu.com>
 <20221116081951.32750-10-lizhijian@fujitsu.com>
 <Y3ziLoRuXFIOpnnl@nvidia.com>
 <9cf2a1c5-2334-dee8-8374-63453e23c5a3@fujitsu.com>
 <Y3+sb3RZkEO3ISpW@nvidia.com>
 <a3a10e6e-c75e-853b-06d3-ce2f67424afc@fujitsu.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <a3a10e6e-c75e-853b-06d3-ce2f67424afc@fujitsu.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?3Y3S80sN3YuqN4zQ9rKm4ViQsOc1VzIiiwg+OBOrRtCBPXMzuWDHQ2hR3Nr6?=
 =?us-ascii?Q?jL+AYfa312S0Lo1JJUmzy0FA19F8sAIUx6Hsn7cH1Ui/IpJrf3ri8YHtIvJN?=
 =?us-ascii?Q?G+N36t4NcXRV6MdpcWKbjuZ7upySf5zEexWT/n//RgliG+PICfoNU17d/iq/?=
 =?us-ascii?Q?zd9KTAEnT1D+I8/ozdNvmQNiEMMjI1TzyaQEVJPN7xhaa1tin+MA8QIbdQIA?=
 =?us-ascii?Q?3FaWswX7jkETFoE/HfAGt5KQfZj207/T8lVLAMYkdeGLLHH1hxNo+0HX1qKX?=
 =?us-ascii?Q?NRJA87cMlhJFN+rBbBhiTmVuD3i0HXIqo5VsJnIrJkw6hM+qME4zs4+B9fTr?=
 =?us-ascii?Q?5Zs1/PdBovXGvS129jg9G/oBESqRPEDlIdALIx3GE8BMa7s6Gtd4jb7OH5Gg?=
 =?us-ascii?Q?xWJBZBS4d70NBsJlw9e1aup6RHqYnc399BlJspQ8XukdD7entJ5lp63RnYqi?=
 =?us-ascii?Q?s+onZWT3EaEups5528en04LwV4/MTSe9/RlRwuzZ8TkPWVK83LPnZomUGgQu?=
 =?us-ascii?Q?PTqS5610sIN0E696r9wvaLPSog9o+b1aEoZS7IfO+tGHi1/go4tZ0WpgaiN4?=
 =?us-ascii?Q?dauqeRHB9Cm9SNWX++X7KNniMAgxpGMFJyGpHejWIGSmSBECy9DKKuIyAig8?=
 =?us-ascii?Q?LXUM31hXKXWaqpdMOyP1m+9J8h2x/6UfKYSt5jbyiXhuumM/3eXk9Ys0W+mj?=
 =?us-ascii?Q?WiBWxUVairVhbxKsOIEKPyYr1w+bVoQXs/nYGqHLIcGtgWT69qMQYxR8WMAU?=
 =?us-ascii?Q?wqf2qdlVxQ9f2sv25LaZwjOjekqff+vih1A+0P9UcHuxBAy2Jzrcu/Gjijlq?=
 =?us-ascii?Q?h7jKyg8IZh9RVgHBhW1UBYI7h4MD45wYUI/GT62DszK1eyxa7M2zRNOWXprl?=
 =?us-ascii?Q?C02wdCScbIph1CKatLYITlnqtacQ1D9GQC1S5Cx7S/SYJTxlBqXgdxHPavxn?=
 =?us-ascii?Q?SaVd0xyi2GDW33lhpLbAVuInqCIKM47O3+W03qOzbabbK6VyNd2TWN/hm5pm?=
 =?us-ascii?Q?6wP9Ruy3HR+1iphCZyjWnPMChjADyqYX/Ia3F5kefDKX27hJODzG9MfeubLx?=
 =?us-ascii?Q?w91VlPernSPx0FUEmxEoeoVkLsXWe7P+AyfY5AU+mhhBWGx53GbhiE/ROJZP?=
 =?us-ascii?Q?+GF9vWStK+xEDdY/qpaTnRzIGC2YJXjBsxoigpFjWf8v6wvBKx9q4RjriO7x?=
 =?us-ascii?Q?RQG6qR+MOJ7mxAAh4BEckKtzA1e7lZETrGYfpvqga0+IPp8oQKHh3aM6ggG9?=
 =?us-ascii?Q?D81xiMJxE10j8mnB5YtfE17/uE694DuGr/Uzwm8lDn1oPJI8VQdKQqzMVWhO?=
 =?us-ascii?Q?YQHnUV+NvV3S/g4pMMjLRZ9yYaoWMXbzk8ojyO6S9JR+hM4gu8JU4Qh3qvEv?=
 =?us-ascii?Q?T6emeQEyMwxcXwlYUSCh8I1uGsrlOwFdouMoy1u+YxTh10By3Lf+LoF/ZRRJ?=
 =?us-ascii?Q?xpU0p8G1+S4JfT8JX3/eRWSM/YIx+ayyMabopZEYYLnh0DipHnx1vjwdx12j?=
 =?us-ascii?Q?pmusUYdgWY6Hbj4rMhpAPFnQyddG/7K62KPdKLMDRtLu8hJT5011RShozne7?=
 =?us-ascii?Q?mEZqQxcpOOIUNhs6yJQ=3D?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: f41ae1dd-65ad-4db0-d674-08daceefa678
X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Nov 2022 14:16:31.4144
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: pTay3tW9DeGZ0mZvV7cS+W0/o7D8BpX13DZkr0B7V2db0QnZaA5M8V3xqLzl/2w9
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL1PR12MB5996
X-Spam-Status: No, score=-1.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO,
        RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE
        autolearn=no autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Fri, Nov 25, 2022 at 02:22:24AM +0000, lizhijian@fujitsu.com wrote:
> 
> 
> On 25/11/2022 01:39, Jason Gunthorpe wrote:
> > On Wed, Nov 23, 2022 at 06:07:37AM +0000, lizhijian@fujitsu.com wrote:
> >>
> >>
> >> On 22/11/2022 22:52, Jason Gunthorpe wrote:
> >>> On Wed, Nov 16, 2022 at 04:19:50PM +0800, Li Zhijian wrote:
> >>>> It enables flushable access flag for qp
> >>>>
> >>>> Reviewed-by: Zhu Yanjun <zyjzyj2000@gmail.com>
> >>>> Signed-off-by: Li Zhijian <lizhijian@fujitsu.com>
> >>>> ---
> >>>> V5: new patch, inspired by Bob
> >>>> ---
> >>>>    drivers/infiniband/core/cm.c | 3 ++-
> >>>>    1 file changed, 2 insertions(+), 1 deletion(-)
> >>>>
> >>>> diff --git a/drivers/infiniband/core/cm.c b/drivers/infiniband/core/cm.c
> >>>> index 1f9938a2c475..58837aac980b 100644
> >>>> --- a/drivers/infiniband/core/cm.c
> >>>> +++ b/drivers/infiniband/core/cm.c
> >>>> @@ -4096,7 +4096,8 @@ static int cm_init_qp_init_attr(struct cm_id_private *cm_id_priv,
> >>>>    		qp_attr->qp_access_flags = IB_ACCESS_REMOTE_WRITE;
> >>>>    		if (cm_id_priv->responder_resources)
> >>>>    			qp_attr->qp_access_flags |= IB_ACCESS_REMOTE_READ |
> >>>> -						    IB_ACCESS_REMOTE_ATOMIC;
> >>>> +						    IB_ACCESS_REMOTE_ATOMIC |
> >>>> +						    IB_ACCESS_FLUSHABLE;
> >>>
> >>> What is the point of this? Nothing checks IB_ACCESS_FLUSHABLE ?
> >>
> >> Previous, responder of RXE will check qp_access_flags in check_op_valid():
> >>    256 static enum resp_states check_op_valid(struct rxe_qp *qp,
> >>
> >>    257                                        struct rxe_pkt_info *pkt)
> >>
> >>    258 {
> >>
> >>    259         switch (qp_type(qp)) {
> >>
> >>    260         case IB_QPT_RC:
> >>
> >>    261                 if (((pkt->mask & RXE_READ_MASK) &&
> >>
> >>    262                      !(qp->attr.qp_access_flags &
> >> IB_ACCESS_REMOTE_READ)) ||
> >>   
> >>
> >>    263                     ((pkt->mask & RXE_WRITE_MASK) &&
> >>
> >>    264                      !(qp->attr.qp_access_flags &
> >> IB_ACCESS_REMOTE_WRITE)) ||
> >>    265                     ((pkt->mask & RXE_ATOMIC_MASK) &&
> >>
> >>    266                      !(qp->attr.qp_access_flags &
> >> IB_ACCESS_REMOTE_ATOMIC))) {
> >>    267                         return RESPST_ERR_UNSUPPORTED_OPCODE;
> >>
> >>    268                 }
> >>
> >> based on this, additional IB_FLUSH_PERSISTENT and IB_ACCESS_FLUSH_GLOBAL
> >> were added in patch7 since V5 suggested by Bob.
> >> Because of this change, QP should become FLUSHABLE correspondingly.
> > 
> > But nothing ever reads IB_ACCESS_FLUSHABLE, so why is it added?
> 
> include/rdma/ib_verbs.h:
> +	IB_ACCESS_FLUSH_GLOBAL = IB_UVERBS_ACCESS_FLUSH_GLOBAL,
> +	IB_ACCESS_FLUSH_PERSISTENT = IB_UVERBS_ACCESS_FLUSH_PERSISTENT,
> +	IB_ACCESS_FLUSHABLE = IB_ACCESS_FLUSH_GLOBAL |
> +			      IB_ACCESS_FLUSH_PERSISTENT,
> 
> IB_ACCESS_FLUSHABLE is a wrapper of IB_ACCESS_FLUSH_GLOBAL | 
> IB_ACCESS_FLUSH_PERSISTENT. With this wrapper, i will write one less 
> line of code :)
> 
> I'm fine to expand it in next version.

OIC, that is why it escaped grep

But this is back to my original question - why is it OK to do this
here in CMA? Shouldn't this cause other drivers to refuse to create
the QP because they don't support the flag?

Jason

> 
> > 
> > Jason