Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates
 216.228.112.34 as permitted sender) receiver=protection.outlook.com;
 client-ip=216.228.112.34; helo=mail.nvidia.com;
Subject: Re: [PATCH v2 rdma-next] RDMA/mlx5: Enable Relaxed Ordering by
 default for kernel ULPs
To:     Leon Romanovsky <leon@kernel.org>
CC:     Doug Ledford <dledford@redhat.com>,
        Jason Gunthorpe <jgg@nvidia.com>,
        Avihai Horon <avihaih@nvidia.com>,
        <linux-kernel@vger.kernel.org>, <linux-rdma@vger.kernel.org>,
        Christoph Hellwig <hch@lst.de>,
        Bart Van Assche <bvanassche@acm.org>,
        Tom Talpey <tom@talpey.com>,
        Santosh Shilimkar <santosh.shilimkar@oracle.com>,
        Chuck Lever III <chuck.lever@oracle.com>,
        Keith Busch <kbusch@kernel.org>,
        David Laight <David.Laight@aculab.com>,
        Honggang LI <honli@redhat.com>
References: <b7e820aab7402b8efa63605f4ea465831b3b1e5e.1623236426.git.leonro@nvidia.com>
 <9c5b7ae5-8578-3008-5e78-02e77e121cda@nvidia.com> <YNQoY7MRdYMNAUPg@unreal>
From:   Max Gurtovoy <mgurtovoy@nvidia.com>
Message-ID: <1ef0ac51-4c7d-d79d-cb30-2e219f74c8c1@nvidia.com>
Date:   Thu, 24 Jun 2021 10:39:16 +0300
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101
 Thunderbird/78.11.0
MIME-Version: 1.0
In-Reply-To: <YNQoY7MRdYMNAUPg@unreal>
Content-Type: text/plain; charset="utf-8"; format=flowed
Content-Transfer-Encoding: 7bit
Content-Language: en-US
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Jun 2021 07:39:24.3400
 (UTC)
X-MS-Exchange-CrossTenant-Network-Message-Id: 994f3af9-ce0a-45cb-bda0-08d936e3303e
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.112.34];Helo=[mail.nvidia.com]
X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT049.eop-nam11.prod.protection.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Anonymous
X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR12MB2806
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


On 6/24/2021 9:38 AM, Leon Romanovsky wrote:
> On Thu, Jun 24, 2021 at 02:06:46AM +0300, Max Gurtovoy wrote:
>> On 6/9/2021 2:05 PM, Leon Romanovsky wrote:
>>> From: Avihai Horon <avihaih@nvidia.com>
>>>
>>> Relaxed Ordering is a capability that can only benefit users that support
>>> it. All kernel ULPs should support Relaxed Ordering, as they are designed
>>> to read data only after observing the CQE and use the DMA API correctly.
>>>
>>> Hence, implicitly enable Relaxed Ordering by default for kernel ULPs.
>>>
>>> Signed-off-by: Avihai Horon <avihaih@nvidia.com>
>>> Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
>>> ---
>>> Changelog:
>>> v2:
>>>    * Dropped IB/core patch and set RO implicitly in mlx5 exactly like in
>>>      eth side of mlx5 driver.
>>> v1: https://lore.kernel.org/lkml/cover.1621505111.git.leonro@nvidia.com
>>>    * Enabled by default RO in IB/core instead of changing all users
>>> v0: https://lore.kernel.org/lkml/20210405052404.213889-1-leon@kernel.org
>>> ---
>>>    drivers/infiniband/hw/mlx5/mr.c | 10 ++++++----
>>>    drivers/infiniband/hw/mlx5/wr.c |  5 ++++-
>>>    2 files changed, 10 insertions(+), 5 deletions(-)
>>>
>>> diff --git a/drivers/infiniband/hw/mlx5/mr.c b/drivers/infiniband/hw/mlx5/mr.c
>>> index 3363cde85b14..2182e76ae734 100644
>>> --- a/drivers/infiniband/hw/mlx5/mr.c
>>> +++ b/drivers/infiniband/hw/mlx5/mr.c
>>> @@ -69,6 +69,7 @@ static void set_mkc_access_pd_addr_fields(void *mkc, int acc, u64 start_addr,
>>>    					  struct ib_pd *pd)
>>>    {
>>>    	struct mlx5_ib_dev *dev = to_mdev(pd->device);
>>> +	bool ro_pci_enabled = pcie_relaxed_ordering_enabled(dev->mdev->pdev);
>>>    	MLX5_SET(mkc, mkc, a, !!(acc & IB_ACCESS_REMOTE_ATOMIC));
>>>    	MLX5_SET(mkc, mkc, rw, !!(acc & IB_ACCESS_REMOTE_WRITE));
>>> @@ -78,10 +79,10 @@ static void set_mkc_access_pd_addr_fields(void *mkc, int acc, u64 start_addr,
>>>    	if (MLX5_CAP_GEN(dev->mdev, relaxed_ordering_write))
>>>    		MLX5_SET(mkc, mkc, relaxed_ordering_write,
>>> -			 !!(acc & IB_ACCESS_RELAXED_ORDERING));
>>> +			 acc & IB_ACCESS_RELAXED_ORDERING && ro_pci_enabled);
>>>    	if (MLX5_CAP_GEN(dev->mdev, relaxed_ordering_read))
>>>    		MLX5_SET(mkc, mkc, relaxed_ordering_read,
>>> -			 !!(acc & IB_ACCESS_RELAXED_ORDERING));
>>> +			 acc & IB_ACCESS_RELAXED_ORDERING && ro_pci_enabled);
>> Jason,
>>
>> If it's still possible to add small change, it will be nice to avoid
>> calculating "acc & IB_ACCESS_RELAXED_ORDERING && ro_pci_enabled" twice.
> The patch is part of for-next now, so feel free to send followup patch.
>
> Thanks
>
> diff --git a/drivers/infiniband/hw/mlx5/mr.c b/drivers/infiniband/hw/mlx5/mr.c
> index c1e70c99b70c..c4f246c90c4d 100644
> --- a/drivers/infiniband/hw/mlx5/mr.c
> +++ b/drivers/infiniband/hw/mlx5/mr.c
> @@ -69,7 +69,8 @@ static void set_mkc_access_pd_addr_fields(void *mkc, int acc, u64 start_addr,
>                                            struct ib_pd *pd)
>   {
>          struct mlx5_ib_dev *dev = to_mdev(pd->device);
> -       bool ro_pci_enabled = pcie_relaxed_ordering_enabled(dev->mdev->pdev);
> +       bool ro_pci_enabled = acc & IB_ACCESS_RELAXED_ORDERING &&
> +                             pcie_relaxed_ordering_enabled(dev->mdev->pdev);
>
>          MLX5_SET(mkc, mkc, a, !!(acc & IB_ACCESS_REMOTE_ATOMIC));
>          MLX5_SET(mkc, mkc, rw, !!(acc & IB_ACCESS_REMOTE_WRITE));
> @@ -78,11 +79,9 @@ static void set_mkc_access_pd_addr_fields(void *mkc, int acc, u64 start_addr,
>          MLX5_SET(mkc, mkc, lr, 1);
>
>          if (MLX5_CAP_GEN(dev->mdev, relaxed_ordering_write))
> -               MLX5_SET(mkc, mkc, relaxed_ordering_write,
> -                        (acc & IB_ACCESS_RELAXED_ORDERING) && ro_pci_enabled);
> +               MLX5_SET(mkc, mkc, relaxed_ordering_write, ro_pci_enabled);
>          if (MLX5_CAP_GEN(dev->mdev, relaxed_ordering_read))
> -               MLX5_SET(mkc, mkc, relaxed_ordering_read,
> -                        (acc & IB_ACCESS_RELAXED_ORDERING) && ro_pci_enabled);
> +               MLX5_SET(mkc, mkc, relaxed_ordering_read, ro_pci_enabled);
>
>          MLX5_SET(mkc, mkc, pd, to_mpd(pd)->pdn);
>          MLX5_SET(mkc, mkc, qpn, 0xffffff);
> (END)
>
Yes this looks good.

Can you/Avihai create a patch from this ? or I'll do it ?


>>