Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <7c5eb785-0fe7-e0e5-8232-403e1d3538ac@intel.com>
Date:   Tue, 11 Apr 2023 16:21:09 -0700
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101
 Thunderbird/102.9.1
Subject: Re: [PATCH rdma-next 0/4] Allow relaxed ordering read in VFs and VMs
Content-Language: en-US
To:     Jason Gunthorpe <jgg@nvidia.com>, Leon Romanovsky <leon@kernel.org>
CC:     Leon Romanovsky <leonro@nvidia.com>,
        Avihai Horon <avihaih@nvidia.com>, Aya Levin <ayal@nvidia.com>,
        Eric Dumazet <edumazet@google.com>,
        "Jakub Kicinski" <kuba@kernel.org>, <linux-kernel@vger.kernel.org>,
        <linux-rdma@vger.kernel.org>, Meir Lichtinger <meirl@mellanox.com>,
        "Michael Guralnik" <michaelgur@mellanox.com>,
        <netdev@vger.kernel.org>, Paolo Abeni <pabeni@redhat.com>,
        Saeed Mahameed <saeedm@nvidia.com>,
        Shay Drory <shayd@nvidia.com>
References: <cover.1681131553.git.leon@kernel.org>
 <ZDVoH0W27xo6mAbW@nvidia.com>
From:   Jacob Keller <jacob.e.keller@intel.com>
In-Reply-To: <ZDVoH0W27xo6mAbW@nvidia.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?bGI1V3Q1ejB4T1lrdEVKTWM0QXVBSWM0UE5ZRitDL1dvMHliNTVRMCt1c0N1?=
 =?utf-8?B?cGEzVXFiVWJyU1pSMzV6QW5lUmowayt6SHRFdXpqRHU2U0YzeTBRTzJieTB4?=
 =?utf-8?B?djBqTUhFYTdpclh2d3ZiM0hhZm9nbWs4RUduSmJKSjZKajd6VHRPYW1GY2xW?=
 =?utf-8?B?d0lIcFM4Qi9XTTUzcG1YSDNCeGEwNDAxMlYrd2JaSFdrT2lqSm0wU3kzS3FQ?=
 =?utf-8?B?L2JleXVyTGpNbURDUTJodnN0bTRRR1AyYjJicXpML2R6MXVaTWZZLzByVzJR?=
 =?utf-8?B?VFhROWswYzNZaVRyS2d0Zk13cHR1RElTV0orOEo0QTRxOWxTS0xkdW9OaG5N?=
 =?utf-8?B?b3VvZFNNRzcrQzlsb21RaWdZUDJWc3I0b1ZLQTV5aHFFckNoSWhrYW5JRlVC?=
 =?utf-8?B?U2d5Q3UzaW1Ma2FwS2E0bjU5U0wva3FSNThERVJ1K2V2bTc5Q1BieDF2NDcv?=
 =?utf-8?B?WXZqemV2bkpPY2dlN3JacUxlbWJCUVl5aHQ1L0NuODNpa2NnVDZSb01MR0FW?=
 =?utf-8?B?MHMyYjRBME9FZGRzZ2VpaEQ3dk9YZjllS3VwS2Fna0hGZElXNEF2dXdJZkFj?=
 =?utf-8?B?aHVFU0JBN2hQWUo4MWZxYlRrYUFTSE1wSHBYcHRtaUZZcW9sNVRRakdxQXBl?=
 =?utf-8?B?VEdzVWJ0Y3Z0bytWWXR5Vm1RZ0JqdjBJZ0hyY3c0VjBuL0dHN1h3NGwyVStT?=
 =?utf-8?B?UGlkKzhqdnFXWFpPS3lNdHNDTnlONS93NnAwWTYraWhpc1JOd09EcW81UUFJ?=
 =?utf-8?B?MlA2NnBKTDZlc3hQSUVFRExkWWEwV0dyYjBkd3pSaHg0MXl4aE1YdEFhaGlD?=
 =?utf-8?B?cW5sU2FXTm51TVY5a1BYSGdoOUxqNFVUeGlZVVFJeVRTdUswazhOL2VqUWcx?=
 =?utf-8?B?RkVYcHNtdnFqcmwrdkVSdDd6aUdISjNyZWxXa1RyTzdPcTJHUFZXelE2R0U0?=
 =?utf-8?B?djZnL3Q3QXBHZVFlNDZ5ZUlReWFkTGNKenZmOWhRMkZpYWhnK3NVd3kyZkFJ?=
 =?utf-8?B?ZlBIWGI4Y2lScm5oTXkxUTgyNkprSXFuOGtnUVUrZWJUUXpUS0h0SGVCZzZl?=
 =?utf-8?B?ODVmWVprQjNNSHhEc1VSTXdLWEVQVVZrc2ZMSWc5OUhYQzdRS0xjR3Z1VW1V?=
 =?utf-8?B?ay83T1NTUWJZajNsUXRlY3BKWFB3NlU3aEhMMnhlcGNuNmZlNnBhc0JRbXUr?=
 =?utf-8?B?b2pLYy83UldHUnZSZTFLOHdzN0dvSDBvYU80UHpuNk1wRC9lc3VlVDVYSW1B?=
 =?utf-8?B?c1VhME9CTW12bHRmeTc5SjhGK0g0cEl1dXZrZ0VhZGJ1T3RVUE9tT3lwak5r?=
 =?utf-8?B?S0FMdzR3ZU5wZTY3ZWNoWEpESEMwaVNUdm9uVWtDTWZmSjh5RnpBbWtGUHFB?=
 =?utf-8?B?c0VCeUtMaE5OSmJnVVd0czY2ZS9OS2llSHNiOWN3U0pqWHNjTXdVUGxTNjAz?=
 =?utf-8?B?eG5NcGxuaGtsZnIvSWpXVzBhZmtSRlpzemZwdlZTMkxnemJjdXZBNklHckpX?=
 =?utf-8?B?bms3U0RoYzc5ditNUENEL3ByeVlnU1JnYmxmTElvb3ppaFN1L04wVzdzbDJm?=
 =?utf-8?B?MGpCSzVjN1U1TjI3VytPNmg5U1V1SjVRMnZDMUoxR0dOK0o0bGRRSUxFQisr?=
 =?utf-8?B?KzJsc0JTZW51elZqeUlnOVhxc01lakdyS3c1WFh5dytMRm1nUXFxL3R3M3ZG?=
 =?utf-8?B?THVJc3F1bDQ0RkxyZS9BbTNSMTFVZGZhU2dXeHZRUHN6OEZ3VWVkeVlsaStZ?=
 =?utf-8?B?WlpqZStYYUxaMnZiR1BWY1F1SjUxanVzN0c2NkRxbmZFeFlickw2eE1UQTJj?=
 =?utf-8?B?QnpnT3oxajIzenFvYW4ranV0czRnUFZrcTBycnlGTkNMdlBDalZhcld5QTJU?=
 =?utf-8?B?bmgyMUJRVzVlMmFrTkdneG51SVZOUkFJcU4xb245VVBjbGVuM0ZSMGs5ZWJQ?=
 =?utf-8?B?enF4cWM3SUZYSTRmUHR4c0dzMk9nejJBSDYzT2hiNVdKb0JnTEQwa2ZkWkwy?=
 =?utf-8?B?VVR4R1pzcFpzejRnaS85QlJCaWpYOVZGcmtPV0Jzeng2WGQ1WXJvWTcvdUhF?=
 =?utf-8?B?R3FNb0lMcEQ4WjJjZVhockk1UkxpMDNMdkZFUVBsN1NCUUppZ2drRUZHQy95?=
 =?utf-8?B?K1VxYzcxZjRCMWlFK2htczc0aTJDOGlJT3lyWmM5TDVVVFdpT0V1VnFJc3I0?=
 =?utf-8?B?a2c9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 07fec917-2a82-49e8-36f5-08db3ae36b94
X-MS-Exchange-CrossTenant-AuthSource: CO1PR11MB5089.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 11 Apr 2023 23:21:04.2355
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: Xudt4SmDBLbJWsv5A/raMcnsVd2VFyeMmS2qG0eaAgHkbQrKy2A2HnyKqp8PHHJQtr92rg41Ul/nIv9UkGkRCmqY8osSp9WotiS1ulXbpf4=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW4PR11MB6984
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-6.6 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,
        RCVD_IN_DNSWL_MED,SPF_HELO_NONE,SPF_NONE autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


On 4/11/2023 7:01 AM, Jason Gunthorpe wrote:
> On Mon, Apr 10, 2023 at 04:07:49PM +0300, Leon Romanovsky wrote:
>> From: Leon Romanovsky <leonro@nvidia.com>
>>
>> From Avihai,
>>
>> Currently, Relaxed Ordering (RO) can't be used in VFs directly and in
>> VFs assigned to QEMU, even if the PF supports RO. This is due to issues
>> in reporting/emulation of PCI config space RO bit and due to current
>> HCA capability behavior.
>>
>> This series fixes it by using a new HCA capability and by relying on FW
>> to do the "right thing" according to the PF's PCI config space RO value.
>>
>> Allowing RO in VFs and VMs is valuable since it can greatly improve
>> performance on some setups. For example, testing throughput of a VF on
>> an AMD EPYC 7763 and ConnectX-6 Dx setup showed roughly 60% performance
>> improvement.
>>
>> Thanks
>>
>> Avihai Horon (4):
>>   RDMA/mlx5: Remove pcie_relaxed_ordering_enabled() check for RO write
>>   RDMA/mlx5: Check pcie_relaxed_ordering_enabled() in UMR
>>   net/mlx5: Update relaxed ordering read HCA capabilities
>>   RDMA/mlx5: Allow relaxed ordering read in VFs and VMs
> 
> This looks OK, but the patch structure is pretty confusing.
> 
> It seems to me there are really only two patches here, the first is to
> add some static inline
> 
> 'mlx5 supports read ro'
> 
> which supports both the cap bits described in
> the PRM, with a little comment to explain that old devices only set
> the old cap.
> 
> And a second patch to call it in all the places we need to check before
> setting the mkc ro read bit.
> 
> Maybe a final third patch to sort out that mistake in the write side.
> 
> But this really doesn't have anything to do with VFs and VMs, this is
> adjusting the code to follow the current PRM because the old one was
> mis-desgined.
> 
> Jason

FWIW I think Jason's outline here makes sense too and might be slightly
better. However, reading through the series I was reasonably able to
understand things enough that I think its fine as-is.

In some sense its not about VF or VM, but fixing this has the result
that it fixes a setup with VF and VM, so I think thats an ok thing to
call out as the goal.