Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Wed, 14 Jun 2023 16:32:41 -0300
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     Alex Williamson <alex.williamson@redhat.com>
Cc:     ankita@nvidia.com, aniketa@nvidia.com, cjia@nvidia.com,
        kwankhede@nvidia.com, targupta@nvidia.com, vsethi@nvidia.com,
        acurrid@nvidia.com, apopple@nvidia.com, jhubbard@nvidia.com,
        danw@nvidia.com, kvm@vger.kernel.org, linux-kernel@vger.kernel.org
Subject: Re: [PATCH v3 1/1] vfio/nvgpu: Add vfio pci variant module for grace
 hopper
Message-ID: <ZIoV2UyZ71cwXroD@nvidia.com>
References: <ZH9p+giEs6bCYfw8@nvidia.com>
 <20230606121348.670229ff.alex.williamson@redhat.com>
 <ZH+DdVIyZ6hHCDaK@nvidia.com>
 <20230606153057.4cbc36a0.alex.williamson@redhat.com>
 <ZH/LzyF/uttviRnQ@nvidia.com>
 <20230607122303.5d25c973.alex.williamson@redhat.com>
 <ZIh+wXFrls7StWzc@nvidia.com>
 <20230613132402.2765b6cb.alex.williamson@redhat.com>
 <ZIn/EHnCg444LJ3i@nvidia.com>
 <20230614132047.519abe95.alex.williamson@redhat.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20230614132047.519abe95.alex.williamson@redhat.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?osDUHYhHmMidBrYAOGJ1BemfBE1HFRQNFVFpr/cQWuWHKeGj00pM80NaVcRR?=
 =?us-ascii?Q?jQJGV1rwiTtHWODf6ZH6aCTqVReMiTrmTvUBodcGh8UoFpYdm19glz7yJthn?=
 =?us-ascii?Q?dCTg0PcCQDJVo+P5EhBYeTbk8YdY0t+661k4Jn6/SKVMGof5yYIywuIce2Ww?=
 =?us-ascii?Q?wLPg5tJc1aFcaU11BE8FUIMPoXz6KUBlt4ZrpNVj5fkr+Kxtk11nn1IaRrAw?=
 =?us-ascii?Q?MMgt9kAbqJIJp71PvHdcqhgJyhFUdX4fpUgcm0P0tQdJV3usE6K0aUCXo6vS?=
 =?us-ascii?Q?spHXJ/N9yri8IiwsuHv0dgEWgJMe7d5Nuocn5H1WJuO5zvkcAjbniyyxJAld?=
 =?us-ascii?Q?Zq2T3gc6spEV3SDtpUgd5O/Frd9+EL1t1GvPf6I2Wlh+FFgFqXAvnVS20o5J?=
 =?us-ascii?Q?fde2fm8o6CuTwTKbBcNyQI1z7bXin1KN5Viu56CiOahhXzTAFBrDbk4/KxBE?=
 =?us-ascii?Q?r8zz73F61n0S9vPKsbDjDpgbVaZXQ5NuCJ2YHH1h+cK0hMdIYcnz9VxCYB4S?=
 =?us-ascii?Q?8JiT4v55X1HKXIqi4pKkWqjFNEbJDr6ASQhDsUqs4DXeU6wwEFDcSlKezBaw?=
 =?us-ascii?Q?/rh2kG3gKg3CwHY3G9Qkea6hlcxqiMxkmT98618tZHendGkW0oIqKTIHERyg?=
 =?us-ascii?Q?W4ZdiQ+cuKwTVWI8Ok28X55zxT4aFRm5ujYVxBeMIM+QPIVp72j5f71YMuFQ?=
 =?us-ascii?Q?/NEiHmL4aMzdutaajp8pBvNdQkCZ+dq5Cu6Mxqla3P3MBSi0V/TdlNeMrygi?=
 =?us-ascii?Q?IbvN9JAH6QyczK/cDCFy6C4fWDc3tJWhbtV+63OmRFyjI1c0I45Sa+HDDySJ?=
 =?us-ascii?Q?D+QDphVnXV4jC5dCQOfha3ZO+bw5ODNltK0wcD8iYvo/CbwaeT+Fn909wR5b?=
 =?us-ascii?Q?dgS/o41oUht5VjLYdT2uH1GDE4peWjDvvgiOP1BxdpDFM6MVcYrBOo/iGQd5?=
 =?us-ascii?Q?SB9sxkMCNHuutXYGkZad+g5L+VselLhzGE1LmAokBR0PQZLUdMnOgx0X9R/B?=
 =?us-ascii?Q?Dr+VppmV4lu+3eiHypBX6BLjb1oAGVzwmVbimN5BmEdznJ2DcYaiogJtbOlR?=
 =?us-ascii?Q?qyH6Dps2xNcqVltTYx8ZFztjEHLBIN+brxbQwJ54YQXA07wwhtCmWl+qKmDy?=
 =?us-ascii?Q?CI2DIXWUtQh0AzDYjJAs1M1AYXjVSWhC+SAp6dEpIYg1cKdUgzWn68MtaTmf?=
 =?us-ascii?Q?dk8DimRJbQHQffR17MbRFkghPbiESfISkC0xd/Yh24LnUYj2tv9uBr6NYtQY?=
 =?us-ascii?Q?rdW8rkBd2TG5xmBM5/voE8XgZMOTKJkgk9l59gqU342Qnng8e+MwfdyP/Cfi?=
 =?us-ascii?Q?vPtLgrPhfvJlUQNd9TbDQpjNweV1+h9d+dDdzs8ENoGVAv6dYZmjYet+nZzQ?=
 =?us-ascii?Q?iizKeYSjEJyYGFuR4Vd0l4RL1UbDIzj80Rwwv0VznpdZGTTpCMzYJ+JT0kCt?=
 =?us-ascii?Q?0mE1lyyPKjO99uvZ3RIOyHEinssc/HHAOxK8SPbCHqcm+KPCOeFKDvqYblPP?=
 =?us-ascii?Q?rgzmGrjTCqECyQX/quwO35jZoVuZQlc6cOLsWIrkN1bFEO+7vWfzBjJC+/dL?=
 =?us-ascii?Q?uxcj4XdhwymmPdgwqkFyrzXjpHoJNf6nVjzONKOu?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 4c266937-0433-45f7-9c24-08db6d0e200e
X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Jun 2023 19:32:43.9189
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: iHM+cq6RHop75EcqQHjV1TMh7sJwJymWxZKJJtSgkvasntNgyjvC7yqpZoxA4pl2
X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA0PR12MB8352
X-Spam-Status: No, score=-1.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO,
        RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE,
        T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=no autolearn_force=no
        version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Wed, Jun 14, 2023 at 01:20:47PM -0600, Alex Williamson wrote:

> Regarding relaxed ordering, are we talking about the 'No RO-enabled
> PR-PR Passing' bit in the devcap2 register?  

I don't think so.. The RO problem is that the RO bit in the config
space is only defined for a SRIOV PF not a SRIOV VF. The VF wires 0 to
that config space.

So if you stick a VF into a VM as a vPCI then it appears to have a 0
RO bit in the config space. A VM driver that relies on it will think
it is not enabled.

I suppose atomic will have the same issue?

> In general, I think we put driver writers in an awkward place if they
> start trying things that are clearly not supported as reported by
> hardware capability bits.  

If you start down this path then IMHO it all has to work the same for
a VF and the VF has to inherent any relevant PF capability bits. This
also means the vPCI emulated bits don't actually work correctly which
is a different mess..

This is why the answer of "don't trust the config space" is appealing
for driver writers because the VMMs are not able to emulate bare metal
very well in these details.

Jason