Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Tue, 16 May 2023 21:28:35 -0300
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     Alex Williamson <alex.williamson@redhat.com>
Cc:     ankita@nvidia.com, aniketa@nvidia.com, cjia@nvidia.com,
        kwankhede@nvidia.com, targupta@nvidia.com, vsethi@nvidia.com,
        acurrid@nvidia.com, apopple@nvidia.com, jhubbard@nvidia.com,
        danw@nvidia.com, kvm@vger.kernel.org, linux-kernel@vger.kernel.org,
        yishaih@nvidia.com, shameerali.kolothum.thodi@huawei.com,
        kevin.tian@intel.com
Subject: Re: [PATCH v2 1/1] vfio/nvgpu: Add vfio pci variant module for grace
 hopper
Message-ID: <ZGQfszAGGKhCp20q@nvidia.com>
References: <20230509040734.24392-1-ankita@nvidia.com>
 <20230516150914.26ae99c3.alex.williamson@redhat.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20230516150914.26ae99c3.alex.williamson@redhat.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?uiH0OUfLxiJyJV9TvKrKUk4VZJuRGCQyl5LMeT7K3bJuTIpWfqfcMD91yLJl?=
 =?us-ascii?Q?ULV3jnYxl4hJgP561fpH6LT2sd7EpxQ5MK9GHlm0Djz/NAOu2MRZRv8Tvqgu?=
 =?us-ascii?Q?lQRNDIE5tQHAg4otuByppLmlLM82wXYvviTWdJZVwhDlXFg1FJN2MWF6mV++?=
 =?us-ascii?Q?txFRpucCgxOVVq9yyUbn3cdm5q0ZGmHoOF1CcnM1qcpP2k6zqc4jwIVGjjlj?=
 =?us-ascii?Q?s/xWdZtm0TH+MENm9o7YIo6Wbqypo1IkmfZI3IRkrMazJJ2tKn6MRHDju4Ns?=
 =?us-ascii?Q?PEIUTh6egbRGm+3WY9lEy5skk1vkH3qqnJXzNSogaubw+zSRgUIqWR5L1ObP?=
 =?us-ascii?Q?fAozzHRCDvsnvqPSd1yp6luZoVWZw5bOLhsn5lLn9cDyBkfFX6BKHu23UpyJ?=
 =?us-ascii?Q?POeG0FEgUsgZHuvqxKUVlsXKI3iHKherkV2j07634yImFLIrBRdmjBPwdyKa?=
 =?us-ascii?Q?cNv8avLb8Muf6yxzXgxw/xI9hUVjs+nNmKekjX2wQWl8BNGoGBnbUF+NSMLq?=
 =?us-ascii?Q?rxnW48SegbAJCAbRCM6c/Hi6zA+7Nm0+8X5EMTXYyst9uFYGfpvWoT8Ea18o?=
 =?us-ascii?Q?sHFxnhUMUVXvsrxnAnou21YVUFGFDjjf83bUxofQmjG8fHbbo2LBbTFtr/O8?=
 =?us-ascii?Q?6b/Myhz1eZX2uk+MeChov3U/KZrZN/uREVYocoYRZtzPcT7mGJALX+437Veq?=
 =?us-ascii?Q?Me2Fop3HU/3aVjP+Ul1LfuxuKm28lNWF7Qaf8JkHQ57wvd788jZqXTtWY7V/?=
 =?us-ascii?Q?y6/XOyPJSOHBSVD1hMb1xAkAKuvwBwP2O1iV5+4fq7Q+baBlvRbkVStAQV+O?=
 =?us-ascii?Q?c+Sit0kPwXmIA18h90nKG4xaGLZm5ocFnk+B7imlVYNFhowIbYSR5M6wqA9M?=
 =?us-ascii?Q?jylIc0nta+EL3Hlw5rpBudqPppAmLN7Tzt6OQykqM9KGmNm7vQBeqRJ+qy3z?=
 =?us-ascii?Q?jGfNaYE6qVuh5JO7TMoUHh6n46F/EN8nMVhQp1BDo2oTbFCrr5imy8Y7I0q/?=
 =?us-ascii?Q?y3TOf/Kkupi9QdusBf7LEtBmdy7IanBajkozpcQNiItGtoPJEC1UxaYN3eCs?=
 =?us-ascii?Q?NgoncvdIJSl4JLKdESxLmNZtBxwohH+9jsPxxsAcw7rQTNJP+wjuTda5CIC0?=
 =?us-ascii?Q?slfgoeu69VliOlxZsD8QpoX24/XNe72NK6UL/ahiwdsx1v29E0BgTZVPLULH?=
 =?us-ascii?Q?NBgvNJFIZfFq6a0zL7mTko6+qbOjB9upMa+Dxry02p506yxE1d5pkjSKKLls?=
 =?us-ascii?Q?QHIzDwQ7Bk0agkvsWFb4UQiCE74AQObIPmDB49wCjKPb+SI9H4zhrKkF+Izx?=
 =?us-ascii?Q?EPcBlLivZZxAhC06oCvWIfQXq9CWPFF0lVncsmMB3tfp2pshPYdgFQ9ZAckh?=
 =?us-ascii?Q?MRN3OBpkX1JhLcRRbm+5FGHWyK1DHbaY2TzGDUtRYAm6wrGaGi5VlCJCPvb1?=
 =?us-ascii?Q?MYU4dJkLYjUrQuPEHbPKwi34HBJ8wg6xgZ8Tp16fPc8x2ho1Or2RE2/YOKIA?=
 =?us-ascii?Q?pv3Po4Z0tKgHYSSkYObsCICNTfHA/GeUHz7ZheJeRCQ0m/pmx2RLNipvz0+U?=
 =?us-ascii?Q?dANE/0I0/qxVoQ9u/Bh9p0rCak1llkuqwKTzVGdl?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: b6e524fc-5764-41e3-a9c4-08db566da796
X-MS-Exchange-CrossTenant-AuthSource: MN0PR12MB5859.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 May 2023 00:28:36.8666
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: Ds11jTz3Lb4/Zyx+5nA4VayuvSIxqSIhThA7psAJf4CSRtilzM47MrAUxUIwXYG2
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR12MB7490
X-Spam-Status: No, score=-1.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO,
        RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE,
        T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=no autolearn_force=no
        version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Tue, May 16, 2023 at 03:09:14PM -0600, Alex Williamson wrote:

> > +# SPDX-License-Identifier: GPL-2.0-only
> > +config NVGPU_VFIO_PCI
> > +	tristate "VFIO support for the GPU in the NVIDIA Grace Hopper Superchip"
> > +	depends on ARM64 || (COMPILE_TEST && 64BIT)
> > +	select VFIO_PCI_CORE
> 
> I think this should be a 'depends on' as well, that's what we have for
> the other vfio-pci variant drivers.

It should be removed completely, AFAICT:

config VFIO_PCI
        tristate "Generic VFIO support for any PCI device"
        select VFIO_PCI_CORE

Ensures it is turned on

if VFIO_PCI
source "drivers/vfio/pci/mlx5/Kconfig"
endif

Autoamtically injects a 'depends on VFIO_PCI' to all the enclosed
kconfig statements (and puts them nicely in the menu)

So we have everything needed already

SELECT is the correct action since it doesn't have a config text.

> Is our test for vm_end < vm_start in vfio-pci-core just paranoia?  I
> don't see an equivalent here.

Yes, mm core will not invoke the op with something incorrect.
 
> Can we also get a comment in the code outlining the various reasons
> that this "BAR" doesn't need the disabled access protections that
> vfio-pci-core implements?  For example outlining the behavior relative
> to BAR access while the memory enable bit is disabled, the bus being in
> reset, or the device being in a low-power state.

The HW has some "isolation" feature that kicks in and safely
disconnects the GPU from the CPU.

A lot of work has been done to make things like VFIO and KVM safe
against machine checks/etc under basically all circumstances.

Jason