Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) client-ip=2620:137:e000::3:1;
Date:   Mon, 18 Sep 2023 11:49:23 -0300
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     Alex Williamson <alex.williamson@redhat.com>
Cc:     ankita@nvidia.com, yishaih@nvidia.com,
        shameerali.kolothum.thodi@huawei.com, kevin.tian@intel.com,
        aniketa@nvidia.com, cjia@nvidia.com, kwankhede@nvidia.com,
        targupta@nvidia.com, vsethi@nvidia.com, acurrid@nvidia.com,
        apopple@nvidia.com, jhubbard@nvidia.com, danw@nvidia.com,
        anuaggarwal@nvidia.com, kvm@vger.kernel.org,
        linux-kernel@vger.kernel.org
Subject: Re: [PATCH v10 1/1] vfio/nvgpu: Add vfio pci variant module for
 grace hopper
Message-ID: <20230918144923.GH13733@nvidia.com>
References: <20230915025415.6762-1-ankita@nvidia.com>
 <20230915082430.11096aa3.alex.williamson@redhat.com>
 <20230918130256.GE13733@nvidia.com>
 <20230918082748.631e9fd9.alex.williamson@redhat.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20230918082748.631e9fd9.alex.williamson@redhat.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?AaZjKIP2/YURX5Xee9o+235ea9LWSOzWgt5vV+IT9dwJIP1uNfryuIOzVs8N?=
 =?us-ascii?Q?6kT5W2M4o2Q3vWYfYnjpEnKbIkDcPAAzmO14Uocy9D1uwDuLY4YrhoK9F6ls?=
 =?us-ascii?Q?3yNk1Mc2z3YfxG1y+t5hbBwYoc9WGFqJAlqhlTSiCgSdPPVLxMXgcVXdsPM7?=
 =?us-ascii?Q?9Iiv0Nf7SvkvxlVmWQDTZPmd7GdiniaQvcKaXmhEd0tWZyYGou6xkQByQ7sv?=
 =?us-ascii?Q?Cc14lExsITyNUf7512T1Spb+9/yfl2rNBxDRDkbqxiFF/oapgNY238DSfTVS?=
 =?us-ascii?Q?9MHju1PhYZKFD/nmWg60n7iGkzXe0Y/da914yJKnZX9CkXNmcZP47A6nBHYV?=
 =?us-ascii?Q?uJvD8BhtV18KsJoQqNgfLEKnks/hVsLL/6Edr88os3qou+Nv8RHmW6o1pzdM?=
 =?us-ascii?Q?P2AmJh8Os46IJIkIw7Vvw9DsJ6MmhUKnTdIMMMgXKlvxinmzFVrNXq8jXDO/?=
 =?us-ascii?Q?N5uJJtveQHGcTcgKPZJMT9Qmj0kkPXDXhkJsicPcJth23piHI10J4JWl1ECQ?=
 =?us-ascii?Q?Jk5LfxEJVjD/hx61oKjraNxM38sA/YGeFGKUeHTCYovvmuUWe1+4REW/RBgd?=
 =?us-ascii?Q?/JFu+PSA1dEhN9ejIkJi/k0SRxoNCzM+cjXsVG/q//JTcD0wA4uNrDruUl3S?=
 =?us-ascii?Q?t0/w8qSau9eVw4fVy3f6NTTMDfYCQqTltEMu3Z0YuHMNh/3bdRfpxqfzrVOj?=
 =?us-ascii?Q?ptF0oiBxTsf9Fs+6tCVsKkM0ZJ4o8x9WqXcjMX9HHRJmZMGf9eYrDdMi8Y8N?=
 =?us-ascii?Q?OtF+8pF6rFGxSUmS0emgcDPT7h6asvLyWddbFupPJ6CbESk00zgMvJMOCcck?=
 =?us-ascii?Q?ib7b5WgVomRcNBqjvs+n+yeEUKmDAn6FoG0NVDZhd26q2BvTIpmnYVgF6Gn1?=
 =?us-ascii?Q?T0Pbkw1ILPm6dTMjFKVnJA8sT4ABKkXcMPxgBc5sTcrMPcfXmICRSq18tyct?=
 =?us-ascii?Q?bMTasT8p7E6AwEKPSyFs+sh7Yz9MUQpj7Y6adSgSjhxvJUzrA+bo3J+GtmXY?=
 =?us-ascii?Q?btZHvDBN+/DkSRan33saVx9Gkr4u4l29RpoUTXkNXdTCvL2B5Gm90PZqRQza?=
 =?us-ascii?Q?FYlkegoIHVF1+akOIBa7zyaXboi9rA7ytT2at826PKuyVdNvS5UR+6k7UYQY?=
 =?us-ascii?Q?KClN4YYMZn7q5YT44sgJrxMyklUjvPlrvSXUDKOgqTojNh6ffH/+01KQUQCu?=
 =?us-ascii?Q?Zttc6BuNPhGjWbyQONJxh4EJIopqiNoV7An1Shmspk8ZkG9cAZuhWrVW9o62?=
 =?us-ascii?Q?9npTvCw2AJJTHMWlG1p1sTjs8xoU7NiaNaKyCUMxKJKxiZXIuWmQwNAEkRNX?=
 =?us-ascii?Q?7CgW629e5RamXQB5x8lRL9sydfKSd0HH25I6+Kj4Hgn5rw8JXqyO7q1ZxG6x?=
 =?us-ascii?Q?deP5+Vpm5r40N3xXnGUi+QTCK2ivrTfyBmwZJ3tfEpJge9hrsc9h81BGPuik?=
 =?us-ascii?Q?y9y50gaqsRH6cjxISh/5NBTW2rk+cbmJS9yL9wxuf6jGUkHtv3pnn4L5b2eC?=
 =?us-ascii?Q?GE4rnD44pIU1Gr19+khH1nNtb0adUkqsUraq/Mkuskpxs5Zbr4IyVAhO3gXc?=
 =?us-ascii?Q?jBKXJ2aQKfzskY0IAcecB0BW/sWjSoJ1s1/IBVDh?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 7a1f7219-3d1e-4991-ffc3-08dbb8567342
X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Sep 2023 14:49:24.5221
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: me4mUtKHJMzD28WJAe/gP8urccus0LBOwXKONXjfjWqqQCEIeGn/czqkt+/QYZF4
X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV8PR12MB9357
X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable
	autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on morse.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (morse.vger.email [0.0.0.0]); Mon, 18 Sep 2023 08:20:49 -0700 (PDT)

On Mon, Sep 18, 2023 at 08:27:48AM -0600, Alex Williamson wrote:

> > > This looks like a giant red flag that this approach of masquerading the
> > > coherent memory as a PCI BAR is the wrong way to go.  If the VMM needs
> > > to know about this coherent memory, it needs to get that information
> > > in-band.   
> > 
> > The VMM part doesn't need this flag, nor does the VM. The
> > orchestration needs to know when to setup the pxm stuff.
> 
> Subject: [PATCH v1 1/4] vfio: new command line params for device memory NUMA nodes
> --- a/hw/vfio/pci.c
> +++ b/hw/vfio/pci.c
> ...
> +static bool vfio_pci_read_cohmem_support_sysfs(VFIODevice *vdev)
> +{
> +    gchar *contents = NULL;
> +    gsize length;
> +    char *path;
> +    bool ret = false;
> +    uint32_t supported;
> +
> +    path = g_strdup_printf("%s/coherent_mem", vdev->sysfsdev);
> +    if (g_file_get_contents(path, &contents, &length, NULL) && length > 0) {
> +        if ((sscanf(contents, "%u", &supported) == 1) && supported) {
> +            ret = true;
> +        }
> +    }

Yes, but it drives the ACPI pxm auto configuration stuff, not really
vfio stuff.

> > I think we should drop the sysfs for now until the qemu thread about
> > the pxm stuff settles into an idea.
> > 
> > When the qemu API is clear we can have a discussion on what component
> > should detect this driver and setup the pxm things, then answer the
> > how should the detection work from the kernel side.
> > 
> > > be reaching out to arbitrary sysfs attributes.  Minimally this
> > > information should be provided via a capability on the region info
> > > chain,   
> > 
> > That definitely isn't suitable, eg libvirt won't have access to inband
> > information if it turns out libvirt is supposed to setup the pxm qemu
> > arguments?
> 
> Why would libvirt look for a "coherent_mem" attribute in sysfs when it
> can just look at the driver used by the device.  

Sure, if that is consensus. Also I think coherent_mem is a terrible
sysfs name for this, it should be more like 'num_pxm_nodes' or
something.

> Part of the QEMU series is also trying to invoke the VM
> configuration based only on this
> device being attached to avoid libvirt orchestration changes:

Right, that is where it gets confusing - it mixes the vfio world in
qemu with the pxm world. That should be cleaned up somehow.

> > > A "coherent_mem" attribute on the device provides a very weak
> > > association to the memory region it's trying to describe.  
> > 
> > That's because it's use has nothing to do with the memory region :)
> 
> So we're creating a very generic sysfs attribute, which is meant to be
> used by orchestration to invoke device specific configuration, but is
> currently only proposed for use by the VMM.  The orchestration problem
> doesn't really exist, libvirt could know simply by the driver name that
> the device requires this configuration.  

Yep

> And the VMM usage is self inflicted because we insist on
> masquerading the coherent memory as a nondescript PCI BAR rather
> than providing a device specific region to enlighten the VMM to this
> unique feature.

I see it as two completely seperate things.

1) VFIO and qemu creating a vPCI device. Here we don't need this
   information.

2) This ACPI pxm stuff to emulate the bare metal FW.
   Including a proposal for auto-detection what kind of bare metal FW
   is being used.

This being a poor idea for #2 doesn't jump to problems with #1, it
just says more work is needed on the ACPI PXM stuff.

So lets remove the sysfs, reach a conclusion with the qemu folks on
how the pxm is supposed to be modeled and then Ankit can come back
with some suggestion how to do auto configure. If it needs a sysfs or
not we can see.

Jason