Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.38 as permitted sender) client-ip=23.128.96.38;
From:   "Tian, Kevin" <kevin.tian@intel.com>
To:     Jason Gunthorpe <jgg@nvidia.com>
CC:     "ankita@nvidia.com" <ankita@nvidia.com>,
        "alex.williamson@redhat.com" <alex.williamson@redhat.com>,
        "yishaih@nvidia.com" <yishaih@nvidia.com>,
        "shameerali.kolothum.thodi@huawei.com" 
        <shameerali.kolothum.thodi@huawei.com>,
        "aniketa@nvidia.com" <aniketa@nvidia.com>,
        "cjia@nvidia.com" <cjia@nvidia.com>,
        "kwankhede@nvidia.com" <kwankhede@nvidia.com>,
        "targupta@nvidia.com" <targupta@nvidia.com>,
        "vsethi@nvidia.com" <vsethi@nvidia.com>,
        "Currid, Andy" <acurrid@nvidia.com>,
        "apopple@nvidia.com" <apopple@nvidia.com>,
        "jhubbard@nvidia.com" <jhubbard@nvidia.com>,
        "danw@nvidia.com" <danw@nvidia.com>,
        "anuaggarwal@nvidia.com" <anuaggarwal@nvidia.com>,
        "dnigam@nvidia.com" <dnigam@nvidia.com>,
        "udhoke@nvidia.com" <udhoke@nvidia.com>,
        "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
Subject: RE: [PATCH v11 1/1] vfio/nvgpu: Add vfio pci variant module for grace
 hopper
Thread-Topic: [PATCH v11 1/1] vfio/nvgpu: Add vfio pci variant module for
 grace hopper
Thread-Index: AQHZ+VwgDWGk0VCFcUeltplt5s48drBCtrFAgAAx5oCAAPC30A==
Date:   Wed, 11 Oct 2023 02:00:30 +0000
Message-ID: <BN9PR11MB5276ECF96BAC7F59C93B5E4A8CCCA@BN9PR11MB5276.namprd11.prod.outlook.com>
References: <20231007202254.30385-1-ankita@nvidia.com>
 <BN9PR11MB52762EE10CBBDF8AB98A53788CCDA@BN9PR11MB5276.namprd11.prod.outlook.com>
 <20231010113357.GG3952@nvidia.com>
In-Reply-To: <20231010113357.GG3952@nvidia.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?5rhfcotzoGW/yB1HYvqfcsTm3PFx+Up8JG3iKTj7gI8y55Lc7lTZQTcwCOeW?=
 =?us-ascii?Q?tzROCsgzamrOOH82b3/YjM8LqrZhsX3xpPLv2AD5zeiU36GTBYucUq5lnPk2?=
 =?us-ascii?Q?Jns1IirjpftRPSMIoqOyRiQqEq9b24q/Y/yazOSrdujl1KRWWVq3N2C+wu6d?=
 =?us-ascii?Q?Ms6FNKxGigepCmufN0HlemIFM6VtyyDzuH3D/VAVL0hnmevolAl5TTmUeq0i?=
 =?us-ascii?Q?ZfrrxbRIhr1hJJqIFk4h5KVMelwMyGva4HSY40P8fUH11Y/vzCZVRn7ZwLZ4?=
 =?us-ascii?Q?C8PyDbGQHYxbQH+ju5SXYGZKr3JRmmh/jQNsAwiYHasDGEcBMUFoj68PDwm5?=
 =?us-ascii?Q?sOrEAgiirtB5uiUizUOGBhlc9CNkaIkCaL0pJ0eSLCOuitme4t72AfYTbvZu?=
 =?us-ascii?Q?f4ZI97TVL6kiqMk7MPGl968s2YR1LUUZ4TpRXdpU/887r/wELkYX6ACtCf9O?=
 =?us-ascii?Q?/OET1Y5Qi+feGw6IVj15E/hTg05YSh1w/VUgwM/b3zzw7BWa0b9NPQAaBzKO?=
 =?us-ascii?Q?XtJ0qmenTsCB5hpuiiOx4vr7EoPIpIuSR0cWwPIF1u6xMP1CzGXj5KIjiSvX?=
 =?us-ascii?Q?xFpcxNa9HEZHjlaVfzcEp+NXdlpwDyKERFIE9CJOGcFVVZ6g3KpzBMFzkL1w?=
 =?us-ascii?Q?r47KDyoz0CECTbXbviJIZVF9Z2u46bTMy6oXqLqneSBl7uC18dRS8eamtRXG?=
 =?us-ascii?Q?MB4WuwOUz9WChpjrCFmYIOw7j1sipsKAgl6IjtdX9wlEkLfQMKFe7Mw2tPzY?=
 =?us-ascii?Q?0gCng6c6rX35akHcTiJa/WP8gdEIPuMuvL61dDNMtL1+7S6x284efa7GdZge?=
 =?us-ascii?Q?b3AUesx2N8PzjW4kaZOJqGkzgMEV5JDd4t6598DE2z0egt/9cGM2YQ1OQg2q?=
 =?us-ascii?Q?AnbFhPTAhJaJ/0vT8AmESuYiJWoCEYku9zfqvijiW1V6v7a8/mx+w0xlBMK9?=
 =?us-ascii?Q?QMihTBO7tA1o3W4SPVjjEf27JXiBYixiVhebvY1AtmwHiJBHckrYzTuq4OR5?=
 =?us-ascii?Q?//T97WzVS4ek+RsSpc6do23DbHyb27uN14+oRXLSux/P3zODSBWuoCksn0mh?=
 =?us-ascii?Q?uUPKH2SBCtyYER8Yh944EMfEFlapcqFJyVAG6eHoeqdq0iHu8y42IzWCS5zO?=
 =?us-ascii?Q?RfMAIPwoUvVzTg7yXAwbqesrmSV3tVVpGSOh+QY2RHGjwWwE8DsqJ7tHjWKL?=
 =?us-ascii?Q?IN2rcxkffMmt17zc0qimJpDo8/fTCdQ99zx3Ti8oPczajYSwguXVMIkDNhP5?=
 =?us-ascii?Q?OxGplYh2uqAlJUCnCMXM9pvwMKSPetjvp+l3PzhVAK8pY9dTbbpOkRUm/fIJ?=
 =?us-ascii?Q?0lVRmdIcjpTaqEgsq7Hd0x5AohwSljmkUrmHJuTxRMivW3aFx/WDmycpnLZ0?=
 =?us-ascii?Q?3sbOqglmSmyUb7zMWnn5RlxvBiOfw6efcjV3KgaOgA0BHxGiBVYnRUCS4kFz?=
 =?us-ascii?Q?ox+WWbqVw8+whoWz0EpboTC2oAADGhpUKNewy+1nS8od1O/hNj3McejgPz8r?=
 =?us-ascii?Q?O1fvug8LRQgTnbF1IgWdw/kSsSAzAUESA40+X0FyiBPkV9Q57D+Bs+MVFGK3?=
 =?us-ascii?Q?JgRUokT6GetrSgKHu/zImKlGMIjVZEyn6m2z/WuG?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN9PR11MB5276.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 6e25c71b-bc20-4563-9ed0-08dbc9fdd8d7
X-MS-Exchange-CrossTenant-originalarrivaltime: 11 Oct 2023 02:00:30.4408
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: gkRYWFhe3HHHts+/GTXW0nA3LZ2NTAFHs8tEgYK6zz570sxoYdjQh1Vyh5mj03Dc41BtlV1sU4GP4EAuBYLIkw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CO1PR11MB4787
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=2.7 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,RCVD_IN_SBL_CSS,SPF_HELO_NONE,SPF_PASS,
	URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on fry.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (fry.vger.email [0.0.0.0]); Tue, 10 Oct 2023 19:00:56 -0700 (PDT)
X-Spam-Level: **

> From: Jason Gunthorpe <jgg@nvidia.com>
> Sent: Tuesday, October 10, 2023 7:34 PM
>=20
> On Tue, Oct 10, 2023 at 08:42:13AM +0000, Tian, Kevin wrote:
> > > From: ankita@nvidia.com <ankita@nvidia.com>
> > > Sent: Sunday, October 8, 2023 4:23 AM
> > >
> > > PCI BAR are aligned to the power-of-2, but the actual memory on the
> > > device may not. A read or write access to the physical address from t=
he
> > > last device PFN up to the next power-of-2 aligned physical address
> > > results in reading ~0 and dropped writes.
> > >
> >
> > my question to v10 was not answered. posted again:
> > --
> > Though the variant driver emulates the access to the offset beyond
> > the available memory size, how does the userspace driver or the guest
> > learn to know the actual size and avoid using the invalid hole to hold
> > valid data?
>=20
> The device FW knows and tells the VM.
>=20

This driver reads the size info from ACPI and records it as nvdev->memlengt=
h.

But nvdev->memlength is not exposed to the userspace. How does the virtual
FW acquires this knowledge and then report it to the VM?

One option is to indirectly decide the size based on sparse mmap info. But
that is kind of a misuse of sparse mmap then such misuse better fits to be
a device specific region as Alex suggested.

Did I overlook an important fact in the overall picture?