Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Tue, 6 Jun 2023 14:16:42 -0300
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     Alex Williamson <alex.williamson@redhat.com>
Cc:     ankita@nvidia.com, aniketa@nvidia.com, cjia@nvidia.com,
        kwankhede@nvidia.com, targupta@nvidia.com, vsethi@nvidia.com,
        acurrid@nvidia.com, apopple@nvidia.com, jhubbard@nvidia.com,
        danw@nvidia.com, kvm@vger.kernel.org, linux-kernel@vger.kernel.org
Subject: Re: [PATCH v3 1/1] vfio/nvgpu: Add vfio pci variant module for grace
 hopper
Message-ID: <ZH9p+giEs6bCYfw8@nvidia.com>
References: <20230606025320.22647-1-ankita@nvidia.com>
 <20230606083238.48ea50e9.alex.williamson@redhat.com>
 <ZH9RfXhbuED2IUgJ@nvidia.com>
 <20230606110510.0f87952c.alex.williamson@redhat.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20230606110510.0f87952c.alex.williamson@redhat.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?iJu6teF7LSK1u/m8HKIwYET7hNba3kJdyhgQ6UAiFsBNLVUgFYcnRpNb8kXW?=
 =?us-ascii?Q?7PdfokJG7A9jDeDe2fvHkQoAxPS7haXSJWFjVNvE8xBdrM/AEHnvAFeWHRT6?=
 =?us-ascii?Q?A9o7yNSnjkzjd77I5qQaZ+sDX/HOlZGgcEWptndinfTkDyBGtkPtw8FTyxmU?=
 =?us-ascii?Q?s3vTse29hYmQ4h+rCUNcdNrSkz9+zo62ZwgjAhGgoD1uF5ypWH1A1bFcyWU1?=
 =?us-ascii?Q?NTCkVxvuuZxvYlm5LICzYMJKDYA4wPn4MPCB1xTurj2vkc0lU5HxjVEwnqff?=
 =?us-ascii?Q?yrZunPS1LMbajA7KZd6TZALZ3rCRp6l2F0vbuWrNwHpYeJtwYanILqwHsKvu?=
 =?us-ascii?Q?RqrHrO5hxVJm8iine+micNzBT8pg3ih+QDczwJfVgSFnLCfX7gJiHwD/b168?=
 =?us-ascii?Q?Dzl0XnnqL02K4bn83Hriy/UnjKrnzWghai618/MvirQiru2T3S5a80LsHqYJ?=
 =?us-ascii?Q?w1RVLqooear65abgiyMBHRZKvHAhwINy8JO6yqTP2l6U92XJ3Jn7C87uNdsi?=
 =?us-ascii?Q?XCsERQk1AuzhBw7eVZPHjxMrw9+CyGCCJvHH0eI9KP5wORv/aDjFAHRtyHs9?=
 =?us-ascii?Q?Hk+FdnmTBehRq9SKxCoSq1HH6TMB9/xsQV1gnyna4FAx4OU8wzsQvbG16BsT?=
 =?us-ascii?Q?TGHaU/kX8lC6STvtxYy5mxG258UcDbykhTSXQfzmmSbXcUBfrWWG9ylniWG9?=
 =?us-ascii?Q?w0yHJ6FMUipYI/u+TG03/GwDPgwrs6JTusA4nx37pt2hw3FQanRHhv3rvH7L?=
 =?us-ascii?Q?vumKwPA0b1/klbTQqFVIM3NsTRBjeoBsdSqBnFLn4yFWT3JqwCA3JQIhvP59?=
 =?us-ascii?Q?XOnPHxsgoxpGU3p0jn2VjaSXToxnW0Eer1cb4aYmFmVMF76B3qlQ2lNAftpk?=
 =?us-ascii?Q?dcFZfjC9hAPqN3tcei5GaRouTabPDRkzytV7HeCHtEKTz51s7AEa62ZaZelT?=
 =?us-ascii?Q?7SoVfl/Y+T2UEozljT2k7RgcCaO+TzAaopbyqZecoyIOmZ/3WLchRY5ACQid?=
 =?us-ascii?Q?uwh76QXrLEyfy5D1nFlH9XWr8+1dQ6kHNxsfQNIAzF8YFWrggnljxQSqdWWF?=
 =?us-ascii?Q?V5Iul8jalMnGMMSxwqxvZq38Lbau6O/iJnpOy/jqRNkxMRYwUYqg9ZLry4ru?=
 =?us-ascii?Q?oANfqbZZOVGYjnQrwZTeDphdJm3M+okrZB9SgKGGlbC3xJP19HrrkD5/83v/?=
 =?us-ascii?Q?2Puafv7ltnF1wUpMr1MkaskIhU+94qmmakvAk+TrOuyWi6uxdjqblg2IRIEa?=
 =?us-ascii?Q?B3Z9mNjqZLfCwks4B9zAA0TDCZcXlImpsq7v+XgM/mvN/b+SF/usIeR9j2Cx?=
 =?us-ascii?Q?w8DgsaHSv+3cEN1vJyNCSQItkA1MsZbowIIdj6dVBYPTXAzJY4gXB27VE/c/?=
 =?us-ascii?Q?6FYgVgBIPprw/vmqKRvMxVqqvl8I7BzHaDHfdyZ+WMPidBUgz6U3QDjhFx4G?=
 =?us-ascii?Q?s2JKsGXEABL+lCdUx8UoOr1Y3XdvqHhag4Kou4PBvrtDvTJN3/34KesOegZF?=
 =?us-ascii?Q?Q87qMRm31W8DZVjIw0md9f8I6djTLE06/V7DEJySZ1+vHw598SK7imERRm7L?=
 =?us-ascii?Q?oJLWuIoCDz6TYPWke/PlTBbgc2H+BJ4uI6a5rSRf?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 50763ff7-91c4-4691-c49f-08db66b1ce36
X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Jun 2023 17:16:46.0436
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: sCCVvMhrRBL4X4V2XViufDOWXHuGf6vpJTWpy+ysdqggSlHpbm+v66kU/EnbpVSb
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR12MB5487
X-Spam-Status: No, score=-1.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO,
        RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE,
        T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=no autolearn_force=no
        version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Tue, Jun 06, 2023 at 11:05:10AM -0600, Alex Williamson wrote:

> It actually seems more complicated this way.  We're masquerading this
> region as a BAR, but then QEMU needs to know based on device IDs that
> it's really not a BAR, it has special size properties, mapping
> attributes, error handling, etc.  

This seems like something has gone wrong then. ie the SIGUBS error
handling stuff should be totally generic in the qemu side. Mapping
attributes are set by the kernel, qemu shouldn't know, doesn't need to
know.

The size issue is going to a be a problem in future anyhow, I expect
some new standards coming to support non-power-two sizes and they will
want to map to PCI devices in VMs still.

It seems OK to me if qemu can do this generically for any "BAR"
region, at least creating an entire "nvidia only" code path just for
non power 2 BAR sizing seems like a bad ABI choice.

> I'm not privy to a v1, the earliest I see is this (v3):
> 
> https://lore.kernel.org/all/20230405180134.16932-1-ankita@nvidia.com/
> 
> That outlines that we have a proprietary interconnect exposing cache
> coherent memory which requires use of special mapping attributes vs a
> standard PCI BAR and participates in ECC.  All of which seems like it
> would be easier to setup in QEMU if the vfio-pci representation of the
> device didn't masquerade this regions as a standard BAR.  In fact it
> also reminds me of NVlink2 coherent RAM on POWER machines that was
> similarly handled as device specific regions.  

It wasn't so good on POWER and if some of that stuff has been done
more generally we would have been further ahead here..

Jason