Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Fri, 14 Jul 2023 09:37:48 -0300
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     Lorenzo Pieralisi <lpieralisi@kernel.org>
Cc:     Benjamin Herrenschmidt <benh@kernel.crashing.org>,
        kvm@vger.kernel.org, linux-kernel@vger.kernel.org,
        alex.williamson@redhat.com, osamaabb@amazon.com,
        linux-pci@vger.kernel.org, Clint Sbisa <csbisa@amazon.com>,
        catalin.marinas@arm.com, maz@kernel.org
Subject: Re: VFIO (PCI) and write combine mapping of BARs
Message-ID: <ZLFBnACjoTbDmKuU@nvidia.com>
References: <2838d716b08c78ed24fdd3fe392e21222ee70067.camel@kernel.crashing.org>
 <ZLD1l1274hQQ54RT@lpieralisi>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <ZLD1l1274hQQ54RT@lpieralisi>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?UJnx4m/SMDCc0VfdfI8gGDBNX9cJArlmNM0qv1PGXt2Of1hFm6Yr9oUYJAL7?=
 =?us-ascii?Q?hdr2mSKTT1nfjeANHKZCzQt8qVayvCnUDGxNZdv4Zx3/4mLWyVRIeqFSnaay?=
 =?us-ascii?Q?ngh2j53bkzwYln6X7ETTQUjSzaGSqVcYnlMXwlUHH4FI1msuujgAl3pAZBjJ?=
 =?us-ascii?Q?Hkig1LVoZ/4ViLJkYWS6kBhvqLdxNl4pRkjKTrvzpRGjKncmqxRwOX0V+8Nn?=
 =?us-ascii?Q?gYiyy+24yQ3IzVrdDbCXBSY01voK9e/Lbvqlly1NsRYOal4pRuM7Bb4KhLr8?=
 =?us-ascii?Q?W8mucMrRwjVQVWIO7o1GnTH5j0cI/yVV4dfVPBetKETvjZqFoe1591IenmJq?=
 =?us-ascii?Q?CgBAcC+AAGidgwDAypvCJHf7sONRhFM2wFbN1tU5ChVSLxzwHMO5RGDsjNy0?=
 =?us-ascii?Q?hhDYXz2hVlg7vtI/SRMtr5wGJV/+DFG9vBR8m6605CCbBgXIqOYWyqRZqHXB?=
 =?us-ascii?Q?5gqs9NbxT6ElP2luJspbs9zvlwmHmzPqMqkT3l9M1u1O4z1IRuVA8g9rln25?=
 =?us-ascii?Q?FfChtRwixcd0XoVRtUq+s0dOlzanVq9+dYJmEYf/UCaiG0iUV/tGogm3C0sI?=
 =?us-ascii?Q?m2oVr8BfGmk9QfYHoMZgi30QwzFYpsmb2d9ym8F9+IlHBHksyWbAW+1hmVWN?=
 =?us-ascii?Q?ayyQNDRrLykzY1mlMQkrikJelT2ueNSGtmVwM1Z61iEIbCyvRPxDs3wlJitT?=
 =?us-ascii?Q?BOkIjByE34DKP/IyMnZDeiS1LoybN9jEiENMpyDb+2eBfTaeunWOThMeGXkF?=
 =?us-ascii?Q?Jw9inZBn7rFOJ1hdUnsJtvwdzF22BMWTFbvra331yWpED7YP36AeRm3X36Wa?=
 =?us-ascii?Q?G7cEl4rAD7kU/f3PaSY01Azsgr6MGxJF0Wl51x+lpTPDSU/RdeoAt3uRKlwq?=
 =?us-ascii?Q?zs0G/QXjYVX2Lx5m7bgZcVuqst0PH/m+Kj4jmVZBL2xQDn5COKM6Amec+jE5?=
 =?us-ascii?Q?TzSzcjZasqq2qIHscPNpk2sR7AAED1y2pJ18gBF66ZCJYTjqQnUgih5N41S4?=
 =?us-ascii?Q?0KBCEgsBnFCQ//B20AmloeAjomSsOBpwHEFFML7OCi4geV8KN61aRQ7FvS7/?=
 =?us-ascii?Q?pkM9wsGMIG9ylEetRBdQK2BeMhcnHcyFN6JsxbG8MH8JvxXcj8vuApwVE9Ol?=
 =?us-ascii?Q?J+hJz8dqPiua8GH8axOy0tYjSiuHLTgmFCx/NmtE5fVM1OzE2UQZT2+bdUoZ?=
 =?us-ascii?Q?zRLHy62RsGf/Y7aJbxkAl6Ow4D/4EmVA/5TdDGY+uqb8emOq4tKXFGCcGZgI?=
 =?us-ascii?Q?lTf9DooRT8TpeYSA7jfzeUFW4o9x6nmPQn3mP1xedHtTDNXwuqlYRFUNF/jq?=
 =?us-ascii?Q?ivTVVoSxnJCzSjKOjyuiAYEiiXwd3nvf+hrIUYQfLQEAFZ/84bAkq2paZa4g?=
 =?us-ascii?Q?PlgR+OjVarfYGrGIjSbX3vAyn7mZMFvtjLrzC9azZRvUU1dpVZ6TU5gQQkUO?=
 =?us-ascii?Q?PklTn+KjpmgEiX+82FjuBMeFeIMklbKF6j6sVdrMFll2yHciIkoks7sQwo6H?=
 =?us-ascii?Q?t+UvcBsk6jvE7gsfgUVVWo+o+l3TQTnYeeV5kXny8p8c0YLibnwHUolrHQLJ?=
 =?us-ascii?Q?riD/oVuXhrqSKWLlvQ2Cymazy1rUkUqIr7cMGYrT?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 01bab31c-5bf1-4bb3-7bf9-08db84672315
X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Jul 2023 12:37:51.0153
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: O3L10blXGiLrolaf3WAJCaERUZ4HyiBAwvi+v8YRm5gJePDWs1nnZccNJWnc2Gz6
X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR12MB7615
X-Spam-Status: No, score=-1.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO,
        RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE,
        T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=no autolearn_force=no
        version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Fri, Jul 14, 2023 at 09:13:27AM +0200, Lorenzo Pieralisi wrote:
> [+Catalin, Marc, Jason]
> 
> On Fri, Jul 14, 2023 at 12:32:49PM +1000, Benjamin Herrenschmidt wrote:
> > Hi Folks !
> > 
> > I'd like to revive an old discussion as we (Amazon Linux) have been
> > getting asks for it.
> > 
> > What's the best interface to provide the option of write combine mmap's
> > of BARs via VFIO ?
> 
> There is an ongoing thread on this topic that we should use to
> bring this discussion to completion:
> 
> https://lore.kernel.org/linux-arm-kernel/ZHcxHbCb439I1Uk2@arm.com

There are two topics here

1) Make ARM KVM allow the VM to select WC for its MMIO. This has
   evolved in a way that is not related to VFIO

2) Allow VFIO to create mmaps with WC for non-VM use cases like DPDK.

We have a draft patch for #1, and I think a general understanding with
ARM folks that this is the right direction.

2 is more like what this email talks about - providing mmaps with
specific flags.

Benjamin, which are you interested in?

> > The problem isn't so much the low level implementation, we just have to
> > play with the pgprot, the question is more around what API to present
> > to control this.

Assuming this is for #2, I think VFIO has fallen into a bit of a trap
by allowing userspace to form the mmap offset. I've seen this happen
in other subsystems too. It seems like a good idea then you realize
you need more stuff in the mmap space and become sad.

Typically the way out is to covert the mmap offset into a cookie where
userspace issues some ioctl and then the ioctl returns an opaque mmap
offset to use.

eg in the vfio context you'd do some 'prepare region for mmap' ioctl
where you could specify flags. The kernel would encode the flags in
the cookie and then mmap would do the right thing. Adding more stuff
is done by enhancing the prepare ioctl.

Legacy mmap offsets are kept working.

> > This is still quite specific to PCI, but so is the entire regions
> > mechanism, so I don't see an easy path to something more generic at
> > this stage.

Regions are general, but the encoding of the mmap cookie has various
PCI semantics when used with the PCI interface..

We'd want the same ability with platform devices too, for instance.

Jason