Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <2838d716b08c78ed24fdd3fe392e21222ee70067.camel@kernel.crashing.org>
Subject: VFIO (PCI) and write combine mapping of BARs
From:   Benjamin Herrenschmidt <benh@kernel.crashing.org>
To:     kvm@vger.kernel.org
Cc:     linux-kernel@vger.kernel.org, alex.williamson@redhat.com,
        osamaabb@amazon.com, linux-pci@vger.kernel.org,
        Clint Sbisa <csbisa@amazon.com>
Date:   Fri, 14 Jul 2023 12:32:49 +1000
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 8BIT
User-Agent: Evolution 3.44.4-0ubuntu1 
MIME-Version: 1.0
Precedence: bulk

Hi Folks !

I'd like to revive an old discussion as we (Amazon Linux) have been
getting asks for it.

What's the best interface to provide the option of write combine mmap's
of BARs via VFIO ?

The problem isn't so much the low level implementation, we just have to
play with the pgprot, the question is more around what API to present
to control this.

One trivial way would be to have an ioctl to set a flag for a given
region/BAR that cause subsequent mmap's to use write-combine. We would
have to keep a bitmap for the "legacy" regions, and use a flag in
struct vfio_pci_region for the others.

One potentially better way is to make it strictly an attribute of
vfio_pci_region, along with an ioctl that creates a "subregion". The
idea here is that we would have an ioctl to create a region from an
existing region dynamically, which represents a subset of the original
region (typically a BAR), with potentially different attributes (or we
keep the attribute get/set separate).

I like the latter more because it will allow to more easily define that
portions of a BAR can need different attributes without causing
state/race issues between setting the attribute and mmap.

This will also enable other attributes than write-combine if/when the
need arises.

Any better idea ? thoughs ? objections ?

This is still quite specific to PCI, but so is the entire regions
mechanism, so I don't see an easy path to something more generic at
this stage.

Cheers,
Ben.