Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Date:   Tue, 17 Jul 2018 09:22:07 -0600
From:   Alex Williamson <alex.williamson@redhat.com>
To:     Srinath Mannam <srinath.mannam@broadcom.com>
Cc:     ray.jui@broadcom.com, vikram.prakash@broadcom.com,
        scott.branden@broadcom.com, kvm@vger.kernel.org,
        bcm-kernel-feedback-list@broadcom.com,
        linux-kernel@vger.kernel.org, <aik@ozlabs.ru>,
        David Gibson <david@gibson.dropbear.id.au>,
        <benh@kernel.crashing.org>
Subject: Re: [RFC PATCH] vfio/pci: map prefetchble bars as writecombine
Message-ID: <20180717092207.775e381c@t450s.home>
In-Reply-To: <1531457777-11825-1-git-send-email-srinath.mannam@broadcom.com>
References: <1531457777-11825-1-git-send-email-srinath.mannam@broadcom.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On Fri, 13 Jul 2018 10:26:17 +0530
Srinath Mannam <srinath.mannam@broadcom.com> wrote:

> By default all BARs map with VMA access permissions
> as pgprot_noncached.
> 
> In ARM64 pgprot_noncached is MT_DEVICE_nGnRnE which
> is strongly ordered and allows aligned access.
> This type of mapping works for NON-PREFETCHABLE bars
> containing EP controller registers.
> But it restricts PREFETCHABLE bars from doing
> unaligned access.
> 
> In CMB NVMe drives PREFETCHABLE bars are required to
> map as MT_NORMAL_NC to do unaligned access.
> 
> Signed-off-by: Srinath Mannam <srinath.mannam@broadcom.com>
> Reviewed-by: Ray Jui <ray.jui@broadcom.com>
> Reviewed-by: Vikram Prakash <vikram.prakash@broadcom.com>
> ---

This has been discussed before:

https://www.spinics.net/lists/kvm/msg156548.html

CC'ing the usual suspects from the previous thread.  I'm not convinced
that the patch here has considered anything other than the ARM64
implications and it's not clear that it considers compatibility with
existing users or devices at all.  Can we guarantee for all devices and
use cases that WC is semantically equivalent and preferable to UC?  If
not then we need to device an extension to the interface that allows
the user to specify WC.  Thanks,

Alex

>  drivers/vfio/pci/vfio_pci.c | 5 ++++-
>  1 file changed, 4 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/vfio/pci/vfio_pci.c b/drivers/vfio/pci/vfio_pci.c
> index b423a30..eff6b65 100644
> --- a/drivers/vfio/pci/vfio_pci.c
> +++ b/drivers/vfio/pci/vfio_pci.c
> @@ -1142,7 +1142,10 @@ static int vfio_pci_mmap(void *device_data, struct vm_area_struct *vma)
>  	}
>  
>  	vma->vm_private_data = vdev;
> -	vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);
> +	if (pci_resource_flags(pdev, index) & IORESOURCE_PREFETCH)
> +		vma->vm_page_prot = pgprot_writecombine(vma->vm_page_prot);
> +	else
> +		vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);
>  	vma->vm_pgoff = (pci_resource_start(pdev, index) >> PAGE_SHIFT) + pgoff;
>  
>  	return remap_pfn_range(vma, vma->vm_start, vma->vm_pgoff,