Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
MIME-Version: 1.0
In-Reply-To: <20180717092207.775e381c@t450s.home>
References: <1531457777-11825-1-git-send-email-srinath.mannam@broadcom.com> <20180717092207.775e381c@t450s.home>
From:   Srinath Mannam <srinath.mannam@broadcom.com>
Date:   Thu, 19 Jul 2018 00:05:18 +0530
Message-ID: <CABe79T7eVS5Z5+CkNRfWik=jFeejq54xheUOS=Jc9oU6qX8=kA@mail.gmail.com>
Subject: Re: [RFC PATCH] vfio/pci: map prefetchble bars as writecombine
To:     Alex Williamson <alex.williamson@redhat.com>
Cc:     Ray Jui <ray.jui@broadcom.com>,
        Vikram Prakash <vikram.prakash@broadcom.com>,
        Scott Branden <scott.branden@broadcom.com>,
        kvm@vger.kernel.org,
        BCM Kernel Feedback <bcm-kernel-feedback-list@broadcom.com>,
        Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
        aik@ozlabs.ru, David Gibson <david@gibson.dropbear.id.au>,
        benh@kernel.crashing.org
Content-Type: text/plain; charset="UTF-8"
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

Hi Alex,

On Tue, Jul 17, 2018 at 8:52 PM, Alex Williamson
<alex.williamson@redhat.com> wrote:
> On Fri, 13 Jul 2018 10:26:17 +0530
> Srinath Mannam <srinath.mannam@broadcom.com> wrote:
>
>> By default all BARs map with VMA access permissions
>> as pgprot_noncached.
>>
>> In ARM64 pgprot_noncached is MT_DEVICE_nGnRnE which
>> is strongly ordered and allows aligned access.
>> This type of mapping works for NON-PREFETCHABLE bars
>> containing EP controller registers.
>> But it restricts PREFETCHABLE bars from doing
>> unaligned access.
>>
>> In CMB NVMe drives PREFETCHABLE bars are required to
>> map as MT_NORMAL_NC to do unaligned access.
>>
>> Signed-off-by: Srinath Mannam <srinath.mannam@broadcom.com>
>> Reviewed-by: Ray Jui <ray.jui@broadcom.com>
>> Reviewed-by: Vikram Prakash <vikram.prakash@broadcom.com>
>> ---
>
> This has been discussed before:
>
> https://www.spinics.net/lists/kvm/msg156548.html
Thank you for inputs.. I have gone through the long list of mail chain
discussion.
>
> CC'ing the usual suspects from the previous thread.  I'm not convinced
> that the patch here has considered anything other than the ARM64
> implications and it's not clear that it considers compatibility with
> existing users or devices at all.  Can we guarantee for all devices and
> use cases that WC is semantically equivalent and preferable to UC?  If
> not then we need to device an extension to the interface that allows
> the user to specify WC.  Thanks,
>
To implement with user specified WC flags, many changes need to be done.
Suppose In DPDK, prefetcable BARs map using WC flag, then also same
question comes
that WC may be different for different CPUs.
As per functionality, both WC and PREFETCHABLE are same, like merging writes and
typically WC is uncached.
So, based on prefetchable BARs behavior and usage we need to map bar memory.
Is it right to map prefetchable BARs as strongly ordered, aligned
access and uncached?

Please correct me if my understanding is wrong.

Regards,
Srinath.

> Alex
>
>>  drivers/vfio/pci/vfio_pci.c | 5 ++++-
>>  1 file changed, 4 insertions(+), 1 deletion(-)
>>
>> diff --git a/drivers/vfio/pci/vfio_pci.c b/drivers/vfio/pci/vfio_pci.c
>> index b423a30..eff6b65 100644
>> --- a/drivers/vfio/pci/vfio_pci.c
>> +++ b/drivers/vfio/pci/vfio_pci.c
>> @@ -1142,7 +1142,10 @@ static int vfio_pci_mmap(void *device_data, struct vm_area_struct *vma)
>>       }
>>
>>       vma->vm_private_data = vdev;
>> -     vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);
>> +     if (pci_resource_flags(pdev, index) & IORESOURCE_PREFETCH)
>> +             vma->vm_page_prot = pgprot_writecombine(vma->vm_page_prot);
>> +     else
>> +             vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);
>>       vma->vm_pgoff = (pci_resource_start(pdev, index) >> PAGE_SHIFT) + pgoff;
>>
>>       return remap_pfn_range(vma, vma->vm_start, vma->vm_pgoff,
>