Received-SPF: pass (google.com: domain of linux-kernel+bounces-168099-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) client-ip=2604:1380:40f1:3f00::1;
Message-ID: <8c28a1d5-ac84-445b-80e6-a705e6d7ff1b@linux.ibm.com>
Date: Sat, 4 May 2024 00:33:53 +0530
Precedence: bulk
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
From: Shivaprasad G Bhat <sbhat@linux.ibm.com>
Subject: Re: [RFC PATCH v2 0/6] powerpc: pSeries: vfio: iommu: Re-enable
 support for SPAPR TCE VFIO
To: Alexey Kardashevskiy <aik@amd.com>, Jason Gunthorpe <jgg@ziepe.ca>
Cc: mpe@ellerman.id.au, tpearson@raptorengineering.com,
        alex.williamson@redhat.com, linuxppc-dev@lists.ozlabs.org,
        npiggin@gmail.com, christophe.leroy@csgroup.eu,
        aneesh.kumar@kernel.org, naveen.n.rao@linux.ibm.com,
        gbatra@linux.vnet.ibm.com, brking@linux.vnet.ibm.com, aik@ozlabs.ru,
        ruscur@russell.cc, robh@kernel.org, linux-kernel@vger.kernel.org,
        joel@jms.id.au, kvm@vger.kernel.org, msuchanek@suse.de,
        oohall@gmail.com, mahesh@linux.ibm.com, jroedel@suse.de,
        vaibhav@linux.ibm.com, svaidy@linux.ibm.com
References: <171450753489.10851.3056035705169121613.stgit@linux.ibm.com>
 <20240501140942.GA1723318@ziepe.ca>
 <703f15b0-d895-4518-9886-0827a6c4e769@amd.com>
Content-Language: en-US
In-Reply-To: <703f15b0-d895-4518-9886-0827a6c4e769@amd.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit

On 5/2/24 06:59, Alexey Kardashevskiy wrote:
>
>
> On 2/5/24 00:09, Jason Gunthorpe wrote:
>> On Tue, Apr 30, 2024 at 03:05:34PM -0500, Shivaprasad G Bhat wrote:
>>> RFC v1 was posted here [1]. As I was testing more and fixing the
>>> issues, I realized its clean to have the table_group_ops implemented
>>> the way it is done on PowerNV and stop 'borrowing' the DMA windows
>>> for pSeries.
>>>
>>> This patch-set implements the iommu table_group_ops for pSeries for
>>> VFIO SPAPR TCE sub-driver thereby enabling the VFIO support on POWER
>>> pSeries machines.
>>
>> Wait, did they previously not have any support?
> >
>> Again, this TCE stuff needs to go away, not grow. I can grudgingly
>> accept fixing it where it used to work, but not enabling more HW that
>> never worked before! :(
>
>
> This used to work when I tried last time 2+ years ago, not a new 
> stuff. Thanks,
>
Thanks Alexey for pitching in.


Hi Jason,


As Alexey implied, this used to work in the past.


The support for pSeries VFIO exists for a long time, and the support
for VFIO_SPAPR_TCE_v2_IOMMU also was added with
9d67c9433509 ("powerpc/iommu: Add "borrowing" iommu_table_group_ops")


The commit 090bad39b237a ("powerpc/powernv: Add indirect levels to 
it_userspace")
broke the userspace view for pSeries, which the Patch 6 here tries to
bring back.


We found more issues with 9d67c9433509 and I felt its
better to stop "borrowing" the DMA windows as that would be
cleaner which is what is done in Patch 6.


In this process we discovered few bugs in upstream as well, which
we have been trying to fix and have posted few of fixes earlier like,
d2d00e15808 powerpc: iommu: Bring back table group release_ownership() call
83b3836bf83 iommu: Allow ops->default_domain to work when !CONFIG_IOMMU_DMA


So, this patch series tries to fix some more issues(patch 2, 4, 6)
coupled with some code refactoring(1, 3, 5 & 6) to stop "borrowing"
DMA windows.


We have legacy workloads using VFIO in userspace/kvm guests running
on downstream distro kernels. We want these workloads to be able to
continue running on our arch.


Going forward we are planning to have the IOMMUFD support for PPC64,
I firmly believe the refactoring in this patch series is a step in
that direction.


Thanks,
Shivaprasad