Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:18 as permitted sender) client-ip=2620:137:e000::1:18;
Message-ID: <920cf807-efcd-319b-560a-ed3a1c6fdc85@linux.ibm.com>
Date:   Tue, 8 Feb 2022 16:37:55 -0500
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101
 Thunderbird/91.5.0
Subject: Re: [PATCH v3 24/30] vfio-pci/zdev: wire up group notifier
Content-Language: en-US
To:     Jason Gunthorpe <jgg@nvidia.com>
Cc:     Alex Williamson <alex.williamson@redhat.com>,
        linux-s390@vger.kernel.org, cohuck@redhat.com,
        schnelle@linux.ibm.com, farman@linux.ibm.com, pmorel@linux.ibm.com,
        borntraeger@linux.ibm.com, hca@linux.ibm.com, gor@linux.ibm.com,
        gerald.schaefer@linux.ibm.com, agordeev@linux.ibm.com,
        frankja@linux.ibm.com, david@redhat.com, imbrenda@linux.ibm.com,
        vneethv@linux.ibm.com, oberpar@linux.ibm.com, freude@linux.ibm.com,
        thuth@redhat.com, pasic@linux.ibm.com, kvm@vger.kernel.org,
        linux-kernel@vger.kernel.org
References: <20220204211536.321475-1-mjrosato@linux.ibm.com>
 <20220204211536.321475-25-mjrosato@linux.ibm.com>
 <20220208104319.4861fb22.alex.williamson@redhat.com>
 <20220208185141.GH4160@nvidia.com>
 <20220208122624.43ad52ef.alex.williamson@redhat.com>
 <438d8b1e-e149-35f1-a8c9-ed338eb97430@linux.ibm.com>
 <20220208204041.GK4160@nvidia.com>
From:   Matthew Rosato <mjrosato@linux.ibm.com>
In-Reply-To: <20220208204041.GK4160@nvidia.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk

On 2/8/22 3:40 PM, Jason Gunthorpe wrote:
> On Tue, Feb 08, 2022 at 03:33:58PM -0500, Matthew Rosato wrote:
> 
>>> Is the purpose of IOAT to associate the device to a set of KVM page
>>> tables?  That seems like a container or future iommufd operation.  I
>>
>> Yes, here we are establishing a relationship with the DMA table in the guest
>> so that once mappings are established guest PCI operations (handled via
>> special instructions in s390) don't need to go through the host but can be
>> directly handled by firmware (so, effectively guest can keep running on its
>> vcpu vs breaking out).
> 
> Oh, well, certainly sounds like a NAK on that - anything to do with
> the DMA translation of a PCI device must go through the iommu layer,
> not here.
> 
> Lets not repeat the iommu subsytem bypass mess power made please.
> 
>> It's more that non-KVM userspace doesn't care about what these ioctls are
>> doing...  The enabling of 'interp, aif, ioat' is only pertinent when there
>> is a KVM userspace, specifically because the information being shared /
>> actions being performed as a result are only relevant to properly enabling
>> zPCI features when the zPCI device is being passed through to a VM
>> guest.
> 
> Then why are they KVM ioctls?

Well, the primary reason I ended up here was that I need to ensure the 
the operation is only performed when guest X owns host zPCI device Y. 
The vfio-pci device ioctl had the benefit of acting on device 
granularity + also already being aware of the host PCI (and thus zPCI) 
device association -- so I already know exactly what hostdev is being 
referenced for the operation.  All that was needed was the KVM notifier 
to ensure the vfio device was associated to a KVM guest.

I think moving over to a KVM ioctl is doable; I might still need to rely 
on VFIO_GROUP_NOTIFY_SET_KVM though, not sure yet.

Based on prior comments in this thread I'm assuming Alex shares that 
view too (don't use vfio device ioctl for something only being used for 
VM passthrough) so I'll start looking at using KVM ioctls instead.