Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932263AbcJZPUH (ORCPT ); Wed, 26 Oct 2016 11:20:07 -0400 Received: from mx1.redhat.com ([209.132.183.28]:36746 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753631AbcJZPUE (ORCPT ); Wed, 26 Oct 2016 11:20:04 -0400 Date: Wed, 26 Oct 2016 09:19:38 -0600 From: Alex Williamson To: "Tian, Kevin" Cc: Kirti Wankhede , "pbonzini@redhat.com" , "kraxel@redhat.com" , "cjia@nvidia.com" , "qemu-devel@nongnu.org" , "kvm@vger.kernel.org" , "Song, Jike" , "bjsdjshi@linux.vnet.ibm.com" , "linux-kernel@vger.kernel.org" Subject: Re: [PATCH v9 04/12] vfio iommu: Add support for mediated devices Message-ID: <20161026091938.399fb73e@t450s.home> In-Reply-To: References: <1476739332-4911-1-git-send-email-kwankhede@nvidia.com> <1476739332-4911-5-git-send-email-kwankhede@nvidia.com> <20161019150231.19e32b84@t450s.home> MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.32]); Wed, 26 Oct 2016 15:19:39 +0000 (UTC) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 2817 Lines: 64 On Wed, 26 Oct 2016 07:54:56 +0000 "Tian, Kevin" wrote: > > From: Tian, Kevin > > Sent: Wednesday, October 26, 2016 3:54 PM > > > > > From: Alex Williamson [mailto:alex.williamson@redhat.com] > > > Sent: Thursday, October 20, 2016 5:03 AM > > > > @@ -83,6 +92,21 @@ struct vfio_group { > > > > }; > > > > > > > > /* > > > > + * Guest RAM pinning working set or DMA target > > > > + */ > > > > +struct vfio_pfn { > > > > + struct rb_node node; > > > > + unsigned long vaddr; /* virtual addr */ > > > > + dma_addr_t iova; /* IOVA */ > > > > + unsigned long pfn; /* Host pfn */ > > > > + int prot; > > > > + atomic_t ref_count; > > > > +}; > > > > > > Somehow we're going to need to fit an invalidation callback here too. > > > How would we handle a case where there are multiple mdev devices, from > > > different vendor drivers, that all have the same pfn pinned? I'm > > > already concerned about the per pfn overhead we're introducing here so > > > clearly we cannot store an invalidation callback per pinned page, per > > > vendor driver. Perhaps invalidations should be done using a notifier > > > chain per vfio_iommu, the vendor drivers are required to register on > > > that chain (fail pinning with empty notifier list) user unmapping > > > will be broadcast to the notifier chain, the vendor driver will be > > > responsible for deciding if each unmap is relevant to them (potentially > > > it's for a pinning from another driver). > > > > > > I expect we also need to enforce that vendors perform a synchronous > > > unmap such that after returning from the notifier list call, the > > > vfio_pfn should no longer exist. If it does we might need to BUG_ON. > > > Also be careful to pay attention to the locking of the notifier vs > > > unpin callbacks to avoid deadlocks. > > > > > > > What about just requesting vendor driver to provide a callback in parent > > device ops? > > > > Curious in which scenario the user application (say Qemu here) may > > unmap memory pages which are still pinned by vendor driver... Is it > > purely about a corner case which we want to handle elegantly? > > > > If yes, possibly a simpler way is to force destroying mdev instead of > > asking vendor driver to take care of each invalidation request under > > such situation. Since anyway the mdev device won't be in an usable > > state anymore... (sorry if I missed the key problem here.) > > > > or calling reset callback of parent device driver, if we don't want to > break libvirt's expectation by blindly removing mdev device... I think we're going off into the weeds here. mdev devices need to honor the existing API, therefore an unmap should result in preventing the device from further access to the unmapped pages, nothing more, nothing less. Thanks, Alex