Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:6 as permitted sender) client-ip=2620:137:e000::3:6;
Date:   Tue, 5 Dec 2023 09:52:33 +0800
From:   Yan Zhao <yan.y.zhao@intel.com>
To:     Jason Gunthorpe <jgg@nvidia.com>
CC:     <iommu@lists.linux.dev>, <kvm@vger.kernel.org>,
        <linux-kernel@vger.kernel.org>, <alex.williamson@redhat.com>,
        <pbonzini@redhat.com>, <seanjc@google.com>, <joro@8bytes.org>,
        <will@kernel.org>, <robin.murphy@arm.com>, <kevin.tian@intel.com>,
        <baolu.lu@linux.intel.com>, <dwmw2@infradead.org>,
        <yi.l.liu@intel.com>
Subject: Re: [RFC PATCH 00/42] Sharing KVM TDP to IOMMU
Message-ID: <ZW6CYYKU7F7wBNj+@yzhao56-desk.sh.intel.com>
Reply-To: Yan Zhao <yan.y.zhao@intel.com>
References: <20231202091211.13376-1-yan.y.zhao@intel.com>
 <20231204150800.GD1493156@nvidia.com>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <20231204150800.GD1493156@nvidia.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?AQWTuKgLFbzo8o1FqI50Yv+K2HSrnJjaX9W52IZJXM6ITdCnviVbVHC3ucPP?=
 =?us-ascii?Q?zNaNt5VgfwVnESpTcUS27Nh0015pfJqGDqJyWE+ay3xHKazOPP18o50OdhDB?=
 =?us-ascii?Q?aOrKGmGYmQXhzO6uZVEYjL4lm0xz731SeEcl62Ua+pEkp6x23F9r6TM1zdTa?=
 =?us-ascii?Q?gpVf7fRiYN2iwy5bA7u7axdvtqFhmh7cHBrZXcKt4+v8bfHLl9pRRlbYNiJp?=
 =?us-ascii?Q?tvXO259zAkCZXRTy7H2l4NGN7gVoKLpJueTlzegWRoyce+BvIh8jllCCX3HO?=
 =?us-ascii?Q?uwDWeDQJ/4o7gKjYFulMdDarHudJ4wF2uI/e0WWROqxS+URyBW4UUJ8n+4Mx?=
 =?us-ascii?Q?NrFeDsJxyt7gqvvyl5Gd2cS9Zr8NrpcC84o3EGQJbCxmAweAURu91md3BIZA?=
 =?us-ascii?Q?UQ1MonTY1Ev8TxrPWaX+vYVN4N8qoY2gmiE8sRA99xaZWQseE0nxsr6TEgyN?=
 =?us-ascii?Q?O6fGCRuhyZPF3gEzf7T+lfeK2kUoNriMg223/VG3rWptqDoS93VOo04KDQKm?=
 =?us-ascii?Q?ovgK6P6huJkX3HrgeRLgZgk3HbBtQL8aocEZSCVDA7J8NASbgRpkv93UoqFm?=
 =?us-ascii?Q?j8NItB/RsLLvz+k4rgSXboDk6L8CNwlDheJOhMudKk6tsYCbHv4cENnDFVD8?=
 =?us-ascii?Q?OA4ua3ldfdYiK76fRa6El0VyFrXO0rmrWI5134fWtJ+w9nrAfnTMoFVXlGjv?=
 =?us-ascii?Q?p+OT4+Lyf4/LOyXpo99f4MXr8RqmcX43keKaKyEMIDQPvBDHyMb8G3ihjQgY?=
 =?us-ascii?Q?1WNOrueRGDHfdgPJa/q7o0bYA0fdOjgf0iYmGCW659CVCK05rm+fqwOy9e5x?=
 =?us-ascii?Q?M8xXdqQZ1sGDCf3tp5EcnzHHK8QmkDWhCxJLxGyWHjy5RiBFekwT0+KHODs+?=
 =?us-ascii?Q?7FTCgAyWMOHr/TlLeaACfqyMl+vAdLHlYOZ7U6Ib5hDIWuUvJEzQz0ZwzFF1?=
 =?us-ascii?Q?h91Apb3GyxTgInzesIyKvsIT8MIZZlb2A+8sQqErTwOckrPzqhAT40niEr5p?=
 =?us-ascii?Q?FJrt/zZIiiX/F3WSiXZM91N51MqRJvm6cEhucakydI829AdmEIHCersmwKOO?=
 =?us-ascii?Q?KKEUGj674cQEVHt2QQMXMUV78CNA8jVqNLMnudJULYu2xq/60Z4hI9FxtEwq?=
 =?us-ascii?Q?mG+WskwoDNc3n6dsFmBvexx06x6LXcBs5sWYWuC834JWLXfJ5RdLV3txhJeN?=
 =?us-ascii?Q?W8nyS6M/3rZz1mzLevD/nk6HLUGfmGdVidWH+TSyJ7Oly69U8ASwYS+OK1lX?=
 =?us-ascii?Q?6UTzUewmhuhirF9oJGmRdMSgAqjEHdmxRYWGwchYhE2aMEJiMq16yGaf0yk6?=
 =?us-ascii?Q?5HL6vUrNdF/xIsKcqGFowfv1hAo3RV+yVtL8/fAWpn1Lfa+xq1yZkbTcoWtK?=
 =?us-ascii?Q?J+GLmME8oZq1F3k7CxPSwmA87l4gDS65W8Imre2Xj2R8lOwQib68j4b0oCpy?=
 =?us-ascii?Q?fMan8nIREijr2WV32/NMDdwp4CrEOKCQuIfqW6/E7bk90wNnDN/QtHoD/ENW?=
 =?us-ascii?Q?3y7oo1UzXL7SycOjKbe7pWmP2BBR26YYbJbBySSa0fFZHQWtHBOdWCHxBFOl?=
 =?us-ascii?Q?GwHM35fvmh6UYDXKj1WMXwyWag2fpHaLYVsyNhKt?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 031711a7-1cdf-44f8-bb5b-08dbf538e7a3
X-MS-Exchange-CrossTenant-AuthSource: DS7PR11MB5966.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Dec 2023 02:21:35.7856
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: QB10IGPXDp22D0URuhPyJg/CK03jub47blrHJ8WCkax/X15J/P8Anam5hs96+UyEvIz8OmVZgCVPoKxfk71PSg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR11MB7213
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE
	autolearn=unavailable autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on pete.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (pete.vger.email [0.0.0.0]); Mon, 04 Dec 2023 18:21:49 -0800 (PST)

On Mon, Dec 04, 2023 at 11:08:00AM -0400, Jason Gunthorpe wrote:
> On Sat, Dec 02, 2023 at 05:12:11PM +0800, Yan Zhao wrote:
> > In this series, term "exported" is used in place of "shared" to avoid
> > confusion with terminology "shared EPT" in TDX.
> > 
> > The framework contains 3 main objects:
> > 
> > "KVM TDP FD" object - The interface of KVM to export TDP page tables.
> >                       With this object, KVM allows external components to
> >                       access a TDP page table exported by KVM.
> 
> I don't know much about the internals of kvm, but why have this extra
> user visible piece? Isn't there only one "TDP" per kvm fd? Why not
> just use the KVM FD as a handle for the TDP?
As explained in a parallel mail, the reason to introduce KVM TDP FD is to let
KVM know which TDP the user wants to export(share).
And another reason is wrap the exported TDP with its exported ops in a
single structure. So, components outside of KVM can query meta data and
request page fault, register invalidate callback through the exported ops. 

struct kvm_tdp_fd {
        /* Public */
        struct file *file;
        const struct kvm_exported_tdp_ops *ops;

        /* private to KVM */
        struct kvm_exported_tdp *priv;
};
For KVM, it only needs to expose this struct kvm_tdp_fd and two symbols
kvm_tdp_fd_get() and kvm_tdp_fd_put().


> 
> > "IOMMUFD KVM HWPT" object - A proxy connecting KVM TDP FD to IOMMU driver.
> >                             This HWPT has no IOAS associated.
> > 
> > "KVM domain" in IOMMU driver - Stage 2 domain in IOMMU driver whose paging
> >                                structures are managed by KVM.
> >                                Its hardware TLB invalidation requests are
> >                                notified from KVM via IOMMUFD KVM HWPT
> >                                object.
> 
> This seems broadly the right direction
> 
> > - About device which partially supports IOPF
> > 
> >   Many devices claiming PCIe PRS capability actually only tolerate IOPF in
> >   certain paths (e.g. DMA paths for SVM applications, but not for non-SVM
> >   applications or driver data such as ring descriptors). But the PRS
> >   capability doesn't include a bit to tell whether a device 100% tolerates
> >   IOPF in all DMA paths.
> 
> The lack of tolerance for truely DMA pinned guest memory is a
> significant problem for any real deployment, IMHO. I am aware of no
> device that can handle PRI on every single DMA path. :(
DSA actaully can handle PRI on all DMA paths. But it requires driver to turn on
this capability :(

> >   A simple way is to track an allowed list of devices which are known 100%
> >   IOPF-friendly in VFIO. Another option is to extend PCIe spec to allow
> >   device reporting whether it fully or partially supports IOPF in the PRS
> >   capability.
> 
> I think we need something like this.
> 
> > - How to map MSI page on arm platform demands discussions.
> 
> Yes, the recurring problem :(
> 
> Probably the same approach as nesting would work for a hack - map the
> ITS page into the fixed reserved slot and tell the guest not to touch
> it and to identity map it.
Ok.