Received: by 2002:a05:6a10:17d3:0:0:0:0 with SMTP id hz19csp415924pxb; Fri, 16 Apr 2021 08:42:30 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxSwxHbtTZAXMQVSLxtF/kfOopNk+z3q7rcwZgb1QwzR4xHHF/ecZd90Mo2cPk4ZplGMjzg X-Received: by 2002:a17:90b:147:: with SMTP id em7mr8964763pjb.138.1618587750317; Fri, 16 Apr 2021 08:42:30 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1618587750; cv=none; d=google.com; s=arc-20160816; b=ovfqrX54oeEy+wuwgoWGeuUzQAvmeGoO7oRgxawH+FK9GPPABUwrw21514Sb4eenzD +/IWpOYvbqBEa3De7l+O3sEg+mrHmY79w8Yo8TiJaKV1U9VVfxAKFKd5kzxCr0eNNOt9 9AnU7tRgUh/WrKnk8/EjNLpds8ngeqkKmMiafmXpUkJJZtNstPzhLKsOSrfPy82GwyEW JakUO5MItORhcyo0bnGwRA9c8MDW3pVpREZfBjZmRmgQnD14F7eCo6hlQ4qDTDcHWXCQ yiLRjbiMeCqT86rQrmlOCcU6dcA5lNzC1dZxO7PoOEy3ss4gVfPuMBQQN9e5WhrVH7yO xiyw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:content-language :in-reply-to:mime-version:user-agent:date:message-id:from:references :cc:to:subject:dkim-signature; bh=ORKl8cG3pXQFZT6dMTWhjbu3j7T6kEqKkWZLykCofkk=; b=Purt4x+S1vjWzakLPfdA6OipjlC44VbaUay5OpB4+nUZh8f2xvWT7gTXN5MuNOMxzq cA7qM6x7MviCVpYo1ko48T9/AdAmUIXRJD1c6MYUHi2QZNtOXw7FUN+BEB4Kqoi22/WN h8Zk+ZTuna5RoYcFMYefXep1Lqk9jhg9v3Lbk/3YuDLPcexfXR1/oRLbOEo0TeZ8x1NT QjSWQsKlYHvcqJHLTIa7ahrHl/mPo8f9GKaUgkxnqhGsYztpaOKZelqqNwWl1vZYtjax B2NhmtREqcBrs1w0wEATdbJLPLCu/iwxl6Z/GrcDL7UblXgiEN5J+SOHu8NYcJMWt5lS Lwhg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=A677PCkG; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id c15si7378419pjv.63.2021.04.16.08.42.18; Fri, 16 Apr 2021 08:42:30 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=A677PCkG; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244083AbhDPNix (ORCPT + 99 others); Fri, 16 Apr 2021 09:38:53 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:50360 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235535AbhDPNis (ORCPT ); Fri, 16 Apr 2021 09:38:48 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1618580303; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ORKl8cG3pXQFZT6dMTWhjbu3j7T6kEqKkWZLykCofkk=; b=A677PCkGXoMrtrw2EifpBvHZ3IlbQ/abd6eu3vZnRHODEX1WMLDtjkYa1w0A6yTYKUrHeF istzAxTU86oEQVH0fyiCsrEz9YwSho0JWPWvv3eGdiu9qTxkavO028wLMp+VMmsvv+eEuR SfgctE7vGjBPmwAwZyJ0S1bDYKWl2kA= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-269-rdYtHHQlPP-nLVltZYYZvA-1; Fri, 16 Apr 2021 09:38:21 -0400 X-MC-Unique: rdYtHHQlPP-nLVltZYYZvA-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 3FFCA107ACCA; Fri, 16 Apr 2021 13:38:18 +0000 (UTC) Received: from [10.36.113.21] (ovpn-113-21.ams2.redhat.com [10.36.113.21]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 3276A19D9F; Fri, 16 Apr 2021 13:38:04 +0000 (UTC) Subject: Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs To: Jason Gunthorpe Cc: "Liu, Yi L" , Jean-Philippe Brucker , "Tian, Kevin" , Jacob Pan , LKML , Joerg Roedel , Lu Baolu , David Woodhouse , "iommu@lists.linux-foundation.org" , "cgroups@vger.kernel.org" , Tejun Heo , Li Zefan , Johannes Weiner , Jean-Philippe Brucker , Alex Williamson , Jonathan Corbet , "Raj, Ashok" , "Wu, Hao" , "Jiang, Dave" References: <20210331124038.GE1463678@nvidia.com> <20210401134236.GF1463678@nvidia.com> <20210401160337.GJ1463678@nvidia.com> <4bea6eb9-08ad-4b6b-1e0f-c97ece58a078@redhat.com> <20210415230732.GG1370958@nvidia.com> From: Auger Eric Message-ID: Date: Fri, 16 Apr 2021 15:38:02 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.8.0 MIME-Version: 1.0 In-Reply-To: <20210415230732.GG1370958@nvidia.com> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hi Jason, On 4/16/21 1:07 AM, Jason Gunthorpe wrote: > On Thu, Apr 15, 2021 at 03:11:19PM +0200, Auger Eric wrote: >> Hi Jason, >> >> On 4/1/21 6:03 PM, Jason Gunthorpe wrote: >>> On Thu, Apr 01, 2021 at 02:08:17PM +0000, Liu, Yi L wrote: >>> >>>> DMA page faults are delivered to root-complex via page request message and >>>> it is per-device according to PCIe spec. Page request handling flow is: >>>> >>>> 1) iommu driver receives a page request from device >>>> 2) iommu driver parses the page request message. Get the RID,PASID, faulted >>>> page and requested permissions etc. >>>> 3) iommu driver triggers fault handler registered by device driver with >>>> iommu_report_device_fault() >>> >>> This seems confused. >>> >>> The PASID should define how to handle the page fault, not the driver. >> >> In my series I don't use PASID at all. I am just enabling nested stage >> and the guest uses a single context. I don't allocate any user PASID at >> any point. >> >> When there is a fault at physical level (a stage 1 fault that concerns >> the guest), this latter needs to be reported and injected into the >> guest. The vfio pci driver registers a fault handler to the iommu layer >> and in that fault handler it fills a circ bugger and triggers an eventfd >> that is listened to by the VFIO-PCI QEMU device. this latter retrives >> the faault from the mmapped circ buffer, it knowns which vIOMMU it is >> attached to, and passes the fault to the vIOMMU. >> Then the vIOMMU triggers and IRQ in the guest. >> >> We are reusing the existing concepts from VFIO, region, IRQ to do that. >> >> For that use case, would you also use /dev/ioasid? > > /dev/ioasid could do all the things you described vfio-pci as doing, > it can even do them the same way you just described. > > Stated another way, do you plan to duplicate all of this code someday > for vfio-cxl? What about for vfio-platform? ARM SMMU can be hooked to > platform devices, right? vfio regions and IRQ related APIs are common user interfaces exposed by all vfio drivers, including platform. Then the actual circular buffer implementation details can be put in a common lib. as for the thin vfio iommu wrappers, the ones you don't like, they are implemented in type1 code. Maybe the need for /dev/ioasid is more crying for PASID management but for the nested use case, that's not obvious to me and in your different replies, it was not crystal clear where the use case belongs to. The redesign requirement came pretty late in the development process. The iommu user API is upstream for a while, the VFIO interfaces have been submitted a long time ago and under review for a bunch of time. Redesigning everything with a different API, undefined at this point, is a major setback for our work and will have a large impact on the introduction of features companies are looking forward, hence our frustration. Thanks Eric > > I feel what you guys are struggling with is some choice in the iommu > kernel APIs that cause the events to be delivered to the pci_device > owner, not the PASID owner. > > That feels solvable. > > Jason >