Received: by 2002:a05:6a11:4021:0:0:0:0 with SMTP id ky33csp4237140pxb; Mon, 27 Sep 2021 12:21:06 -0700 (PDT) X-Google-Smtp-Source: ABdhPJz1vb5VSq+wGt9CCEZ5HrmT2FYLDxNs0RdK27YJf7IT6ESEdMbqNMAlEAfi2ZMaVZxgSrkO X-Received: by 2002:a63:4606:: with SMTP id t6mr1088490pga.388.1632770466151; Mon, 27 Sep 2021 12:21:06 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1632770466; cv=none; d=google.com; s=arc-20160816; b=XQYQL/N4qj24brWIPqXCuV6ZiV7ImtE9/QaDYQjITLoFVWh0uWiPjMrdqYTEf6LXbP LanyWM/lLAlvSul/9X/6vcL7P/aLw3JG19rJTMEtj/Dd4TXJ2PlpJ8bBNP5XRZEMuHRW 28acCXvpmTBtUZ2PGA962nlnTT6dfB1p73kFifACbHUkRb6v+TmDkug8TdMsMgkqTP40 T4rwhy19ncoItM046NpoJHKLoAH8RqkyfC2LdPX50OukuZNqG0Na4W7X8JK+qnKotjbs sEyrVgOEIJWZ2bXe6LcjduF2MDkvTP1ObZbqgODpvm1BTKNY5SHKhW9qFLE36Wxni8Gm MCPg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :organization:references:in-reply-to:message-id:subject:cc:to:from :date:dkim-signature; bh=Ytbf7mkDtKm4hXqfmsIh+zBdQwECAHFtYv2udg+9N/s=; b=fBZdVQ7zkSqSpmqhbSwXF+FLkqp+tEd0EDeEyHJsPZsR8EJJkv1Q57Ka/NLmQsmCfx sq5lP29Y9hQ/OMmzwk3rYFgUa2MHsgZUz0JKJV9Whxn0kB8sFp2B/irA7jXlyNuFM8ax /WpXUlKOBtsGsj277UbLuiOawOAjwJ/MoIJ8ex+ZI2vMr5aESqxaEmaCozZQBchElA+b IxIl4dpLfwOjCA9SH5X+y2ppod5vYVZM1wOybSTVainy0v+xUI2V7xG7k9lLsYMGOi2A YTrar7RdQ+vnedUQ0IClwsLIBW3wMe2ALmUCDq1rqg/rZkUvqxQndGVmCT5abdRZ+93m cYNw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=gycSX+ht; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id j1si20578968plr.208.2021.09.27.12.20.51; Mon, 27 Sep 2021 12:21:06 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=gycSX+ht; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236399AbhI0TVe (ORCPT + 99 others); Mon, 27 Sep 2021 15:21:34 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:36452 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236260AbhI0TVd (ORCPT ); Mon, 27 Sep 2021 15:21:33 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1632770394; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Ytbf7mkDtKm4hXqfmsIh+zBdQwECAHFtYv2udg+9N/s=; b=gycSX+ht0iq0vyVLZMvUrd6m9OFSS6/vtex6QTrxw80cq1UjkzzFqGeqHLKgfg80A9vI8c z6f4R4QBi4GBCREDP25acLiudTXUceNX6FFjBtTc7xG8VgX6d1g79S2AG/wSdIzALHZxSF nPgVnOnF76fJ+2iZat6whiVvSuk5nVA= Received: from mail-oi1-f200.google.com (mail-oi1-f200.google.com [209.85.167.200]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-40-dO9ZhWkfPfuseyZeC6h0ig-1; Mon, 27 Sep 2021 15:19:52 -0400 X-MC-Unique: dO9ZhWkfPfuseyZeC6h0ig-1 Received: by mail-oi1-f200.google.com with SMTP id y185-20020acaafc2000000b0027359453ad4so14338523oie.6 for ; Mon, 27 Sep 2021 12:19:52 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:in-reply-to :references:organization:mime-version:content-transfer-encoding; bh=Ytbf7mkDtKm4hXqfmsIh+zBdQwECAHFtYv2udg+9N/s=; b=eE39WIDD8HVNkb1yceH3fB4k6Dw3VRJfGI50Aji1DJyRSKj7xg82EcwJldIrt5/ypS D5WYeJoHcXJgVKrGOXexlMcyPhBYFQW0Lw7FoTHXEWPntSg4p5NPBG3NHRaQ37O7znxl njwj0YFhh7i+HyHQepaVTCOMuZi6+k5Xk1zCDQko9AsOlhURJNI3p4CI1bM3+i3fImQ/ pZT4MwMldlnH1XUnRHAS2U8r2jLI177N4PjClqZi1vGCwtu9mzgqNPienLdqOpyVVpbe tgu6Rmx00UEoCzsjK7WyNQml/mrF7PDramLh87VdPt0zE/dVfwrXVM037rLrOeMetZHK jWMA== X-Gm-Message-State: AOAM532N3z9gfw4cBX6UidWIPz5yK9tnBn/FiMfKq8iTPKLqofojq0QA LroomH8rrGHKUWNNQsJh9R/2mpDnQbbPVVYo4e5LQJvScv8F/TdoJ2PU78zhT+cJ1klLhSsOSUK QOFvlG8x9LC/ii2zO2n3q7Ol6 X-Received: by 2002:a05:6808:d53:: with SMTP id w19mr565220oik.135.1632770392054; Mon, 27 Sep 2021 12:19:52 -0700 (PDT) X-Received: by 2002:a05:6808:d53:: with SMTP id w19mr565205oik.135.1632770391818; Mon, 27 Sep 2021 12:19:51 -0700 (PDT) Received: from redhat.com ([198.99.80.109]) by smtp.gmail.com with ESMTPSA id bg36sm1238777oib.2.2021.09.27.12.19.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 27 Sep 2021 12:19:51 -0700 (PDT) Date: Mon, 27 Sep 2021 13:19:49 -0600 From: Alex Williamson To: "Tian, Kevin" Cc: Jason Gunthorpe , "kvm@vger.kernel.org" , "jasowang@redhat.com" , "kwankhede@nvidia.com" , "hch@lst.de" , "jean-philippe@linaro.org" , "Jiang, Dave" , "Raj, Ashok" , "corbet@lwn.net" , "parav@mellanox.com" , "lkml@metux.net" , "david@gibson.dropbear.id.au" , "dwmw2@infradead.org" , "Tian, Jun J" , "linux-kernel@vger.kernel.org" , "lushenming@huawei.com" , "pbonzini@redhat.com" , "robin.murphy@arm.com" Subject: Re: [RFC 06/20] iommu: Add iommu_device_init[exit]_user_dma interfaces Message-ID: <20210927131949.052d8481.alex.williamson@redhat.com> In-Reply-To: References: <20210919063848.1476776-1-yi.l.liu@intel.com> <20210919063848.1476776-7-yi.l.liu@intel.com> <20210921170943.GS327412@nvidia.com> <20210922123931.GI327412@nvidia.com> <20210927115342.GW964074@nvidia.com> <20210927130935.GZ964074@nvidia.com> Organization: Red Hat X-Mailer: Claws Mail 3.18.0 (GTK+ 2.24.33; x86_64-redhat-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Mon, 27 Sep 2021 13:32:34 +0000 "Tian, Kevin" wrote: > > From: Jason Gunthorpe > > Sent: Monday, September 27, 2021 9:10 PM > > > > On Mon, Sep 27, 2021 at 01:00:08PM +0000, Tian, Kevin wrote: > > > > > > I think for such a narrow usage you should not change the struct > > > > device_driver. Just have pci_stub call a function to flip back to user > > > > mode. > > > > > > Here we want to ensure that kernel dma should be blocked > > > if the group is already marked for user-dma. If we just blindly > > > do it for any driver at this point (as you commented earlier): > > > > > > + ret = iommu_set_kernel_ownership(dev); > > > + if (ret) > > > + return ret; > > > > > > how would pci-stub reach its function to indicate that it doesn't > > > do dma and flip back? > > > > > Do you envision a simpler policy that no driver can be bound > > > to the group if it's already set for user-dma? what about vfio-pci > > > itself? > > > > Yes.. I'm not sure there is a good use case to allow the stub drivers > > to load/unload while a VFIO is running. At least, not a strong enough > > one to justify a global change to the driver core.. > > I'm fine with not loading pci-stub. From the very 1st commit msg > looks pci-stub was introduced before vfio to prevent host driver > loading when doing device assignment with KVM. I'm not sure > whether other usages are built on pci-stub later, but in general it's > not good to position devices in a same group into different usages. IIRC, pci-stub was invented for legacy KVM device assignment because KVM was never an actual device driver, it just latched onto and started using the device. If there was an existing driver for the device then KVM would fail to get device resources. Therefore the device needed to be unbound from its standard host driver, but that left it susceptible to driver loads usurping the device. Therefore pci-stub came along to essentially claim the device on behalf of KVM. With vfio, there are a couple use cases of pci-stub that can be interesting. The first is that pci-stub is generally built into the kernel, not as a module, which provides users the ability to specify a list of ids for pci-stub to claim on the kernel command line with higher priority than loadable modules. This can prevent default driver bindings to devices until tools like driverctl or boot time scripting gets a shot to load the user designated driver for a device. The other use case, is that if a group is composed of multiple devices and all those devices are bound to vfio drivers, then the user can gain direct access to each of those devices. If we wanted to insert a barrier to restrict user access to certain devices within a group, we'd suggest binding those devices to pci-stub. Obviously within a group, it may still be possible to manipulate the device via p2p DMA, but the barrier is much higher and device, if not platform, specific to manipulate such devices. An example use case might be a chipset Ethernet controller grouped among system management function in a multi-function root complex integrated endpoint. > but I'm little worried that even vfio-pci itself cannot be bound now, > which implies that all devices in a group which are intended to be > used by the user must be bound to vfio-pci in a breath before the > user attempts to open any of them, i.e. late-binding and device- > hotplug is disallowed after the initial open. I'm not sure how > important such an usage would be, but it does cause user-tangible > semantics change. Yep, a high potential to break userspace, especially as pci-stub has been recommended for the cases noted above. I don't expect that tools like libvirt manage unassigned devices within a group, but that probably means that there are all sorts of ad-hoc user mechanisms beyond simply assigning all the devices. Thanks, Alex