Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755468AbcKOH1M (ORCPT ); Tue, 15 Nov 2016 02:27:12 -0500 Received: from mail-pg0-f66.google.com ([74.125.83.66]:36643 "EHLO mail-pg0-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752217AbcKOH1J (ORCPT ); Tue, 15 Nov 2016 02:27:09 -0500 Subject: Re: [PATCH v11 10/22] vfio iommu type1: Add support for mediated devices To: Kirti Wankhede , alex.williamson@redhat.com, pbonzini@redhat.com, kraxel@redhat.com, cjia@nvidia.com References: <1478293856-8191-1-git-send-email-kwankhede@nvidia.com> <1478293856-8191-11-git-send-email-kwankhede@nvidia.com> <93c0ace9-bf1f-3d94-56ab-3513f7183a1b@ozlabs.ru> <5219635a-011b-ed6a-9535-867085b072b0@nvidia.com> Cc: qemu-devel@nongnu.org, kvm@vger.kernel.org, kevin.tian@intel.com, jike.song@intel.com, bjsdjshi@linux.vnet.ibm.com, linux-kernel@vger.kernel.org From: Alexey Kardashevskiy Message-ID: Date: Tue, 15 Nov 2016 18:27:02 +1100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.4.0 MIME-Version: 1.0 In-Reply-To: <5219635a-011b-ed6a-9535-867085b072b0@nvidia.com> Content-Type: text/plain; charset=koi8-r Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 3552 Lines: 96 On 15/11/16 17:33, Kirti Wankhede wrote: > > > On 11/15/2016 10:47 AM, Alexey Kardashevskiy wrote: >> On 08/11/16 17:52, Alexey Kardashevskiy wrote: >>> On 05/11/16 08:10, Kirti Wankhede wrote: >>>> VFIO IOMMU drivers are designed for the devices which are IOMMU capable. >>>> Mediated device only uses IOMMU APIs, the underlying hardware can be >>>> managed by an IOMMU domain. >>>> >>>> Aim of this change is: >>>> - To use most of the code of TYPE1 IOMMU driver for mediated devices >>>> - To support direct assigned device and mediated device in single module >>>> >>>> This change adds pin and unpin support for mediated device to TYPE1 IOMMU >>>> backend module. More details: >>>> - vfio_pin_pages() callback here uses task and address space of vfio_dma, >>>> that is, of the process who mapped that iova range. >>>> - Added pfn_list tracking logic to address space structure. All pages >>>> pinned through this interface are trached in its address space. >>>> - Pinned pages list is used to verify unpinning request and to unpin >>>> remaining pages while detaching the group for that device. >>>> - Page accounting is updated to account in its address space where the >>>> pages are pinned/unpinned. >>>> - Accouting for mdev device is only done if there is no iommu capable >>>> domain in the container. When there is a direct device assigned to the >>>> container and that domain is iommu capable, all pages are already pinned >>>> during DMA_MAP. >>>> - Page accouting is updated on hot plug and unplug mdev device and pass >>>> through device. >>>> >>>> Tested by assigning below combinations of devices to a single VM: >>>> - GPU pass through only >>> >>> This does not require this patchset, right? >>> > > Sorry I missed this earlier. > This testing is required for this patch, because this patch touches code > that is used for direct device assignment. Also for page accounting, all > cases are considered i.e. when there is only pass through device in a > container, when there is pass through device + vGPU device in a > container. Also have to test that pages are pinned properly when device > is hotplugged. In that case vfio_iommu_replay() is called to take > necessary action. So in this particular test you are only testing that the patchset did not break the already existing functionality, is that correct? > >>>> - vGPU device only >>> >>> Out of curiosity - how exactly did you test this? The exact GPU, how to >>> create vGPU, what was the QEMU command line and the guest does with this >>> passed device? Thanks. >> >> ping? >> > > I'm testing this code with M60, with custom changes in our driver. Is this shared anywhere? What does the mediated driver do? Can Tesla K80 do the same thing, or [10de:15fe] (whatever its name is)? > Steps how to create mediated device are listed in > Documentation/vfio-mediated-device.txt for sample mtty driver. Same > steps I'm following for GPU. Quoting those steps here for you: Nah, I saw this, I was wondering about actual hardware :) Like when you say "tested with vGPU" - I am wondering what is passed to the guest and how the guest is actually using it. > > 2. Create a mediated device by using the dummy device that you created > in the > previous step. > > # echo "83b8f4f2-509f-382f-3c1e-e6bfe0fa1001" > \ > > /sys/devices/virtual/mtty/mtty/mdev_supported_types/mtty-2/create > > 3. Add parameters to qemu-kvm. > > -device vfio-pci,\ > sysfsdev=/sys/bus/mdev/devices/83b8f4f2-509f-382f-3c1e-e6bfe0fa1001 -- Alexey