Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S965188AbcKOH7V (ORCPT ); Tue, 15 Nov 2016 02:59:21 -0500 Received: from hqemgate16.nvidia.com ([216.228.121.65]:5273 "EHLO hqemgate16.nvidia.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S934210AbcKOH7T (ORCPT ); Tue, 15 Nov 2016 02:59:19 -0500 X-PGP-Universal: processed; by hqpgpgate101.nvidia.com on Mon, 14 Nov 2016 23:59:16 -0800 Subject: Re: [PATCH v11 10/22] vfio iommu type1: Add support for mediated devices To: Alexey Kardashevskiy , , , , References: <1478293856-8191-1-git-send-email-kwankhede@nvidia.com> <1478293856-8191-11-git-send-email-kwankhede@nvidia.com> <93c0ace9-bf1f-3d94-56ab-3513f7183a1b@ozlabs.ru> <5219635a-011b-ed6a-9535-867085b072b0@nvidia.com> CC: , , , , , X-Nvconfidentiality: public From: Kirti Wankhede Message-ID: Date: Tue, 15 Nov 2016 13:26:55 +0530 MIME-Version: 1.0 In-Reply-To: X-Originating-IP: [10.24.216.210] X-ClientProxiedBy: DRUKMAIL102.nvidia.com (10.25.59.20) To bgmail102.nvidia.com (10.25.59.11) Content-Type: text/plain; charset="koi8-r" Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 4028 Lines: 113 On 11/15/2016 12:57 PM, Alexey Kardashevskiy wrote: > On 15/11/16 17:33, Kirti Wankhede wrote: >> >> >> On 11/15/2016 10:47 AM, Alexey Kardashevskiy wrote: >>> On 08/11/16 17:52, Alexey Kardashevskiy wrote: >>>> On 05/11/16 08:10, Kirti Wankhede wrote: >>>>> VFIO IOMMU drivers are designed for the devices which are IOMMU capable. >>>>> Mediated device only uses IOMMU APIs, the underlying hardware can be >>>>> managed by an IOMMU domain. >>>>> >>>>> Aim of this change is: >>>>> - To use most of the code of TYPE1 IOMMU driver for mediated devices >>>>> - To support direct assigned device and mediated device in single module >>>>> >>>>> This change adds pin and unpin support for mediated device to TYPE1 IOMMU >>>>> backend module. More details: >>>>> - vfio_pin_pages() callback here uses task and address space of vfio_dma, >>>>> that is, of the process who mapped that iova range. >>>>> - Added pfn_list tracking logic to address space structure. All pages >>>>> pinned through this interface are trached in its address space. >>>>> - Pinned pages list is used to verify unpinning request and to unpin >>>>> remaining pages while detaching the group for that device. >>>>> - Page accounting is updated to account in its address space where the >>>>> pages are pinned/unpinned. >>>>> - Accouting for mdev device is only done if there is no iommu capable >>>>> domain in the container. When there is a direct device assigned to the >>>>> container and that domain is iommu capable, all pages are already pinned >>>>> during DMA_MAP. >>>>> - Page accouting is updated on hot plug and unplug mdev device and pass >>>>> through device. >>>>> >>>>> Tested by assigning below combinations of devices to a single VM: >>>>> - GPU pass through only >>>> >>>> This does not require this patchset, right? >>>> >> >> Sorry I missed this earlier. >> This testing is required for this patch, because this patch touches code >> that is used for direct device assignment. Also for page accounting, all >> cases are considered i.e. when there is only pass through device in a >> container, when there is pass through device + vGPU device in a >> container. Also have to test that pages are pinned properly when device >> is hotplugged. In that case vfio_iommu_replay() is called to take >> necessary action. > > So in this particular test you are only testing that the patchset did not > break the already existing functionality, is that correct? > > Yes. >> >>>>> - vGPU device only >>>> >>>> Out of curiosity - how exactly did you test this? The exact GPU, how to >>>> create vGPU, what was the QEMU command line and the guest does with this >>>> passed device? Thanks. >>> >>> ping? >>> >> >> I'm testing this code with M60, with custom changes in our driver. > > > Is this shared anywhere? What does the mediated driver do? Can Tesla K80 do > the same thing, or [10de:15fe] (whatever its name is)? > > No, we are still in the process of upstreaming this patchset. After that we would prepare official drivers. There is no timeline for the driver release yet. >> Steps how to create mediated device are listed in >> Documentation/vfio-mediated-device.txt for sample mtty driver. Same >> steps I'm following for GPU. Quoting those steps here for you: > > > Nah, I saw this, I was wondering about actual hardware :) Like when you say > "tested with vGPU" - I am wondering what is passed to the guest and how the > guest is actually using it. > > Its similar to mtty sample driver, that simulates serial port and so you see serial ports in guest. For vGPU, you see a virtual GPU in guest. Thanks, Kirti >> >> 2. Create a mediated device by using the dummy device that you created >> in the >> previous step. >> >> # echo "83b8f4f2-509f-382f-3c1e-e6bfe0fa1001" > \ >> >> /sys/devices/virtual/mtty/mtty/mdev_supported_types/mtty-2/create >> >> 3. Add parameters to qemu-kvm. >> >> -device vfio-pci,\ >> sysfsdev=/sys/bus/mdev/devices/83b8f4f2-509f-382f-3c1e-e6bfe0fa1001 > > > >