Received: by 2002:a25:6193:0:0:0:0:0 with SMTP id v141csp1437886ybb; Thu, 9 Apr 2020 02:04:15 -0700 (PDT) X-Google-Smtp-Source: APiQypL1P1vxtPtN2uD3Uj2IwaW7ntYTCn5lbCvgDiobVgN92fM05klmOnm1baThklzQBBnFPym1 X-Received: by 2002:a4a:d21e:: with SMTP id c30mr9238586oos.0.1586423055319; Thu, 09 Apr 2020 02:04:15 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1586423055; cv=none; d=google.com; s=arc-20160816; b=rSG8Xc/4dzLksWyp1VZ+tZBeuSrQU6EVbUx8SnPES7wUXEf2Gbf0XUpQ/Gqj/1gJya yTt54N+os8C46dIRMi5dyD8ivVzIWF7q6NuouoKY3XJsRL9mc2DXQen2rq6NXh62tKqQ YrTj+lBxeXVkHhrDup0MQ9gnJW4cFO99p0ipdH3KvHVTK7cosgrLiaUpK4c4u9IrDOuW 9iGkY2N/qHi4jGqMDpGhr93f/Crl3VJDkojC8GzvWPFyeSWMoJE8ueBNC9n9hv8nKtCD GoH0zDQrP5CzbmwInkYvS7FwdJoNyfIOzfo90kxgAXlMJdN/E5s62JbRjtXCx2iC+kDH 5uFQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding :content-language:in-reply-to:mime-version:user-agent:date :message-id:from:references:cc:to:subject:dkim-signature; bh=msTkXhnouvwVtlF7/78TkiH9nVYNya7vCxTaw1gbKMk=; b=RmdaixhBSJdRD2FfTNyFIe3q51qHmnU4tOPrnbNU8eH0DekVuSiySf27C6c8gJzMqT 4Y2yFDcQqN/cASB3JyxUWBgqgrwIpybPGFlAXGvYrzWtHgQSSAyg8fdc/YayuvFnHhpj QP4CsJAF4tu94uvOcqmDystsRxigkRNDRsbyrAvdaUmjjoq0O6WArIxdBp2KxHphjkdv OZE/Bltz6xJBbij6TlkwnVDGJHKgrfYuM/SLQSTSIpocM7Hzt0IxP9lfjxbFkeLMB+tY XrecaYp/1i7FTtwfSz9+crNnUU1YhcCcMUcQfYy6gsrR/dg+xyD/j3dMlXB2YJ7UTiKd XGiw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=Mn4dIaBi; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id t13si3377724otp.249.2020.04.09.02.03.30; Thu, 09 Apr 2020 02:04:15 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=Mn4dIaBi; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726663AbgDIJBk (ORCPT + 99 others); Thu, 9 Apr 2020 05:01:40 -0400 Received: from us-smtp-delivery-1.mimecast.com ([205.139.110.120]:47699 "EHLO us-smtp-1.mimecast.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726082AbgDIJBk (ORCPT ); Thu, 9 Apr 2020 05:01:40 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1586422899; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=msTkXhnouvwVtlF7/78TkiH9nVYNya7vCxTaw1gbKMk=; b=Mn4dIaBiUGlkQQvGjJ50BEnVgvcmpzeZ/NjCnn7/pz8HCIjuWyQqx3OorYHtIUp6fIo2/n Y3cCnd9rnOIqULb85bfdjbG1gRoYx3W+u4mSDQC11Q/p98WY5CQ81ueCLBZ75FsCwDJhWI DexeS/qkuigD+QaBdx5ollJOwMN8Nno= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-20-4d7R36GENiiwKgaxvAcgYQ-1; Thu, 09 Apr 2020 05:01:37 -0400 X-MC-Unique: 4d7R36GENiiwKgaxvAcgYQ-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.phx2.redhat.com [10.5.11.15]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 88F18107ACCA; Thu, 9 Apr 2020 09:01:35 +0000 (UTC) Received: from [10.36.115.53] (ovpn-115-53.ams2.redhat.com [10.36.115.53]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 8122AA63D6; Thu, 9 Apr 2020 09:01:26 +0000 (UTC) Subject: Re: [PATCH v1 5/8] vfio/type1: Report 1st-level/stage-1 format to userspace To: Jean-Philippe Brucker Cc: "Liu, Yi L" , "alex.williamson@redhat.com" , "Tian, Kevin" , "jacob.jun.pan@linux.intel.com" , "joro@8bytes.org" , "Raj, Ashok" , "Tian, Jun J" , "Sun, Yi Y" , "peterx@redhat.com" , "iommu@lists.linux-foundation.org" , "kvm@vger.kernel.org" , "linux-kernel@vger.kernel.org" , "Wu, Hao" References: <1584880325-10561-1-git-send-email-yi.l.liu@intel.com> <1584880325-10561-6-git-send-email-yi.l.liu@intel.com> <20200403082305.GA1269501@myrica> <20200409081442.GD2435@myrica> From: Auger Eric Message-ID: Date: Thu, 9 Apr 2020 11:01:24 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.4.0 MIME-Version: 1.0 In-Reply-To: <20200409081442.GD2435@myrica> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.79 on 10.5.11.15 Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hi Jean, On 4/9/20 10:14 AM, Jean-Philippe Brucker wrote: > On Wed, Apr 08, 2020 at 12:27:58PM +0200, Auger Eric wrote: >> Hi Yi, >> >> On 4/7/20 11:43 AM, Liu, Yi L wrote: >>> Hi Jean, >>> >>>> From: Jean-Philippe Brucker >>>> Sent: Friday, April 3, 2020 4:23 PM >>>> To: Auger Eric >>>> userspace >>>> >>>> On Wed, Apr 01, 2020 at 03:01:12PM +0200, Auger Eric wrote: >>>>>>>> header = vfio_info_cap_add(caps, sizeof(*nesting_cap), >>>>>>>> VFIO_IOMMU_TYPE1_INFO_CAP_NESTING, 1); >>>> @@ -2254,6 +2309,7 >>>>>>>> @@ static int vfio_iommu_info_add_nesting_cap(struct >>>>>>> vfio_iommu *iommu, >>>>>>>> /* nesting iommu type supports PASID requests (alloc/free) */ >>>>>>>> nesting_cap->nesting_capabilities |= VFIO_IOMMU_PASID_REQS; >>>>>>> What is the meaning for ARM? >>>>>> >>>>>> I think it's just a software capability exposed to userspace, on >>>>>> userspace side, it has a choice to use it or not. :-) The reason >>>>>> define it and report it in cap nesting is that I'd like to make the >>>>>> pasid alloc/free be available just for IOMMU with type >>>>>> VFIO_IOMMU_TYPE1_NESTING. Please feel free tell me if it is not good >>>>>> for ARM. We can find a proper way to report the availability. >>>>> >>>>> Well it is more a question for jean-Philippe. Do we have a system wide >>>>> PASID allocation on ARM? >>>> >>>> We don't, the PASID spaces are per-VM on Arm, so this function should consult the >>>> IOMMU driver before setting flags. As you said on patch 3, nested doesn't >>>> necessarily imply PASID support. The SMMUv2 does not support PASID but does >>>> support nesting stages 1 and 2 for the IOVA space. >>>> SMMUv3 support of PASID depends on HW capabilities. So I think this needs to be >>>> finer grained: >>>> >>>> Does the container support: >>>> * VFIO_IOMMU_PASID_REQUEST? >>>> -> Yes for VT-d 3 >>>> -> No for Arm SMMU >>>> * VFIO_IOMMU_{,UN}BIND_GUEST_PGTBL? >>>> -> Yes for VT-d 3 >>>> -> Sometimes for SMMUv2 >>>> -> No for SMMUv3 (if we go with BIND_PASID_TABLE, which is simpler due to >>>> PASID tables being in GPA space.) >>>> * VFIO_IOMMU_BIND_PASID_TABLE? >>>> -> No for VT-d >>>> -> Sometimes for SMMUv3 >>>> >>>> Any bind support implies VFIO_IOMMU_CACHE_INVALIDATE support. >>> >>> good summary. do you expect to see any >>> >>>> >>>>>>>> + nesting_cap->stage1_formats = formats; >>>>>>> as spotted by Kevin, since a single format is supported, rename >>>>>> >>>>>> ok, I was believing it may be possible on ARM or so. :-) will rename >>>>>> it. >>>> >>>> Yes I don't think an u32 is going to cut it for Arm :( We need to describe all sorts of >>>> capabilities for page and PASID tables (granules, GPA size, ASID/PASID size, HW >>>> access/dirty, etc etc.) Just saying "Arm stage-1 format" wouldn't mean much. I >>>> guess we could have a secondary vendor capability for these? >>> >>> Actually, I'm wondering if we can define some formats to stands for a set of >>> capabilities. e.g. VTD_STAGE1_FORMAT_V1 which may indicates the 1st level >>> page table related caps (aw, a/d, SRE, EA and etc.). And vIOMMU can parse >>> the capabilities. >> >> But eventually do we really need all those capability getters? I mean >> can't we simply rely on the actual call to VFIO_IOMMU_BIND_GUEST_PGTBL() >> to detect any mismatch? Definitively the error handling may be heavier >> on userspace but can't we manage. > > I think we need to present these capabilities at boot time, long before > the guest triggers a bind(). For example if the host SMMU doesn't support > 16-bit ASID, we need to communicate that to the guest using vSMMU ID > registers or PROBE properties. Otherwise a bind() will succeed, but if the > guest uses 16-bit ASIDs in its CD, DMA will result in C_BAD_CD events > which we'll inject into the guest, for no apparent reason from their > perspective. OK I understand this case as in this situation we may be able to change the way to iommu is exposed to the guest. > > In addition some VMMs may have fallbacks if shared page tables are not > available. They could fall back to a MAP/UNMAP interface, or simply not > present a vIOMMU to the guest. fair enough, there is a need for such capability checker in the mid term. But this patch introduces the capability to check whether system wide PASID alloc is supported and this may not be requested at that stage for the whole vSVM integration? Thanks Eric > > Thanks, > Jean > >> My fear is we end up with an overly >> complex series. This capability getter may be interesting if we can >> switch to a fallback implementation but here I guess we don't have any >> fallback. With smmuv3 nested stage we don't have any fallback solution >> either. For the versions, it is different because the userspace shall be >> able to adapt (or not) to the max version supported by the kernel. >> >> Thanks >> >> Eric >>> >>> Regards, >>> Yi Liu >>> >> >