Received: by 2002:a25:4158:0:0:0:0:0 with SMTP id o85csp698777yba; Wed, 15 May 2019 08:27:12 -0700 (PDT) X-Google-Smtp-Source: APXvYqz9h1aEcYNeI96+wb4tBivVViAPUe3VilGaY66dDiNZeFqTA7oxOwDFbXVbVaDd8GEUQMrG X-Received: by 2002:a62:f247:: with SMTP id y7mr47319476pfl.18.1557934032637; Wed, 15 May 2019 08:27:12 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1557934032; cv=none; d=google.com; s=arc-20160816; b=W2KAePM4EoDoHFXB2Z+uhdUK497iuOIkkMRH/oXL5dbTRD0fQwgZjKYfngb1SESSKx jZw2val8vjCw/b9s6cqseUUZovP54ib/vhDx4uGcLpZFO8VOvxvmZCf+/kXRtmWovLv2 EcoqhkkgM2NRSulhyv3Tl06SStGLLs0tYUL/S69WJvsG0XMFzCWaIgBsRh3YbMvmIABm IwcfldX+UAR1kcRGFJtbSjoBVbr+SeCLMT2gULby9EnqEmcW7d4jf9rPeaAmhmVNgnBd +5vcVs9oJVYeaP/AyD3QCm5i2nnB8L56ZxI/8bgfmT45vnq2d6FuXHHJRhuflFroPCeq izSw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding :content-language:in-reply-to:mime-version:user-agent:date :message-id:from:references:cc:to:subject; bh=Z6DPE7mBYaxgCacZpJnbsEXbchJnVDEAoHs5TRfyxVI=; b=sJtetV5nImpYZWiHLhP460tBGlyGSwjYh9HxIp3Mna9Xspc46gQZJNKJnfH/JByx2e 9sf6WXv8UxkJIH+cuSVP9+mq4pYdb6q/ZWwzJCKSip5xfSF/yzjiajdd10PW7mpAZ7Tl Gr+BJG7nh78aOWjNUXjgTIPp0buTK+j/HzJ03hAbSGisG7Bou4Xb7pTgQrPTdhCHTH0K tVtDarZkM/oSb3TseB1ET5//YKZd28Z8detCfiWMBgVFQF5FWN9mGNLiZ6kQDR7USP+y TvBDkRyYDda8qrCY8lui/ChIsrwtP/KKYVOYHCtbwsepU8P2oDghFRR9TNHOZs3Ma38u dDVg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id q3si2425661pfc.89.2019.05.15.08.26.56; Wed, 15 May 2019 08:27:12 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727833AbfEOPZ2 (ORCPT + 99 others); Wed, 15 May 2019 11:25:28 -0400 Received: from foss.arm.com ([217.140.101.70]:46702 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726501AbfEOPZ2 (ORCPT ); Wed, 15 May 2019 11:25:28 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 7E9B7374; Wed, 15 May 2019 08:25:27 -0700 (PDT) Received: from [10.1.196.129] (ostrya.cambridge.arm.com [10.1.196.129]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 15DEF3F703; Wed, 15 May 2019 08:25:25 -0700 (PDT) Subject: Re: [PATCH v3 02/16] iommu: Introduce cache_invalidate API To: "Tian, Kevin" , Jacob Pan Cc: "Raj, Ashok" , Alex Williamson , LKML , "iommu@lists.linux-foundation.org" , Andriy Shevchenko , David Woodhouse References: <1556922737-76313-1-git-send-email-jacob.jun.pan@linux.intel.com> <1556922737-76313-3-git-send-email-jacob.jun.pan@linux.intel.com> <44d5ba37-a9e9-cc7a-2a3a-d32b840afa29@arm.com> <7807afe9-efab-9f48-4ca0-2332a7a54950@redhat.com> <1a5a5fad-ed21-5c79-9a9e-ff21fadfb95f@arm.com> <20190513151637.79c273e2@jacob-builder> <0da76e57-76f6-06fa-d34e-30cd0c294984@redhat.com> <20190514104401.79d563f4@jacob-builder> From: Jean-Philippe Brucker Message-ID: <78baa435-7e6a-505e-a8d1-d16c6c6aa1b9@arm.com> Date: Wed, 15 May 2019 16:25:04 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.6.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 15/05/2019 15:47, Tian, Kevin wrote: >> From: Jean-Philippe Brucker >> Sent: Wednesday, May 15, 2019 7:04 PM >> >> On 14/05/2019 18:44, Jacob Pan wrote: >>> Hi Thank you both for the explanation. >>> >>> On Tue, 14 May 2019 11:41:24 +0100 >>> Jean-Philippe Brucker wrote: >>> >>>> On 14/05/2019 08:36, Auger Eric wrote: >>>>> Hi Jacob, >>>>> >>>>> On 5/14/19 12:16 AM, Jacob Pan wrote: >>>>>> On Mon, 13 May 2019 18:09:48 +0100 >>>>>> Jean-Philippe Brucker wrote: >>>>>> >>>>>>> On 13/05/2019 17:50, Auger Eric wrote: >>>>>>>>> struct iommu_inv_pasid_info { >>>>>>>>> #define IOMMU_INV_PASID_FLAGS_PASID (1 << 0) >>>>>>>>> #define IOMMU_INV_PASID_FLAGS_ARCHID (1 << 1) >>>>>>>>> __u32 flags; >>>>>>>>> __u32 archid; >>>>>>>>> __u64 pasid; >>>>>>>>> }; >>>>>>>> I agree it does the job now. However it looks a bit strange to >>>>>>>> do a PASID based invalidation in my case - SMMUv3 nested stage - >>>>>>>> where I don't have any PASID involved. >>>>>>>> >>>>>>>> Couldn't we call it context based invalidation then? A context >>>>>>>> can be tagged by a PASID or/and an ARCHID. >>>>>>> >>>>>>> I think calling it "context" would be confusing as well (I >>>>>>> shouldn't have used it earlier), since VT-d uses that name for >>>>>>> device table entries (=STE on Arm SMMU). Maybe "addr_space"? >>>>>>> >>>>>> I am still struggling to understand what ARCHID is after scanning >>>>>> through SMMUv3.1 spec. It seems to be a constant for a given SMMU. >>>>>> Why do you need to pass it down every time? Could you point to me >>>>>> the document or explain a little more on ARCHID use cases. >>>>>> We have three fileds called pasid under this struct >>>>>> iommu_cache_invalidate_info{} >>>>>> Gets confusing :) >>>>> archid is a generic term. That's why you did not find it in the >>>>> spec ;-) >>>>> >>>>> On ARM SMMU the archid is called the ASID (Address Space ID, up to >>>>> 16 bits. The ASID is stored in the Context Descriptor Entry (your >>>>> PASID entry) and thus characterizes a given stage 1 translation >>>>> "context"/"adress space". >>>> >>>> Yes, another way to look at it is, for a given address space: >>>> * PASID tags device-IOTLB (ATC) entries. >>>> * ASID (here called archid) tags IOTLB entries. >>>> >>>> They could have the same value, but it depends on the guest's >>>> allocation policy which isn't in our control. With my PASID patches >>>> for SMMUv3, they have different values. So we need both fields if we >>>> intend to invalidate both ATC and IOTLB with a single call. >>>> >>> For ASID invalidation, there is also page/address selective within an >>> ASID, right? I guess it is CMD_TLBI_NH_VA? >>> So the single call to invalidate both ATC & IOTLB should share the same >>> address information. i.e. >>> struct iommu_inv_addr_info {} >>> >>> Just out of curiosity, what is the advantage of having guest tag its >>> ATC with its own PASID? I thought you were planning to use custom >>> ioasid allocator to get PASID from host. >> >> Hm, for the moment I mostly considered the custom ioasid allocator for >> Intel platforms. On Arm platforms the SR-IOV model where each VM has its >> own PASID space is still very much on the table. This would be the only >> model supported by a vSMMU emulation for example, since the SMMU >> doesn't >> have PASID allocation commands. >> > > I didn't get how ATS works in such case, if device ATC PASID is different > from IOTLB ASID. Who will be responsible for translation in-between? ATS with the SMMU works like this: * The PCI function sends a Translation Request with PASID. * The SMMU walks the PASID table (which we call context descriptor table), finds the context descriptor indexed by PASID. This context descriptor has an ASID field, and a page directory pointer. * After successfully walking the page tables, the SMMU may add an IOTLB entry tagged by ASID and address, then returns a Translation Completion. * The PCI function adds an ATC entry tagged by PASID and address. I think the ASID on Arm CPUs is roughly equivalent to Intel PCID. One reason we use ASIDs for IOTLBs is that with SVA, the ASID of an address space is the same on the CPU side. And when the CPU executes a TLB invalidation instructions, it also invalidates the corresponding IOTLB entries. It's nice for vSVA because you don't need to context-switch to the host to send an IOTLB invalidation. But only non-PCI devices that implement SVA benefit from this at the moment, because ATC invalidations still have to go through the SMMU command queue. Thanks, Jean