Received: by 2002:a05:6a10:6744:0:0:0:0 with SMTP id w4csp3228412pxu; Mon, 19 Oct 2020 07:12:52 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxklq7D3xRT4asqB0VF/N9kvkbiAHhMJO6kH+Y2k8ul3N1/N+XqZk+2fIUxswQ62QV3q4Hb X-Received: by 2002:a17:907:2079:: with SMTP id qp25mr52871ejb.347.1603116772652; Mon, 19 Oct 2020 07:12:52 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1603116772; cv=none; d=google.com; s=arc-20160816; b=pNLLOJ3zzsUl6Ep5LV48i+WXG01ytgghaxRdPxTxcrryC3buBVbN7MUBFR+iG0aosP FrVVaMKgfJkKA2b7s8VT2C054XLUwAKmE3caAgaB9juCkqs9MkFv4N7Ai/AhiZChCrFj t0wDy47PThj/Rr+X7uoQrk9UX+aSlb7IDe95c1dDdmN9tA3Wk/fB4VnClPftHE30TI+F NEk8Md4hJ4LXjOD7VqwoMx6vbQOiBF8B+LDhtTaYF5mpiHq4upvb7Ar0yEID0rPDK/Qb Ehe/zxRfQJJ8KnpdxmzNVRrGqchzxXvXz/0mbWr4MhM1M2QlOUpzGnRK2qNhwUodnE4h PF2A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:in-reply-to:content-disposition:mime-version :references:message-id:subject:cc:to:from:date:dkim-signature; bh=/KfoISsApaN0NSURJOvvfJb6dwcjjrFUhXMB/UsO5u8=; b=fxEZFZKFlssrj0j9fpUafw9gBZqqXX4qD5tRNhtiM/GicxZ0A3HXAb+6o89puzj5Ks 83Yg4vIUETMz7GQOpY8LSG4iV6OkXAp+YvxJhcWA/DDS351QylsVm5wccSDWls240V3O 1Wkm7xD7SJPYGpSYIG/asvlNWjBtN8As1L3YKAarMvJA5LtDKjGnmMn+InzEdM4aycvH mQ+/1ygSuXInumYRILoYpR4/Mo6fieKHS2P1d00d2IdWY+6bdkbZMkODzS4lHF3DQpYi 88hVzG/wLC1h78IyQk6uJ7Y1/aTafauzuJg1xaMNu05bqHrFW7FhYjPUI0VC4ixEEMrn HPwQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=NdZ9UN+y; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id lh7si46617ejb.309.2020.10.19.07.12.29; Mon, 19 Oct 2020 07:12:52 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=NdZ9UN+y; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729061AbgJSOIr (ORCPT + 99 others); Mon, 19 Oct 2020 10:08:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38172 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728810AbgJSOIr (ORCPT ); Mon, 19 Oct 2020 10:08:47 -0400 Received: from mail-ej1-x644.google.com (mail-ej1-x644.google.com [IPv6:2a00:1450:4864:20::644]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BCEA3C0613CE for ; Mon, 19 Oct 2020 07:08:46 -0700 (PDT) Received: by mail-ej1-x644.google.com with SMTP id h24so14101404ejg.9 for ; Mon, 19 Oct 2020 07:08:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=/KfoISsApaN0NSURJOvvfJb6dwcjjrFUhXMB/UsO5u8=; b=NdZ9UN+yHyzobun2pSlu10EnMwdoI5kP57Dg+V9m5zdFRPjXDmTawgoX/3q3hFLOr7 TR5GP/k/Lp6kZRQsnuow8o3X97atocZ1Gg0S7CVRhxF9sCC+PDl3TXsHlCw5ZYr214ny d39+ustjfCO8YQY2hsjzo1g621AuJAVzgi+iHCt05Y95N7+cvHBoiwfmQS5qyExSnd6U VNNSXIdBmX7tr2OAhm0mMmhtrkIqePNqGUavbKyC7caHTGUlbxS42DsK58B8keW2wtLJ 66lBN8ZAkgIhKatEKaGFAMpbsGtj/WH3lUykXJ8SFQGCrlYHkigOufghId9ptnzODJmi c9Cg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=/KfoISsApaN0NSURJOvvfJb6dwcjjrFUhXMB/UsO5u8=; b=I7oqEwoxROHhxLkRv3gwISGRcEGYXScHZjshpoVYvXpbmS9XfY/uwNUR08UEcztCPz 1iSwfv5P9MUzfBKAu64HTdGEh0DIJ4JVM8j6eXX+8Gi/AYRbiMAXPkZGITgB8ySYou0n 0fUnzr/BZ/H0SQzXUVWAjHu6Euy3qJsoA/xoSuhdKwbHgfPboZonLauj9YuMU5hf7g/f EsJFQBjFvrm0tHUbBTG2JfiA1NFnef7vnVgaPddGVGuGzwNDXSjinMA8ISUHzPIObmZG /rtiR57UQnCHs5cgOQfn8/swKquf65mtjDNFftzJNegp9b6cqwQzzpPbY2xGWJsqGrWp L3HQ== X-Gm-Message-State: AOAM530jO+js8QqjaBtmcl/KkWvHIGGqgT0Kit7UPenFWiKF8AYcK70u KalSAZM34++BPmEy8m5sdUcV0w== X-Received: by 2002:a17:906:1150:: with SMTP id i16mr96740eja.82.1603116524907; Mon, 19 Oct 2020 07:08:44 -0700 (PDT) Received: from myrica ([2001:1715:4e26:a7e0:116c:c27a:3e7f:5eaf]) by smtp.gmail.com with ESMTPSA id f25sm10868868edy.52.2020.10.19.07.08.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 19 Oct 2020 07:08:44 -0700 (PDT) Date: Mon, 19 Oct 2020 16:08:24 +0200 From: Jean-Philippe Brucker To: "Raj, Ashok" Cc: dwmw2@infradead.org, baolu.lu@linux.intel.com, joro@8bytes.org, zhangfei.gao@linaro.org, wangzhou1@hisilicon.com, arnd@arndb.de, gregkh@linuxfoundation.org, iommu@lists.linux-foundation.org, linux-kernel@vger.kernel.org, linux-accelerators@lists.ozlabs.org, kevin.tian@intel.com, jacob.jun.pan@linux.intel.com, linux-pci@vger.kernel.org, "Lu, Baolu" , Jacon Jun Pan Subject: Re: [RFC PATCH 0/2] iommu: Avoid unnecessary PRI queue flushes Message-ID: <20201019140824.GA1478235@myrica> References: <20201015090028.1278108-1-jean-philippe@linaro.org> <20201015182211.GA54780@otc-nc-03> <20201016075923.GB1309464@myrica> <20201017112525.GA47206@otc-nc-03> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20201017112525.GA47206@otc-nc-03> Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Sat, Oct 17, 2020 at 04:25:25AM -0700, Raj, Ashok wrote: > > For devices that *don't* use a stop marker, the PCIe spec says (10.4.1.2): > > > > To stop [using a PASID] without using a Stop Marker Message, the > > function shall: > > 1. Stop queueing new Page Request Messages for this PASID. > > The device driver would need to tell stop sending any new PR's. > > > 2. Finish transmitting any multi-page Page Request Messages for this > > PASID (i.e. send the Page Request Message with the L bit Set). > > 3. Wait for PRG Response Messages associated any outstanding Page > > Request Messages for the PASID. > > > > So they have to flush their PR themselves. And since the device driver > > completes this sequence before calling unbind(), then there shouldn't be > > any oustanding PR for the PASID, and unbind() doesn't need to flush, > > right? > > I can see how the device can complete #2,3 above. But the device driver > isn't the one managing page-responses right. So in order for the device to > know the above sequence is complete, it would need to get some assist from > IOMMU driver? No the device driver just waits for the device to indicate that it has completed the sequence. That's what the magic stop-PASID mechanism described by PCIe does. In 6.20.1 "Managing PASID TLP Prefix Usage" it says: "A Function must have a mechanism to request that it gracefully stop using a specific PASID. This mechanism is device specific but must satisfy the following rules: [...] * When the stop request mechanism indicates completion, the Function has: [...] * Complied with additional rules described in Address Translation Services (Chapter 10 [10.4.1.2 quoted above]) if Address Translations or Page Requests were issued on the behalf of this PASID." So after the device driver initiates this mechanism in the device, the device must be able to indicate completion of the mechanism, which includes completing all in-flight Page Requests. At that point the device driver can call unbind() knowing there is no pending PR for this PASID. Thanks, Jean > > How does the driver know that everything host received has been responded > back to device? > > > > > > I'm not sure about other IOMMU's how they behave, When there is no space in > > > the PRQ, IOMMU auto-responds to the device. This puts the device in a > > > while (1) loop. The fake successful response will let the device do a ATS > > > lookup, and that would fail forcing the device to do another PRQ. > > > > But in the sequence above, step 1 should ensure that the device will not > > send another PR for any successful response coming back at step 3. > > True, but there could be some page-request in flight on its way to the > IOMMU. By draining and getting that round trip back to IOMMU we gaurantee > things in flight are flushed to PRQ after that Drain completes. > > > > So I agree with the below if we suspect there could be pending PR, but > > given that pending PR are a stop marker thing and we don't know any device > > using stop markers, I wondered why I bothered implementing PRIq flush at > > all for SMMUv3, hence this RFC. > > > > Cheers, > Ashok