Received: by 2002:a05:6a10:6744:0:0:0:0 with SMTP id w4csp708160pxu; Fri, 23 Oct 2020 11:14:45 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxxXHxZYBuFZdZTee0zW7ALjwpL8RqnRtyTo+MHD50Zcn0JZQotEIgan/cUOPwKMVSocgbd X-Received: by 2002:a17:906:cc8b:: with SMTP id oq11mr3446100ejb.116.1603476884866; Fri, 23 Oct 2020 11:14:44 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1603476884; cv=none; d=google.com; s=arc-20160816; b=hL+qkf0O5n9O4517XjCoBgPe4VSdGvl+DvczDui6PGdMKTjb2rkeBZ5ZdZkJxfos45 kSnrMzexd3tMNyi2qXq1qcI1t9DdtjJTWIFpeQUPJ76NEtrOgMQBndyLuQIasSyKawdC 4HBuTPcQLLC75di0GIluR2XgojThteVYU6MHg2uRWV5qxp7R/7XyAX3XezjUepbMY3kX cB8qLX0+CGVbu8V+PhM4gjJYGahXH9d/DIMUjEPimonITv8OtD7hh40W0DMhmgqAL7xg Ig3gKRkSNHqF0HqQZGZ8QT6zpUHAKwisxXkPfRiSoarJAsDB4NotkZnRU4VLAnphwBSI 9mHA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:in-reply-to:content-disposition:mime-version :references:message-id:subject:cc:to:from:date:dkim-signature; bh=ybxYfhP0qOoEv/Z3igcvaiaTXfddpdf7DoEgMqfdT7A=; b=OkfNbqJfKZGwnhTR+X60epGLadtcsTBh7YP8Zyi/RIawZQTekOagM/+ApUBApqDEky M7ExjwH4QXCuCK3MkmbfSOkMIFHo1MTxoDz+nv/7gyUkhf5eKhXUdQEYMZClSBTM0Pk+ Md4muzHfIo+HLMHENKH+l9PIIEwNQKcZTYoMoDv+c9rZc/6dLGXo1tfwezdo3mnk67s/ Ry6Ch2TodqM4jpWMBxZUuhidvvlX0x+ccSsd+g2XNzn7fnbncW90omOQpl7CqEmbnPJ4 zmQpFk5Z0meLih4z4D6tLc9Ap/bVVspg9v4/IwPOwkrd0wn+p5Nlr4Tt5VhntLyFyd/a YRLA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=bNaAR+Oj; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id m18si1329581ejb.604.2020.10.23.11.14.22; Fri, 23 Oct 2020 11:14:44 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=bNaAR+Oj; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S465038AbgJWNew (ORCPT + 99 others); Fri, 23 Oct 2020 09:34:52 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49310 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S464934AbgJWNeq (ORCPT ); Fri, 23 Oct 2020 09:34:46 -0400 Received: from mail-ej1-x644.google.com (mail-ej1-x644.google.com [IPv6:2a00:1450:4864:20::644]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C3EDBC0613D2 for ; Fri, 23 Oct 2020 06:34:45 -0700 (PDT) Received: by mail-ej1-x644.google.com with SMTP id gs25so2442181ejb.1 for ; Fri, 23 Oct 2020 06:34:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=ybxYfhP0qOoEv/Z3igcvaiaTXfddpdf7DoEgMqfdT7A=; b=bNaAR+OjYmeeEzFzzZBJyzDKMbF4do7AhpOBiv9gsSMgNMqyq5/l4Ywkmo2cc+ESzz lH05Ok9PCaR+wHbV3UtzXlRRA8JfrhwHMLbb0Zj4NWXmQ4eNylNbUcFPFq5iZJZ3HE4A 4A9t44PEBK+fF9Uz9ustwTwn9OxP5fS7OhuPeCnuRonbtMuE6/l3mFYgIYjLd95R3VaS dpWDsX5QYk6s/5ms0m/fST6rb3d8AWD5vnd1IhI1k1i/p+mau+CZ/smjsHXHtkBLdSSy daqTJeHfA9fXyZGQ5P3D3zhZtf8qMmTy+xx4KaAg3dJBOtHchBp51fOrGZD//kZ/nWsq 0QAQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=ybxYfhP0qOoEv/Z3igcvaiaTXfddpdf7DoEgMqfdT7A=; b=bqdsJE1rI0afkGq7KapUbCU0tBvLsi7dVaiEUGuTeSzAuAc2AMbEpD6JGPhT4SfGrx iYSQHmEl09TaCHlJPNcku/1V7NgXi3eYFWiOHf1Cy2Uw2LXgiFVk2vvKRVhD/m2Pz50d yQV4PRuBHIfgd0csUEFum/NFlLGWzGGpc3uNRWIpYN9QCXQOx0QK0JhaztIz4uAsT+dO I0mAN/rNDQUy19nuIsQwuDgzMfZdiaSKlF5Q1YhUbcfLGIJJYMyXcviGNkCY2uvJdPn+ VpbINOXCGroOUaHSneivzCxI1h6rrO8orzN+4GWo2QjAlc4BVi8XBLUChPokl2iCcXKd 3S2Q== X-Gm-Message-State: AOAM531Ke3P6LpvJdGReiNoWH0gjuczdxKo2unN45INaPfv9wAR7NJ1f pE5YOTwDizI09lLtFcrS0mZZVA== X-Received: by 2002:a17:906:1418:: with SMTP id p24mr1964123ejc.46.1603460084421; Fri, 23 Oct 2020 06:34:44 -0700 (PDT) Received: from myrica ([2001:1715:4e26:a7e0:116c:c27a:3e7f:5eaf]) by smtp.gmail.com with ESMTPSA id q3sm788808edv.17.2020.10.23.06.34.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 23 Oct 2020 06:34:43 -0700 (PDT) Date: Fri, 23 Oct 2020 15:34:23 +0200 From: Jean-Philippe Brucker To: "Raj, Ashok" Cc: dwmw2@infradead.org, baolu.lu@linux.intel.com, joro@8bytes.org, zhangfei.gao@linaro.org, wangzhou1@hisilicon.com, arnd@arndb.de, gregkh@linuxfoundation.org, iommu@lists.linux-foundation.org, linux-kernel@vger.kernel.org, linux-accelerators@lists.ozlabs.org, kevin.tian@intel.com, jacob.jun.pan@linux.intel.com, linux-pci@vger.kernel.org, "Lu, Baolu" , Jacon Jun Pan Subject: Re: [RFC PATCH 0/2] iommu: Avoid unnecessary PRI queue flushes Message-ID: <20201023133423.GF2265982@myrica> References: <20201015090028.1278108-1-jean-philippe@linaro.org> <20201015182211.GA54780@otc-nc-03> <20201016075923.GB1309464@myrica> <20201017112525.GA47206@otc-nc-03> <20201019140824.GA1478235@myrica> <20201019211608.GA79633@otc-nc-03> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20201019211608.GA79633@otc-nc-03> Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Mon, Oct 19, 2020 at 02:16:08PM -0700, Raj, Ashok wrote: > Hi Jean > > On Mon, Oct 19, 2020 at 04:08:24PM +0200, Jean-Philippe Brucker wrote: > > On Sat, Oct 17, 2020 at 04:25:25AM -0700, Raj, Ashok wrote: > > > > For devices that *don't* use a stop marker, the PCIe spec says (10.4.1.2): > > > > > > > > To stop [using a PASID] without using a Stop Marker Message, the > > > > function shall: > > > > 1. Stop queueing new Page Request Messages for this PASID. > > > > > > The device driver would need to tell stop sending any new PR's. > > > > > > > 2. Finish transmitting any multi-page Page Request Messages for this > > > > PASID (i.e. send the Page Request Message with the L bit Set). > > > > 3. Wait for PRG Response Messages associated any outstanding Page > > > > Request Messages for the PASID. > > > > > > > > So they have to flush their PR themselves. And since the device driver > > > > completes this sequence before calling unbind(), then there shouldn't be > > > > any oustanding PR for the PASID, and unbind() doesn't need to flush, > > > > right? > > > > > > I can see how the device can complete #2,3 above. But the device driver > > > isn't the one managing page-responses right. So in order for the device to > > > know the above sequence is complete, it would need to get some assist from > > > IOMMU driver? > > > > No the device driver just waits for the device to indicate that it has > > completed the sequence. That's what the magic stop-PASID mechanism > > described by PCIe does. In 6.20.1 "Managing PASID TLP Prefix Usage" it > > says: > > The goal is we do this when the device is in a messup up state. So we can't > take for granted the device is properly operating which is why we are going > to wack the device with a flr(). > > The only thing that's supposed to work without a brain freeze is the > invalidation logic. Spec requires devices to respond to invalidations even when > they are in the process of flr(). > > So when IOMMU does an invalidation wait with a Page-Drain, IOMMU waits till > the response for that arrives before completing the descriptor. Due to > the posted semantics it will ensure any PR's issued and in the fabric are flushed > out to memory. > > I suppose you can wait for the device to vouch for all responses, but that > is assuming the device is still functioning properly. Given that we use it > in two places, > > * Reclaiming a PASID - only during a tear down sequence, skipping it > doesn't really buy us much. Yes I was only wondering about normal PASID reclaim operations, in unbind(). Agreed that for FLR we need to properly clean the queue, though I do need to do more thinking about this. Anyway, having a full priq drain in unbind() isn't harmful, just unnecessary delay in my opinion. I'll drop these patches for now but thanks for the discussion. Thanks, Jean > * During FLR this can't be skipped anyway due to the above sequence > requirement. > > > > > "A Function must have a mechanism to request that it gracefully stop using > > a specific PASID. This mechanism is device specific but must satisfy the > > following rules: > > [...] > > * When the stop request mechanism indicates completion, the Function has: > > [...] > > * Complied with additional rules described in Address Translation > > Services (Chapter 10 [10.4.1.2 quoted above]) if Address Translations > > or Page Requests were issued on the behalf of this PASID." > > > > So after the device driver initiates this mechanism in the device, the > > device must be able to indicate completion of the mechanism, which > > includes completing all in-flight Page Requests. At that point the device > > driver can call unbind() knowing there is no pending PR for this PASID. > > > > Cheers, > Ashok