Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Wed, 14 Sep 2022 18:18:18 +0300
From:   "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com>
To:     Ashok Raj <ashok.raj@intel.com>
Cc:     "Kirill A. Shutemov" <kirill@shutemov.name>,
        Ashok Raj <ashok_raj@linux.intel.com>,
        Dave Hansen <dave.hansen@linux.intel.com>,
        Andy Lutomirski <luto@kernel.org>,
        Peter Zijlstra <peterz@infradead.org>, x86@kernel.org,
        Kostya Serebryany <kcc@google.com>,
        Andrey Ryabinin <ryabinin.a.a@gmail.com>,
        Andrey Konovalov <andreyknvl@gmail.com>,
        Alexander Potapenko <glider@google.com>,
        Taras Madan <tarasmadan@google.com>,
        Dmitry Vyukov <dvyukov@google.com>,
        "H . J . Lu" <hjl.tools@gmail.com>,
        Andi Kleen <ak@linux.intel.com>,
        Rick Edgecombe <rick.p.edgecombe@intel.com>,
        linux-mm@kvack.org, linux-kernel@vger.kernel.org,
        Jacon Jun Pan <jacob.jun.pan@intel.com>,
        Jason Gunthorpe <jgg@nvidia.com>,
        Joerg Roedel <joro@8bytes.org>
Subject: Re: [PATCHv8 00/11] Linear Address Masking enabling
Message-ID: <20220914151818.uupzpyd333qnnmlt@box.shutemov.name>
References: <20220830010104.1282-1-kirill.shutemov@linux.intel.com>
 <YxDvpLb77lwb8zaT@araj-dh-work>
 <20220904003952.fheisiloilxh3mpo@box.shutemov.name>
 <20220912224930.ukakmmwumchyacqc@box.shutemov.name>
 <20220914144518.46rhhyh7zmxieozs@box.shutemov.name>
 <YyHvF2K7ELVSTGvB@araj-MOBL2.amr.corp.intel.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <YyHvF2K7ELVSTGvB@araj-MOBL2.amr.corp.intel.com>
Precedence: bulk

On Wed, Sep 14, 2022 at 08:11:19AM -0700, Ashok Raj wrote:
> On Wed, Sep 14, 2022 at 05:45:18PM +0300, Kirill A. Shutemov wrote:
> > On Tue, Sep 13, 2022 at 01:49:30AM +0300, Kirill A. Shutemov wrote:
> > > On Sun, Sep 04, 2022 at 03:39:52AM +0300, Kirill A. Shutemov wrote:
> > > > On Thu, Sep 01, 2022 at 05:45:08PM +0000, Ashok Raj wrote:
> > > > > Hi Kirill,
> > > > > 
> > > > > On Tue, Aug 30, 2022 at 04:00:53AM +0300, Kirill A. Shutemov wrote:
> > > > > > Linear Address Masking[1] (LAM) modifies the checking that is applied to
> > > > > > 64-bit linear addresses, allowing software to use of the untranslated
> > > > > > address bits for metadata.
> > > > > 
> > > > > We discussed this internally, but didn't bubble up here.
> > > > > 
> > > > > Given that we are working on enabling Shared Virtual Addressing (SVA)
> > > > > within the IOMMU. This permits user to share VA directly with the device,
> > > > > and the device can participate even in fixing page-faults and such.
> > > > > 
> > > > > IOMMU enforces canonical addressing, since we are hijacking the top order
> > > > > bits for meta-data, it will fail sanity check and we would return a failure
> > > > > back to device on any page-faults from device. 
> > > > > 
> > > > > It also complicates how device TLB and ATS work, and needs some major
> > > > > improvements to detect device capability to accept tagged pointers, adjust
> > > > > the devtlb to act accordingly. 
> > > > > 
> > > > > 
> > > > > Both are orthogonal features, but there is an intersection of both
> > > > > that are fundamentally incompatible.
> > > > > 
> > > > > Its even more important, since an application might be using SVA under the
> > > > > cover provided by some library that's used without their knowledge.
> > > > > 
> > > > > The path would be:
> > > > > 
> > > > > 1. Ensure both LAM and SVM are incompatible by design, without major
> > > > >    changes.
> > > > >    	- If LAM is enabled already and later SVM enabling is requested by
> > > > > 	  user, that should fail. and Vice versa.
> > > > > 	- Provide an API to user to ask for opt-out. Now they know they
> > > > > 	  must sanitize the pointers before sending to device, or the
> > > > > 	  working set is already isolated and needs no work.
> > > > 
> > > > The patch below implements something like this. It is PoC, build-tested only.
> > > > 
> > > > To be honest, I hate it. It is clearly a layering violation. It feels
> > > > dirty. But I don't see any better way as we tie orthogonal features
> > > > together.
> > > > 
> > > > Also I have no idea how to make forced PASID allocation if LAM enabled.
> > > > What the API has to look like?
> > > 
> > > Jacob, Ashok, any comment on this part?
> > > 
> > > I expect in many cases LAM will be enabled very early (like before malloc
> > > is functinal) in process start and it makes PASID allocation always fail.
> > > 
> > > Any way out?
> > 
> > We need closure on this to proceed. Any clue?
> 
> Failing PASID allocation seems like the right thing to do here. If the
> application is explicitly allocating PASID's it can opt-out using the
> similar mechanism you have for LAM enabling. So user takes
> responsibility for sanitizing pointers. 
> 
> If some library is using an accelerator without application knowledge,
> that would use the failure as a mechanism to use an alternate path if
> one exists.
> 
> I don't know if both LAM and SVM need a separate forced opt-in (or i
> don't have an opinion rather). Is this what you were asking? 
> 
> + Joerg, JasonG in case they have an opinion.

My point is that the patch provides a way to override LAM vs. PASID mutual
exclusion, but only if PASID allocated first. If we enabled LAM before
PASID is allcoated there's no way to forcefully allocate PASID, bypassing
LAM check. I think there should be one, no?

-- 
  Kiryl Shutsemau / Kirill A. Shutemov