Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Fri, 23 Sep 2022 08:28:36 -0700
From:   Ashok Raj <ashok.raj@intel.com>
To:     Dave Hansen <dave.hansen@intel.com>
CC:     Jason Gunthorpe <jgg@nvidia.com>,
        "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com>,
        Jacob Pan <jacob.jun.pan@intel.com>,
        "Kirill A. Shutemov" <kirill@shutemov.name>,
        Ashok Raj <ashok_raj@linux.intel.com>,
        Dave Hansen <dave.hansen@linux.intel.com>,
        "Andy Lutomirski" <luto@kernel.org>,
        Peter Zijlstra <peterz@infradead.org>, <x86@kernel.org>,
        Kostya Serebryany <kcc@google.com>,
        Andrey Ryabinin <ryabinin.a.a@gmail.com>,
        Andrey Konovalov <andreyknvl@gmail.com>,
        "Alexander Potapenko" <glider@google.com>,
        Taras Madan <tarasmadan@google.com>,
        "Dmitry Vyukov" <dvyukov@google.com>,
        "H . J . Lu" <hjl.tools@gmail.com>,
        Andi Kleen <ak@linux.intel.com>,
        Rick Edgecombe <rick.p.edgecombe@intel.com>,
        <linux-mm@kvack.org>, <linux-kernel@vger.kernel.org>,
        Joerg Roedel <joro@8bytes.org>, Ashok Raj <ashok.raj@intel.com>
Subject: Re: [PATCHv8 00/11] Linear Address Masking enabling
Message-ID: <Yy3QpOnsNJQ6MykN@araj-MOBL2.amr.corp.intel.com>
References: <20220915172858.pl62a5w3m5binxrk@box.shutemov.name>
 <562dec4d-a39f-b517-58a3-45f691a2d10a@intel.com>
 <YytFGqv5V4T8lbM8@araj-MOBL2.amr.corp.intel.com>
 <f1aac14e-380c-f76f-84bb-fdff56207d2f@intel.com>
 <YytT1hfRZdb9GYQK@nvidia.com>
 <20220923004239.ma2gfrmoezsff4ro@box.shutemov.name>
 <Yy1D0dLdqTr7ya71@a4bf019067fa.jf.intel.com>
 <20220923093826.kjad4qe3clwybeh6@box.shutemov.name>
 <Yy2cfppqkf85hPT7@nvidia.com>
 <fd1f54b5-010d-da65-3989-d8cf727261c7@intel.com>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <fd1f54b5-010d-da65-3989-d8cf727261c7@intel.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?sNWBEEqCJWFABVlUkqbIGQIN3PP7rbF2KMWA3JnmLXaGzrqt6j2NmIJj/oVS?=
 =?us-ascii?Q?5/unaQUR3ilmRvyfdtThwV1rJP3rK6xLbIIf/IOfPFKjFN1gm22teUjN0aEO?=
 =?us-ascii?Q?GhGEBPzGxPHFr+ildWRRqfWxCY54uk3ZZOX2a+EukWtGsBvRAvA+UCFn8Bpg?=
 =?us-ascii?Q?kk7NFN6b+70Dz5Ff+7SF5ho8uR9M4gh/FqIc+iHmpq4QEQcELFSMbJxEcKzP?=
 =?us-ascii?Q?I45pU1XvPU7/NxAMHsu/TnOc99pLqOPfQ/zfP++xc/EVar43A5qUhbfhsS7s?=
 =?us-ascii?Q?Maw0lxo8SYwd2eUCzxbd7788oYJ6De8hY9jQ5JqDBq1Gubc/pXLKkU+GRe4J?=
 =?us-ascii?Q?cs8xZQkM4+rvolQMpS9Bb3ruOtUxDJqcNU/WAVKWHydq6hDCvPhgbpuVzQq6?=
 =?us-ascii?Q?5h3WzFyOGi44B+Um1qc5Aymn69ESiJq7msBb9bKYNa1CGnolsD/7VO3+Xfqp?=
 =?us-ascii?Q?wMUkYg3pb6DjDn8UdOu4R6qDrIxnrwhEg4hi2nTTlKYZoPheVZ9Fvf9JjKVF?=
 =?us-ascii?Q?q2wrTvjDkYddeTNB+lNOhwsWAlkmOovaGRf5sJcMlIzrb7ClrHQBgPdAiIy7?=
 =?us-ascii?Q?Dc15M0QfF2d/YAOn+C29WBxBgoaRQd+5HJMOjvz4jHS0SB5A52ebi+9vz6oD?=
 =?us-ascii?Q?j6Iyv2YzQOZQcZxciXPPrQHeXtk/SdvHW12A17dnLx2lYJzl1oRcWuv2N0mX?=
 =?us-ascii?Q?iGRQ2jPNzLCTZarHKF5pK+QRpm6BOhW4jj2KiLcffG/BB0pQk48rKvhGn2d1?=
 =?us-ascii?Q?ayllCB+GPyRxbKOBDvDPyvhMjlSOx1hQZB/hwbKhnY1EYaw+ggD/0r2+ZFvh?=
 =?us-ascii?Q?HBt4e0A5MLyWGRM8gxqKGalg1gqgyWbnQC25r8rI+rHXpuVC3xl7386ofbZs?=
 =?us-ascii?Q?EihKUxtKePH8Jzu9iUhE9NhGF1k43H9heSC3C/6dW3+A/7dCX6jEdupTn5rB?=
 =?us-ascii?Q?NCtbSOP1ls0dp66HrOqKWOFRNVh8BaxC/pdBeFb+lYK6WobinVMCbu1lhcZ+?=
 =?us-ascii?Q?QCwBE21QiFh/NHfGzInYs8QoE+B28kOoJMgrl5Os6djTakV1UGqnFGADyTnU?=
 =?us-ascii?Q?/6+oGPXvq2gU65bHMrMDDHjuE96WcYvPtgaqFF16h/GalEgPl1v3ozNkuqwb?=
 =?us-ascii?Q?o9ouNX82OBMO4Cy3+FpBSxKt22fXAAPalBCwo7oPgMPsdjunCKrP7x2g/dvc?=
 =?us-ascii?Q?rQ24B5LsVSKv+PI7y4akpaAXC82AdM7ORIUVzA5I60ElN30y4WcOr4Jp7qUc?=
 =?us-ascii?Q?C6PQALYhQGsO/wHb8Et1AhXZxUPpl65PnCTk6Oq5y0tFye1VYhTjGlDD4xXv?=
 =?us-ascii?Q?8mDwVKKtFzRhaymUUuVKD14db5G4h8WiOMHrbMAOCOs9oUKoVIC2ty++Qls3?=
 =?us-ascii?Q?T9IrIBJvnV1LVKg28GCqrN6c1cN9Np4pThcxY19a7XIkCH0ej8lh66UPo/aC?=
 =?us-ascii?Q?wSChGn5/YpkNXnhrno2Z+x3212LOua95+bmaYAj/yAp/J0lhwGc3cuTq7kLx?=
 =?us-ascii?Q?cXG2Xw0qCO8Wtzv3v7u+CnP4wN3FmZif/Ew5qEwF5T4TROP4g5P08wDYCrhQ?=
 =?us-ascii?Q?f4d3S5CUD8G89Uyzu1D/nmwZap7Xtf7EUsz+OGPc?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 7aab4fa0-43bb-4b3a-37b8-08da9d784b7a
X-MS-Exchange-CrossTenant-AuthSource: SJ1PR11MB6201.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Sep 2022 15:28:41.6002
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: zj9g8URBKQ24v5hEscSwdvys4HR2lgMDGzOx0V8YyuwJxb2XDBmG5JpZAHlNN5l1nbx72r8zVx8mUbrnAKT0/w==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR11MB7272
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,
        SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Fri, Sep 23, 2022 at 07:18:42AM -0700, Dave Hansen wrote:
> On 9/23/22 04:46, Jason Gunthorpe wrote:
> > On Fri, Sep 23, 2022 at 12:38:26PM +0300, Kirill A. Shutemov wrote:
> >>> So I would assume an untagged pointer should just be fine for the IOMMU
> >>> to walk. IOMMU currently wants canonical addresses for VA.
> >> Right. But it means that LAM compatibility can be block on two layers:
> >> IOMMU and device. IOMMU is not the only HW entity that has to be aware of
> >> tagged pointers.
> > Why does a device need to care about this? What do you imagine a
> > device doing with it?
> > 
> > The userspace should program the device with the tagged address, the
> > device should present the tagged address on the bus, the IOMMU should
> > translate the tagged address the same as the CPU by ignoring the upper
> > bits.
> 
> Is this how *every* access works?  Every single device access to the
> address space goes through the IOMMU?
> 
> I thought devices also cached address translation responses from the
> IOMMU and stashed them in their own device-local TLB.  If the device is
> unaware of the tags, then how does device TLB invalidation work?  Would

This is coming a full circle now :-)

Since the device doesn't understand tagging, SVM and tagging aren't
compatible. If you need SVM, you can only send sanitized pointers to the
device period. In fact our page-request even looks for canonical checks
before doing the page-faulting.

> all device TLB flushes be full flushes of the devices TLB?  If something
> tried to use single-address invalidation, it would need to invalidate
> every possible tag alias because the device wouldn't know that the tags
> *are* tags instead of actual virtual addresses.

Once tagging is extended into the PCI SIG, and devices know to work with
them, so will the IOMMU, then they can all play in the same field. Until
then they are isolated, or let SVM only work with untagged VA's.