Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Wed, 21 Sep 2022 10:29:29 -0700
From:   Ashok Raj <ashok.raj@intel.com>
To:     Dave Hansen <dave.hansen@intel.com>
CC:     "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com>,
        Jacob Pan <jacob.jun.pan@intel.com>,
        "Kirill A. Shutemov" <kirill@shutemov.name>,
        Ashok Raj <ashok_raj@linux.intel.com>,
        Dave Hansen <dave.hansen@linux.intel.com>,
        Andy Lutomirski <luto@kernel.org>,
        "Peter Zijlstra" <peterz@infradead.org>, <x86@kernel.org>,
        Kostya Serebryany <kcc@google.com>,
        Andrey Ryabinin <ryabinin.a.a@gmail.com>,
        Andrey Konovalov <andreyknvl@gmail.com>,
        Alexander Potapenko <glider@google.com>,
        "Taras Madan" <tarasmadan@google.com>,
        Dmitry Vyukov <dvyukov@google.com>,
        "H . J . Lu" <hjl.tools@gmail.com>,
        Andi Kleen <ak@linux.intel.com>,
        Rick Edgecombe <rick.p.edgecombe@intel.com>,
        <linux-mm@kvack.org>, <linux-kernel@vger.kernel.org>,
        Jason Gunthorpe <jgg@nvidia.com>,
        "Joerg Roedel" <joro@8bytes.org>, Ashok Raj <ashok.raj@intel.com>
Subject: Re: [PATCHv8 00/11] Linear Address Masking enabling
Message-ID: <YytJ+VbOrZ/gaWmS@araj-MOBL2.amr.corp.intel.com>
References: <YyHvF2K7ELVSTGvB@araj-MOBL2.amr.corp.intel.com>
 <20220914151818.uupzpyd333qnnmlt@box.shutemov.name>
 <YyHz7H0uyqG58b3E@araj-MOBL2.amr.corp.intel.com>
 <20220914154532.mmxfsr7eadgnxt3s@box.shutemov.name>
 <20220914165116.24f82d74@jacob-builder>
 <20220915090135.fpeokbokkdljv7rw@box.shutemov.name>
 <20220915172858.pl62a5w3m5binxrk@box.shutemov.name>
 <562dec4d-a39f-b517-58a3-45f691a2d10a@intel.com>
 <YytFGqv5V4T8lbM8@araj-MOBL2.amr.corp.intel.com>
 <f1aac14e-380c-f76f-84bb-fdff56207d2f@intel.com>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <f1aac14e-380c-f76f-84bb-fdff56207d2f@intel.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?nHx4o9Wkw9ImoZw1FLCKN3rIqaeDR7Nrs1aREu1oQKIIslZp8p7vPYONfzy4?=
 =?us-ascii?Q?zB675wFX4dt+xbPRwot6Vo5TwKayezsvbfC/Kt4Wykb8xxdIXsdTGxrWqdKu?=
 =?us-ascii?Q?fQmKdkoEiS8Njy0+4tSYoILdgW4BXC3/UX261ca4oJHt+E9vn4nGWCq6/afY?=
 =?us-ascii?Q?Yj4VCp6ytppVBiZSl6QEZ7EMi8h2IsAQ2I648G+xT3P5grEGyrUt9z4Q/DM5?=
 =?us-ascii?Q?7TmbCyTN6X9aPe76qPiZiFtRLF8dACMWF21Sw2DJdJPNyxFsCPu/rpQb8o3/?=
 =?us-ascii?Q?Q4c7+eZ4z7b+lEAMnOHbfHaDda837MO0mVEN1L48y1m3cxmZ5e7slW1s+EmX?=
 =?us-ascii?Q?7pnSujWg3rIH2TF6OqOJe6jXbHJxmn1ZoKkgu08xtBNycmz7jGDYux/TM+Mk?=
 =?us-ascii?Q?mQ/fQ8IyGAUDAGTs7sodKONVpSeKrdyUFXU2BaayoRE4ToYyluONaaLSPmz7?=
 =?us-ascii?Q?8qAiNs0R3lYhvYAgPYBPJoT86Fifj28FpVrHaJ+l+IhOBfQ/79GeXngIigxf?=
 =?us-ascii?Q?/pf8wO6HyCjAkc0qRbX4ixz3DykkYsv8h3hF7f+iSNvSFt0Vi2/b1tr3XGOH?=
 =?us-ascii?Q?AWeTgK9XOH0UpTNBLSCQ8eC3XjxxNpR6uqM3Cwb9cvovfwPNTu+ZzIeisU1O?=
 =?us-ascii?Q?/sXIhC1pAi8MvCUleuJIr6mFZZlPczfxdPfA/MzhqWRwJwPUfFgSJ/wAWYwg?=
 =?us-ascii?Q?8jrYXfpqzxIJe5UWM0UrIGhF/JR3z55qWSiFPCpl2YYthM/s4GyDjq8xf6Hk?=
 =?us-ascii?Q?JjTcnssHMeFh50IgQNueK84iyGE1wyR2eRMsVwEOlfkWBJFVzCfzhqcPp8AI?=
 =?us-ascii?Q?S/1FENgX9bvX6gU9vYNmGww1F2QR3tWBQ6JQL5nw7xTHtXl5FynHYEdNWT0L?=
 =?us-ascii?Q?f9C2sNrsauNP6XYybp0LItVakbDlTVZqdVBa9NOd/IY9KfOauTiGiiSudnWk?=
 =?us-ascii?Q?sn1CQhHwWUUVlN44y0xrQXcfCoqTE+6rADs5GMx5jqkTyH7fJ3xN5wins8jJ?=
 =?us-ascii?Q?kIK33jkT9AETlAmpPZ4xmswvW8Yy7fRewfjtZsV+X4374cGw6ZFsjTmgj9Sd?=
 =?us-ascii?Q?Anbw33ozZ3TqModze4h7fiHNN0iPwQkBXZ72OTNv2vAYMwt9pq59YT9gwCzY?=
 =?us-ascii?Q?YTrH+0jj+mPwv0yaK/gyBz+3ifQ+aW6HsTDUIjyTXE/cXDTr+V8NNtrwR40i?=
 =?us-ascii?Q?Yz+M/yhNI+EwprLnLo2KsLniAx5rsLKtv+PXmhCypgwazjS26mWSUghgvIVM?=
 =?us-ascii?Q?AKYaSvvsQG16yav52Pi2LOnQY/SYiQrO2/wU0H5oUi44CDNX8RTfBhltPu1f?=
 =?us-ascii?Q?5FEL2niJDDzCW5dw+TNys9fig5pF5sl9UentdBQBI+/PZe+aa7tH4YjATjEm?=
 =?us-ascii?Q?vMyWmb9PzO0lRrKUljWFehC//XPjr/Z5hzTwISZ1+lxGgtvvukqiucnFHmOR?=
 =?us-ascii?Q?JRtr4TQOdQbxJEam2NCMfML0GhxgdM32QnauV20JJ1x+kvox3/rFBZRduuSy?=
 =?us-ascii?Q?TgVnlt6S8vasEBj0lURdyj4aJu4PB4YYno/kEQ61dBmEwGhr//Boc3t55kTR?=
 =?us-ascii?Q?azLVo+Apl0I8fWXBApamqo1Hg5fnLkRtrjNi9ZgL?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 0ad6d9b9-9c09-445c-c154-08da9bf6db49
X-MS-Exchange-CrossTenant-AuthSource: SJ1PR11MB6201.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 21 Sep 2022 17:29:37.1525
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: hD5LpcfLjrCbCk4ylJlkDk//MlhNnTcLyNMA1CFsgs5izkVTUz6h0pLdgKH2sJGQ0KrPd6GZO4D0tcbpvkRHAQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR11MB7123
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,
        SPF_HELO_PASS,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Wed, Sep 21, 2022 at 10:11:46AM -0700, Dave Hansen wrote:
> On 9/21/22 10:08, Ashok Raj wrote:
> > On Wed, Sep 21, 2022 at 09:57:47AM -0700, Dave Hansen wrote:
> >> On 9/15/22 10:28, Kirill A. Shutemov wrote:> +	/* Serialize against
> >> address tagging enabling *
> >>> +	if (mmap_write_lock_killable(mm))
> >>> +		return -EINTR;
> >>> +
> >>> +	if (!arch_can_alloc_pasid(mm)) {
> >>> +		mmap_write_unlock(mm);
> >>> +		return -EBUSY;
> >>> +	}
> >> Shouldn't this actually be some kind of *device* check?
> > The device will enable svm only when its capable of it, and performs all
> > the normal capability checks like PASID, ATS etc before enabling it.
> > This is the final step before the mm is hooked up with the IOMMU.
> 
> What does that mean, though?
> 
> Are you saying that any device compatibility with an mm is solely
> determined by the IOMMU in play, so the IOMMU code should host the mm
> compatibility checks?
> 

To check if a device supports SVM like capabilities it needs to support
the following PCIe capabilities.

- PASID
- Page Request Interface (PRI) for dynamic page-faulting
- ATS - For quick VA->PA lookups.

The device purely works only with memory addresses and caches them in
its device TLB after a lookup via ATS. 

When device does ATS, it sends a translation request, and IOMMU will
walk the page-tables to give the PA back. It can use it until it gets an
invalidation. 

So the device doesn't need to know page-table formats. but if you use
tagged pointers its something you want to check device support for it. I
don't think there is any plans right now to support something like the
following.

- Check device ability to work with tagged pointers.
- OS should configure the width to ignore etc
- Device TLB's properly handle the tagged portion without creating
  aliasing etc.

In order for LAM and SVM to play nicely you need 

#1 IOMMU support for tagged pointers 
#2 Device ability to handle tagged pointers. 

#2 above is an additional check to perform in addition to PASID,PRI,ATS
   checks we do today.

Cheers,
Ashok