Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
From:   "Tian, Kevin" <kevin.tian@intel.com>
To:     Jason Gunthorpe <jgg@nvidia.com>
CC:     Alex Williamson <alex.williamson@redhat.com>,
        "Liu, Yi L" <yi.l.liu@intel.com>, "hch@lst.de" <hch@lst.de>,
        "jasowang@redhat.com" <jasowang@redhat.com>,
        "joro@8bytes.org" <joro@8bytes.org>,
        "jean-philippe@linaro.org" <jean-philippe@linaro.org>,
        "parav@mellanox.com" <parav@mellanox.com>,
        "lkml@metux.net" <lkml@metux.net>,
        "pbonzini@redhat.com" <pbonzini@redhat.com>,
        "lushenming@huawei.com" <lushenming@huawei.com>,
        "eric.auger@redhat.com" <eric.auger@redhat.com>,
        "corbet@lwn.net" <corbet@lwn.net>,
        "Raj, Ashok" <ashok.raj@intel.com>,
        "yi.l.liu@linux.intel.com" <yi.l.liu@linux.intel.com>,
        "Tian, Jun J" <jun.j.tian@intel.com>, "Wu, Hao" <hao.wu@intel.com>,
        "Jiang, Dave" <dave.jiang@intel.com>,
        "jacob.jun.pan@linux.intel.com" <jacob.jun.pan@linux.intel.com>,
        "kwankhede@nvidia.com" <kwankhede@nvidia.com>,
        "robin.murphy@arm.com" <robin.murphy@arm.com>,
        "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
        "iommu@lists.linux-foundation.org" <iommu@lists.linux-foundation.org>,
        "dwmw2@infradead.org" <dwmw2@infradead.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "baolu.lu@linux.intel.com" <baolu.lu@linux.intel.com>,
        "david@gibson.dropbear.id.au" <david@gibson.dropbear.id.au>,
        "nicolinc@nvidia.com" <nicolinc@nvidia.com>
Subject: RE: [RFC 10/20] iommu/iommufd: Add IOMMU_DEVICE_GET_INFO
Thread-Topic: [RFC 10/20] iommu/iommufd: Add IOMMU_DEVICE_GET_INFO
Thread-Index: AQHXrSGNbNtRgavabUSKJjvt8l12BauwlhaAgAAouwCAACufAIAAEL4QgACKr4CACtdfoIAA3DaAgBUX8GCAAHivgIAAl4FggAl6ZNCAAXETAIAAHLdQ
Date:   Fri, 22 Oct 2021 03:08:06 +0000
Message-ID: <BN9PR11MB5433482C3754A8A383C3B6298C809@BN9PR11MB5433.namprd11.prod.outlook.com>
References: <20210922234954.GB964074@nvidia.com>
 <BN9PR11MB5433409DF766AAEF1BB2CF258CA39@BN9PR11MB5433.namprd11.prod.outlook.com>
 <BN9PR11MB54333BDB1E58387FD9999DF18CA39@BN9PR11MB5433.namprd11.prod.outlook.com>
 <20210923114219.GG964074@nvidia.com>
 <BN9PR11MB5433519229319BA951CA97638CAA9@BN9PR11MB5433.namprd11.prod.outlook.com>
 <20210930222355.GH964074@nvidia.com>
 <BN9PR11MB5433530032DC8400B71FCB788CB89@BN9PR11MB5433.namprd11.prod.outlook.com>
 <20211014154259.GT2744544@nvidia.com>
 <BN9PR11MB543327BB6D58AEF91AD2C9D18CB99@BN9PR11MB5433.namprd11.prod.outlook.com>
 <BL1PR11MB5429973588E4FBCEC8F519A88CBF9@BL1PR11MB5429.namprd11.prod.outlook.com>
 <20211021233036.GN2744544@nvidia.com>
In-Reply-To: <20211021233036.GN2744544@nvidia.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?6CSnk8iE+/Jr+nUk8twzrin0FYvSE7Lnz5V3CDMVdCVmHuW0j0zFZ1XhCkQY?=
 =?us-ascii?Q?szDluwAlTCZgsLFkLiY+fN2NvWsaQabsctbI+pVMJfzNPw/53Ta+0thX5HEc?=
 =?us-ascii?Q?h3Ljzel72xKYEDhbS/+DvgmNophVJ7W7gXyg4FUG7rDEryHjVFgCYYGnjHPs?=
 =?us-ascii?Q?kPNmbon5ZneIGUSb3dzuMjJMy+kC4348DUmMGW2QjS6ytsgGr26UADYFGaf5?=
 =?us-ascii?Q?HHcFxZWrcY4kSEO0/bj0aeo7AtUoPDP8zbjabmbMwuBWW6lki1XNCb5/4dcD?=
 =?us-ascii?Q?M+vP9u9hzeax/m8KUdZRyON8bCGhPMo5p3aoAxPFml3TWsPEntJOijXFGDz8?=
 =?us-ascii?Q?eT/0/Jmqly3roeZeCRVEL3NapSILx2lVaUmGgE6wExVdWOR+iFaREAdEmxDh?=
 =?us-ascii?Q?zbPQ5X4mSbS6oHPBuCApmWfYAbA4F7Hl6R4LtE1dR4vKJHCg9+RaNcdsg/XL?=
 =?us-ascii?Q?pRAGjPqU8VMS1Jpa4xAOynsZWkDfNEoGHpY5E8X8C0FHBrA9NlQmsYVmbdRt?=
 =?us-ascii?Q?3L4pJmeRoVhrbuQb5L5rfR9y6XrjtfuEfi4rQcdU9thONvMQ/EYuY3vvP4L7?=
 =?us-ascii?Q?HebjVu9i95Z7b9IMFUIdfWrfwsBalXy62EqcAP4QwduwqVj4g6jkJBBjgOZ9?=
 =?us-ascii?Q?SkYjyiXaRjJNzRdwV+leYEBkzXVe1hUrCuFkxDXwa06qbv2TRDMKyGHiRFTA?=
 =?us-ascii?Q?EocU/GN1Gsmf6rjVQbyHlAyMGPGxQ4F1b4HR+9TGTOwdj8mIGpdcEjkrnJXx?=
 =?us-ascii?Q?+kxQi+3z2y84Q0eIf29VXcPUFLerAZx73YH5+2g7zxevQbc4Jf4SDO/lTvrv?=
 =?us-ascii?Q?XvNJ/cparqu/nln4OTi2Sw6f1rMEM7WjgvrqTZesBWwKdT2+tO2ro8Z+XsV1?=
 =?us-ascii?Q?XEXnxY92bVCvwZpe2Q6xE+d/1Y/qHsotOB0/mQ81C+iDrpOE+VZq9VCqCNS0?=
 =?us-ascii?Q?9hPcIZ8mEstyhD3+4hMbQFZbHwviVZiBXr31BkfyYOTqWNJX7TRAvpLly5Mr?=
 =?us-ascii?Q?6TCibpmOXGEv8aPYLLW520rlGMB3ia5VV5N1FyT2kUSiwOZ8AsAs9E3FJIR1?=
 =?us-ascii?Q?Z10Iipex+5xo9x28s7MnKdaH9AaWXGvPtfAYwlT9tqOPDlU1FyGqZZixfIUi?=
 =?us-ascii?Q?3bKvlbN2Aav0o6HtMad5bHyk32L9WzM13g7qhHkcT5sEfjKW8P1/gxG/KEFZ?=
 =?us-ascii?Q?DStq1usHR7NOV8hAjviyNuUykGiEDV0hdtjferKZHvKDoEF2YEbC7aIWIZxR?=
 =?us-ascii?Q?wGYn0kzOrd28l67u8EJkirEveXd5p6aRZ/fxkgCUZd2z2j39jbUJgH8RIYLk?=
 =?us-ascii?Q?oJrfWmbBkniz5YGAV0Wccgw8?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN9PR11MB5433.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 0e8203e1-a662-441c-9cba-08d995092b6f
X-MS-Exchange-CrossTenant-originalarrivaltime: 22 Oct 2021 03:08:06.4878
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: kevin.tian@intel.com
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN6PR11MB1538
X-OriginatorOrg: intel.com
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

> From: Jason Gunthorpe <jgg@nvidia.com>
> Sent: Friday, October 22, 2021 7:31 AM
>=20
> On Thu, Oct 21, 2021 at 02:26:00AM +0000, Tian, Kevin wrote:
>=20
> > But in reality only Intel integrated GPUs have this special no-snoop
> > trick (fixed knowledge), with a dedicated IOMMU which doesn't
> > support enforce-snoop format at all. In this case there is no choice
> > that the user can further make.
>=20
> huh? That is not true at all. no-snoop is a PCIe spec behavior, any
> device can trigger it

yes, I should say Intel GPU 'drivers'.

>=20
> What is true today is that only Intel GPU drivers are crazy enough to
> use it on Linux without platform support.
>=20
> > Also per Christoph's comment no-snoop is not an encouraged
> > usage overall.
>=20
> I wouldn't say that, I think Christoph said using it without API
> support through the DMA layer is very wrong.

ok, sounds like I drew out a wrong impression from previous discussion.

>=20
> DMA layer support could be added if there was interest, all the pieces
> are there to do it.
>=20
> > Given that I wonder whether the current vfio model better suites for
> > this corner case, i.e. just let the kernel to handle instead of
> > exposing it in uAPI. The simple policy (as vfio does) is to
> > automatically set enforce-snoop when the target IOMMU supports it,
> > otherwise enable vfio/kvm contract to handle no-snoop requirement.
>=20
> IMHO you need to model it as the KVM people said - if KVM can execute
> a real wbinvd in a VM then an ioctl shoudl be available to normal
> userspace to run the same instruction.
>=20
> So, figure out some rules to add a wbinvd ioctl to iommufd that makes
> some kind of sense and logically kvm is just triggering that ioctl,
> including whatever security model protects it.

wbinvd instruction is x86 specific. Here we'd want a generic cache=20
invalidation ioctl and then need some form of arch callbacks though x86=20
is the only concerned platform for now.=20

>=20
> I have no idea what security model makes sense for wbinvd, that is the
> major question you have to answer.

wbinvd flushes the entire cache in local cpu. It's more a performance
isolation problem but nothing can prevent it once the user is allowed
to call this ioctl. This is the main reason why wbinvd is a privileged=20
instruction and is emulated by kvm as a nop unless an assigned device
has no-snoop requirement. alternatively the user may call clflush
which is unprivileged and can invalidate a specific cache line, though=20
not efficient for flushing a big buffer.

One tricky thing is that the process might be scheduled to different=20
cpus between writing buffers and calling wbinvd ioctl. Since wbvind=20
only has local behavior, it requires the ioctl to call wbinvd on all
cpus that this process has previously been scheduled on.

kvm maintains a dirty cpu mask in its preempt notifier (see=20
kvm_sched_in/out).

Is there any concern if iommufd also follows the same mechanism?
Currently looks preempt notifier is only  used by kvm. Not sure whether
there is strong criteria around using it. and this local behavior may
not apply to all platforms (then better hidden behind arch callback?)

>=20
> And obviously none of this should be hidden behind a private API to
> KVM.
>=20
> > I don't see any interest in implementing an Intel GPU driver fully
> > in userspace. If just talking about possibility, a separate uAPI can
> > be still introduced to allow the userspace to issue wbinvd as Paolo
> > suggested.
> >
> > One side-effect of doing so is that then we may have to support
> > multiple domains per IOAS when Intel GPU and other devices are
> > attached to the same IOAS.
>=20
> I think we already said the IOAS should represent a single IO page
> table layout?

yes. I was just talking about tradeoff possibility if the aforementioned
option is feasible. Now based on above discussion then we will resume
back to this one-ioas-one-layout model.

>=20
> So if there is a new for incompatible layouts then the IOAS should be
> duplicated.
>=20
> Otherwise, I also think the iommu core code should eventually learn to
> share the io page table across HW instances. Eg ARM has a similar
> efficiency issue if there are multiple SMMU HW blocks.
>=20

or we may introduce an alias ioas concept that any change on one=20
ioas is automatically replayed on the alias ioas if two ioas's are created=
=20
just due to incompatible layout.

Thanks
Kevin