Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) client-ip=23.128.96.35;
Date:   Mon, 16 Oct 2023 08:57:36 -0300
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     "Tian, Kevin" <kevin.tian@intel.com>
Cc:     Nicolin Chen <nicolinc@nvidia.com>,
        "Liu, Yi L" <yi.l.liu@intel.com>,
        "joro@8bytes.org" <joro@8bytes.org>,
        "alex.williamson@redhat.com" <alex.williamson@redhat.com>,
        "robin.murphy@arm.com" <robin.murphy@arm.com>,
        "baolu.lu@linux.intel.com" <baolu.lu@linux.intel.com>,
        "cohuck@redhat.com" <cohuck@redhat.com>,
        "eric.auger@redhat.com" <eric.auger@redhat.com>,
        "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
        "mjrosato@linux.ibm.com" <mjrosato@linux.ibm.com>,
        "chao.p.peng@linux.intel.com" <chao.p.peng@linux.intel.com>,
        "yi.y.sun@linux.intel.com" <yi.y.sun@linux.intel.com>,
        "peterx@redhat.com" <peterx@redhat.com>,
        "jasowang@redhat.com" <jasowang@redhat.com>,
        "shameerali.kolothum.thodi@huawei.com" 
        <shameerali.kolothum.thodi@huawei.com>,
        "lulu@redhat.com" <lulu@redhat.com>,
        "suravee.suthikulpanit@amd.com" <suravee.suthikulpanit@amd.com>,
        "iommu@lists.linux.dev" <iommu@lists.linux.dev>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "linux-kselftest@vger.kernel.org" <linux-kselftest@vger.kernel.org>,
        "Duan, Zhenzhong" <zhenzhong.duan@intel.com>,
        "Martins, Joao" <joao.m.martins@oracle.com>
Subject: Re: [PATCH v4 08/17] iommufd: Always setup MSI and anforce cc on
 kernel-managed domains
Message-ID: <20231016115736.GP3952@nvidia.com>
References: <20230921075138.124099-1-yi.l.liu@intel.com>
 <20230921075138.124099-9-yi.l.liu@intel.com>
 <BN9PR11MB527658EAF017FF3576667EEA8CC3A@BN9PR11MB5276.namprd11.prod.outlook.com>
 <ZSnkiKoCspmG1+/j@Asurada-Nvidia>
 <BN9PR11MB52763227866603ED7795AA068CD7A@BN9PR11MB5276.namprd11.prod.outlook.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <BN9PR11MB52763227866603ED7795AA068CD7A@BN9PR11MB5276.namprd11.prod.outlook.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?FsrlPBU7f8zhWFz6B0SQm61A1F7df8ShQLCoWqk9OHmaa+dN/RKrbgVtvYLs?=
 =?us-ascii?Q?DkLyFFSYD8mtrue+7zmynnYXBoP08yIBWNYrCb/Fp0/0gpwEQjyzs8tqV8oV?=
 =?us-ascii?Q?Z474Fh11rPa9PLHrMxTK/NyH8XSacY34IfaktB5ZCDlXz0fTxlXlBNMFXgY/?=
 =?us-ascii?Q?JE6cG3T7bRq+aHY69o3kW7uYv/P2hqPSQ7lykRc+pEYNwLgjp0QduuihNvdM?=
 =?us-ascii?Q?DKaZxaj+UkgF7G9aOL8GQOFS7uFkJWH8o/Vz4/K2VTVxDBa/qzLTnhVckv21?=
 =?us-ascii?Q?omrETqK4y08RGSGYu61enGivQ6cOtlXxzhfuxzj+DoQul2twjCfyJvUD9Xl2?=
 =?us-ascii?Q?fRrvP0uruvSrP/BoJzOiieftnbDRxcDi1n+Q2BeK9f8oBiQohWQ4iRFK6mSI?=
 =?us-ascii?Q?TgJPy5QzibfLde1nXiqo2Bt/GuwCXEsP6Ib0J4k9CfWtwI042c1UhyVH/DdW?=
 =?us-ascii?Q?X2BmXQ3U4c6e/z2ezqSTmEsXF92/uK992GtREoRxmTecVgA0/9r2ilQVZb3s?=
 =?us-ascii?Q?RCl0WASiy8/87kfIz84d2OSoc+z1vBGRpq41H4xyqBlU56UF1+X+fP6irSIi?=
 =?us-ascii?Q?SzfD/9DaSSI/wdYvmaRRnYRWpw3WXOPeIW+DC6wXfWf+4UnAk4JH7eBG6KR7?=
 =?us-ascii?Q?uehEXeKXoXKTxiajMNcp5Nl81uTeuyuvtYtj85i702h1GMioNoAr+09UEMGE?=
 =?us-ascii?Q?/xbKE0hXbXj5WVjG85aAJ8oRZdVw7jrkb9wHuQDA54h8f9ZxP36Y98x/xsn9?=
 =?us-ascii?Q?L/Rj27irkCgOrSeiCaPgaKW2m3w0tlFvTAlWs7vStjvh9kR667KmN0gHUq1c?=
 =?us-ascii?Q?u3NKF28sNxyKbDNzLeXG8LFsjOpnH0NsOqLwbe9Y4+mZE4WlO53wWmUgSrnm?=
 =?us-ascii?Q?Th8oYhjWU0MmVE7P1ZlQqVgXAD56gxqxhlUeuvGJuIKUFdLZqSXAseZrmaB/?=
 =?us-ascii?Q?cqkz3D4NQPKrhN0R8vrS/GFJjaU1C84tX+qz6NF9RXpsyJT2LZC8rPuIWyKG?=
 =?us-ascii?Q?KHw7ZJ84l48Vxtn/FjgiTjr2qouE9SonRYI6/ITpkKup4fZ1amfPtkjAGfjE?=
 =?us-ascii?Q?PGMJPrqa6nFwWL5J00ToKDt6Nrk1KfzuNLWS67NoLUhY4AbXP2WDRvSPDeDh?=
 =?us-ascii?Q?srvPEjsVjJwyxC+wR9eW/Il+oVsfNobxHTP/cFiRktlybrIkPF88Txo6AbHj?=
 =?us-ascii?Q?d5xOqSUg5UX99ta5McbLTLdRDkGbRz3nA6R31TIt2jZWxa26Vf81e1fexI2k?=
 =?us-ascii?Q?0d4oAnQ9iR6Np50cLoHTO1slPJZKjuUE+Z7CuQwsVebGSJAV6v76gdLWQymx?=
 =?us-ascii?Q?uwspDTe6b/3xgJPjhgjqVNAHKlsTUOWeZ6//WRsheO+uwE5vjfLFIhVOQIXD?=
 =?us-ascii?Q?xOY75YXFjYORG/CEWf0U6Gch5Vt2xDtyNxpGBrlhvcNxxaRm1T5alNFHxeQ4?=
 =?us-ascii?Q?1NQuIbsZUvLc5XUbgnDseAvW4Uv4jIOynaH9vgydn6lFcDW0YtgTc6pKkOq5?=
 =?us-ascii?Q?VoYI+meDBLanJqW5a/tiP/aGbATiYMXQAuUvXaTBVeppwaLbv21o1+PXk+TC?=
 =?us-ascii?Q?fW22IIZyA73YmXAqMcE=3D?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 61a481a6-1720-4908-598e-08dbce3f178f
X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Oct 2023 11:57:38.1317
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: YbtT0U1cuRZAuylEEWagtiiw3vNjfg9ikpj/DYeezdqTJgf0LG41PSPybhD8oAdD
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR12MB7368
X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable
	autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on groat.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (groat.vger.email [0.0.0.0]); Mon, 16 Oct 2023 04:58:00 -0700 (PDT)

On Mon, Oct 16, 2023 at 08:48:03AM +0000, Tian, Kevin wrote:
> > From: Nicolin Chen <nicolinc@nvidia.com>
> > Sent: Saturday, October 14, 2023 8:45 AM
> > 
> > On Tue, Sep 26, 2023 at 01:16:35AM -0700, Tian, Kevin wrote:
> > > > From: Liu, Yi L <yi.l.liu@intel.com>
> > > > Sent: Thursday, September 21, 2023 3:51 PM
> > > >
> > > > From: Nicolin Chen <nicolinc@nvidia.com>
> > > >
> > > > Now enforce_cache_coherency and msi_cookie are kernel-managed hwpt
> > > > things.
> > > > So, they should be only setup on kernel-managed domains. If the
> > attaching
> > > > domain is a user-managed domain, redirect the hwpt to hwpt->parent to
> > do
> > > > it correctly.
> > > >
> > >
> > > No redirection. The parent should already have the configuration done
> > > when it's created. It shouldn't be triggered in the nesting path.
> > 
> > iommufd_hw_pagetable_enforce_cc() is not only called in alloc(),
> > but also in hwpt_attach/replace() if cc is not enforced by the
> > alloc() because the idev that initiates the hwpt_alloc() might
> > not have idev->enforce_cache_coherency. Only when another idev
> > that has idev->enforce_cache_coherency attaches to the shared
> > hwpt, the cc configuration would be done.
> 
> is this a bug already? If the 1st device doesn't have enforce_cc in its
> iommu, setting the snp bit in the hwpt would lead to reserved
> bit violation.

I suspect there are technically some gaps in the intel driver, yes..
 
> another problem is that intel_iommu_enforce_cache_coherency()
> doesn't update existing entries. It only sets a domain flag to affect
> future mappings. so it means the 2nd idev is also broken.

This is such a gap, intel driver should not permit that.

> The simplest option is to follow vfio type1 i.e. don't mix devices
> with different enforce_cc in one domain.

This is why I wanted to get rid of this bad mechanism going forward.

Manually created hwpt should have a manual specification of cc and
then we don't have so many problems.

It means userspace needs to compute if they want to use CC or not, but
userspace already needs to figure this out since without autodomains
it must create two hwpts manually anyhow.

Jason