Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) client-ip=23.128.96.32;
From:   "Tian, Kevin" <kevin.tian@intel.com>
To:     Nicolin Chen <nicolinc@nvidia.com>
CC:     "Liu, Yi L" <yi.l.liu@intel.com>,
        "joro@8bytes.org" <joro@8bytes.org>,
        "alex.williamson@redhat.com" <alex.williamson@redhat.com>,
        Jason Gunthorpe <jgg@nvidia.com>,
        "robin.murphy@arm.com" <robin.murphy@arm.com>,
        "baolu.lu@linux.intel.com" <baolu.lu@linux.intel.com>,
        "cohuck@redhat.com" <cohuck@redhat.com>,
        "eric.auger@redhat.com" <eric.auger@redhat.com>,
        "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
        "mjrosato@linux.ibm.com" <mjrosato@linux.ibm.com>,
        "chao.p.peng@linux.intel.com" <chao.p.peng@linux.intel.com>,
        "yi.y.sun@linux.intel.com" <yi.y.sun@linux.intel.com>,
        "peterx@redhat.com" <peterx@redhat.com>,
        "jasowang@redhat.com" <jasowang@redhat.com>,
        "shameerali.kolothum.thodi@huawei.com" 
        <shameerali.kolothum.thodi@huawei.com>,
        "lulu@redhat.com" <lulu@redhat.com>,
        "suravee.suthikulpanit@amd.com" <suravee.suthikulpanit@amd.com>,
        "iommu@lists.linux.dev" <iommu@lists.linux.dev>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "linux-kselftest@vger.kernel.org" <linux-kselftest@vger.kernel.org>,
        "Duan, Zhenzhong" <zhenzhong.duan@intel.com>,
        "Martins, Joao" <joao.m.martins@oracle.com>
Subject: RE: [PATCH v4 08/17] iommufd: Always setup MSI and anforce cc on
 kernel-managed domains
Thread-Topic: [PATCH v4 08/17] iommufd: Always setup MSI and anforce cc on
 kernel-managed domains
Thread-Index: AQHZ7GCNzWxyrCHnVkqPkBvqOzEqrbAsylxggBvMMACAA6lO8A==
Date:   Mon, 16 Oct 2023 08:48:03 +0000
Message-ID: <BN9PR11MB52763227866603ED7795AA068CD7A@BN9PR11MB5276.namprd11.prod.outlook.com>
References: <20230921075138.124099-1-yi.l.liu@intel.com>
 <20230921075138.124099-9-yi.l.liu@intel.com>
 <BN9PR11MB527658EAF017FF3576667EEA8CC3A@BN9PR11MB5276.namprd11.prod.outlook.com>
 <ZSnkiKoCspmG1+/j@Asurada-Nvidia>
In-Reply-To: <ZSnkiKoCspmG1+/j@Asurada-Nvidia>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?kJbYRwptZd7WGm7NJLJlanwWPBGO5LRyy+6RfWHS3I+8jw8SuiPOgok82wEW?=
 =?us-ascii?Q?2P0gzsVNovWZxFidQuhCtzXpSro+faSyp1KTyRETN/Gi8NXfZAs+AjkXBUpe?=
 =?us-ascii?Q?D3adMTmzAHJlXs93bcNn0END97lbNrE/CcTxyPaGYV1uZZtXcG1Mxtj2ypho?=
 =?us-ascii?Q?KEzN92XktL9yZaXms4Qk3whecFaG++DJwBNvDqKwS/xhxTFZeQ90opwcowhH?=
 =?us-ascii?Q?24OSfM9ljBNhCdbDmqZJAuyPorQNLW6CaImfIBKkdH1yqllQm6L5Ie3DIZtS?=
 =?us-ascii?Q?riHLa+C5J0f0Iy4CfM7OiPoN7pTdI9rFlOlnKhkfxFrU55QdFPxVGVUSnQl8?=
 =?us-ascii?Q?Ll6yRbpnC7ZuKjVCm5jbASzcYtVDotYevFnWLAW5t4+TRhwgSkXVdwe24Bda?=
 =?us-ascii?Q?0i1xSuXs2dLxJpqufoigTD+a+kCn3h/wUwVbkvhk1DxYBCul72A0lKqfDQ2Q?=
 =?us-ascii?Q?e5v1tn9eQMsCqp971X622aYRR1BmUZ3cNVQKWVL6xLMI5Bf9DSGTzR/NwOym?=
 =?us-ascii?Q?8cF9BoNe1smIhfTLL++pglzYEb62kEH5Wa1LlkQXLbeXR3zM8U86FJwO+3CQ?=
 =?us-ascii?Q?CrZ8u/uL3GQ+ji5epcbb5TehldR79fGCpgtQvzWrEoGsFZKzbPH4oMetedHG?=
 =?us-ascii?Q?xoWacIyDT5qH9IFyu4jClDU7GL0CvcCC75KdBemshT5vZcFhQrDdR4JAhdHl?=
 =?us-ascii?Q?C32KMOUOpGTZg8DTwPsFFeylo0GjZzRNBhOrpgq9tfiiVQP9Jv5jxzVhw9aY?=
 =?us-ascii?Q?voeqRcQsKhzM1ttlntK4YUeNUdsqN0lpoKp3tjEgFGtGQE2GOIQds0QnEfmt?=
 =?us-ascii?Q?kRcO+Nh/7uA+JMGNmaW6DnFY5lpsh9V5NDioMYMLE2SEdXUIKfnyHUU5Wuli?=
 =?us-ascii?Q?daMV0Z5S7AiIRK1tOZb0APrTRDPz8LEfHr2Gj3Q8MzI7/59+EWdsKRK4Vthm?=
 =?us-ascii?Q?PX7+wUJXvcvF/zBwxWBYXTUxmta80dQKIGMlhDxgvbkoSGTkWVZrfWXZidKJ?=
 =?us-ascii?Q?HIo2PTE44qCUWiIRzj9RUXouVVY+DeADUKY+gUdHfvDFZYTaxwuiMVQlFcra?=
 =?us-ascii?Q?MEipdVfa0LX6U47kgSYFFIwTX1eHe0SWZ9cEC0hHguWjqQHlh3SRXU0S+zHC?=
 =?us-ascii?Q?0MzFimeRITWCT3XRoRS3/fnWUucgLxnoIyDsN+0nzAasR2BTobwbQjnueo8m?=
 =?us-ascii?Q?y6D+1J1pnCw+263MISIsOWx0Xq/rsDAOfnU1JuUiab26UMiHj5XG8Wl7yxdx?=
 =?us-ascii?Q?Br057zvXf9ohT4kE/Dcux85lGQauZvEYBplPMoOZo3MSc7Njo2F+YFURgmJF?=
 =?us-ascii?Q?c4wJasP277ygSISk5xbcuy94HNKtIrjkTvqc156w/qPurjTOy1ZsQPyQr/VY?=
 =?us-ascii?Q?sIVpjeA3hMzcHQYjSuv3UV5BH97UM0XcUGynYFFv8kmFgDOAYkUuZctb1KdK?=
 =?us-ascii?Q?Dx0nKJVpziId2aoczA0m67YSEXECcCDWaAbCW/XJ8N8RQBomT7Cb0QiKiOV/?=
 =?us-ascii?Q?+xwNGtQMcIMRjKuZpzC34zYYv8KK5ep3eUSPxIFwr31lVieddHRCJ58H/Gf+?=
 =?us-ascii?Q?+LF6uU2CGcZGOsaS9dy2qAzF+KTIqqtCefoQYHJm?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN9PR11MB5276.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: f4bb1ef6-c223-481f-ae1a-08dbce249c0b
X-MS-Exchange-CrossTenant-originalarrivaltime: 16 Oct 2023 08:48:03.4715
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: u7KivQLoLjz7pLYIiT+7f6+3xnh8laDT9VMcqCJXnq/XZFxhZha98nb4RvQTzmfhlio2P0V3w1Xnf3gWRSBZrQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW4PR11MB6862
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable
	autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on agentk.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (agentk.vger.email [0.0.0.0]); Mon, 16 Oct 2023 01:49:03 -0700 (PDT)

> From: Nicolin Chen <nicolinc@nvidia.com>
> Sent: Saturday, October 14, 2023 8:45 AM
>=20
> On Tue, Sep 26, 2023 at 01:16:35AM -0700, Tian, Kevin wrote:
> > > From: Liu, Yi L <yi.l.liu@intel.com>
> > > Sent: Thursday, September 21, 2023 3:51 PM
> > >
> > > From: Nicolin Chen <nicolinc@nvidia.com>
> > >
> > > Now enforce_cache_coherency and msi_cookie are kernel-managed hwpt
> > > things.
> > > So, they should be only setup on kernel-managed domains. If the
> attaching
> > > domain is a user-managed domain, redirect the hwpt to hwpt->parent to
> do
> > > it correctly.
> > >
> >
> > No redirection. The parent should already have the configuration done
> > when it's created. It shouldn't be triggered in the nesting path.
>=20
> iommufd_hw_pagetable_enforce_cc() is not only called in alloc(),
> but also in hwpt_attach/replace() if cc is not enforced by the
> alloc() because the idev that initiates the hwpt_alloc() might
> not have idev->enforce_cache_coherency. Only when another idev
> that has idev->enforce_cache_coherency attaches to the shared
> hwpt, the cc configuration would be done.
>=20

is this a bug already? If the 1st device doesn't have enforce_cc in its
iommu, setting the snp bit in the hwpt would lead to reserved
bit violation.

another problem is that intel_iommu_enforce_cache_coherency()
doesn't update existing entries. It only sets a domain flag to affect
future mappings. so it means the 2nd idev is also broken.

The simplest option is to follow vfio type1 i.e. don't mix devices
with different enforce_cc in one domain.