Received-SPF: pass (google.com: domain of linux-kernel+bounces-140030-linux.lists.archive=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99;
From: "Tian, Kevin" <kevin.tian@intel.com>
To: Jacob Pan <jacob.jun.pan@linux.intel.com>, LKML
	<linux-kernel@vger.kernel.org>, X86 Kernel <x86@kernel.org>, Peter Zijlstra
	<peterz@infradead.org>, "iommu@lists.linux.dev" <iommu@lists.linux.dev>,
	Thomas Gleixner <tglx@linutronix.de>, Lu Baolu <baolu.lu@linux.intel.com>,
	"kvm@vger.kernel.org" <kvm@vger.kernel.org>, "Hansen, Dave"
	<dave.hansen@intel.com>, Joerg Roedel <joro@8bytes.org>, "H. Peter Anvin"
	<hpa@zytor.com>, Borislav Petkov <bp@alien8.de>, Ingo Molnar
	<mingo@redhat.com>
CC: "Luse, Paul E" <paul.e.luse@intel.com>, "Williams, Dan J"
	<dan.j.williams@intel.com>, Jens Axboe <axboe@kernel.dk>, "Raj, Ashok"
	<ashok.raj@intel.com>, "maz@kernel.org" <maz@kernel.org>, "seanjc@google.com"
	<seanjc@google.com>, Robin Murphy <robin.murphy@arm.com>,
	"jim.harris@samsung.com" <jim.harris@samsung.com>, "a.manzanares@samsung.com"
	<a.manzanares@samsung.com>, Bjorn Helgaas <helgaas@kernel.org>, "Zeng, Guang"
	<guang.zeng@intel.com>, "robert.hoo.linux@gmail.com"
	<robert.hoo.linux@gmail.com>
Subject: RE: [PATCH v2 08/13] x86/irq: Install posted MSI notification handler
Thread-Topic: [PATCH v2 08/13] x86/irq: Install posted MSI notification
 handler
Thread-Index: AQHah6hfZiCSRy/rT0igaUfC7YsDdrFiuYWA
Date: Thu, 11 Apr 2024 07:52:28 +0000
Message-ID: <BN9PR11MB5276C4932F6CFD217CC50AD78C052@BN9PR11MB5276.namprd11.prod.outlook.com>
References: <20240405223110.1609888-1-jacob.jun.pan@linux.intel.com>
 <20240405223110.1609888-9-jacob.jun.pan@linux.intel.com>
In-Reply-To: <20240405223110.1609888-9-jacob.jun.pan@linux.intel.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?6+FjnbMCa6UoRUR8XQfSH+u1nFykDLe/ghxthzn5EKq7qm/V06RkJxzOUTQ+?=
 =?us-ascii?Q?cXBRqE/lL0mZ3WSUp61bYZh2ykKRFkx2wr1wDc1IHFVvbrGk3rbZ17XWh+l2?=
 =?us-ascii?Q?19il2onVeYr9R+fnXunVmME5utCAL+PNY6a/kTr3g70P2R+LImkRdl+L6NSb?=
 =?us-ascii?Q?Nw+YHXImex6reSf5tGfQGdDEDvGbNTmJjeIlqeWvH83hwPrFE5DwMxHAs3HK?=
 =?us-ascii?Q?IzEw8Q+HTpY+ez5GWISGCZvltg1KakqzaEj1r5vzmlSa5uXwOt7tIxNjTLI2?=
 =?us-ascii?Q?gEOislJ1wmS99EigD11y2pVv/u6wFUfuKF5xlJo5vrpwJ+J1EIX73QUlwnWP?=
 =?us-ascii?Q?rwlShikXAhwM8z0hr4QbFJcJW/DxZmp9FKFQnhWwZSyzXGcyzUBXkaenVMRz?=
 =?us-ascii?Q?1yHs8nIiA8rv/mu3hrwv6DR6XXDFKLqQpBwgcjrBcI1Yv2IXw1Iid8i0GRWO?=
 =?us-ascii?Q?I1QfTYS5ddg3PljvtXW9mdK/Oi5XoajXRgf+rjxcXgp3nE8mXMpFQryuFgRH?=
 =?us-ascii?Q?x8O8T9x3V/4VDek6DR+R0aL+wRbBLyP5qUxHo5BghKAIPg7KCXTaDNdHuKPQ?=
 =?us-ascii?Q?dvCZWrJ1EbzpcTh2n/6YzSB2bNJDWnjBA68ewHjpRtHYXVM6OYWJXMfcpEk9?=
 =?us-ascii?Q?M1rqesClPTGzdMNNYJgYk3B4keneX2wxCOXlMfwZvGTgN6a/o/xf5YG0HkWy?=
 =?us-ascii?Q?STj4v6lePV9tUYH1DzzxtOiAyyvkB7f74t5e1y+/DVr5hEw7YKUa1CcAcXdV?=
 =?us-ascii?Q?RK2Ig5XOvLHAAc3CLdYXD7/9x8d6pSa8Ua/lrksdGmn02wMPo2bTstGWOSKx?=
 =?us-ascii?Q?pDAsj6FaqJuipgeyCtfDFQ3EckY0I8tl5RQS+3xThfI7F79aRHn/UFWyDlN/?=
 =?us-ascii?Q?ZG4QLs86kc1ipxtN191uzYwi6JqjBcybct1xKv3Y1RtjKesJTkz1kt39qMme?=
 =?us-ascii?Q?Gm7x7txhL6buTMBt1cEoa37dcAQ6dlkARx5Q6JME7XKTAxkXWCALL9wqvR8z?=
 =?us-ascii?Q?H4ck2tr57gqnCDuTqspMKhgRp+rohVqOGhDctaOW7R1vWWWSr1XDBcRvMOr2?=
 =?us-ascii?Q?nOavaUou7g2eSt44iHpuWxWlzeIBVOHoqDDyIa1CqbujDelLlUGDnZ4aeBLn?=
 =?us-ascii?Q?VbTV2p4Ja+qxZYGB3GW7V9AgR2KVtLlRrbx59i0APohYe64zIXLBo5UfhcFV?=
 =?us-ascii?Q?xz1GY5ULlTxluOdVWyADrDyO06XnFxd5Passr852pwHhGc86/NGVXLeYi0+/?=
 =?us-ascii?Q?c/jL1oF66D9KIyXTUI6AlfHtA94Nk18cH6iKXL9g5B4yz4eJxacOvKhz9Fre?=
 =?us-ascii?Q?DC/CZM2X7x0NvrC6vX7kZxvq5agVchtpCXKsyBlsiRl4xZyx5d9cuV7Zb+FO?=
 =?us-ascii?Q?YHPVmkfOtjAp2UiA2jyxAIQfwqc1aR0Av77AaIGVnK0iNfxvwiKolPslDiou?=
 =?us-ascii?Q?iKWWN5iLJHKcf2rXZd0Dqzyx7Q++Yi3/aU1HMt0/5SrVMw/mwNqyLGuMjAy3?=
 =?us-ascii?Q?cQxM2gbJ8lspsNaAWAzFFqVaOgrHIPrhhYLbUkigUjFNrRqW2lmXLp21Vn2A?=
 =?us-ascii?Q?8qByeSc2uTJYGzILAlDEwtVJ4azD3GsjGnm4GtdQ?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk
X-Mailing-List: linux-kernel@vger.kernel.org
List-Id: <linux-kernel.vger.kernel.org>
List-Subscribe: <mailto:linux-kernel+subscribe@vger.kernel.org>
List-Unsubscribe: <mailto:linux-kernel+unsubscribe@vger.kernel.org>
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN9PR11MB5276.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 197fd6fe-3abf-47b6-d986-08dc59fc55d4
X-MS-Exchange-CrossTenant-originalarrivaltime: 11 Apr 2024 07:52:28.5993
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: gUW/szwC1ouo2qhdTe3rkQj5pnAPqFuuyg1zPaZ9F+EbQtvUllLlpIy6uhGFAfPQqkSyJk9MHNDZlKqMCN49MQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW4PR11MB7078
X-OriginatorOrg: intel.com

> From: Jacob Pan <jacob.jun.pan@linux.intel.com>
> Sent: Saturday, April 6, 2024 6:31 AM
>=20
> +
> +/*
> + * De-multiplexing posted interrupts is on the performance path, the cod=
e
> + * below is written to optimize the cache performance based on the
> following
> + * considerations:
> + * 1.Posted interrupt descriptor (PID) fits in a cache line that is freq=
uently
> + *   accessed by both CPU and IOMMU.
> + * 2.During posted MSI processing, the CPU needs to do 64-bit read and
> xchg
> + *   for checking and clearing posted interrupt request (PIR), a 256 bit=
 field
> + *   within the PID.
> + * 3.On the other side, the IOMMU does atomic swaps of the entire PID
> cache
> + *   line when posting interrupts and setting control bits.
> + * 4.The CPU can access the cache line a magnitude faster than the IOMMU=
.
> + * 5.Each time the IOMMU does interrupt posting to the PIR will evict th=
e
> PID
> + *   cache line. The cache line states after each operation are as follo=
ws:
> + *   CPU		IOMMU			PID Cache line state
> + *   ---------------------------------------------------------------
> + *...read64					exclusive
> + *...lock xchg64				modified
> + *...			post/atomic swap	invalid
> + *...-------------------------------------------------------------
> + *

According to VT-d spec: 5.2.3 Interrupt-Posting Hardware Operation:

"
- Read contents of the Posted Interrupt Descriptor, claiming exclusive
  ownership of its hosting cache-line.
  ...
- Modify the following descriptor field values atomically:
  ...
- Promote the cache-line to be globally observable, so that the modificatio=
ns
  are visible to other caching agents. Hardware may write-back the cache-li=
ne
  anytime after this step.
"

sounds that the PID cache line is not evicted after IOMMU posting?