Received-SPF: pass (google.com: domain of linux-kernel+bounces-135688-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) client-ip=2604:1380:40f1:3f00::1;
Date: Mon, 8 Apr 2024 09:39:30 -0700
From: Jacob Pan <jacob.jun.pan@linux.intel.com>
To: "Tian, Kevin" <kevin.tian@intel.com>
Cc: Baolu Lu <baolu.lu@linux.intel.com>, Dimitri Sivanich
 <sivanich@hpe.com>, Thomas Gleixner <tglx@linutronix.de>, Joerg Roedel
 <joro@8bytes.org>, Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>,
 Will Deacon <will@kernel.org>, Robin Murphy <robin.murphy@arm.com>, David
 Woodhouse <dwmw2@infradead.org>, Mark Rutland <mark.rutland@arm.com>, Peter
 Zijlstra <peterz@infradead.org>, Arnd Bergmann <arnd@arndb.de>, YueHaibing
 <yuehaibing@huawei.com>, "iommu@lists.linux.dev" <iommu@lists.linux.dev>,
 "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>, Steve Wahl
 <steve.wahl@hpe.com>, "Anderson, Russ" <russ.anderson@hpe.com>,
 jacob.jun.pan@linux.intel.com
Subject: Re: [PATCH v2] iommu/vt-d: Allocate DMAR fault interrupts locally
Message-ID: <20240408093930.45422bc1@jacob-builder>
In-Reply-To: <BN9PR11MB527627689CCBBFEAFC6745568C002@BN9PR11MB5276.namprd11.prod.outlook.com>
References: <Zfydpp2Hm+as16TY@hpe.com>
	<BN9PR11MB52765241E3261B58BE93E8A88C002@BN9PR11MB5276.namprd11.prod.outlook.com>
	<ec0ff625-855c-4dba-b19f-db050671fbfb@linux.intel.com>
	<BN9PR11MB527627689CCBBFEAFC6745568C002@BN9PR11MB5276.namprd11.prod.outlook.com>
Organization: OTC
Precedence: bulk
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit

Hi Kevin,

On Mon, 8 Apr 2024 09:00:05 +0000, "Tian, Kevin" <kevin.tian@intel.com>
wrote:

> > From: Baolu Lu <baolu.lu@linux.intel.com>
> > Sent: Monday, April 8, 2024 3:22 PM
> > 
> > On 2024/4/8 14:54, Tian, Kevin wrote:  
> > >> From: Dimitri Sivanich <sivanich@hpe.com>
> > >> Sent: Friday, March 22, 2024 4:51 AM
> > >>
> > >> The Intel IOMMU code currently tries to allocate all DMAR fault
> > >> interrupt vectors on the boot cpu.  On large systems with high DMAR
> > >> counts this results in vector exhaustion, and most of the vectors
> > >> are not initially allocated socket local.
> > >>
> > >> Instead, have a cpu on each node do the vector allocation for the
> > >> DMARs  
> > on  
> > >> that node.  The boot cpu still does the allocation for its node
> > >> during its boot sequence.
> > >>
> > >> Signed-off-by: Dimitri Sivanich <sivanich@hpe.com>  
> > >
> > > Reviewed-by: Kevin Tian <kevin.tian@intel.com>
> > >  
> > 
> > Kevin,
> > 
> > Jacob has another proposal which shares the irq among all IOMMUs.
> > 
> > https://lore.kernel.org/linux-iommu/20240403234548.989061-1-
> > jacob.jun.pan@linux.intel.com/
> > 
> > How do you like this?
> >   
> 
> I'm a bit concerning about the need of looping all IOMMU's in DMAR
> irqchip mask/unmask handlers. this one sounds simpler to me.
The difference is that with this patch, we still burn a few vectors on BSP
and the leading CPU of each socket.

e.g. on sapphire rapids, we lose 8 vectors to DMAR fault IRQ on BSP.

Thanks,

Jacob