Received-SPF: pass (google.com: domain of linux-kernel+bounces-135649-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1;
Date: Mon, 8 Apr 2024 09:05:56 -0700
From: Jacob Pan <jacob.jun.pan@linux.intel.com>
To: "Tian, Kevin" <kevin.tian@intel.com>
Cc: "sivanich@hpe.com" <sivanich@hpe.com>, Thomas Gleixner
 <tglx@linutronix.de>, LKML <linux-kernel@vger.kernel.org>,
 "iommu@lists.linux.dev" <iommu@lists.linux.dev>, Lu Baolu
 <baolu.lu@linux.intel.com>, Joerg Roedel <joro@8bytes.org>, "Liu, Yi L"
 <yi.l.liu@intel.com>, "steve.wahl@hpe.com" <steve.wahl@hpe.com>, "Anderson,
 Russ" <russ.anderson@hpe.com>, "Peter Zijlstra" <peterz@infradead.org>,
 Will Deacon <will@kernel.org>, Robin Murphy <robin.murphy@arm.com>,
 jacob.jun.pan@linux.intel.com
Subject: Re: [PATCH 2/2] iommu/vt-d: Share DMAR fault IRQ to prevent vector
 exhaustion
Message-ID: <20240408090556.6165e603@jacob-builder>
In-Reply-To: <BN9PR11MB527677A69A2004A951165CC38C002@BN9PR11MB5276.namprd11.prod.outlook.com>
References: <20240403234548.989061-1-jacob.jun.pan@linux.intel.com>
	<20240403234548.989061-2-jacob.jun.pan@linux.intel.com>
	<BN9PR11MB527677A69A2004A951165CC38C002@BN9PR11MB5276.namprd11.prod.outlook.com>
Organization: OTC
Precedence: bulk
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit

Hi Kevin,

On Mon, 8 Apr 2024 08:48:54 +0000, "Tian, Kevin" <kevin.tian@intel.com>
wrote:

> > From: Jacob Pan <jacob.jun.pan@linux.intel.com>
> > Sent: Thursday, April 4, 2024 7:46 AM
> > 
> > DMAR fault interrupt is used for per-IOMMU unrecoverable fault
> > reporting, it occurs only if there is a kernel programming error or
> > serious hardware failure. In other words, they should never occur under
> > normal circumstances.  
> 
> this is not accurate. When a device is assigned to a malicious guest then
> it's not unusual to observe faults.
> 
Right, a malicious guest kernel could cause unrecoverable faults, e.g.
wrong privilege.

> in this context you probably meant that it's not a performance path hence
> sharing the vector is acceptable.
> 
Yes.
> >
> > @@ -1182,7 +1182,6 @@ static void free_iommu(struct intel_iommu
> > *iommu)
> >  			iommu->pr_irq = 0;
> >  		}
> >  		free_irq(iommu->fault_irq, iommu);
> > -		dmar_free_hwirq(iommu->fault_irq);  
> 
> You still want to free the vector for the iommu which first gets the
> vector allocated.
> 
I think we always want to keep this vector since the system always needs
one vector to share. We will never offline all the IOMMUs, right?

> > @@ -1956,9 +1955,8 @@ void dmar_msi_mask(struct irq_data *data)
> >  	raw_spin_unlock_irqrestore(&iommu->register_lock, flag);
> >  }
> > 
> > -void dmar_msi_write(int irq, struct msi_msg *msg)
> > +static void dmar_msi_write_msg(struct intel_iommu *iommu, int irq,
> > struct msi_msg *msg)
> >  {  
> 
> what about iommu_msi_write_msg() to match the first parameter?
> 
> otherwise it leads to a slightly circled calltrace:
> 	dmar_msi_write_msg()
> 		dmar_msi_write()
> 			dmar_msi_write_msg()
> 
Good point, will do.

> > +
> > +	/*
> > +	 * Only the owner IOMMU of the shared IRQ has its fault event
> > +	 * interrupt unmasked after request_irq(), the rest are
> > explicitly
> > +	 * unmasked.
> > +	 */
> > +	if (!(iommu->flags & VTD_FLAG_FAULT_IRQ_OWNER))
> > +		dmar_fault_irq_unmask(iommu);
> > +  
> 
> em there is a problem in dmar_msi_mask() and dmar_msi_mask()
> which only touches the owner IOMMU. With this shared vector
> approach we should mask/unmask all IOMMU's together. 
I thought about this as well, in addition to fault_irq,
dmar_msi_mask/unmask() are used for other DMAR irqs, page request and
perfmon. So we need a special case for fault_irq there, it is not pretty.

I added a special case here in this patch, thinking we never mask the
fault_irq since we need to cover the lifetime of the system. I have looked
at:
1.IOMMU suspend/resume, no mask/unmask
2.IRQ migration, added IRQF_NOBALANCING

maybe I missed some cases?


Thanks,

Jacob