Received-SPF: pass (google.com: domain of linux-kernel+bounces-40846-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) client-ip=2604:1380:40f1:3f00::1;
Date: Fri, 26 Jan 2024 15:30:47 -0800
From: Jacob Pan <jacob.jun.pan@linux.intel.com>
To: Thomas Gleixner <tglx@linutronix.de>
Cc: LKML <linux-kernel@vger.kernel.org>, X86 Kernel <x86@kernel.org>,
 iommu@lists.linux.dev, Lu Baolu <baolu.lu@linux.intel.com>,
 kvm@vger.kernel.org, Dave Hansen <dave.hansen@intel.com>, Joerg Roedel
 <joro@8bytes.org>, "H. Peter Anvin" <hpa@zytor.com>, Borislav Petkov
 <bp@alien8.de>, Ingo Molnar <mingo@redhat.com>, Raj Ashok
 <ashok.raj@intel.com>, "Tian, Kevin" <kevin.tian@intel.com>,
 maz@kernel.org, peterz@infradead.org, seanjc@google.com, Robin Murphy
 <robin.murphy@arm.com>, jacob.jun.pan@linux.intel.com
Subject: Re: [PATCH RFC 12/13] iommu/vt-d: Add a helper to retrieve PID
 address
Message-ID: <20240126153047.5e42e5d0@jacob-builder>
In-Reply-To: <874jgvuls0.ffs@tglx>
References: <20231112041643.2868316-1-jacob.jun.pan@linux.intel.com>
 <20231112041643.2868316-13-jacob.jun.pan@linux.intel.com>
 <874jgvuls0.ffs@tglx>
Organization: OTC
Precedence: bulk
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit

Hi Thomas,

On Wed, 06 Dec 2023 21:19:11 +0100, Thomas Gleixner <tglx@linutronix.de>
wrote:

> On Sat, Nov 11 2023 at 20:16, Jacob Pan wrote:
> > From: Thomas Gleixner <tglx@linutronix.de>
> >
> > When programming IRTE for posted mode, we need to retrieve the
> > physical  
> 
> we need .... I surely did not write this changelog.
> 
Will delete this.

> > address of the posted interrupt descriptor (PID) that belongs to it's
> > target CPU.
> >
> > This per CPU PID has already been set up during cpu_init().  
> 
> This information is useful because?
ditto.

> > +static u64 get_pi_desc_addr(struct irq_data *irqd)
> > +{
> > +	int cpu =
> > cpumask_first(irq_data_get_effective_affinity_mask(irqd));  
> 
> The effective affinity mask is magically correct when this is called?
> 
My understanding is that remappable device MSIs have the following
hierarchy,e.g.

parent:                              
    domain:  INTEL-IR-5-13            
     hwirq:   0x20000                 
     chip:    INTEL-IR-POST           
      flags:   0x0                    
     parent:                          
        domain:  VECTOR            
         hwirq:   0x3c             
         chip:    APIC         

When irqs are allocated and activated, parents domain op is always called
first. Effective affinity mask is set up by the parent domain, i.e. VECTOR.
Example call stack for alloc:
	irq_data_update_effective_affinity
	apic_update_irq_cfg
	x86_vector_alloc_irqs
	intel_irq_remapping_alloc
	msi_domain_alloc

x86_vector_activate also changes the effective affinity mask before calling
intel_irq_remapping_activate() where a posted interrupt is configured for
its destination CPU.

At runtime, when IRQ affinity is changed by userspace Intel interrupt
remapping code also calls parent data/chip to update the effective affinity
map before changing IRTE.

intel_ir_set_affinity(struct irq_data *data, const struct cpumask *mask,
		      bool force)
{
	ret = parent->chip->irq_set_affinity(parent, mask, force);

..
}
Here the parent APIC chip does apic_set_affinity() which will set up
effective mask before posted MSI affinity change.

Maybe I missed some cases?

I will also add a check if the effective affinity mask is not set up.

static phys_addr_t get_pi_desc_addr(struct irq_data *irqd)
{
	int cpu = cpumask_first(irq_data_get_effective_affinity_mask(irqd));

	if (WARN_ON(cpu >= nr_cpu_ids))
		return 0;

	return __pa(per_cpu_ptr(&posted_interrupt_desc, cpu));
}


Thanks,

Jacob